An image optimization method, device, equipment and storage medium

Abstract

Embodiments of the present application disclose an image optimization method, device and equipment and a storage medium. The method comprises: determining a predicted classification of a noisy image based on a pre-training network, wherein the pre-training network is used for image classification; determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image; performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss; determining a loss function according to the classification loss and the regularization loss; performing optimization processing on the noisy image, calculating a function value of the loss function according to the processed noisy image, and determining the processed noisy image as an optimized image when the function value of the loss function meets a preset condition. The optimized image obtained by optimizing the noisy image can be used as a training image to retrain the pre-training network, thereby improving the classification accuracy of the pre-training network and solving the problem that it is difficult to obtain real images for model training in the prior art.

Description

TECHNICAL FIELD

[0001] Embodiments of the present application relate to the image processing technology, and particularly relate to an image optimization method, device, equipment and storage medium. BACKGROUND

[0002] Image classification is a hot research topic, which has a wide range of applications in many fields. In recent years, deep learning technology has been widely used in image classification. Deep learning requires powerful hardware computing power, large amount of training data and deep network layers to extract data, which is both the key to its advantages and the factor limiting its use.

[0003] In the prior art, a large number of labeled real images are needed to train a deep learning model to improve the accuracy of the deep learning model classification.

[0004] In the process of implementing the present application, it is found that at least the following technical problems exist in the prior art:

[0005] Due to privacy, storage and transmission restrictions, it is often difficult to obtain real images for model training. SUMMARY

[0006] The present application provides an image optimization method, device, equipment and storage medium to optimize noise images to obtain optimized images for training a model, thereby solving the problem that it is difficult to obtain real images for model training in the prior art.

[0007] In a first aspect, embodiments of the present application provide an image optimization method, comprising:

[0008] determining a predicted classification of a noise image based on a pre-trained network, and determining a classification loss of the noise image according to the predicted classification and a classification label set for the noise image; wherein the pre-trained network is used for image classification;

[0009] performing distortion constraint and regularization processing on the noise image to obtain a regularization loss; and determining a loss function according to the classification loss and the regularization loss;

[0010] performing optimization processing on the noise image, calculating a function value of the loss function according to the processed noise image, and determining the processed noise image as an optimized image when the function value of the loss function meets a preset condition.

[0011] In a second aspect, embodiments of the present application also provide an image optimization device, comprising:

[0012] The determining module is configured to determine a predicted classification of the noisy image based on a pre-trained network, and determine a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image, wherein the pre-trained network is used for image classification.

[0013] The processing module is configured to perform distortion constraint and regularization processing on the noisy image to obtain a regularization loss.

[0014] The execution module is configured to determine a loss function according to the classification loss and the regularization loss.

[0015] The optimization module is configured to perform optimization processing on the noisy image, calculate a function value of the loss function according to the processed noisy image, and determine the processed noisy image as an optimized image when the function value of the loss function meets a preset condition.

[0016] In a third aspect, an embodiment of the present application further provides a computer device, and the device comprises:

[0017] at least one processor; and a memory connected with the at least one processor in communication;

[0018] The memory stores a computer program executable by the at least one processor, and the computer program is executed by the at least one processor to enable the at least one processor to execute the image optimization method in any one of the first aspect.

[0019] In a fourth aspect, an embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the program is executed by a processor to implement the image optimization method in any one of the first aspect.

[0020] In a fifth aspect, an embodiment of the present application further provides a computer program product, and the computer program product comprises a computer program, and the computer program is executed by a processor to implement the image optimization method in any one of the first aspect.

[0021] The embodiments of the above application have the following advantages or beneficial effects:

[0022] The embodiment of the present application provides an image optimization method, comprising: determining a predicted classification of a noisy image based on a pre-training network, wherein the pre-training network is used for image classification; determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image; performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss; determining a loss function according to the classification loss and the regularization loss; performing optimization processing on the noisy image, calculating a function value of the loss function according to the processed noisy image, and determining the processed noisy image as an optimized image when the function value of the loss function meets a preset condition. The above technical solution, the image classification accuracy of the pre-training network is low, and a large number of real images are needed for retraining, but the real images are difficult to obtain. First, the predicted classification of the noisy image can be determined according to the pre-training network. Second, the classification loss of the noisy image is determined according to the predicted classification and the randomly set classification label of the noisy image. After the regularization loss of the noisy image is determined, the loss function can be determined according to the classification loss and the regularization loss, the noisy image is optimized, and the function value of the loss function is calculated according to the processed noisy image. When the function value of the loss function meets the preset condition, the processed noisy image is determined as the optimized image. When the loss function meets the preset condition, the predicted classification is close to the classification label, the texture distribution of the image is natural, and the image feature distribution is close to the real image feature. Therefore, the optimized image can be determined as the training image, the pre-training network is retrained to improve the classification accuracy of the pre-training network, and the problem that the real image used for model training is difficult to obtain in the prior art is solved. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A flowchart of an image optimization method provided by the embodiment of the present application is provided.

[0024] Figure 2 A flowchart of another image optimization method provided by the embodiment of the present application is provided.

[0025] Figure 3 A flowchart of step 240 in another image optimization method provided by the embodiment of the present application is provided.

[0026] Figure 4 A schematic diagram of a determination mode of a loss function in another image optimization method provided by the embodiment of the present application is provided.

[0027] Figure 5 A structural schematic diagram of an image optimization device provided by the embodiment of the present application is provided.

[0028] Figure 6 A structural schematic diagram of a computer device provided by the embodiment of the present application is provided. DETAILED DESCRIPTION

[0029] The application will be further described below in conjunction with the drawings and embodiments. It should be understood that the specific embodiments described herein are intended to be illustrative only and are not intended to limit the scope of the application. In addition, it should be noted that the drawings are not drawn to scale and only show the parts relevant to the application.

[0030] The term "and / or" in the present application is merely used to describe associated objects, and can represent three relationships, for example, A and / or B can represent three cases of A alone, A and B, and B alone.

[0031] The terms "first" and "second" and the like in the description of the present application and the drawings are used to distinguish different objects or different treatments of the same object, and are not used to describe the specific order of the objects.

[0032] In addition, the terms "comprising" and "having" and any variations thereof mentioned in the description of the present application are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include other steps or units not listed or other steps or units inherent to the process, method, product or device.

[0033] Before some example embodiments are discussed in more detail, it should be noted that some example embodiments are described as processes or methods depicted as flowcharts. Although the flowcharts describe the operations (or steps) as sequential processes, many of the operations can be performed in parallel, concurrently or simultaneously. In addition, the order of the operations can be rearranged. The processes can be terminated when their operations are completed, but can also have additional steps not included in the drawings. The processes can correspond to methods, functions, procedures, subroutines, etc. In addition, the embodiments and features in the embodiments can be combined with each other without conflict.

[0034] It should be noted that in the embodiments of the present application, the words "exemplary" or "for example" are used to mean serving as an example, instance, or illustration. Any embodiment or design scheme described as "exemplary" or "for example" in the embodiments of the present application should not be construed as being more preferred or advantageous than other embodiments or design schemes. Rather, the use of "exemplary" or "for example" is intended to present concepts in a concrete manner.

[0035] In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise specified.

[0036] Figure 1A flowchart of an image optimization method provided by an embodiment of the present application, the embodiment can be applicable to a case where it is difficult to obtain real images for training of an image classification model, and the method can be executed by an image optimization device, as shown in Figure 1 The method specifically includes the following steps:

[0037] Step 110: determining a predicted classification of the noisy image based on a pre-trained network, and determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image.

[0038] The pre-trained network is used for image classification and can be a pre-trained convolutional neural network.

[0039] Specifically, the pre-trained network can be used to classify images, and when the noisy image is input as input information into the pre-trained network, the output information obtained can be a predicted classification corresponding to the noisy image. While determining the predicted classification of the noisy image based on the pre-trained network, a classification label can also be randomly set for the noisy image, which can be any category, for example, a cat. Since the pre-trained network has only been pre-trained, the accuracy of the classification is not high when used to classify images. Therefore, the predicted classification of the noisy image determined according to the pre-trained network is not accurate.

[0040] The classification label and the predicted classification can be used to determine the classification loss of the noisy image according to a cross-entropy loss function, so that the noisy image is continuously optimized to be an image that meets the classification label judgment domain in the optimization process. In the embodiment of the present application, the predicted classification of the noisy image determined by the pre-trained network and the classification label set for the noisy image can determine the classification loss, and the noisy image can be optimized based on the classification loss, so that the noisy image is optimized to be an image that is judged as the classification label by the image classification model.

[0041] Step 120: performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss.

[0042] Specifically, the regularization loss can include two parts, a prior part and a regularization part, wherein the prior part can perform distortion constraint on the noisy image, and the regularization part can perform regularization processing on the noisy image. The noisy image can be subjected to distortion constraint based on a TV regularization loss function and an L2 regularization loss function, and the distortion constraint makes the noisy image optimized to be an image that does not have distortion and other textures that do not conform to the distribution of natural images. The noisy image can be subjected to regularization processing based on BN regularization, and the regularization processing makes the noisy image optimized to be an image that conforms to the feature distribution of real images, thereby generating a meaningful image.

[0043] In the embodiment of the present application, the regular loss can be determined to ensure that the noise image does not appear distortion and other textures that do not conform to the natural image distribution in the optimization process, and the noise is optimized to the feature distribution and meaningful that conforms to the real image.

[0044] In step 130, the loss function is determined according to the classification loss and the regular loss.

[0045] Specifically, the classification loss and the regular loss can be summed, and the sum is determined as the loss function.

[0046] In step 140, the noise image is optimized, the function value of the loss function is calculated according to the processed noise image, and the processed noise image is determined as the optimized image when the function value of the loss function meets the preset condition.

[0047] Specifically, the color feature, texture feature, shape feature and spatial relationship feature of the noise image can be adjusted based on the pixel to optimize the noise image. After the optimization processing is completed, the function value of the loss function can be calculated according to the processed noise image. If the function value of the loss function meets the preset condition, the processed noise image is determined as the optimized image. If the function value of the loss function does not meet the preset condition, the optimization processing of the noise image is continued until the function value of the loss function meets the preset condition.

[0048] In the embodiment of the present application, the pre-training network is back-propagated to optimize the noise image to obtain the optimized image, and the optimized image can be used as the training image to retrain the pre-training network to improve the classification accuracy of the pre-training network, thereby solving the problem that it is difficult to obtain the real image for model training in the prior art.

[0049] The embodiment of the present application provides an image optimization method, comprising: determining a predicted classification of a noisy image based on a pre-training network, wherein the pre-training network is used for image classification; determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image; performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss; determining a loss function according to the classification loss and the regularization loss; performing optimization processing on the noisy image, calculating a function value of the loss function according to the processed noisy image, and determining the processed noisy image as an optimized image when the function value of the loss function meets a preset condition. The above technical solution has the advantages that the image classification accuracy of the pre-training network is low, and a large number of real images are needed for retraining, but the real images are difficult to obtain. First, the predicted classification of the noisy image can be determined according to the pre-training network. Second, the classification loss of the noisy image can be determined according to the predicted classification and the randomly set classification label of the noisy image. After the regularization loss of the noisy image is determined, the loss function can be determined according to the classification loss and the regularization loss, the noisy image is optimized, and the function value of the loss function is calculated according to the processed noisy image. When the function value of the loss function meets the preset condition, the processed noisy image is determined as the optimized image. When the loss function converges, the predicted classification is close to the classification label, the texture distribution of the image is natural, and the image feature distribution is close to the real image feature. Therefore, the optimized image can be determined as the training image, the pre-training network is retrained to improve the classification accuracy of the pre-training network, and the problem that the real image used for model training is difficult to obtain in the prior art is solved.

[0050] Figure 2 The flowchart of another image optimization method provided by the embodiment of the present application can be applied to the case that it is difficult to obtain real images used for image classification model training. The embodiment of the present application is based on the above-mentioned embodiment, and before determining the predicted classification of the noisy image based on the pre-training network, "constructing a network model used for image classification, and pre-training the network model to obtain the pre-training network" is added. Before determining the loss function according to the classification loss and the regularization loss, "determining the diversification loss of the noisy image according to the simplest positive sample, the most difficult positive sample and the most difficult negative sample of the noisy image" is added. The explanations of the same or corresponding terms in the above-mentioned embodiments are not repeated here. As Figure 2 The image optimization method provided by the embodiment of the present application specifically comprises the following steps:

[0051] Step 210, constructing a network model used for image classification, and pre-training the network model to obtain the pre-training network.

[0052] In an implementation manner, the step 210 can specifically include:

[0053] The network model for image classification is constructed based on a convolutional neural network, the network model is pre-trained through a public data set, and a pre-loss function is calculated, network optimization is performed based on a back propagation algorithm until the pre-loss function converges, and the pre-training network is obtained.

[0054] Specifically, the convolutional neural network of deep learning can be used for image classification, therefore, the network model for image classification can be constructed based on the convolutional neural network, after the network model is constructed, the network model can be pre-trained according to a public data set, specifically, the network model can be pre-trained based on an image Net data set, and a pre-loss function is calculated, of course, network optimization can be performed based on a back propagation algorithm until the pre-loss function converges, and the pre-training network is determined according to the network parameters when the pre-loss function converges.

[0055] In the embodiment of the application, before the predicted classification of the noise image is determined, the network model for image classification can be constructed, of course, the network model can also be pre-trained based on a public data set to improve the classification accuracy, but the classification accuracy of the pre-training network obtained by pre-training the network model is still low and cannot be directly applied to image classification.

[0056] Step 220, determining the predicted classification of the noise image based on the pre-training network, and determining the classification loss of the noise image according to the predicted classification and the classification label set for the noise image.

[0057] Wherein, x represents the noise image, x∈R H×W×C H represents the width of the noise image, W represents the height of the noise image, and C represents the channel number of the noise image.

[0058] In one embodiment, step 220 can specifically include:

[0059] Determining the cross-entropy loss function of the classification label and the predicted classification, and determining the cross-entropy loss function as the classification loss.

[0060] Specifically, the classification loss can be determined based on formula 1.

[0061] L CE (m, n) = distance(n, p(m|x)) formula 1

[0062] Wherein, m represents the predicted classification, n represents the classification label, and x represents the noise image.

[0063] In the embodiment of the present application, the classification loss of the noise image is determined according to the cross-entropy loss function and the determination of the predicted classification, so that the noise image is continuously optimized to be an image conforming to the classification label discriminant domain in the optimization process. For example, when the classification label is a cat, the noise image is finally optimized to be an image that can make the image classification model determine the image as a cat.

[0064] Step 230, distortion constraint and regularization processing are performed on the noise image to obtain a regularization loss.

[0065] In one implementation, step 230 can specifically include:

[0066] determining a regularization loss function of the noise image, determining a distortion constraint result according to the regularization loss function; determining an input statistical value and a storage statistical value of the image feature of the noise image according to the pre-training network, and determining a regularization processing result according to the input statistical value and the storage statistical value, wherein the input statistical value includes the mean and / or variance of the image feature of each image in the batch of noise images to which the noise image belongs, and the storage statistical value includes the mean and / or variance of the image feature stored in the pre-training network; and determining the sum of the distortion constraint result and the regularization processing result as the regularization loss.

[0067] The regularization loss can include two parts, a prior part R prior and a regularization part R BN . The prior part can perform distortion constraint on the noise image, and the regularization part can perform regularization processing on the noise image.

[0068] Specifically, the noise image can be distortion constrained based on a regularization loss function, and the regularization loss function includes a TV regularization loss function and an L2 regularization loss function. The noise image can be distortion constrained based on the TV regularization loss function and the L2, and specifically, the distortion constraint can be determined based on formula 2. The noise image can be regularized based on BN regularization, and specifically, the regularization processing can be determined based on formula 3.

[0069] R prior (x)=α TV R TV (x)+α L2 R L2 (x) Formula 2

[0070] Wherein, α TV represents a preset coefficient of the TV regularization loss function in the prior part, R TV (x) represents the TV regularization loss function of the noise image, α L2 represents a preset coefficient of the L2 regularization loss function in the prior part, R L2(x) represents an L2 regular loss function of a noise image. In practical applications, α TV and the specific numerical value of α L2 may be determined according to actual requirements, for example, α TV may be 10 -2 , and α L2 may be 10 -3 .

[0071]

[0072] wherein α BN represents a preset coefficient of a regularization part, L represents the number of BN layers in a pre-training network, f(x) represents image features of a noise image output by a pooling layer after the noise image is input into the pre-training network, μ l f(x) represents an input mean of image features, μ l BN represents a storage mean stored in the pre-training network, μ l 2 f(x) represents an input variance of image features, μ l 2BN represents a storage variance stored in the pre-training network. In practical applications, the specific numerical value of α BN may also be determined according to actual requirements, for example, α BN may be 10.

[0073] Since the mean and variance information stored in the pre-training network represent the feature distribution of real images learned by the pre-training network, the image noise is optimized to be a feature conforming to the distribution, so that an image that can be used to train the pre-training network is obtained, thereby generating a meaningful image.

[0074] Therefore, it can be determined that the regular loss R(x) = R prior (x) + R BN f(x).

[0075] In the embodiment of the application, the distortion constraint makes the noise image be optimized to be a texture that does not appear distortion and does not conform to the natural image distribution, and the regularization processing makes the noise image be optimized to be a feature conforming to the real image, thereby generating a meaningful image.

[0076] Step 240, determining a diversity loss of the noise image according to the simplest positive sample, the most difficult positive sample and the most difficult negative sample of the noise image.

[0077] To ensure the diversity of the optimized images, the diversity loss can maintain the diversity of the boundaries and the interiors in the batch of optimized images. The diversity loss can enable each noise image in the batch of noise images to be optimized into images of multiple categories, ensure the correctness of the image categories in the batch of optimized images obtained by optimizing the batch of noise images, and ensure the semantic features of the optimized images.

[0078] Figure 3 For another image optimization method provided in the embodiments of the present application, a flowchart of step 240 is shown in FIG. 24. Figure 3 As shown in FIG. 24, in an embodiment, step 240 can specifically include:

[0079] Step 2410, determining the simplest positive sample, the most difficult positive sample, and the most difficult negative sample of the noise image.

[0080] The simplest positive sample represents an image with the smallest Euclidean distance to the feature of the current noise image among other images of the batch of noise images to which the current noise image belongs; the most difficult positive sample represents an image with the largest Euclidean distance to the feature of the current noise image among other noise images of the same category as the current noise image in the batch of noise images; and the most difficult negative sample represents an image with the largest Euclidean distance to the feature of the current noise image among other noise images of a different category from the current noise image in the batch of noise images.

[0081] Specifically, in the batch of noise images input into the pre-trained network, for each noise image, the simplest positive sample can be determined based on formula 4, the most difficult positive sample can be determined based on formula 5, and the most difficult negative sample can be determined based on formula 6.

[0082]

[0083] wherein x ep represents the simplest positive sample, x a represents the current noise image, x b represents other noise images in the batch of noise images except the current noise image, f(x a ) represents the image feature of the current noise image output by the pooling layer after the current noise image is input into the pre-trained network, f(x b ) represents the image feature of x b output by the pooling layer after x b is input into the pre-trained network, and dist(f(x a ), f(x b )) = ||f(x a )-f(x b )||2 can represent the Euclidean distance between the image feature of the current noise image and the image feature of other noise images, i.e., the Euclidean distance between the features of two noise images.

[0084]

[0085] wherein x p represents other noise images of the same class as the current noise image in the batch of noise images, f(x p ) represents the image feature of x p output by the pooling layer after inputting x p into the pre-trained network, and dist(f(x a ), f(x p )) = ||f(x a )-f(x p )||2 can represent the Euclidean distance of the image features of the current noise image and other noise images of the same class, i.e., the feature Euclidean distance of the two noise images.

[0086]

[0087] wherein x n represents other noise images of different classes from the current noise image in the batch of noise images, f(x n ) represents the image feature of x n output by the pooling layer after inputting x n into the pre-trained network, and dist(f(x a ), f(x n )) = ||f(x a )-f(x n )||2 can represent the Euclidean distance of the image features of the current noise image and other noise images of different classes, i.e., the feature Euclidean distance of the two noise images.

[0088] In the embodiment of the application, for the batch of noise images input into the pre-trained network, the simplest positive sample, the most difficult positive sample and the most difficult negative sample of each noise image can be determined, so as to prepare data basis for the diversified loss of the noise image.

[0089] Step 2420, determining a first diversified loss according to the noise image and the simplest positive sample; and determining a second diversified loss according to the noise image, the most difficult positive sample and the most difficult negative sample.

[0090] In an implementation, step 2420 can specifically include:

[0091] determine a first feature Euclidean distance between the noise image and the easiest positive sample, and determine the first diversity loss according to the first feature Euclidean distance; determine a second feature Euclidean distance between the noise image and the hardest positive sample, and a third feature Euclidean distance between the noise image and the hardest negative sample, and determine the second diversity loss according to the second feature Euclidean distance and the third feature Euclidean distance.

[0092] Specifically, the closer the images are in the latent space, the more similar the images represent, and therefore increasing the distance between the images can increase the diversity of the images. After obtaining the easiest positive sample of the current noise image in the batch of noise images, the feature Euclidean distance between the current noise image and the easiest positive sample can be increased, so as to ensure the diversity of the batch of optimized images obtained by optimizing the batch of noise images. Specifically, the diversity increase result, i.e., the first diversity loss, can be determined based on formula 7.

[0093] L ep (x) = -dist(f(x a ), f(x ep )) formula 7

[0094] In an implementation, the second diversity loss is determined according to the second feature Euclidean distance and the third feature Euclidean distance, including:

[0095] The difference between the second feature Euclidean distance and the third feature Euclidean distance is determined, and the sum of the difference and a preset hyperparameter is calculated; if the sum is greater than zero, the sum is determined as the second diversity loss; if the sum is less than zero, the second diversity loss is determined as zero.

[0096] Specifically, in order to ensure that the batch of optimized images is still in the discriminant domain of the classifier after the distance is increased, so as to ensure that the semantic discriminant information of the batch of optimized images is not destroyed, the semantic optimization result, i.e., the second diversity loss, can be introduced. Specifically, the second diversity loss can be determined based on formula 8.

[0097] L triplet (x) = max(0, dist ap -dist an + margin) formula 8

[0098] wherein dist ap = dist(f(x a ), f(x p )) = ||f(x a )-f(x p )||2, dist an = dist(f(x a ), f(x n )) = ||f(xa )-f(x n )||2, margin is a preset hyperparameter.

[0099] The distance between a noisy image and its farthest positive sample and its nearest negative sample in the latent space is greater than a preset hyperparameter. That is, the sum of the distances between the noisy image and the most difficult positive sample and the distance between the noisy image and the most difficult negative sample is greater than the preset hyperparameter. This ensures that the noisy image will not be incorrectly optimized into the discrimination space of other categories, thereby ensuring the correctness of the category of the optimized image. This allows the semantic features of the optimized image to be guaranteed while increasing diversity.

[0100] In this embodiment of the invention, a first diversification loss can be determined based on the simplest positive sample of the current noise image in the batch of noise images to which the current noise image belongs and the current noise image; a second diversification loss can be determined based on the most difficult positive sample and the most difficult negative sample of the current noise image in the batch of noise images to which the current noise image belongs and the current noise image.

[0101] Step 2430: Determine the diversification loss using the first diversification loss and the second diversification loss.

[0102] Specifically, the diversification loss can be determined based on Formula 9.

[0103] L intra-div (x)=α ep L ep (x)+α triplet L triplet (x) Formula 9

[0104] Where, α ep α represents the preset coefficient for the first diversification loss. triplet This represents the preset coefficient for the first diversification loss. In practical applications, α... ep and α triplet The specific value can also be determined according to actual needs, for example, α ep It can be 50, α triplet It can be 0.5.

[0105] Diversity loss can be combined with first diversification loss to increase the diversity of batch optimized images while ensuring the semantic features of each optimized image in the batch optimized images.

[0106] In the embodiment of the present application, in order to increase the diversity of the batch optimization image, ensure the correctness of the image category of the batch optimization image, increase the diversity while ensuring the semantic features of the batch optimization image, the diversification loss can be determined. The diversification loss ensures that the sample is still within the boundary range of the classification domain while increasing the intra-class sample distance, thereby generating high-quality diverse images.

[0107] Step 250, determining a loss function according to the classification loss, the regularization loss and the diversification loss.

[0108] Figure 4 Another determination method of the loss function in the image optimization method provided by the embodiment of the present application is shown in the schematic diagram, and one of the determination methods is exemplarily given. Figure 4 As shown in the figure, it includes:

[0109] Step 410, calculating the prior regularization loss of the noise image, that is, the distortion constraint R prior (x).

[0110] Step 420, inputting the noise image into the pre-training model, and determining the regularization loss R BN f(x) according to the image feature f(x) output by the pooling layer included in the pre-training model.

[0111] Step 430, performing sample mining on the noise image, determining the simplest sample, the most difficult positive sample and the most difficult negative sample of the noise image in the batch of noise images to which the noise image belongs, and determining the diversification loss L intra-div (x) according to the noise image, the simplest positive sample, the most difficult positive sample and the most difficult negative sample.

[0112] Step 440, determining the classification loss L CE (m, n) according to the predicted classification of the noise image output by the pre-training model and the classification label set for the noise image.

[0113] Step 450, determining the loss function LOSS(x) according to the prior regularization loss R prior (x), the regularization loss R BN f(x), the diversification loss L intra-div (x) and the classification loss L CE (m, n).

[0114] Specifically, the loss function LOSS(x) can be determined after summing the classification loss, the regularization loss and the diversification loss, that is, LOSS(x) = L CE (m, n) + R(x) + L diversity (x).

[0115] In step 260, the noise image is optimized, a function value of the loss function is calculated according to the processed noise image, and the processed noise image is determined as the optimized image when the function value of the loss function meets a preset condition.

[0116] In an implementation, step 260 can specifically include:

[0117] The noise image is optimized, a function value of the loss function is calculated according to the processed noise image, and the processed noise image is determined as the optimized image when the function value of the loss function converges.

[0118] After the loss function is determined, the optimized image can be determined based on formula 10.

[0119]

[0120] wherein x y represents the optimized image.

[0121] Specifically, the color feature, the texture feature, the shape feature and the spatial relationship feature of the noise image can be adjusted based on the pixel, so as to optimize the noise image. After the optimization is completed, the function value of the loss function can be calculated according to the processed noise image. If the function value of the loss function indicates that the loss function converges, the processed noise image is determined as the optimized image; if the function value of the loss function indicates that the loss function has not converged, the noise image is continuously optimized until the loss function converges.

[0122] In actual application, if the function value of the loss function indicates that the loss function has not converged, the noise image is continuously optimized to obtain a processed noise image, and the function value of the loss function is continuously calculated according to the processed noise image. If the function value indicates that the loss function converges, the processed noise image is determined as the optimized image; if the function value indicates that the loss function has not converged, the noise image is continuously optimized.

[0123] In the embodiments of the present application, the classification loss is used to determine the loss function of the classification label and the predicted classification, the regularization loss is used to determine the regularization term of the noise image, and the diversity loss is used to ensure the diversity of the optimized image generated by the optimized noise image.

[0124] The embodiment of the present application provides an image optimization method, comprising: constructing a network model for image classification, and pre-training the network model to obtain a pre-training network; determining a predicted classification of a noisy image based on the pre-training network, determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image; performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss; determining a diversification loss of the noisy image according to the simplest positive sample, the most difficult positive sample and the most difficult negative sample of the noisy image; determining a loss function according to the classification loss, the regularization loss and the diversification loss; performing optimization processing on the noisy image, calculating a function value of the loss function according to the processed noisy image, and determining the processed noisy image as an optimized image when the function value of the loss function meets a preset condition. The above technical solution, after constructing the network model for image classification, the network model can be pre-trained to obtain the pre-training network, which can be used for image classification, but the image classification accuracy of the pre-training network is low, and a large amount of real data is needed for retraining, and real data is difficult to obtain. First, the predicted classification of the noisy image can be determined according to the pre-training network. Secondly, the classification loss of the noisy image is determined according to the predicted classification and the classification label randomly set for the noisy image. After the regularization loss and the diversification loss of the noisy image are determined, the loss function can be determined according to the classification loss, the regularization loss and the diversification loss, the noisy image is optimized, and the function value of the loss function is calculated according to the processed noisy image. When the function value indicates that the loss loss function converges, the processed noisy image is determined as an optimized image. When the loss function converges, the predicted classification is close to the classification label, the texture distribution of the image is natural, the image feature distribution is close to the real image feature, and the image diversity is strong. Therefore, the optimized image can be determined as a training image, the pre-training network is retrained to improve the classification accuracy of the pre-training network, and the problem that real images used for model training are difficult to obtain in the prior art is solved.

[0125] Figure 5 The structure diagram of an image optimization device provided by the embodiment of the present application is provided. The device and the foregoing image optimization method belong to the same inventive concept. Details not described in the embodiment can be referred to the related description of the image optimization method.

[0126] The specific structure of the image optimization device is shown in Figure 5 The image optimization device comprises:

[0127] The determining module 510 is configured to determine a predicted classification of a noisy image based on a pre-training network, and determine a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image. The pre-training network is used for image classification.

[0128] The processing module 520 is configured to perform distortion constraint and regularization processing on the noise image to obtain a regularization loss.

[0129] The execution module 530 is configured to determine a loss function according to the classification loss and the regularization loss.

[0130] The optimization module 540 is configured to perform optimization processing on the noise image, calculate a function value of the loss function according to the processed noise image, and determine the processed noise image as an optimized image when the function value of the loss function meets a preset condition.

[0131] On the basis of the above embodiment, the device further includes a construction module, which is configured to:

[0132] construct a network model for image classification, and pre-train the network model to obtain the pre-trained network.

[0133] Specifically, the network model for image classification is constructed based on a convolutional neural network, the network model is pre-trained through a public data set, and a pre-loss function is calculated; network optimization is performed based on a back propagation algorithm until the pre-loss function converges, and the pre-trained network is obtained.

[0134] On the basis of the above embodiment, the determination module 510 is specifically configured to:

[0135] determine a predicted classification of the noise image based on the pre-trained network, determine a cross-entropy loss function of the classification label and the predicted classification, and determine the cross-entropy loss function as the classification loss.

[0136] On the basis of the above embodiment, the processing module 520 is specifically configured to:

[0137] determine a regularization loss function of the noise image, and determine a distortion constraint result according to the regularization loss function.

[0138] determine an input statistical value and a storage statistical value of the image feature of the noise image based on the pre-trained network, and determine a regularization processing result according to the input statistical value and the storage statistical value, wherein the input statistical value includes a mean value and / or a variance of the image feature of each image in a batch of noise images to which the noise image belongs, and the storage statistical value includes a mean value and / or a variance of the image feature stored in the pre-trained network.

[0139] determine a sum value of the distortion constraint result and the regularization processing result as the regularization loss.

[0140] On the basis of the above embodiment, the device further includes:

[0141] a diversity module configured to determine a diversity loss of the noisy image according to the easiest positive sample, the hardest positive sample and the hardest negative sample of the noisy image.

[0142] Correspondingly, the execution module 530 is specifically configured to:

[0143] The classification loss, the regularization loss and the diversity loss are summed, and a sum result is determined as the loss function.

[0144] On the basis of the above embodiment, the diversity module is specifically configured to:

[0145] determine the easiest positive sample, the hardest positive sample and the hardest negative sample of the noisy image;

[0146] determine a first diversity loss according to the noisy image and the easiest positive sample, and determine a second diversity loss according to the noisy image, the hardest positive sample and the hardest negative sample;

[0147] determine the diversity loss through the first diversity loss and the second diversity loss.

[0148] In an implementation, the determination of the easiest positive sample, the hardest positive sample and the hardest negative sample of the noisy image comprises:

[0149] determination of a feature Euclidean distance between an image feature of the noisy image and image features of other images in a batch of noisy images to which the noisy image belongs;

[0150] determination of an image with the smallest feature Euclidean distance as the easiest positive sample, determination of a same-class image with the largest feature Euclidean distance as the hardest positive sample, and determination of a different-class image with the smallest feature Euclidean distance as the hardest negative sample.

[0151] In an implementation, the determination of a first diversity loss according to the noisy image and the easiest positive sample, and the determination of a second diversity loss according to the noisy image, the hardest positive sample and the hardest negative sample comprise:

[0152] determination of a first feature Euclidean distance between the noisy image and the easiest positive sample, and determination of the first diversity loss according to the first feature Euclidean distance;

[0153] determination of a second feature Euclidean distance between the noisy image and the hardest positive sample, and a third feature Euclidean distance between the noisy image and the hardest negative sample, and determination of the second diversity loss according to the second feature Euclidean distance and the third feature Euclidean distance.

[0154] On the basis of the above-mentioned embodiments, the optimization module 540 is specifically configured to:

[0155] The noise image is subjected to an optimization process, a function value of the loss function is calculated according to the processed noise image, and the processed noise image is determined as the optimized image when the function value of the loss function converges.

[0156] The image optimization apparatus provided in the embodiments of the present application can execute the image optimization method provided in any of the embodiments of the present application, and has the corresponding function modules and beneficial effects of the execution method.

[0157] It is worth noting that, in the embodiments of the above-mentioned image optimization apparatus, each unit and module included is only divided according to the function logic, but is not limited to the above-mentioned division, as long as the corresponding function can be realized; in addition, the specific name of each functional unit is only for the convenience of mutual differentiation, and is not used to limit the protection scope of the present application.

[0158] Figure 6 A structural schematic diagram of a computer device provided in the embodiments of the present application is provided. Figure 6 A block diagram of an exemplary computer device 6 suitable for implementing embodiments of the present application is shown. Figure 6 The computer device 6 shown is merely an example, and should not bring any limitation to the function and use range of the embodiments of the present application.

[0159] As shown in Figure 6 The computer device 6 is shown in the form of a general-purpose computing device. The components of the computer device 6 can include, but are not limited to, one or more processors or processing units 16, a system memory 28, and a bus 18 that connects the various system components, including the system memory 28 and the processing unit 16.

[0160] The bus 18 represents one or more of several types of bus structures, including a memory bus or memory controller, a peripheral bus, a graphics acceleration bus, a processor or local bus using any of a variety of bus architectures. By way of example, these architectures include Industry Standard Architecture (ISA) bus, Micro Channel Architecture (MAC) bus, Enhanced ISA bus, Video Electronics Standards Association (VESA) local bus, and Peripheral Component Interconnect (PCI) bus.

[0161] The computer device 6 typically includes a variety of computer system readable media. These media can be any available media that can be accessed by the computer device 6 and includes both volatile and nonvolatile media, removable and non-removable media.

[0162] System memory 28 can include computer system readable media in the form of volatile memory, such as random access memory (RAM) 30 and / or cache memory 32. Computer device 6 can further include other removable / non-removable, volatile / non-volatile computer system storage media. By way of example only, storage system 34 can be provided for reading from and writing to a non-removable, non-volatile magnetic media (not shown and typically called a "hard drive"). Figure 6 Although not shown, computer device 6 can employ other, different or Figure 6 additional hardware and / or software modules that can be used in conjunction with the computer device 6, including but not limited to a microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc.

[0163] Program / utility 40, having a set (at least one) of program modules 42, can be stored in, for example, system memory 28 by way of example, and can include an operating system, one or more application programs, other program modules, and program data, each of which

[0164] Computer device 6 can also communicate with one or more external devices 14 such as a keyboard, a pointing device, a display 24, etc.; one or more devices that enable a user to interact with computer device 6; and / or one or more devices that enable computer device 6 to communicate with one or more other computing devices. Such communication can be via input / output (I / O) interfaces 22. Still yet, computer device 6 can communicate with one or more networks (such as a local area network (LAN), a wide area network (WAN), and / or the Internet) through network adapter 20. As Figure 6 illustrated, network adapter 20 can communicate with the other components of computer device 6 via bus 18. It should be understood that although not shown, other hardware and / or software components could be used in conjunction with computer device 6. Examples, include, but are not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data archival storage systems, etc. Figure 6

[0165] ​The processing unit 16 executes various function applications and page displays by running programs stored in the system memory 28, such as implementing the image optimization method provided by the embodiments of the present application, which comprises:

[0166] determining a predicted classification of the noisy image based on a pre-trained network, determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image, wherein the pre-trained network is used for image classification;

[0167] performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss, and determining a loss function according to the classification loss and the regularization loss;

[0168] performing optimization processing on the noisy image, calculating a function value of the loss function according to the processed noisy image, and determining the processed noisy image as an optimized image when the function value of the loss function meets a preset condition.

[0169] Of course, those skilled in the art can understand that the processor can also implement the technical solutions of the image optimization method provided by any of the embodiments of the present application.

[0170] The embodiments of the present application also provide a computer readable storage medium having a computer program stored thereon, which is executed by a processor to implement, for example, the image optimization method provided by the embodiments of the present application, which comprises:

[0171] determining a predicted classification of the noisy image based on a pre-trained network, determining a classification loss of the noisy image according to the predicted classification and a classification label set for the noisy image, wherein the pre-trained network is used for image classification;

[0172] performing distortion constraint and regularization processing on the noisy image to obtain a regularization loss, and determining a loss function according to the classification loss and the regularization loss;

[0173] performing optimization processing on the noisy image, calculating a function value of the loss function according to the processed noisy image, and determining the processed noisy image as an optimized image when the function value of the loss function meets a preset condition.

[0174] The computer storage medium of the embodiments of the present application can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium may, for example, but is not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or apparatus, or any combination thereof. More specific examples (non-exhaustive list) of the computer-readable storage medium include an electrical connection having one or more wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in connection with an instruction execution system, apparatus or device.

[0175] The computer-readable signal medium can include a data signal propagated in baseband or propagated as a carrier wave, in which the computer-readable program code is contained. Such propagated data signal can take a variety of forms, including but not limited to electro-magnetic, optical or any suitable combination thereof. The computer-readable signal medium can also be any computer-readable medium that is not a storage medium and that can be used to carry or propagate program code that is used by or in connection with an instruction execution system, apparatus or device.

[0176] The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to wireless, wire line, optical fiber cable, RF, etc., or any suitable combination of the above.

[0177] The computer program code for carrying out operations of the present application can be written in one or more programming languages or combinations of languages including object oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer, or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0178] Those of ordinary skill in the art should understand that the modules or steps of the present application described above can be implemented by a general computing device, which can be centralized on a single computing device or distributed on a network composed of multiple computing devices. Alternatively, the modules or steps can be implemented by computer-executable program codes, which can be stored in a storage device and executed by a computing device, or implemented by individual integrated circuit modules, or implemented by multiple modules or steps in a single integrated circuit module. Thus, the present application is not limited to any particular combination of hardware and software.

[0179] Note that the above only describes the preferred embodiments of the present application and the principles of the applied technology. Those of ordinary skill in the art will understand that the present application is not limited to the specific embodiments described herein, and that various obvious changes, reconfigurations and substitutions can be made without departing from the scope of the present application. Therefore, although the present application has been described in detail through the above embodiments, the present application is not limited to the above embodiments, and can include more other equivalent embodiments without departing from the concept of the present application, and the scope of the present application is determined by the appended claims.

Claims

1. An image optimization method, characterized in that, include: A predicted classification of a noisy image is determined based on a pre-trained network, and a classification loss of the noisy image is determined based on the predicted classification and a classification label set for the noisy image; wherein, the pre-trained network is used for image classification; A regularization loss function for the noisy image is determined, and a distortion constraint result is determined based on the regularization loss function. Input statistics and stored statistics of the image features of the noisy image are determined based on the pre-trained network, and a regularization processing result is determined based on the input statistics and the stored statistics. The input statistics include the mean and / or variance of the image features of each image in the batch to which the noisy image belongs, and the stored statistics include the mean and / or variance of the image features stored in the pre-trained network. The sum of the distortion constraint result and the regularization processing result is determined as the regularization loss. The loss function is determined based on the classification loss and the regularization loss; The noisy image is optimized by calculating the value of the loss function based on the optimized image, and when the value of the loss function meets a preset condition, the optimized image is determined.

2. The image optimization method according to claim 1, characterized in that, Before determining the predicted classification of noisy images based on pre-trained networks, the following steps are also included: A network model for image classification is constructed, and the network model is pre-trained to obtain the pre-trained network.

3. The image optimization method according to claim 1, characterized in that, Determining the classification loss of the noisy image based on the predicted classification and the classification label set for the noisy image includes: Determine the cross-entropy loss function for the classification label and the predicted classification, and use the cross-entropy loss function as the classification loss.

4. The image optimization method according to claim 1, characterized in that, Before determining the loss function based on the classification loss and the regularization loss, the method further includes: The diversification loss of the noisy image is determined based on the simplest positive sample, the most difficult positive sample, and the most difficult negative sample of the noisy image. The simplest sample is the image with the smallest feature Euclidean distance to the noisy image among the noisy images in the same batch. The most difficult positive sample is the image with the largest feature Euclidean distance to the noisy image among the noisy images of the same class in the same batch. The most difficult negative sample is the image with the largest feature Euclidean distance to the noisy image among the noisy images of a different class in the same batch. Accordingly, the loss function is determined based on the classification loss and the regularization loss, including: The classification loss, the regularization loss, and the diversification loss are summed, and the summation result is determined as the loss function.

5. The image optimization method according to claim 4, characterized in that, The diversity loss of the noisy image is determined based on the simplest positive sample, the most difficult positive sample, and the most difficult negative sample of the noisy image, including: Determine the simplest positive sample, the most difficult positive sample, and the most difficult negative sample of the noisy image; A first diversification loss is determined based on the noisy image and the simplest positive sample; a second diversification loss is determined based on the noisy image, the most difficult positive sample, and the most difficult negative sample. The diversification loss is determined by the first diversification loss and the second diversification loss.

6. The image optimization method according to claim 5, characterized in that, Determining the simplest positive sample, the most difficult positive sample, and the most difficult negative sample of the noisy image includes: Determine the feature Euclidean distance between the image features of the noisy image and the image features of other images in the same batch of noisy images; The image with the smallest feature Euclidean distance is determined as the simplest positive sample, the image of the same class with the largest feature Euclidean distance is determined as the most difficult positive sample, and the image of a different class with the smallest feature Euclidean distance is determined as the most difficult negative sample.

7. The image optimization method according to claim 5, characterized in that, The first diversification loss is determined based on the noisy image and the simplest positive sample; The second diversification loss is determined based on the noisy image, the most difficult positive sample, and the most difficult negative sample, including: Determine the first feature Euclidean distance between the noisy image and the simplest positive sample, and determine the first diversification loss based on the first feature Euclidean distance; Determine the second feature Euclidean distance between the noisy image and the most difficult positive sample, and the third feature Euclidean distance between the noisy image and the most difficult negative sample, and determine the second diversification loss based on the second feature Euclidean distance and the third feature Euclidean distance.

8. The image optimization method according to claim 1, characterized in that, When the value of the loss function meets a preset condition, the processed noisy image is determined as the optimized image, including: When the function value of the loss function converges, the processed noisy image is determined as the optimized image.

9. The image optimization method according to claim 2, characterized in that, Constructing a network model for image classification and pre-training the network model to obtain the pre-trained network includes: The network model for image classification is constructed based on a convolutional neural network; The network model was pre-trained using a public dataset, and a pre-loss function was calculated. The network is optimized using the backpropagation algorithm until the pre-loss function converges, thus obtaining the pre-trained network.

10. An image optimization device, characterized in that, include: A determination module is used to determine the predicted classification of a noisy image based on a pre-trained network, and to determine the classification loss of the noisy image based on the predicted classification and the classification label set for the noisy image; wherein, the pre-trained network is used for image classification; A processing module is configured to determine a regularization loss function for the noisy image, determine a distortion constraint result based on the regularization loss function, determine input statistics and stored statistics of the image features of the noisy image based on the pre-trained network, determine a regularization processing result based on the input statistics and the stored statistics, wherein the input statistics include the mean and / or variance of the image features of each image in the batch of noisy images to which the noisy image belongs, and the stored statistics include the mean and / or variance of the image features stored in the pre-trained network; and determine the regularization loss as the sum of the distortion constraint result and the regularization processing result. The execution module is used to determine the loss function based on the classification loss and the regularization loss; An optimization module is used to optimize the noisy image, calculate the function value of the loss function based on the processed noisy image, and determine the processed noisy image as an optimized image when the function value of the loss function meets a preset condition.

11. A computer device, characterized in that, The device includes: At least one processor; and a memory communicatively connected to said at least one processor; The memory stores a computer program that can be executed by the at least one processor, which is then executed by the at least one processor to enable the at least one processor to perform the image optimization method according to any one of claims 1-9.

12. A computer-readable storage medium having a computer program stored thereon, characterized in that, When executed by the processor, the program implements the image optimization method as described in any one of claims 1-9.

13. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the image optimization method according to any one of claims 1-9.

Images