Image processing method, device and storage medium

Abstract

The present disclosure relates to an image processing method, device and storage medium, and belongs to the field of image processing. The method comprises: acquiring a to-be-processed image; compressing the to-be-processed image to obtain a compressed image; generating a plurality of prediction tensors of the compressed image, wherein the plurality of prediction tensors comprise a slope tensor and an intercept tensor corresponding to each channel of three primary color channels of the compressed image; enlarging each prediction tensor to obtain a plurality of target tensors, so that the number of pixels corresponding to each target tensor is the same as the number of pixels of the to-be-processed image, and the plurality of target tensors comprise a target slope tensor and a target intercept tensor of each channel of the three primary color channels; and generating a de-shadowing image of the to-be-processed image according to each target slope tensor, each target intercept tensor and the to-be-processed image.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the field of image processing, and particularly to the field of image shadow removal. BACKGROUND

[0002] In the related art, traditional shadow removal methods use prior information to remove shadows, such as gradient information, brightness information, and region information. In the face of complex shadow shapes and lighting conditions, the robustness of traditional algorithms is insufficient, and satisfactory results cannot be achieved. With the development of deep learning technology, many scholars use deep learning technology to remove shadows and have achieved good results. For example, scholars propose an ST-CGAN network that combines two stacked conditional generative adversarial networks for shadow detection and shadow removal.

[0003] However, in the related art, most deep learning-based shadow removal methods are picture-to-picture processes. Since the pixels of pictures taken by cameras nowadays are large, the calculation amount is large and the time consumption is long, making it difficult to deploy to terminals with small computing power. SUMMARY

[0004] To overcome the problems in the related art, the present disclosure provides an image processing method, device, and storage medium.

[0005] According to a first aspect of an embodiment of the present disclosure, an image processing method is provided, comprising:

[0006] obtaining a to-be-processed image;

[0007] compressing the to-be-processed image to obtain a compressed image;

[0008] generating a plurality of prediction tensors of the compressed image, the plurality of prediction tensors comprising a slope tensor and an intercept tensor corresponding to each channel of three primary color channels of the compressed image;

[0009] enlarging each of the prediction tensors to obtain a plurality of target tensors, so that each of the target tensors corresponds to the same number of pixels as the to-be-processed image, the plurality of target tensors comprising a target slope tensor and a target intercept tensor of each of the three primary color channels;

[0010] generating a shadow-removed image of the to-be-processed image according to the target slope tensors, the target intercept tensors, and the to-be-processed image.

[0011] Optionally, the generating the shadow-removed image of the to-be-processed image according to the target slope tensors, the target intercept tensors, and the to-be-processed image comprises:

[0012] multiplying the pixel value of the red channel of the pixel point by the target slope tensor corresponding to the red channel, and adding the target intercept tensor corresponding to the red channel, to obtain the pixel value of the red channel of the pixel point in the shadow-removed image; and

[0013] multiplying the pixel value of the green channel of the pixel point by the target slope tensor corresponding to the green channel, and adding the target intercept tensor corresponding to the green channel, to obtain the pixel value of the green channel of the pixel point in the shadow-removed image; and

[0014] multiplying the pixel value of the blue channel of the pixel point by the target slope tensor corresponding to the blue channel, and adding the target intercept tensor corresponding to the blue channel, to obtain the pixel value of the blue channel of the pixel point in the shadow-removed image.

[0015] Optionally, the method further comprises:

[0016] generating a plurality of prediction tensors of the shadow-removed image;

[0017] generating an optimized image of the shadow-removed image without boundary residues according to the plurality of prediction tensors and the image to be processed.

[0018] Optionally, the plurality of prediction tensors further comprises a mask tensor corresponding to a shadow mask of the compressed image, and the plurality of target tensors further comprises a target mask tensor.

[0019] Before the step of generating the plurality of prediction tensors of the shadow-removed image, the method further comprises:

[0020] determining a brightness value of a shadow of the image to be processed according to the image to be processed and the target mask tensor; and

[0021] determining that the brightness value is greater than a preset threshold.

[0022] Optionally, the step of generating the plurality of prediction tensors of the compressed image is performed by a first generation model, and training of the first generation model comprises:

[0023] compressing a sample image to obtain a sample compressed image, wherein the sample image is an image obtained by adding a preset shadow image to a non-shadow image;

[0024] generating a plurality of sample tensors of the sample compressed image by the first generation model;

[0025] enlarging each of the sample tensors to obtain a plurality of target sample tensors;

[0026] After obtaining a de-shadow sample image of the sample image based on the plurality of target sample tensors, a first loss value is calculated according to a target mask sample tensor in the plurality of target sample tensors and the preset shadow image, and a second loss value is calculated according to the shadow-free image and the de-shadow sample image, and the first loss value and the second loss value are used to update parameters of the first generation model.

[0027] Optionally, the step of generating the plurality of predicted tensors of the de-shadow image is performed by a second generation model, and the method further includes:

[0028] After obtaining a de-shadow sample image of the sample image based on the plurality of target sample tensors, a plurality of optimized sample tensors of the de-shadow sample image are generated by the second generation model;

[0029] According to the plurality of optimized sample tensors and the de-shadow sample image, a predicted optimized sample image is generated;

[0030] A third loss value is calculated according to the optimized sample image and the shadow-free image, and the third loss value is used to update parameters of the second generation model and / or the first generation model.

[0031] Optionally, the method further includes:

[0032] After the predicted optimized sample image is generated, the optimized sample image is input into a discriminant model to obtain a discriminant value, wherein the discriminant value is used to represent a degree of authenticity of the optimized sample image.

[0033] A fourth loss value is calculated according to the discriminant value, and the fourth loss value is used to update parameters of the second generation model and / or the first generation model.

[0034] Optionally, the discriminant model includes a first discriminant model and a second discriminant model, a first discriminant value output by the first discriminant model is a scalar used to represent a degree of authenticity of global features of the optimized sample image, and a second discriminant value output by the second discriminant model is a tensor used to represent a degree of authenticity of a plurality of local features of the optimized sample image.

[0035] According to a second aspect of the embodiments of the present disclosure, an image processing apparatus is provided, including:

[0036] An acquisition module configured to acquire a to-be-processed image;

[0037] A compression module configured to compress the to-be-processed image to obtain a compressed image;

[0038] a first generating module configured to generate a plurality of prediction tensors of the compressed image, the plurality of prediction tensors comprising a slope tensor and an intercept tensor corresponding to each of three primary color channels of the compressed image;

[0039] an amplifying module configured to amplify each of the prediction tensors to obtain a plurality of target tensors, so that each of the target tensors corresponds to a same number of pixels as the to-be-processed image, the plurality of target tensors comprising a target slope tensor and a target intercept tensor of each of the three primary color channels;

[0040] a second generating module configured to generate a de-shading image of the to-be-processed image according to the target slope tensors, the target intercept tensors, and the to-be-processed image.

[0041] According to a third aspect of embodiments of the present disclosure, an image processing apparatus is provided, comprising:

[0042] a processor;

[0043] a memory for storing processor-executable instructions;

[0044] wherein the processor is configured to:

[0045] obtain a to-be-processed image;

[0046] compress the to-be-processed image to obtain a compressed image;

[0047] generate a plurality of prediction tensors of the compressed image, the plurality of prediction tensors comprising a slope tensor and an intercept tensor corresponding to each of three primary color channels of the compressed image;

[0048] amplify each of the prediction tensors to obtain a plurality of target tensors, so that each of the target tensors corresponds to a same number of pixels as the to-be-processed image, the plurality of target tensors comprising a target slope tensor and a target intercept tensor of each of the three primary color channels;

[0049] generate a de-shading image of the to-be-processed image according to the target slope tensors, the target intercept tensors, and the to-be-processed image.

[0050] According to a fourth aspect of embodiments of the present disclosure, a computer-readable storage medium is provided, which stores computer program instructions, the program instructions being executed by a processor to implement the steps of the image processing method provided in the first aspect of the present disclosure.

[0051] The technical scheme provided by the embodiment of the present disclosure can have the following beneficial effects: by compressing the to-be-processed image, using a smaller image input to obtain a prediction tensor, and then processing the to-be-processed image after enlarging the prediction tensor, a clear shadow-removed image with the same size as the to-be-processed image can be obtained. Since the amount of data input into the model is small, the calculation amount of the model is effectively reduced, and since the shadow in the image has a uniform characteristic, the shadow-removed image obtained by processing the to-be-processed image after enlarging the prediction tensor obtained by predicting the smaller input still has a good effect, so that the calculation amount can be reduced while ensuring the quality of the processed image, and the method can be deployed on a terminal with low computing power.

[0052] It should be understood that the foregoing general description and the following detailed description are only exemplary and explanatory, and are not limiting to the present disclosure. BRIEF DESCRIPTION OF DRAWINGS

[0053] The accompanying drawings, which are incorporated into and form part of the specification, illustrate an embodiment consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0054] Figure 1 is a flowchart of an image processing method according to an exemplary embodiment.

[0055] Figure 2 is a schematic diagram of an image processing network according to an exemplary embodiment.

[0056] Figure 3 is a flowchart of a training method of an image processing network according to an exemplary embodiment.

[0057] Figure 4 is another flowchart of an image processing method according to an exemplary embodiment.

[0058] Figure 5 is a block diagram of an image device according to an exemplary embodiment.

[0059] Figure 6 is a block diagram of a device according to an exemplary embodiment.

[0060] Figure 7 is a block diagram of a device according to an exemplary embodiment. DETAILED DESCRIPTION

[0061] The exemplary embodiments will be described in detail below with reference to the drawings. The following description is only exemplary and is not intended to limit the present disclosure, as described in the appended claims, in all aspects. Rather, they are merely examples of apparatuses and methods consistent with some aspects of the present disclosure, as described in the appended claims.

[0062] Figure 1 An image processing method according to an exemplary embodiment is shown, which can be applied to electronic devices such as mobile phones, computers, etc., and can also be applied to servers. The method comprises the steps of:

[0063] S101, obtaining a to-be-processed image.

[0064] S102, compressing the to-be-processed image to obtain a compressed image.

[0065] Specifically, for example, the pixel size of the to-be-processed image is 1000*1000, and after the processing of step S102, a compressed image with a pixel size of 500*500 can be obtained. In the present disclosure, the compression ratio of the to-be-processed image is not limited, for example, it can be 1:0.5, or 1:1, i.e. no compression is performed on the to-be-processed image.

[0066] S103, generating a plurality of prediction tensors of the compressed image, the plurality of prediction tensors comprising a slope tensor and an intercept tensor corresponding to each channel of the three primary color channels of the compressed image.

[0067] Each of the prediction tensors can be represented as a one-dimensional matrix, and each element in the matrix can correspond to a pixel. In some possible implementations, the plurality of prediction tensors can be combined into a multi-dimensional tensor, and the slope tensor or the intercept tensor can be regarded as a dimension in the tensor.

[0068] It can be understood that for any pixel value x of any color channel of any pixel of the compressed image, a slope a and an intercept b can be predicted. Combining the corresponding slope a and the intercept b of each color channel of each pixel of the compressed image, a and b are the lighting parameters of the pixel, and the slope tensor and the intercept tensor of each channel of the three primary color channels of the compressed image can be obtained.

[0069] S104, magnifying each of the prediction tensors to obtain a plurality of target tensors, so that each of the target tensors corresponds to the same number of pixels as the to-be-processed image, and the plurality of target tensors comprise target slope tensors and target intercept tensors of each channel of the three primary color channels.

[0070] The magnification of the predicted tensor in step S104 may be, for example, magnifying a 500*500 matrix to a 1000*1000 matrix, so that each element in the magnified target tensor can correspond to each pixel in the image to be processed, that is, the target slope tensor and the target intercept tensor of each channel in the three primary color channels corresponding to the image to be processed are obtained. The algorithm for magnifying the predicted tensor is not limited in the present disclosure, and may be, for example, nearest neighbor, bilinear, bicubic, pixel region relationship-based, or Lanczos interpolation algorithm.

[0071] S105, generating a shadow-removed image of the image to be processed according to the target slope tensors, the target intercept tensors, and the image to be processed.

[0072] In the embodiments of the present disclosure, by compressing the image to be processed, using a smaller image input, predicting a predicted tensor, and then magnifying the predicted tensor to process the image to be processed, a clear shadow-removed image with the same size as the image to be processed can be obtained. Since the amount of data input into the model is small, the computational load of the model is effectively reduced. Moreover, since the shadows in the image have uniform characteristics, the shadow-removed image obtained by processing the image to be processed using the magnified predicted tensor obtained from the smaller input still has good results, so that the quality of the processed image can be ensured while the computational load is reduced, and thus the model can be deployed on a terminal with low computational load.

[0073] In some optional embodiments, the generating a shadow-removed image of the image to be processed according to the target slope tensors, the target intercept tensors, and the image to be processed in step S105 includes:

[0074] For a pixel in the image to be processed, the pixel value of the red channel of the pixel is multiplied by the target slope tensor corresponding to the red channel, and then the target intercept tensor corresponding to the red channel is added to obtain the pixel value of the red channel of the pixel in the shadow-removed image; and,

[0075] For a pixel in the image to be processed, the pixel value of the red channel of the pixel is multiplied by the target slope tensor corresponding to the red channel, and then the target intercept tensor corresponding to the red channel is added to obtain the pixel value of the red channel of the pixel in the shadow-removed image; and,

[0076] For a pixel in the image to be processed, the pixel value of the red channel of the pixel is multiplied by the target slope tensor corresponding to the red channel, and then the target intercept tensor corresponding to the red channel is added to obtain the pixel value of the red channel of the pixel in the shadow-removed image; and,

[0077] For example, a pixel point A of a to-be-processed image of a three-channel three-primary color image can be represented as (R, G, B), where R represents a pixel value of a red channel of the pixel point A, G represents a pixel value of a green channel of the pixel point A, and B represents a pixel value of a blue channel of the pixel point A. The value of an element in the target slope tensor corresponding to the red channel of the pixel point A is ar, and the value of an element in the target intercept tensor corresponding to the red channel of the pixel point A is br; the value of an element in the target slope tensor corresponding to the green channel of the pixel point A is ag, and the value of an element in the target intercept tensor corresponding to the green channel of the pixel point A is bg; the value of an element in the target slope tensor corresponding to the blue channel of the pixel point A is ab, and the value of an element in the target intercept tensor corresponding to the blue channel of the pixel point A is bb. Thus, the pixel values of the three primary color channels of the pixel point A in the shadow-removed image corresponding to the to-be-processed image are obtained, that is, (ar*R+br, ag*G+bg, ab*B+bb), and the pixel values of each channel of each pixel point in the shadow-removed image are obtained.

[0078] By predicting the target slope tensor and the target intercept tensor and performing linear operation on the to-be-processed image after the prediction to obtain the shadow-removed image, the pixel values of each channel of each pixel point of the shadow-removed image with a large pixel size do not need to be predicted through a neural network, the calculation amount is effectively reduced, the applicability is improved, and the shadow-removed image can be deployed on a terminal with low computing capability.

[0079] In some other optional embodiments, the method further includes:

[0080] generating a plurality of predicted tensors of the shadow-removed image;

[0081] generating an optimized image in which boundary residues are removed from the shadow-removed image according to the plurality of predicted tensors and the to-be-processed image.

[0082] It can be understood that the shadow-removed image obtained after the processing of steps S101-S105 can still have some boundary residues at a boundary part of a shadow region and a non-shadow region.

[0083] By designing a model and taking the generated shadow-removed image as an input of the model to process the shadow-removed image, the boundary residues at the boundary part of the shadow region and the non-shadow region of the shadow-removed image are eliminated, and the obtained optimized image is more realistic.

[0084] In some other optional embodiments, the plurality of predicted tensors further include a mask tensor corresponding to a shadow mask of the compressed image, and the plurality of target tensors further include a target mask tensor.

[0085] Before the plurality of predicted tensors of the shadow-removed image are generated, the method further includes:

[0086] determine a brightness value of the shadow of the to-be-processed image according to the to-be-processed image and the target mask tensor; and

[0087] determine that the brightness value is greater than a preset threshold.

[0088] The mask tensor corresponding to the shadow mask of the compressed image can be a one-dimensional matrix composed of 0 and 1, and the area with an element value of 1 in the matrix can be understood as the area of the shadow in the compressed image, and the area with an element value of 0 can be regarded as the area without the shadow in the compressed image. The target mask tensor is the same.

[0089] Those skilled in the art should understand that in the area of a particularly dense shadow, the pixel values of each channel tend to be zero, and the image has lost the original color and texture information, and after processing, the texture and color may not be real.

[0090] By predicting the mask tensor corresponding to the compressed image, the target mask tensor corresponding to the to-be-processed image is obtained, it can be determined that there is a shadow in the to-be-processed image, and then according to the to-be-processed image and the target mask tensor, the brightness value of the shadow part in the to-be-processed image can be determined, and whether to process the shadow part is determined according to the brightness value, thereby avoiding the negative effect of image distortion caused by the low brightness value after processing.

[0091] In some embodiments, the step of generating a plurality of prediction tensors of the compressed image is performed by a first generation model, and the training of the first generation model comprises:

[0092] compressing a sample image to obtain a sample compressed image, wherein the sample image is an image obtained by adding a preset shadow image to a non-shadow image;

[0093] generating a plurality of sample tensors of the sample compressed image by using the first generation model;

[0094] enlarging each sample tensor to obtain a plurality of target sample tensors;

[0095] After obtaining a de-shadow sample image of the sample image based on the plurality of target sample tensors, a first loss value is calculated according to a target mask sample tensor in the plurality of target sample tensors and the preset shadow image, and a second loss value is calculated according to the non-shadow image and the de-shadow sample image, and the first loss value and the second loss value are used to update the parameters of the first generation model.

[0096] The first loss value and the second loss value can be L1 loss. It can be understood that, since the compressed image obtained through the compression process of the image and the target sample tensor obtained through the tensor amplification process are calculated based on a preset algorithm, and the target mask sample tensor and the shadow-removed sample image of the sample image are obtained based on linear operations, the parameters of the first generator have low correlation, therefore, whether the sample image is compressed has little effect on the training of the parameters of the first generation model, and thus in some possible implementation manners, the compression ratio of the sample image in the training process of the first generation model can be 1:1, that is, the sample image is not compressed.

[0097] According to the scheme, the sample image is obtained by adding a shadow image to a non-shadow image, and then the first generation model is input, and a loss value is calculated according to the comparison between the non-shadow image and the shadow-removed sample image generated by the first generation model, and a loss value is calculated according to the comparison between the shadow image and the target mask sample tensor generated by the first generation model, and the parameters of the first generation model are adjusted according to the calculated loss value, so that the first generation model can more accurately determine the shadow part of the input image, and improve the ability of shadow removal.

[0098] In yet some alternative implementations, the step of generating the plurality of predicted tensors of the shadow-removed image is performed by a second generation model, and the method further comprises:

[0099] After obtaining the shadow-removed sample image of the sample image based on the plurality of target sample tensors, a plurality of optimized sample tensors of the shadow-removed sample image are generated by the second generation model;

[0100] According to the plurality of optimized sample tensors and the shadow-removed sample image, a predicted optimized sample image is generated;

[0101] A third loss value is calculated according to the optimized sample image and the non-shadow image, and the third loss value is used to update the parameters of the second generation model and / or the first generation model.

[0102] According to the scheme, the shadow-removed sample image generated by the first generation model is input into the second generator to obtain a predicted optimized sample image, and a loss value is calculated based on the predicted optimized sample image and the pre-labeled non-shadow image, and the parameters of the second generation model are adjusted based on the loss value, so that the second generation model can more accurately remove the boundary residues of the shadow-removed image generated by the first generation model, and the optimized image can be more realistic.

[0103] Further, the method further comprises:

[0104] After the predicted optimized sample image is generated, the optimized sample image is input into a discriminant model to obtain a discriminant value, wherein the discriminant value is used to represent the authenticity of the optimized sample image.

[0105] A fourth loss value is calculated according to the discriminant value, and the fourth loss value is used to update the parameters of the second generation model and / or the first generation model.

[0106] The discriminant model can be pre-trained, and the sample image output by the first generation model and the second generation model is discriminated for authenticity, and the parameters of the first generation model and / or the second generation model are adjusted according to the discrimination result, so that the authenticity of the image predicted and generated by the first generation model and / or the second generation model is improved, and a more authentic image without shadows is obtained.

[0107] Further, the discriminant model includes a first discriminant model and a second discriminant model, the first discriminant value output by the first discriminant model is a scalar used to represent the authenticity of the global feature of the optimized sample image, and the second discriminant value output by the second discriminant model is a tensor used to represent the authenticity of the multiple local features of the optimized sample image.

[0108] The first discriminant model can be a CGAN discriminator, and the second discriminant model can be a PatchGAN discriminator. The two discriminators are used to discriminate the authenticity of the image generated by the first generation model and the second generation model, which not only makes the image generated by the generation model more authentic globally, but also removes different types of shadows by predicting the lighting parameters (i.e. slope and intercept) of each pixel point. Therefore, the PatchGAN discriminator is used to judge the authenticity of the local features, which can make the authenticity of each small area of the generated image higher, and the removal effect of different types of shadows better.

[0109] Based on the above inventive concept, the first generation model, the second generation model, the first discriminant model and the second discriminant model can constitute a whole neural network, as shown in Figure 2 Figure 2 is a schematic diagram of an image processing network 20 according to an example embodiment, as shown in Figure 2 The network 20 includes a first generation model G1, a second generation model G2, a first discriminant model D1, and a second discriminant model D2.

[0110] The training of the image processing network 20 can be according to Figure 3 ​The flowchart shown illustrates a training method for an image processing network according to an exemplary embodiment. The execution entity of this method may be a mobile phone, computer, tablet device, or server; this disclosure does not limit this to any particular device. Figure 3 As shown, the method includes the following steps:

[0111] S301. A sample image obtained by adding a preset shadow image to an image without shadow.

[0112] The shadowless image can be as follows: Figure 2 The ground truth shown can be preset to a groundtruth(mask) image.

[0113] S302. Compress the sample image to obtain a compressed sample image.

[0114] in, Figure 2 The training of the network shown is based on a compression ratio of 1:1. The pixel size of the compressed sample image can be the same as that of the sample image, which is W*H.

[0115] S303. Generate seven sample tensors of the compressed image of the sample using the first generative model.

[0116] The sample compressed image is used as input to the first generative model G1, resulting in seven matrices of size W*H.

[0117] S304. Magnify each of the sample tensors to obtain seven target sample tensors.

[0118] In this model, the number of elements in any one of the seven target sample tensors is the same as the number of pixels in the corresponding sample image input to the first generation model, which is W*H. The first three tensors are target slope tensors corresponding to the three primary color channels of the sample image, the fourth to sixth tensors are target intercept tensors corresponding to the three primary color channels of the sample image, and the seventh tensor is the target mask sample tensor. Figure 2 The Predicted mask in [the context of the text].

[0119] S305. Multiply the first three tensors of the seven target sample tensors with the corresponding sample images, and then add them to the fourth to seventh tensors of the seven sample tensors to obtain the shaded sample images of the sample images.

[0120] The shaded sample image is a three-channel, three-primary-color image, which can be used as... Figure 2 Predictedcoarse in the context of [the text].

[0121] S306, generating six optimization sample tensors of the de-shading sample image by the second generation model.

[0122] As shown in Figure 2 , each of the optimization sample tensors can have a size of W*H.

[0123] S307, generating a predicted optimization sample image according to the six optimization sample tensors and the de-shading sample image.

[0124] The specific execution manner of step S307 is similar to that of step S305, and thus will not be described herein again. The optimization sample image can be Predicted refine in Figure 2

[0125] S308, inputting the optimization sample image into the first discrimination model to obtain a first discrimination value.

[0126] The first discrimination value is Real / False information output by the first discrimination model D1. The first discrimination model is a CGAN discriminator, and the first discrimination value is a scalar used to represent the authenticity of the global feature of the optimization sample image.

[0127] S309, inputting the optimization sample image into the second discrimination model to obtain a second discrimination value.

[0128] The second discrimination value is Real / False information output by the second discrimination model D2. The second discrimination model is a PatchGAN discriminator, and the second discrimination value is a tensor used to represent the authenticity of the multiple local features of the optimization sample image.

[0129] S310, calculating a first loss value according to a target mask sample tensor in the seven target sample tensors and a preset shadow image.

[0130] S311, calculating a second loss value according to the shadow-free image and the de-shading sample image.

[0131] S312, calculating a third loss value according to the optimization sample image and the shadow-free image.

[0132] The first loss value, the second loss value, and the third loss value can all be L1 loss.

[0133] S313, calculating a fourth loss value and a fifth loss value according to the first discrimination value and the second discrimination value, respectively.

[0134] S314, back-propagating the first loss value and the second loss value to the first generation model to adjust the parameters of the first generation model.

[0135] ​S315, back-propagating the third loss value, the fourth loss value, and the fifth loss value to the first generation model and the second generation model to adjust parameters of the first generation model and the second generation model.

[0136] S316, when it is determined that the first discriminant value and the second discriminant value both satisfy the preset condition, determining that the image processing network is trained, and fixing the parameters of the first generation model and the second generation model.

[0137] Based on the same inventive concept, Figure 4 is an image processing method according to an example embodiment, which can be performed by the trained image processing network 20 as shown in Figure 2 , and the execution subject of the method is an electronic device with information processing capability, such as Figure 4 , the method comprises the steps of:

[0138] S401, obtaining a to-be-processed image.

[0139] S402, compressing the to-be-processed image to obtain a compressed image.

[0140] S403, generating seven predicted tensors of the compressed image by the first generation model.

[0141] S404, determining a brightness value of a shadow of the to-be-processed image according to a target mask tensor in the seven predicted tensors and the to-be-processed image.

[0142] S405, determining whether the brightness value is greater than a preset threshold.

[0143] In a case where it is determined that the brightness value is greater than the preset threshold, step S406 is performed.

[0144] S406, obtaining a shadow-removed image according to tensors other than the target mask tensor in the seven predicted tensors and the to-be-processed image.

[0145] S407, generating an optimized image of the shadow-removed image by the second generation model.

[0146] In the embodiments of the present disclosure, the to-be-processed image is first compressed and then input into the trained image processing network, so that the image processing network outputs a shadow-removed image with the same size as the to-be-processed image. The image processing network has a small amount of calculation and low deployment difficulty, and can be easily deployed on various electronic devices. In addition, the concentration of the shadow can be determined according to the mask tensor output by the image processing network, so as to avoid the network from having a reverse effect and improve the authenticity of the shadow-removed image.

[0147] The trained image processing network processes the to-be-processed image, first compresses

[0148] based on the same inventive concept, Figure 5 is a block diagram of an image processing apparatus according to an exemplary embodiment. Referring to Figure 5 The apparatus comprises:

[0149] an acquisition module 51 configured to acquire a to-be-processed image;

[0150] a compression module 52 configured to compress the to-be-processed image to obtain a compressed image;

[0151] a first generation module 53 configured to generate a plurality of prediction tensors of the compressed image, the plurality of prediction tensors comprising a slope tensor and an intercept tensor corresponding to each of three primary color channels of the compressed image;

[0152] an amplification module 54 configured to amplify each of the prediction tensors to obtain a plurality of target tensors, so that each of the target tensors corresponds to a same number of pixels as the to-be-processed image, the plurality of target tensors comprising a target slope tensor and a target intercept tensor of each of the three primary color channels;

[0153] a second generation module 55 configured to generate a de-shadowing image of the to-be-processed image according to the target slope tensors, the target intercept tensors, and the to-be-processed image.

[0154] Optionally, the second generation module 55 is specifically configured to:

[0155] for a pixel point of the to-be-processed image, multiply a pixel value of a red channel of the pixel point by a target slope tensor corresponding to the red channel, and add a target intercept tensor corresponding to the red channel to obtain a pixel value of a red channel corresponding to the pixel point in the de-shadowing image; and

[0156] multiply a pixel value of a green channel of the pixel point by a target slope tensor corresponding to the green channel, and add a target intercept tensor corresponding to the green channel to obtain a pixel value of a green channel corresponding to the pixel point in the de-shadowing image; and

[0157] multiply a pixel value of a blue channel of the pixel point by a target slope tensor corresponding to the blue channel, and add a target intercept tensor corresponding to the blue channel to obtain a pixel value of a blue channel corresponding to the pixel point in the de-shadowing image.

[0158] Optionally, the apparatus 50 further comprises:

[0159] a third generation module configured to generate a plurality of prediction tensors of the de-shadowing image;

[0160] an optimization module configured to generate an optimized image removing boundary residues of the shadow-removed image according to the plurality of predicted tensors and the image to be processed.

[0161] Optionally, the plurality of predicted tensors further comprises a mask tensor corresponding to a shadow mask of the compressed image, and the plurality of target tensors further comprises a target mask tensor.

[0162] The apparatus 50 further comprises:

[0163] a first determination module configured to determine a brightness value of a shadow of the image to be processed according to the image to be processed and the target mask tensor.

[0164] a second determination module configured to determine that the brightness value is greater than a preset threshold.

[0165] Optionally, the step of generating the plurality of predicted tensors of the compressed image is performed by a first generation model, and the apparatus 50 further comprises a training module configured to:

[0166] compress a sample image to obtain a sample compressed image, wherein the sample image is an image obtained by adding a preset shadow image to a non-shadow image;

[0167] generate a plurality of sample tensors of the sample compressed image by the first generation model;

[0168] enlarge each of the sample tensors to obtain a plurality of target sample tensors;

[0169] after obtaining a shadow-removed sample image of the sample image based on the plurality of target sample tensors, calculate a first loss value according to a target mask sample tensor in the plurality of target sample tensors and the preset shadow image, and calculate a second loss value according to the non-shadow image and the shadow-removed sample image, and the first loss value and the second loss value are used to update parameters of the first generation model.

[0170] Optionally, the step of generating the plurality of predicted tensors of the shadow-removed image is performed by a second generation model, and the training module is further configured to:

[0171] after obtaining a shadow-removed sample image of the sample image based on the plurality of target sample tensors, generate a plurality of optimized sample tensors of the shadow-removed sample image by the second generation model;

[0172] generate a predicted optimized sample image according to the plurality of optimized sample tensors and the shadow-removed sample image;

[0173] A third loss value is calculated according to the optimized sample image and the shadow-free image, and the third loss value is used to update parameters of the second generation model and / or the first generation model.

[0174] Optionally, the training module is further configured to:

[0175] After the predicted optimized sample image is generated, the optimized sample image is input into a discriminant model to obtain a discriminant value, wherein the discriminant value is used to represent a degree of authenticity of the optimized sample image.

[0176] A fourth loss value is calculated according to the discriminant value, and the fourth loss value is used to update parameters of the second generation model and / or the first generation model.

[0177] Optionally, the training module is further configured to:

[0178] The discriminant model includes a first discriminant model and a second discriminant model, wherein a first discriminant value output by the first discriminant model is a scalar used to represent a degree of authenticity of global features of the optimized sample image, and a second discriminant value output by the second discriminant model is a tensor used to represent a degree of authenticity of multiple local features of the optimized sample image.

[0179] As to the apparatus in the above-mentioned embodiments, the specific manners in which various modules perform operations have been described in detail in the embodiments of the method, and thus will not be described in detail here.

[0180] The present disclosure also provides a computer-readable storage medium having computer program instructions stored thereon, wherein the program instructions are executed by a processor to implement the steps of the image processing method provided by the present disclosure.

[0181] Figure 6 is a block diagram of an image processing apparatus 600 according to an exemplary embodiment. For example, the apparatus 600 can be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, and the like.

[0182] Referring to Figure 6 , the apparatus 600 can include one or more of the following components: a processing component 602, a memory 604, a power supply component 606, a multimedia component 608, an audio component 610, an input / output (I / O) interface 612, a sensor component 614, and a communication component 616.

[0183] The processing component 602 generally controls the overall operations of the device 600, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 602 can include one or more processors 620 to execute instructions and to complete the steps of the methods described above, in whole or in part. Moreover, the processing component 602 can include one or more modules to facilitate the interaction between the processing component 602 and other components. For example, the processing component 602 can include a multimedia module to facilitate the interaction between the multimedia component 608 and the processing component 602.

[0184] The memory 604 is configured to store various types of data to support the operations of the device 600. Examples of such data include instructions for any application or methods operating on the device 600, contact data, phonebook data, messages, pictures, videos, and so on. The memory 604 can be implemented by any type of volatile or non-volatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disks, or optical disks.

[0185] The power component 606 supplies the various components of the device 600 with power. The power component 606 can include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 600.

[0186] The multimedia component 608 includes a screen providing an output interface between the device 600 and a user. In some embodiments, the screen can include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touch, swiping, and gestures on the touch panel. The touch sensors can not only sense a boundary of a touch or swiping action, but also detect duration and pressure associated with the touch or swiping action. In some embodiments, the multimedia component 608 includes a front camera and / or a rear camera. The front and / or rear camera can receive external multimedia data when the device 600 is in an operation mode, such as a shooting mode or a video mode. Each of the front and rear camera can be a fixed optical lens system or have a focal length and optical zoom capability.

[0187] The audio component 610 is configured to output and / or input audio signals. For example, the audio component 610 includes a microphone (MIC) that is configured to receive an external audio signal when the device 600 is in an operation mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory 604 or transmitted via the communication component 616. In some embodiments, the audio component 610 also includes a speaker for outputting audio signals.

[0188] The I / O interface 612 provides an interface between the processing component 602 and peripheral interface modules, which can be a keypad, a click wheel, buttons, and the like. The buttons can include, but are not limited to, a home button, a volume button, a start button, and a lock button.

[0189] The sensor component 614 includes one or more sensors for providing status assessments of various aspects of the device 600. For example, the sensor component 614 can detect an open / closed position of the device 600, relative positioning of components, such as a display and a keypad of the device 600, a change of position of the device 600 or a component of the device 600, presence or absence of user contact with the device 600, changes in orientation or acceleration / deceleration

[0190] The communication component 616 is configured to facilitate wired or wireless communication between the device 600 and other devices. The device 600 can access a wireless network based on a corresponding communication standard, such as WiFi, 2G, or 3G, or a combination thereof. In an example embodiment, the communication component 616 receives broadcast signals or broadcast-related information from external broadcast management systems via a broadcast channel. In an example embodiment, the communication component 616 also includes a Near Field Communication (NFC) module to promote short-range communication. For example, the NFC module can be implemented based on Radio Frequency Identification (RFID) techniques, infrared data association (IrDA) techniques, ultra-wideband (UWB) techniques, Bluetooth (BT) techniques, and other techniques.

[0191] In an exemplary embodiment, the apparatus 600 can be implemented using one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components, for performing the above-described methods.

[0192] In an exemplary embodiment, a non-transitory computer-readable storage medium comprising instructions, such as the memory 604 comprising instructions, is also provided, which can be executed by the processor 620 of the apparatus 600 to complete the above-described methods. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disc, and an optical data storage device, etc.

[0193] In another exemplary embodiment, a computer program product is also provided, which contains a computer program capable of being executed by a programmable apparatus, and the computer program has code portions for executing the above-described image processing method when executed by the programmable apparatus.

[0194] Figure 7 is a block diagram of an image processing apparatus 700 according to an exemplary embodiment. For example, the apparatus 700 can be provided as a server. Referring to Figure 7 , the apparatus 700 includes a processing component 722, which further includes one or more processors, and a memory resource represented by a memory 732, for storing instructions, such as application programs, executable by the processing component 722. The application programs stored in the memory 732 can include one or more than one module each corresponding to a set of instructions. In addition, the processing component 722 is configured to execute the instructions to perform the above-described image processing method.

[0195] The apparatus 700 can also include a power component 726 configured to perform power management of the apparatus 700, a wired or wireless network interface 750 configured to connect the apparatus 700 to a network, and an input / output (I / O) interface 758. The apparatus 700 can operate based on an operating system stored in the memory 732, such as Windows Server TM , Mac OS X TM , Unix TM , Linux TM , FreeBSD TM or the like.

[0196] Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

[0197] It is to be understood that the disclosure is not limited to the precise construction herein described and as shown in the attached drawings, and that various modifications and changes can be made by those skilled in the art without departing from the scope of the disclosure. The scope of the disclosure is limited only by the claims that follow.

Claims

1. An image processing method, characterized in that, include: Obtain the image to be processed; The image to be processed is compressed to obtain a compressed image; The first generation model generates multiple first prediction tensors of the compressed image. The multiple first prediction tensors include illumination parameter tensors corresponding to each channel in the three primary color channels of the compressed image. The three primary color channels include the red channel, the green channel, and the blue channel. Based on the plurality of first prediction tensors, a plurality of target tensors are obtained such that the number of pixels corresponding to each target tensor is the same as the number of pixels in the image to be processed. The plurality of target tensors include the target illumination parameter tensors of each channel in the three primary color channels. Based on the target illumination parameter tensors and the image to be processed, a shadow-removed image of the image to be processed, including the three primary color channels, is generated. The second generation model generates multiple second prediction tensors for the deshaded image. Based on the plurality of second prediction tensors and the image to be processed, an optimized image with the removed shadows and boundary residues is generated. Wherein, the first generative model is obtained by updating the first initial generative model based on the first loss value, the second loss value and / or the third loss value, and the second generative model is obtained by updating the second initial generative model based on the third loss value; The first loss value is calculated based on the target sample tensor and the preset shadow image; the second loss value is calculated based on the shadowless image and the shadowless sample image; and the third loss value is calculated based on the optimized sample image and the shadowless image. The target sample tensor and the deshaded sample image are generated based on the sample image using the first initial generation model. The optimized sample image is generated based on the deshaded sample image using the second initial generation model. The sample image is an image obtained by adding the preset shadow image to the shadowless image.

2. The method according to claim 1, characterized in that, The lighting parameter tensor includes a slope tensor and an intercept tensor, and the target lighting parameter tensor includes a target slope tensor and a target intercept tensor. The step of generating the deshaded image of the image to be processed based on each of the target illumination parameter tensors and the image to be processed includes: For each pixel in the image to be processed, the pixel value of the red channel is multiplied by the target slope tensor corresponding to the red channel, and then added to the target intercept tensor corresponding to the red channel to obtain the pixel value of the red channel in the deshaded image corresponding to that pixel; and, Multiply the pixel value of the green channel of the pixel by the target slope tensor corresponding to the green channel, and then add the target intercept tensor corresponding to the green channel to obtain the pixel value of the green channel of the deshaded image corresponding to that pixel; and, Multiply the pixel value of the blue channel of the pixel by the target slope tensor corresponding to the blue channel, and then add the target intercept tensor corresponding to the blue channel to obtain the pixel value of the blue channel of the deshaded image corresponding to the pixel.

3. The method according to claim 1, characterized in that, The plurality of second prediction tensors also include a mask tensor corresponding to the shadow mask of the compressed image, and the plurality of target tensors also include a target mask tensor; Prior to generating the plurality of second prediction tensors for the deshaded image, the method further includes: The brightness value of the shadow in the image to be processed is determined based on the image to be processed and the target mask tensor; and, The brightness value is determined to be greater than a preset threshold.

4. The method according to claim 3, characterized in that, The method includes: The sample image is compressed to obtain a compressed sample image; Multiple sample tensors of the sample compressed image are generated using the first initial generation model; Based on the multiple sample tensors, multiple target sample tensors are obtained; After obtaining the deshaded sample image based on the plurality of target sample tensors, the first loss value is calculated based on the target mask sample tensor in the plurality of target sample tensors and the preset shadow image, and the second loss value is calculated based on the shadowless image and the deshaded sample image. The parameters of the first initial generative model are updated based on the first loss value and the second loss value to obtain the first generative model.

5. The method according to claim 4, characterized in that, The method further includes: After obtaining the shaded sample image based on the multiple target sample tensors, multiple optimized sample tensors of the shaded sample image are generated through the second initial generation model; Based on the multiple optimized sample tensors and the shaded sample images, a predicted optimized sample image is generated; The third loss value is calculated based on the optimized sample image and the shadowless image; The parameters of the second initial generative model and / or the first generative model are updated based on the third loss value to obtain the second generative model and / or the updated first generative model.

6. The method according to claim 5, characterized in that, The method further includes: After generating the optimized sample image for prediction, the optimized sample image is input into the discriminant model to obtain a discriminant value, wherein the discriminant value is used to characterize the degree of realism of the optimized sample image; Calculate the fourth loss value based on the discriminant value; The parameters of the second generative model and / or the updated first generative model are updated based on the fourth loss value to obtain the updated second generative model and / or the second-updated first generative model.

7. The method according to claim 6, characterized in that, The discriminant model includes a first discriminant model and a second discriminant model. The first discriminant model outputs a first discriminant value, which is a scalar used to characterize the realism of the global features of the optimized sample image. The second discriminant model outputs a second discriminant value, which is a tensor used to characterize the realism of multiple local features of the optimized sample image.

8. An image processing apparatus, characterized in that, include: The acquisition module is configured to acquire the image to be processed. A compression module is configured to compress the image to be processed to obtain a compressed image; The first generation module is configured to generate multiple first prediction tensors of the compressed image through a first generation model. The multiple first prediction tensors include slope tensors and intercept tensors corresponding to each channel in the three primary color channels of the compressed image. The amplification module is configured to amplify each of the first prediction tensors to obtain a plurality of target tensors, such that the number of pixels corresponding to each target tensor is the same as the number of pixels in the image to be processed, wherein the plurality of target tensors include the target slope tensor and the target intercept tensor of each of the three primary color channels. The second generation module is configured to generate a deshaded image of the image to be processed based on each of the target slope tensors, each of the target intercept tensors, and the image to be processed. The third generation module is configured to generate multiple second prediction tensors of the deshaded image using the second generation model; The optimization module is configured to generate an optimized image of the deshaded image with boundary residue removed, based on the plurality of second prediction tensors and the image to be processed; Wherein, the first generative model is obtained by updating the first initial generative model based on the first loss value, the second loss value and / or the third loss value, and the second generative model is obtained by updating the second initial generative model based on the third loss value; The first loss value is calculated based on the target sample tensor and the preset shadow image; the second loss value is calculated based on the shadowless image and the shadowless sample image; and the third loss value is calculated based on the optimized sample image and the shadowless image. The target sample tensor and the deshaded sample image are generated based on the sample image using the first initial generation model. The optimized sample image is generated based on the deshaded sample image using the second initial generation model. The sample image is an image obtained by adding the preset shadow image to the shadowless image.

9. An imaging device, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured as follows: Obtain the image to be processed; The image to be processed is compressed to obtain a compressed image; The first generation model generates multiple first prediction tensors of the compressed image, the multiple first prediction tensors including the slope tensor and the intercept tensor of each channel in the three primary color channels of the compressed image. Each of the first prediction tensors is amplified to obtain multiple target tensors, such that the number of pixels corresponding to each target tensor is the same as the number of pixels in the image to be processed. The multiple target tensors include the target slope tensor and the target intercept tensor of each of the three primary color channels. Based on the target slope tensor, the target intercept tensor, and the image to be processed, a deshaded image of the image to be processed is generated; The second generation model generates multiple second prediction tensors for the deshaded image. Based on the plurality of second prediction tensors and the image to be processed, an optimized image with the removed shadows and boundary residues is generated. Wherein, the first generative model is obtained by updating the first initial generative model based on the first loss value, the second loss value and / or the third loss value, and the second generative model is obtained by updating the second initial generative model based on the third loss value; The first loss value is calculated based on the target sample tensor and the preset shadow image; the second loss value is calculated based on the shadowless image and the shadowless sample image; and the third loss value is calculated based on the optimized sample image and the shadowless image. The target sample tensor and the deshaded sample image are generated based on the sample image using the first initial generation model. The optimized sample image is generated based on the deshaded sample image using the second initial generation model. The sample image is an image obtained by adding the preset shadow image to the shadowless image.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When executed by a processor, the program instructions implement the steps of the method described in any one of claims 1-7.

Images