Shadow elimination method and device, storage medium and program product
By predicting shadow images through shadow prediction models and combining the original image to remove shadows, the shadow removal problem in complex shadows and high-resolution images in the prior art is solved, achieving a more efficient and accurate shadow removal effect.
Patent Information
- Application Number
- CN202310525253.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-10
- Publication Date
- 2024-11-12
AI Technical Summary
The prior art is difficult to effectively remove shadows in complex shadows and high-resolution images, especially complex color images, and the neural network model is computationally large, time-consuming and low accuracy.
By obtaining the original image to be processed, the shadow image is predicted using the shadow prediction model, and the original image is combined with the shadow image to remove shadows, reducing model complexity and calculation volume.
Improves shadow removal effect and efficiency, especially in high resolution and complex shadow images, reducing calculation time and improving accuracy.
Smart Images

Figure CN118941482A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present disclosure relate to the field of computer and network communication technology, and in particular, to a shadow elimination method, device, storage medium, and program product. Background Art
[0002] When users use mobile phones, cameras and other image acquisition devices to take pictures of documents, books, photos, etc., the images taken may have shadows due to the influence of light. In order to obtain an effect similar to that of white text and black text scanned by a professional scanner, it is usually necessary to remove the shadows from the images.
[0003] One related technology is to use low-level features such as color and image gradient to eliminate shadows in images, but it can only process simple shadows, not complex shadows and complex color images; another related technology is to use a neural network model to directly predict shadow-free images by processing shadowed images through the neural network model, that is, the input is an image with shadows, and the output is an image without shadows, but its model is complex, the amount of calculation is large, and it takes a long time, especially for high-resolution images and images with complex shadows, it takes longer and the accuracy is not high. Summary of the invention
[0004] The embodiments of the present disclosure provide a shadow removal method, device, storage medium and program product to improve the shadow removal effect of an image and reduce time consumption.
[0005] In a first aspect, an embodiment of the present disclosure provides a shadow elimination method, comprising:
[0006] Get the original image including the shadow to be processed;
[0007] Acquire a shadow image corresponding to the original image according to the original image and the shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image;
[0008] A shadow-free image corresponding to the original image is obtained according to the original image and the shadow image.
[0009] In a second aspect, an embodiment of the present disclosure provides a shadow removal device, comprising:
[0010] An acquisition unit, used for acquiring an original image containing shadows to be processed;
[0011] A shadow extraction unit, configured to obtain a shadow image corresponding to the original image according to the original image and a shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image;
[0012] The shadow removal unit is used to obtain a shadow-free image corresponding to the original image according to the original image and the shadow image.
[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;
[0014] The memory stores computer-executable instructions;
[0015] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the shadow removal method described in the first aspect and various possible designs of the first aspect.
[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the shadow elimination method described in the first aspect and various possible designs of the first aspect is implemented.
[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including computer execution instructions. When a processor executes the computer execution instructions, the shadow elimination method described in the first aspect and various possible designs of the first aspect is implemented.
[0018] The shadow removal method, device, storage medium and program product provided by the embodiments of the present disclosure obtain an original image to be processed that contains a shadow; obtain a shadow image corresponding to the original image based on the original image and a shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain the image content of the original image; and obtain a shadow-free image corresponding to the original image based on the original image and the shadow image. In the embodiments of the present disclosure, a shadow prediction model is used to predict a shadow image corresponding to the original image, and then the original image and the shadow image are combined to remove the shadow from the original image. There is no need to use a model to directly predict a shadow-free image for the original image, which reduces the complexity of the model, reduces the amount of calculation of the model, reduces the time spent on calculation, and has a better shadow removal effect for images with high resolution and complex shadows. BRIEF DESCRIPTION OF THE DRAWINGS
[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.
[0020] Figure 1 This is an example of removing shadows from an image;
[0021] Figure 2 A flowchart of a shadow elimination method provided by an embodiment of the present disclosure;
[0022] Figure 3 A flowchart of a shadow elimination method provided by another embodiment of the present disclosure;
[0023] Figure 4 A schematic diagram of a model architecture of a shadow elimination method provided in an embodiment of the present disclosure;
[0024] Figure 5 A flowchart of a shadow elimination method provided by another embodiment of the present disclosure;
[0025] Figure 6 A schematic diagram of a preset mapping curve provided in an embodiment of the present disclosure;
[0026] Figure 7 A flowchart of a shadow prediction model training method provided by another embodiment of the present disclosure;
[0027] Figure 8a A schematic diagram of a shadow shape mask provided by an embodiment of the present disclosure;
[0028] Figure 8b A schematic diagram of a shadow shape mask using random transparency provided by an embodiment of the present disclosure;
[0029] Fig. 9 A schematic diagram of multiple shadows provided by an embodiment of the present disclosure;
[0030] Fig.10 A schematic diagram of blurring a real shadow image provided by an embodiment of the present disclosure;
[0031] Fig.11 A schematic diagram of a random color change of a shadow edge provided by an embodiment of the present disclosure;
[0032] Fig.12 A schematic diagram showing the comparison before and after adding Perlin noise to a real shadow image provided by an embodiment of the present disclosure;
[0033] Fig.13 A flowchart of a shadow prediction model training method provided by another embodiment of the present disclosure;
[0034] Fig.14 A schematic diagram of obtaining a second loss provided by an embodiment of the present disclosure;
[0035] Fig.15 A structural block diagram of a shadow elimination device provided in an embodiment of the present disclosure;
[0036] Fig.16 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.
[0038] When users use mobile phones, cameras and other image acquisition devices to take pictures of documents, books, photos, etc., the images taken may have shadows due to the influence of light. In order to obtain the effect of white text and black text similar to that of professional scanners, it is usually necessary to remove the shadows of the images with shadows. Figure 1 shown.
[0039] One related technology is to use low-level features such as color and image gradient to eliminate shadows in images, but it can only process simple shadows, and cannot process complex shadows (for example, irregular blurring of shadow boundaries, complex shadows caused by paper wrinkles, etc.) and complex color images; another related technology is to use a neural network model to directly predict shadow-free images by processing shadowed images with the neural network model, that is, the input is an image with shadows, and the output is an image without shadows, but its model is complex, the amount of calculation is large, and it takes a long time, especially for high-resolution images and images with complex shadows. It takes longer and the accuracy is not high. In addition, the neural network model usually uses rendering technology to synthesize virtual training data sets, which cannot simulate complex real lighting conditions, resulting in inaccurate models and difficult to remove heavy shadow areas.
[0040] In order to solve the above technical problems, the present disclosure provides a shadow removal method, which is provided by obtaining an original image to be processed that contains a shadow; obtaining a shadow image corresponding to the original image according to the original image and a shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain the image content of the original image; and obtaining a shadow-free image corresponding to the original image according to the original image and the shadow image. In the embodiment of the present disclosure, a shadow prediction model is used to predict a shadow image corresponding to the original image, and then the original image and the shadow image are combined to remove the shadow of the original image. There is no need to use a model to directly predict a shadow-free image for the original image, which reduces the complexity of the model, reduces the amount of calculation of the model, reduces the time consumption of calculation, and has a better shadow removal effect for images with high resolution and complex shadows.
[0041] The shadow elimination method disclosed in the present invention may be applicable to a terminal device or a server.
[0042] If the shadow elimination method disclosed in the present invention is applicable to a terminal device, the camera module of the terminal device can be used to shoot to obtain the original image containing shadows to be processed, or an image containing shadows transmitted from other devices can be received, or an image containing shadows pre-stored in a local storage unit can be read as the original image containing shadows to be processed, and then the terminal device can execute a specific shadow elimination method to finally obtain a shadow-free image.
[0043] If the shadow elimination method disclosed in the present invention is applicable to a server, the camera module of a terminal device or other image acquisition device can be used to take a picture, and the original image containing the shadow to be processed can be uploaded to the server. The server executes a specific shadow elimination method, and finally obtains a shadow-free image, which is then sent to the terminal device or other device.
[0044] The shadow elimination method disclosed in the present invention will be described in detail below in conjunction with specific embodiments.
[0045] refer to Figure 2 , Figure 2 A flowchart of a shadow elimination method provided by an embodiment of the present disclosure is provided. The method of this embodiment can be applied in a terminal device or a server, and the shadow elimination method includes:
[0046] S201, obtaining an original image containing shadows to be processed.
[0047] In this embodiment, you can use a mobile phone, camera or other image acquisition device to shoot paper books, documents, photos, etc., or shoot electronic documents and pictures displayed on the display of an electronic device to obtain an image. Due to the influence of lighting, the captured image may have shadows, such as Figure 1 As shown on the left, the original image containing shadows to be processed in this embodiment. Alternatively, an image containing shadows transmitted by other devices or an image containing shadows pre-stored in a local storage unit may be read as the original image containing shadows to be processed in this embodiment.
[0048] S202. Acquire a shadow image corresponding to the original image according to the original image and a shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image.
[0049] In this embodiment, a shadow prediction model can be pre-trained. The input of the shadow prediction model can be any image containing shadows, and the output is an image presented by the shadows contained in the image. The shadow image does not contain image content, and the shadow image can characterize the shape, position, color and other information of the shadow in the image containing the shadow. The shadow prediction model can be any machine learning model that can perform image semantic recognition or image segmentation, such as a neural network model.
[0050] In this embodiment, after obtaining the original image containing shadows to be processed, the original image can be input into the shadow prediction model to obtain the shadow image corresponding to the original image. In this embodiment, the shadow image contained in the original image is predicted by the model, which reduces the complexity of the model, the amount of model calculation, and the time consumption compared to directly predicting the shadow-free image by the model. In particular, it takes a long time to process high-resolution images by directly predicting the shadow-free image by the model. In addition, it is also impossible to handle complex scenes such as heavy shadows and complex colors. However, the shadow image predicted by the shadow prediction model can be applied to complex scenes and high-resolution images, and a shadow image with higher accuracy can be obtained, which takes less time.
[0051] S203: Obtain a shadow-free image corresponding to the original image according to the original image and the shadow image.
[0052] In this embodiment, after obtaining the shadow image, the influence of the shadow image can be eliminated on the basis of the original image to obtain a shadow-free image. Specifically, the original image can be divided by the shadow image, that is, the value of each pixel in the original image is divided by the value of the pixel corresponding to the shadow image, so as to obtain the shadow-free image corresponding to the original image, which can eliminate the shadow boundary cleanly and ensure better color reproduction.
[0053] The shadow removal method of this embodiment obtains an original image to be processed that contains shadows; obtains a shadow image corresponding to the original image based on the original image and a shadow prediction model, wherein the shadow image is an image presented by the shadows in the original image, and the shadow image does not contain the image content of the original image; and obtains a shadow-free image corresponding to the original image based on the original image and the shadow image. In this embodiment, the shadow image corresponding to the original image is predicted by the shadow prediction model, and then the shadow is removed from the original image by combining the original image and the shadow image. There is no need to use a model to directly predict a shadow-free image for the original image, which reduces the complexity of the model, reduces the amount of calculation of the model, reduces the time spent on calculation, and has a better shadow removal effect for images with high resolution and complex shadows.
[0054] On the basis of the above embodiment, the step of acquiring a shadow image corresponding to the original image according to the original image and the shadow prediction model in S202 may specifically include:
[0055] S301, scaling the original image from an original size to a first target size to obtain an intermediate image;
[0056] S302, inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size;
[0057] S303 . Up-sample the shadow image of the first target size to the original size to obtain a shadow image of the original size.
[0058] In this embodiment, in order to further reduce the amount of calculation of the model, Figure 4 As shown, in this embodiment, the original image is scaled (resized) from the original size (resolution) to the first target size to obtain an intermediate image, especially for a larger original image, such as an original image of 1125×1500, it is reduced to 512×512, so that the shadow prediction model only needs to process an intermediate image of a relatively smaller size, thereby reducing the computational complexity of the model. For an original image smaller than the first target size, it is also enlarged to the first target size, so that the shadow prediction model only needs to process images of the same size, which is convenient for deployment and training. The shadow prediction model does not need to support a variety of resolutions, and scaling the size of the original image does not affect the shape of the shadow image. It is only necessary to upsample the shadow image of the first target size output by the shadow prediction model and scale it to the original size to obtain a shadow image of the same size as the original image. In addition, it also avoids the need to prepare training data of different resolutions during the training process of the shadow prediction model in order to ensure that the image quality is not lost at different resolutions.
[0059] It should be noted that since the first target size is fixed, the aspect ratio of the original size of the original image may not be necessarily the same as the aspect ratio of the first target size. In the case of difference, the original image is scaled from the original size to the first target size. The width and height of the original image are not scaled proportionally and does not affect the shape of the shadow image. The final upsampling can also obtain a shadow image of the same size as the original image.
[0060] In the above embodiments, see Figure 4 In the shadow prediction model in S302, the intermediate image is input into the shadow prediction model for processing to obtain a shadow image of the first target size, such as Figure 5 As shown, it may specifically include:
[0061] S401, inputting the intermediate image into the shadow prediction model, performing downsampling layer by layer based on the intermediate image to obtain a feature map of a second target size, and performing global pooling on the feature map of the second target size to obtain global features;
[0062] S402 , up-sampling is performed layer by layer based on the global features, and feature fusion is performed on each up-sampling result and the corresponding down-sampling result, so as to finally obtain a shadow image of the first target size.
[0063] In this embodiment, the intermediate image is first downsampled layer by layer, that is, features are extracted. Each downsampling layer can obtain a downsampling result, that is, a feature map, which is then input to the next layer until a feature map of a second target size is obtained. For example, the second target size is 32×32. Further, global pooling is performed on the feature map of the second target size to obtain global features. The global features are 1×1, which is conducive to learning the global illumination information of the image, so that the shadow prediction model can accurately predict complex shadow images, thereby improving the ability to eliminate complex shadows.
[0064] After extracting the global features, upsampling is performed layer by layer based on the global features. The number of upsampling layers is the same as the number of downsampling layers. The first layer directly upsamples the global features, and the upsampling result is concatenated with the downsampling result of the last layer. The fused result is then upsampled to the second layer, and the upsampling result is concatenated with the downsampling result of the second-to-last layer, and so on, until the first target size is upsampled, that is, the shadow image of the first target size is obtained. For example, Figure 4 The shadow prediction model shown in the figure includes 5 downsampling layers and 5 upsampling layers. The first upsampling layer directly upsamples the global features, and the upsampling result is feature fused with the downsampling result of the fifth downsampling layer, and the feature fusion is input into the second upsampling layer; the second upsampling layer upsamples the feature fusion result of the first upsampling layer, and the upsampling result is feature fused with the downsampling result of the fourth downsampling layer, and the feature fusion is input into the third upsampling layer, and so on. That is, the first upsampling layer corresponds to the fifth downsampling layer, the second upsampling layer corresponds to the fourth downsampling layer, the third upsampling layer corresponds to the third downsampling layer, the fourth upsampling layer corresponds to the second downsampling layer, and the fifth upsampling layer corresponds to the first downsampling layer, and the feature fusion of the upsampling result and the downsampling result is performed.
[0065] In the above embodiment, optionally, the basic structure of the shadow prediction model can be a residual network composed of BasicBlocks, or can be a VGG (Visual Geometry Group) network. Of course, other neural networks can also be used for implementation.
[0066] On the basis of any of the above embodiments, after obtaining the shadow-free image corresponding to the original image, the method further includes:
[0067] The brightness of the shadow-free image is adjusted, and the brightness of pixels in the shadow-free image whose brightness values are greater than a preset threshold is increased, and the brightness of pixels in the shadow-free image whose brightness values are less than the preset threshold is reduced.
[0068] In this embodiment, in order to make the contrast between the text and the background in the shadow-free image more distinct and the text clearer, the brightness of the shadow-free image corresponding to the original image obtained in the above embodiments can be adjusted, wherein the pixels in the shadow-free image whose brightness value is greater than the preset threshold are the background part, and the brightness of the pixels in the background part can be increased, and the pixels in the shadow-free image whose brightness value is less than the preset threshold are the dark font part, and the brightness of the pixels in the font part can be reduced.
[0069] More specifically, in this embodiment, the brightness of each pixel of the shadow-free image can be adjusted according to a preset mapping curve, wherein in the preset mapping curve, if the brightness value before adjustment is greater than a preset threshold, the corresponding brightness value after adjustment is greater than the brightness value before adjustment, and if the brightness value before adjustment is less than the preset threshold, the corresponding brightness value after adjustment is less than the brightness value before adjustment. The preset mapping curve is an S-shaped curve such as Figure 6 As shown, the mapping curve is expressed by mathematical formula as follows:
[0070]
[0071] in, is the brightness value before adjustment (i.e., the brightness value of any pixel in the shadow-free image obtained in S203); and I p is the adjusted brightness value; pow represents the power function; t represents the preset threshold value, which can be selected as 0.96 in this embodiment; gamma represents the curvature of the curve. The larger the gamma value, the greater the curvature, that is, the more concave the part of the curve less than t is, the effect of reducing the brightness is achieved, and the more convex the part greater than t is, the effect of increasing the brightness is achieved. In this embodiment, 2.2 can be selected.
[0072] In the above formula, In the range of t~1, The purpose of dividing by (1-t) is to normalize it to between 0 and 1, and then perform a power function operation. When the power is less than 1, the curve is convex upward, and then multiply it by (1-t), and then add t to restore it to the range of t to 1; In the range of 0 to t, the power of the power function operation is a value greater than 1, and the curve is concave downward.
[0073] Based on any of the above embodiments, this embodiment also provides a method for training a shadow prediction model, such as Figure 7 As shown, the training process is as follows:
[0074] S501, acquiring different real shadow images and different real shadow-free images, wherein the real shadow images are images presented by shadows;
[0075] S502, multiplying any true shadow-free image with any true shadow image to obtain a shadowed image, and using the true shadow-free image, the true shadowed image, and the shadowed image as a set of training data;
[0076] S503: Train the shadow prediction model according to the training data to obtain a trained shadow prediction model.
[0077] In this embodiment, in order to train the shadow prediction model, it is necessary to prepare training data, and the training data includes a real shadow-free image, a real shadow image, and a shadowed image. The real shadow image is an image presented by the shadow, that is, it only presents the shape, position, color and other information of the shadow, but does not include other elements. Different real shadow images and different real shadow-free images can be obtained first, and any real shadow-free image can be multiplied with any real shadow image to obtain a shadowed image, and then the real shadow-free image, the real shadow image and the shadowed image are used as a set of training data. In the above manner, multiple sets of training data can be obtained, and then the shadow prediction model is trained according to the training data to obtain the trained shadow prediction model. It should be noted that the training process can be executed on the same device as the above-mentioned shadow prediction model application process; it can also be executed on different devices, and then the trained shadow prediction model is deployed to the device of the shadow prediction model application process.
[0078] Optionally, when acquiring different real shadow images, the real shadow images may be photographed by a camera; and / or, the real shadow images may be generated by a preset algorithm, thereby reducing the workload of photographing.
[0079] Optionally, generating a real shadow image by a preset algorithm may include at least one of the following methods:
[0080] Method 1: Generate a shadow shape mask and synthesize the shadow shape mask onto the paper texture image with random transparency and random displacement to generate a real shadow image.
[0081] In order to make the shadow shape richer, the shape masks of various objects can be generated by image segmentation algorithm when generating shadow shape masks, such as Figure 8aAs shown in FIG. 1 , as the shadow shape mask of this embodiment, the shadow shape mask is further transformed into a random transparency (such as Figure 8b As shown in FIG. 1 , the paper texture image is synthesized with random displacement to obtain a real shadow image. The paper texture image can be obtained by photographing a blank paper or by other means.
[0082] Method 2: Generate a shadow shape mask, randomly offset the shadow shape mask, and synthesize the shadow shape masks with different offsets onto the paper texture image with random transparency to generate a real shadow image with multiple shadows.
[0083] Considering that there may be multiple light sources in the real environment, thus forming a multi-layer ghosting phenomenon, in order to simulate this phenomenon, in method 2, the shadow shape mask can be randomly offset, and the shadow shape masks with different offsets can be superimposed with random transparency (such as Fig. 9 The three multiple shadows obtained by superimposing the three different transparency values shown in FIG. 1 are synthesized onto the paper texture image to generate a real shadow image with multiple shadows. The acquisition of the shadow shape mask and the paper texture image can be referred to in method 1.
[0084] Based on any of the above embodiments and on the basis of the real shadow image, at least one of the following methods is also included:
[0085] Method 3: Use a blur kernel to blur each pixel of any real shadow image.
[0086] Among them, through blur processing, soft shadows in real lighting environments can be simulated, such as Fig.10 As shown, the synthesized real shadow image is more realistic and the training data is enriched. The blur kernel can be implemented by using Gaussian kernels of different sizes, or other blur kernels can also be used.
[0087] Method 4: Use a Gaussian kernel to convolve the color channels of any real shadow image to make the shadow edge undergo random color changes.
[0088] Among them, considering that there are many rough micro-surfaces on the surface of the object in the real lighting environment, the light is refracted and transmitted on these micro-surfaces. In order to simulate this phenomenon, the color channels of the real shadow image (such as at least one channel in the RGB channel) can be convolved with Gaussian kernels respectively. Different Gaussian kernels can be used for convolution respectively, so that the color of the shadow edge changes randomly, such as Fig.11 As shown in the figure, since the real shadow image is processed, the inner area of the shadow is black (dark) and the outer area of the shadow is white (light), and the color change after processing is not obvious, so only the color change of the shadow edge is obvious.
[0089] Method 5: Generate a random Perlin noise image and synthesize the Perlin noise image onto a real shadow image or a paper texture image.
[0090] Perlin noise is a natural noise. A random Perlin noise image can be generated by using the Perlin noise generation algorithm. The Perlin noise image is synthesized onto a real shadow image or a paper texture image to form light and dark changes in the image space, such as Fig.12 As shown, the shadow image with added Perlin noise has light and dark changes compared to the shadow image without Perlin noise, which is more realistic and natural.
[0091] It should be noted that methods three to five can be further processing of the real shadow image, and any one of the methods can be used alone, or two or three methods can be mixed for processing. Through the above methods, a series of real shadow images with rich forms and more realism can be obtained, the model training effect can be improved, and the model's robustness in processing different shadow situations can be increased, so that the shadow prediction model can be applied to various complex scenes, and can perform good shadow removal on images with shadows of different forms, especially complex scenes such as heavy shadows and colors, and can remove shadow boundaries cleanly while ensuring better color restoration.
[0092] On the basis of the above embodiment, the shadow prediction model is trained according to the training data in S503 to obtain a trained shadow prediction model, such as Fig.13 As shown, including:
[0093] S601. In a first training stage, after each forward propagation, a first loss and a second loss of the shadow prediction model are obtained, and the shadow prediction model is back-propagated according to the first loss and the second loss; wherein the first loss is used to characterize the error between the prediction result of the shadow prediction model and the real image, and the second loss is used to characterize the error of the feature extraction process of the shadow prediction model;
[0094] S602. In a second training stage, the first loss of the shadow prediction model is obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss to obtain a trained shadow prediction model.
[0095] In this embodiment, in order to make the shadow prediction model have a better shadow removal effect on the heavy shadow area, a first loss function is designed to characterize the error between the prediction result of the shadow prediction model and the real image, and a second loss function is designed to characterize the error of the feature extraction process of the shadow prediction model; and a training strategy of two training stages is adopted. In the first training stage, the total loss is obtained according to the first loss and the second loss after each forward propagation, and the shadow prediction model is back-propagated based on the total loss to realize model training, so that the first training stage can optimize the degree of shadow removal of the area background and produce stronger constraints on the large area background; through the first training stage, the shadow prediction model has a certain shadow removal ability for the large area background, and the second loss is removed in the second training stage. After each forward propagation, the shadow prediction model is back-propagated only according to the first loss to achieve fine-tuning of the shadow prediction model, focusing on optimizing heavy shadows, edges and other areas, so that the shadow prediction model can accurately identify the shadow boundary so as to remove the shadow boundary cleanly.
[0096] More specifically, when obtaining the first loss of the shadow prediction model, the predicted shadow image output by the shadow prediction model may be obtained. With the real shadow image I shadow The first error between the groundtruth corresponding to the predicted shadow image and the predicted shadow-free image (The result of dividing the shadow image by the predicted shadow image) and the real shadow-free image I clean (as the groundtruth corresponding to the predicted shadow-free image); and then obtaining the first loss of the shadow prediction model according to the first error and the second error. The specific calculation formula of the first loss is as follows:
[0097]
[0098] Among them, L1(a,b)=|ab|.
[0099] The first loss is equivalent to the L1 loss (mean absolute error), in the area with heavier shadows (i.e. the real shadow image I shadow The smaller the value is, the greater the weight of the loss function is, so that the shadow prediction model has a stronger de-shadowing ability for the heavily shadowed area.
[0100] More specifically, when obtaining the second loss of the shadow prediction model, the predicted shadow-free image and the real shadow-free image can be respectively input into the shadow prediction model, and the downsampling result of the same target layer can be obtained in the layer-by-layer downsampling stage, wherein the target layer is one or more layers in the downsampling stage; according to the error between the downsampling results of the same target layer, the second loss of the shadow prediction model is determined. For example, if there are 5 layers in the downsampling stage, the 1st, 3rd, and 5th layers can be used as the target layers, that is, in the predicted shadow-free image and the true shadow-free image I clean When input into the shadow prediction model respectively, the errors (e.g., L1 loss) between the downsampling results of the 1st, 3rd, and 5th layers are obtained respectively, and the sum of the errors of each target layer is used as the second loss of the shadow prediction model.
[0101] Taking the VGG network as an example, assuming that its downsampling stage has 5 layers, use the 1st, 3rd, and 5th layers as the target layers, such as Fig.14 , then the specific calculation formula of the second loss is as follows:
[0102]
[0103] Among them, vgg1(I clean )、vgg3(I clean )、vgg5(I clean ) are the true shadow-free images I clean The downsampling results (features) of the 1st, 3rd, and 5th layers are input to the shadow prediction model respectively. Similarly, Predict shadow-free images The downsampling results (features) of the 1st, 3rd, and 5th layers are input into the shadow prediction model respectively. The second loss has a stronger constraint on the semantics of the colossus and can produce a stronger constraint on the large area of the background, making the text clearer and the background cleaner after removing the shadow.
[0104] Based on the above embodiment, the first loss Loss1 and the second loss Loss2 are obtained in the first training stage, and the total loss calculation formula is as follows:
[0105] Loss = Loss1 + 0.001 * Loss2
[0106] In the first training stage, the shadow prediction model is trained based on the total loss, so that the first training stage optimizes the degree of de-shadowing of the area background and generates stronger constraints on the large area background;
[0107] In the second training stage, the second loss Loss2 is removed, and the shadow prediction model is fine-tuned only according to the first loss Loss1. That is, the loss calculation formula at this time is:
[0108] Loss = Loss1
[0109] In the second training stage, since the shadow prediction model has a certain ability to remove shadows for large areas of background through the first training stage, the second training stage focuses on optimizing areas such as heavy shadows and edges, so that the shadow prediction model can accurately identify the shadow boundaries so that the shadow boundaries can be removed cleanly.
[0110] Corresponding to the shadow elimination method of the above embodiment, Fig.15 This is a structural block diagram of a shadow removal device provided by an embodiment of the present disclosure. For ease of explanation, only the parts related to the embodiment of the present disclosure are shown. Fig.15 The shadow removal device 700 includes: an acquisition unit 701, a shadow extraction unit 702 and a shadow removal unit 703.
[0111] The acquisition unit 701 is used to acquire the original image containing the shadow to be processed;
[0112] A shadow extraction unit 702 is used to obtain a shadow image corresponding to the original image according to the original image and the shadow prediction model, wherein the shadow image is an image presented by the shadow in the original image, and the shadow image does not contain the image content of the original image;
[0113] The shadow removal unit 703 is configured to obtain a shadow-free image corresponding to the original image according to the original image and the shadow image.
[0114] In one or more embodiments of the present disclosure, when the shadow extraction unit 702 obtains the shadow image corresponding to the original image according to the original image and the shadow prediction model, it is configured to:
[0115] Scaling the original image from an original size to a first target size to obtain an intermediate image;
[0116] Inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size;
[0117] The shadow image of the first target size is upsampled to the original size to obtain a shadow image of the original size.
[0118] In one or more embodiments of the present disclosure, when the shadow extraction unit 702 inputs the intermediate image into the shadow prediction model for processing to obtain a shadow image of the first target size, it is configured to:
[0119] Inputting the intermediate image into the shadow prediction model, downsampling layer by layer based on the intermediate image to obtain a feature map of a second target size, and performing global pooling on the feature map of the second target size to obtain global features;
[0120] Upsampling is performed layer by layer based on the global features, and feature fusion is performed on the upsampling result of each layer with the corresponding downsampling result, so as to finally obtain a shadow image of the first target size.
[0121] In one or more embodiments of the present disclosure, when the shadow removal unit 703 obtains the shadow-free image corresponding to the original image according to the original image and the shadow image, it is configured to:
[0122] The original image is divided by the shadow image to obtain a shadow-free image corresponding to the original image.
[0123] In one or more embodiments of the present disclosure, after obtaining the shadow-free image corresponding to the original image, the shadow removal unit 703 is further configured to:
[0124] The brightness of the shadow-free image is adjusted, and the brightness of pixels in the shadow-free image whose brightness values are greater than a preset threshold is increased, and the brightness of pixels in the shadow-free image whose brightness values are less than the preset threshold is reduced.
[0125] In one or more embodiments of the present disclosure, when the shadow removal unit 703 performs brightness adjustment on the shadow-free image, it is configured to:
[0126] According to a preset mapping curve, the brightness of each pixel of the shadow-free image is adjusted respectively, wherein in the preset mapping curve, if the brightness value before adjustment is greater than a preset threshold, the corresponding brightness value after adjustment is greater than the brightness value before adjustment, and if the brightness value before adjustment is less than the preset threshold, the corresponding brightness value after adjustment is less than the brightness value before adjustment.
[0127] In one or more embodiments of the present disclosure, the device further includes a training unit 704, configured to:
[0128] Acquire different real shadow images and different real shadow-free images; wherein the real shadow images are images presented by shadows;
[0129] Multiply any true shadow-free image with any true shadow image to obtain a shadowed image, and use the true shadow-free image, the true shadowed image, and the shadowed image as a set of training data;
[0130] The shadow prediction model is trained according to the training data to obtain a trained shadow prediction model.
[0131] In one or more embodiments of the present disclosure, when the training unit 704 trains the shadow prediction model according to the training data to obtain the trained shadow prediction model, it is used to:
[0132] In a first training stage, a first loss and a second loss of the shadow prediction model are obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss and the second loss; wherein the first loss is used to characterize the error between the prediction result of the shadow prediction model and the real image, and the second loss is used to characterize the error of the feature extraction process of the shadow prediction model;
[0133] In the second training stage, the first loss of the shadow prediction model is obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss to obtain a trained shadow prediction model.
[0134] In one or more embodiments of the present disclosure, when acquiring the first loss of the shadow prediction model, the training unit 704 is used to:
[0135] Obtaining a first error between a predicted shadow image output by the shadow prediction model and a real shadow image, and a second error between a predicted shadow-free image and a real shadow-free image;
[0136] A first loss of the shadow prediction model is obtained according to the first error and the second error.
[0137] In one or more embodiments of the present disclosure, when acquiring the second loss of the shadow prediction model, the training unit 704 is used to:
[0138] Inputting the predicted shadow-free image and the real shadow-free image into the shadow prediction model respectively, and obtaining the downsampling result of the same target layer in the layer-by-layer downsampling stage, wherein the target layer is one or more layers in the downsampling stage;
[0139] A second loss of the shadow prediction model is determined according to an error between downsampling results of the same target layer.
[0140] In one or more embodiments of the present disclosure, the training unit 704 performs at least one of the following when acquiring different real shadow images:
[0141] Generate a shadow shape mask, and synthesize the shadow shape mask onto the paper texture image with random transparency and random displacement to generate a real shadow image; or
[0142] A shadow shape mask is generated, the shadow shape mask is randomly offset, and the shadow shape masks with different offsets are synthesized onto the paper texture image with random transparency to generate a real shadow image with multiple shadows.
[0143] In one or more embodiments of the present disclosure, when acquiring different real shadow images, the training unit 704 is further configured to:
[0144] Blur each pixel of any true shadow image using a blur kernel; or
[0145] Use a Gaussian kernel to convolve the color channels of any real shadow image to make the shadow edges change randomly; or
[0146] Generate a random Perlin noise image and synthesize the Perlin noise image onto any real shadow image or paper texture image.
[0147] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effect are similar, and this embodiment will not be repeated here.
[0148] refer to Fig.16 , which shows a schematic diagram of the structure of an electronic device 800 suitable for implementing the embodiment of the present disclosure, and the electronic device 800 may be a terminal device or a server. The terminal device may include but is not limited to mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle terminals (such as vehicle navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.16 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.
[0149] like Fig.16As shown, the electronic device 800 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. Various programs and data required for the operation of the electronic device 800 are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0150] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device 800 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.16 The electronic device 800 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.
[0151] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 809, or installed from a storage device 808, or installed from a ROM 802. When the computer program is executed by the processing device 801, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.
[0152] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.
[0153] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.
[0154] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.
[0155] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0156] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.
[0157] The units involved in the embodiments described in the present disclosure may be implemented by software or hardware. The name of a unit does not limit the unit itself in some cases. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses".
[0158] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.
[0159] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0160] It should be noted that the user information (including but not limited to various images, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.
[0161] In a first aspect, according to one or more embodiments of the present disclosure, a shadow removal method is provided, comprising:
[0162] Get the original image including the shadow to be processed;
[0163] Acquire a shadow image corresponding to the original image according to the original image and the shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image;
[0164] A shadow-free image corresponding to the original image is obtained according to the original image and the shadow image.
[0165] According to one or more embodiments of the present disclosure, acquiring a shadow image corresponding to the original image according to the original image and the shadow prediction model includes:
[0166] Scaling the original image from an original size to a first target size to obtain an intermediate image;
[0167] Inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size;
[0168] The shadow image of the first target size is upsampled to the original size to obtain a shadow image of the original size.
[0169] According to one or more embodiments of the present disclosure, the step of inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size includes:
[0170] Inputting the intermediate image into the shadow prediction model, downsampling layer by layer based on the intermediate image to obtain a feature map of a second target size, and performing global pooling on the feature map of the second target size to obtain global features;
[0171] Upsampling is performed layer by layer based on the global features, and feature fusion is performed on the upsampling result of each layer with the corresponding downsampling result, so as to finally obtain a shadow image of the first target size.
[0172] According to one or more embodiments of the present disclosure, obtaining a shadow-free image corresponding to the original image according to the original image and the shadow image includes:
[0173] The original image is divided by the shadow image to obtain a shadow-free image corresponding to the original image.
[0174] According to one or more embodiments of the present disclosure, after obtaining the shadow-free image corresponding to the original image, the method further includes:
[0175] The brightness of the shadow-free image is adjusted, and the brightness of pixels in the shadow-free image whose brightness values are greater than a preset threshold is increased, and the brightness of pixels in the shadow-free image whose brightness values are less than the preset threshold is reduced.
[0176] According to one or more embodiments of the present disclosure, the step of adjusting the brightness of the shadow-free image includes:
[0177] According to a preset mapping curve, the brightness of each pixel of the shadow-free image is adjusted respectively, wherein in the preset mapping curve, if the brightness value before adjustment is greater than a preset threshold, the corresponding brightness value after adjustment is greater than the brightness value before adjustment, and if the brightness value before adjustment is less than the preset threshold, the corresponding brightness value after adjustment is less than the brightness value before adjustment.
[0178] According to one or more embodiments of the present disclosure, the shadow prediction model is trained through the following process:
[0179] Acquire different real shadow images and different real shadow-free images; wherein the real shadow images are images presented by shadows;
[0180] Multiply any true shadow-free image with any true shadow image to obtain a shadowed image, and use the true shadow-free image, the true shadowed image, and the shadowed image as a set of training data;
[0181] The shadow prediction model is trained according to the training data to obtain a trained shadow prediction model.
[0182] According to one or more embodiments of the present disclosure, training the shadow prediction model according to the training data to obtain the trained shadow prediction model includes:
[0183] In a first training stage, a first loss and a second loss of the shadow prediction model are obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss and the second loss; wherein the first loss is used to characterize the error between the prediction result of the shadow prediction model and the real image, and the second loss is used to characterize the error of the feature extraction process of the shadow prediction model;
[0184] In the second training stage, the first loss of the shadow prediction model is obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss to obtain a trained shadow prediction model.
[0185] According to one or more embodiments of the present disclosure, obtaining the first loss of the shadow prediction model includes:
[0186] Obtaining a first error between a predicted shadow image output by the shadow prediction model and a real shadow image, and a second error between a predicted shadow-free image and a real shadow-free image;
[0187] A first loss of the shadow prediction model is obtained according to the first error and the second error.
[0188] According to one or more embodiments of the present disclosure, obtaining the second loss of the shadow prediction model includes:
[0189] Inputting the predicted shadow-free image and the real shadow-free image into the shadow prediction model respectively, and obtaining the downsampling result of the same target layer in the layer-by-layer downsampling stage, wherein the target layer is one or more layers in the downsampling stage;
[0190] A second loss of the shadow prediction model is determined according to an error between downsampling results of the same target layer.
[0191] According to one or more embodiments of the present disclosure, obtaining different real shadow images includes at least one of the following:
[0192] Generate a shadow shape mask, and synthesize the shadow shape mask onto the paper texture image with random transparency and random displacement to generate a real shadow image; or
[0193] A shadow shape mask is generated, the shadow shape mask is randomly offset, and the shadow shape masks with different offsets are synthesized onto the paper texture image with random transparency to generate a real shadow image with multiple shadows.
[0194] According to one or more embodiments of the present disclosure, the step of obtaining a shadow image corresponding to the original image further includes at least one of the following:
[0195] Blur each pixel of any true shadow image using a blur kernel; or
[0196] Use a Gaussian kernel to convolve the color channels of any real shadow image to make the shadow edges change randomly; or
[0197] Generate a random Perlin noise image and synthesize the Perlin noise image onto any real shadow image or paper texture image.
[0198] In a second aspect, according to one or more embodiments of the present disclosure, a shadow removal device is provided, comprising:
[0199] An acquisition unit, used for acquiring an original image containing shadows to be processed;
[0200] A shadow extraction unit, configured to obtain a shadow image corresponding to the original image according to the original image and a shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image;
[0201] The shadow removal unit is used to obtain a shadow-free image corresponding to the original image according to the original image and the shadow image.
[0202] According to one or more embodiments of the present disclosure, when the shadow extraction unit obtains the shadow image corresponding to the original image according to the original image and the shadow prediction model, it is used to:
[0203] Scaling the original image from an original size to a first target size to obtain an intermediate image;
[0204] Inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size;
[0205] The shadow image of the first target size is upsampled to the original size to obtain a shadow image of the original size.
[0206] According to one or more embodiments of the present disclosure, when the shadow extraction unit inputs the intermediate image into the shadow prediction model for processing to obtain a shadow image of the first target size, it is configured to:
[0207] Inputting the intermediate image into the shadow prediction model, downsampling layer by layer based on the intermediate image to obtain a feature map of a second target size, and performing global pooling on the feature map of the second target size to obtain global features;
[0208] Upsampling is performed layer by layer based on the global features, and feature fusion is performed on the upsampling result of each layer with the corresponding downsampling result, so as to finally obtain a shadow image of the first target size.
[0209] According to one or more embodiments of the present disclosure, the shadow removal unit obtains a shadow-free image corresponding to the original image according to the original image and the shadow image, and is used to:
[0210] The original image is divided by the shadow image to obtain a shadow-free image corresponding to the original image.
[0211] According to one or more embodiments of the present disclosure, after obtaining the shadow-free image corresponding to the original image, the shadow removal unit is further configured to:
[0212] The brightness of the shadow-free image is adjusted, and the brightness of pixels in the shadow-free image whose brightness values are greater than a preset threshold is increased, and the brightness of pixels in the shadow-free image whose brightness values are less than the preset threshold is reduced.
[0213] According to one or more embodiments of the present disclosure, when the shadow removal unit performs brightness adjustment on the shadow-free image, it is configured to:
[0214] According to a preset mapping curve, the brightness of each pixel of the shadow-free image is adjusted respectively, wherein in the preset mapping curve, if the brightness value before adjustment is greater than a preset threshold, the corresponding brightness value after adjustment is greater than the brightness value before adjustment, and if the brightness value before adjustment is less than the preset threshold, the corresponding brightness value after adjustment is less than the brightness value before adjustment.
[0215] According to one or more embodiments of the present disclosure, the device further includes a training unit, configured to:
[0216] Acquire different real shadow images and different real shadow-free images; wherein the real shadow images are images presented by shadows;
[0217] Multiply any true shadow-free image with any true shadow image to obtain a shadowed image, and use the true shadow-free image, the true shadowed image, and the shadowed image as a set of training data;
[0218] The shadow prediction model is trained according to the training data to obtain a trained shadow prediction model.
[0219] According to one or more embodiments of the present disclosure, when the training unit trains the shadow prediction model according to the training data to obtain the trained shadow prediction model, it is used to:
[0220] In a first training stage, a first loss and a second loss of the shadow prediction model are obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss and the second loss; wherein the first loss is used to characterize the error between the prediction result of the shadow prediction model and the real image, and the second loss is used to characterize the error of the feature extraction process of the shadow prediction model;
[0221] In the second training stage, the first loss of the shadow prediction model is obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss to obtain a trained shadow prediction model.
[0222] According to one or more embodiments of the present disclosure, when acquiring the first loss of the shadow prediction model, the training unit is used to:
[0223] Obtaining a first error between a predicted shadow image output by the shadow prediction model and a real shadow image, and a second error between a predicted shadow-free image and a real shadow-free image;
[0224] A first loss of the shadow prediction model is obtained according to the first error and the second error.
[0225] According to one or more embodiments of the present disclosure, when acquiring the second loss of the shadow prediction model, the training unit is used to:
[0226] Inputting the predicted shadow-free image and the real shadow-free image into the shadow prediction model respectively, and obtaining the downsampling result of the same target layer in the layer-by-layer downsampling stage, wherein the target layer is one or more layers in the downsampling stage;
[0227] A second loss of the shadow prediction model is determined according to an error between downsampling results of the same target layer.
[0228] According to one or more embodiments of the present disclosure, when acquiring different real shadow images, the training unit performs at least one of the following:
[0229] Generate a shadow shape mask, and synthesize the shadow shape mask onto the paper texture image with random transparency and random displacement to generate a real shadow image; or
[0230] A shadow shape mask is generated, the shadow shape mask is randomly offset, and the shadow shape masks with different offsets are synthesized onto the paper texture image with random transparency to generate a real shadow image with multiple shadows.
[0231] According to one or more embodiments of the present disclosure, when acquiring different real shadow images, the training unit is further configured to perform at least one of the following:
[0232] Blur each pixel of any true shadow image using a blur kernel; or
[0233] Use a Gaussian kernel to convolve the color channels of any real shadow image to make the shadow edges change randomly; or
[0234] Generate a random Perlin noise image and synthesize the Perlin noise image onto any real shadow image or paper texture image.
[0235] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;
[0236] The memory stores computer-executable instructions;
[0237] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the shadow removal method described in the first aspect and various possible designs of the first aspect.
[0238] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the shadow elimination method described in the first aspect and various possible designs of the first aspect is implemented.
[0239] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the shadow removal method as described in the first aspect and various possible designs of the first aspect is implemented.
[0240] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.
[0241] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.
[0242] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims.
Claims
1. A shadow elimination method, characterized in that: include: Get the original image including the shadow to be processed; Acquire a shadow image corresponding to the original image according to the original image and the shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image; A shadow-free image corresponding to the original image is obtained according to the original image and the shadow image.
2. The method according to claim 1, characterized in that The step of acquiring a shadow image corresponding to the original image according to the original image and the shadow prediction model includes: Scaling the original image from an original size to a first target size to obtain an intermediate image; Inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size; The shadow image of the first target size is upsampled to the original size to obtain a shadow image of the original size.
3. The method according to claim 2, characterized in that The step of inputting the intermediate image into the shadow prediction model for processing to obtain a shadow image of a first target size includes: Inputting the intermediate image into the shadow prediction model, downsampling layer by layer based on the intermediate image to obtain a feature map of a second target size, and performing global pooling on the feature map of the second target size to obtain global features; Upsampling is performed layer by layer based on the global features, and feature fusion is performed on the upsampling result of each layer with the corresponding downsampling result, so as to finally obtain a shadow image of the first target size.
4. The method according to claim 1, characterized in that: The step of obtaining a shadow-free image corresponding to the original image according to the original image and the shadow image includes: The original image is divided by the shadow image to obtain a shadow-free image corresponding to the original image.
5. The method according to any one of claims 1 to 4, characterized in that: After obtaining the shadow-free image corresponding to the original image, the method further includes: The brightness of the shadow-free image is adjusted, and the brightness of pixels in the shadow-free image whose brightness values are greater than a preset threshold is increased, and the brightness of pixels in the shadow-free image whose brightness values are less than the preset threshold is reduced.
6. The method according to claim 5, characterized in that The step of adjusting the brightness of the shadow-free image comprises: According to a preset mapping curve, the brightness of each pixel of the shadow-free image is adjusted respectively, wherein in the preset mapping curve, if the brightness value before adjustment is greater than a preset threshold, the corresponding brightness value after adjustment is greater than the brightness value before adjustment, and if the brightness value before adjustment is less than the preset threshold, the corresponding brightness value after adjustment is less than the brightness value before adjustment.
7. The method according to any one of claims 1 to 4, characterized in that: The shadow prediction model is trained through the following process: Acquire different real shadow images and different real shadow-free images; wherein the real shadow images are images presented by shadows; Multiply any true shadow-free image with any true shadow image to obtain a shadowed image, and use the true shadow-free image, the true shadowed image, and the shadowed image as a set of training data; The shadow prediction model is trained according to the training data to obtain a trained shadow prediction model.
8. The method according to claim 7, characterized in that The step of training the shadow prediction model according to the training data to obtain the trained shadow prediction model includes: In a first training stage, a first loss and a second loss of the shadow prediction model are obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss and the second loss; wherein the first loss is used to characterize the error between the prediction result of the shadow prediction model and the real image, and the second loss is used to characterize the error of the feature extraction process of the shadow prediction model; In the second training stage, the first loss of the shadow prediction model is obtained after each forward propagation, and the shadow prediction model is back-propagated according to the first loss to obtain a trained shadow prediction model.
9. The method according to claim 7, characterized in that: The obtaining the first loss of the shadow prediction model includes: Obtaining a first error between a predicted shadow image output by the shadow prediction model and a real shadow image, and a second error between a predicted shadow-free image and a real shadow-free image; A first loss of the shadow prediction model is obtained according to the first error and the second error.
10. The method according to claim 7, characterized in that The obtaining the second loss of the shadow prediction model comprises: Inputting the predicted shadow-free image and the real shadow-free image into the shadow prediction model respectively, and obtaining the downsampling result of the same target layer in the layer-by-layer downsampling stage, wherein the target layer is one or more layers in the downsampling stage; A second loss of the shadow prediction model is determined according to an error between downsampling results of the same target layer.
11. The method according to claim 7, characterized in that The obtaining of different real shadow images includes at least one of the following: Generate a shadow shape mask, and synthesize the shadow shape mask onto the paper texture image with random transparency and random displacement to generate a real shadow image; or A shadow shape mask is generated, the shadow shape mask is randomly offset, and the shadow shape masks with different offsets are synthesized onto the paper texture image with random transparency to generate a real shadow image with multiple shadows.
12. The method according to claim 11, characterized in that The obtaining of different real shadow images further includes at least one of the following: Blur each pixel of any true shadow image using a blur kernel; or Use Gaussian kernel to convolve the color channels of any real shadow image to make the shadow edge change randomly. or Generate a random Perlin noise image and synthesize the Perlin noise image onto any real shadow image or paper texture image.
13. A shadow removal device, characterized in that: include: An acquisition unit, used for acquiring an original image containing shadows to be processed; A shadow extraction unit, configured to obtain a shadow image corresponding to the original image according to the original image and a shadow prediction model, wherein the shadow image is an image presented by a shadow in the original image, and the shadow image does not contain image content of the original image; The shadow removal unit is used to obtain a shadow-free image corresponding to the original image according to the original image and the shadow image.
14. An electronic device, characterized in that: include: at least one processor and memory; The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 12.
15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.
16. A computer program product, characterized in that The method comprises computer-executable instructions, and when a processor executes the computer-executable instructions, the method according to any one of claims 1 to 12 is implemented.
