IMAGE PROCESSING METHOD, IMAGE PROCESSING APPARATUS, STORAGE MEDIUM, AND ELECTRONIC APPARATUS
A two-stage neural network with image downsampling and reconstruction techniques addresses shadow removal issues on mobile devices, ensuring efficient and high-quality shadow removal without affecting the background layer.
Patent Information
- Application Number
- JP2024523517
- Authority / Receiving Office
- JP · JP
- Patent Type
- Applications
- Current Assignee / Owner
- Priority Date
- 2021-10-18
- Filing Date
- 2022-10-17
- Publication Date
- 2024-11-28
- Estimated Expiration
- 2042-10-17
AI Technical Summary
Existing shadow removal methods in image processing cause side effects on the background layer, require high computational resources, and have slow execution speeds, making them unsuitable for mobile devices.
A two-stage neural network architecture with a first stage for shadow area mask generation and a second stage for final image processing, combined with image downsampling and reconstruction using a Laplacian pyramid, and a multi-scale pyramid pooling module to enhance feature fusion, along with loss functions for precise shadow removal.
The method effectively removes shadows while preserving the background layer, achieving fast processing speeds suitable for mobile devices without noticeable side effects.
Smart Images

Figure 2024544118000001_ABST
Abstract
Description
[Technical field]
[0001] The present application relates to image processing technology, and more particularly to an image processing method, device, storage medium, and electronic device. [Background technology]
[0002] When people take pictures of documents with their mobile phones, shadows will always remain on the document due to the blocking of light by hands or mobile phones and the blocking of light by other objects in the environment, which will affect the visual experience of the captured image. By processing the captured image with computer vision processing technology, the shadows can be removed and the content of the characters or pictures behind the shadows can be restored, effectively improving the quality of the image. Therefore, document shadow removal is a technology of great significance, which can greatly improve the quality of captured images and has broad market potential.
[0003] The basic need and main challenge for applying a shadow removal method to a mobile phone is to effectively remove the shadow layer without causing significant side effects to the background layer, while having a fast execution speed and an acceptable hardware configuration. Current shadow removal methods either cannot completely remove the shadow, lose background layer information, or have a slow execution speed, all of which are disadvantageous for general users.
[0004] The conventional shadow removal method uses a neural network including a global positioning module, an appearance modeling module, and a semantic modeling module. The global positioning module detects shadow areas and obtains the location features of the shadow areas, the appearance modeling module learns the features of the non-shadow areas so that the output of the network and the labeled data (Ground Truth, GT) match in the non-shadow areas, and the semantic modeling module restores the original content behind the shadow. However, the method does not directly output the background image with the shadow removed, but outputs the ratio of the shadow image to the background image, and further requires dividing the shadow image and the output of the network pixel by pixel to obtain the background image, which introduces a larger amount of calculation and may affect the calculation stability due to the problem of divide by zero in the division.
[0005] Therefore, there is a need to propose an image processing technique that can effectively remove shadows without causing noticeable side effects in the background layer, and that has fast execution speed and acceptable hardware configuration requirements. Summary of the Invention [Problem to be solved by the invention]
[0006] The embodiments of the present application provide an image processing method, apparatus, storage medium, and electronic device to solve at least the technical problems that, in the prior art, removing shadow areas is prone to causing side effects in the background layer of an image, and has high requirements for the hardware platform. [Means for solving the problem]
[0007] According to one aspect of an embodiment of the present application, there is provided an image processing method including: obtaining a processed image including a shadow area; and inputting the processed image into a trained neural network to obtain a shadow-removed image, wherein the neural network includes a first-stage network and a second-stage network cascaded in two stages, wherein the first-stage network receives the processed image and outputs a shadow area mask image, and the second-stage network simultaneously receives the processed image and the shadow area mask image, and outputs the shadow-removed image.
[0008] Optionally, the first stage network includes a first feature extraction module including a first encoder and extracting features of the processed image layer by layer to obtain a first set of feature data, and a shadow area estimation module connected to the output of the first feature extraction module, including a first decoder, for estimating a shadow area based on the first set of feature data and outputting a shadow area mask image.
[0009] Optionally, the second stage network includes a second encoder, a second feature extraction module connected to the output of the first stage network to receive the processed image and the shadow area mask image output from the first stage network to obtain a second set of feature data, and a result image output module connected to the output of the second feature extraction module, including a second decoder, to output a shadow-removed image based on the second set of feature data.
[0010] Optionally, the output of each layer of the first decoder or the second decoder is spliced to the output of a corresponding layer in the first encoder or the second encoder along the channel axis through a cross-layer connection, and a multi-scale pyramid pooling module that fuses features of different scales is added to the cross-layer connection between the first decoder or the second decoder and the first encoder or the second encoder.
[0011] Optionally, after obtaining a processed image including a shadow region, the image processing method further includes downsampling the processed image using an image pyramid algorithm, retaining gradient information of each stage of image layer during downsampling to form a Laplacian pyramid, feeding the minimum size image layer into a trained neural network to obtain an output image, and performing low-resolution to high-resolution reconstruction on the output image using the Laplacian pyramid to obtain a shadow-removed image.
[0012] Optionally, the image processing method further includes constructing an initial neural network; and training the initial neural network with sample data to obtain a trained neural network, the sample data including real-life images and synthetic shaded images, and the synthetic shaded images are synthesized with the plain shaded image and the unshaded image using an image synthesis method.
[0013] Optionally, synthesizing the composite shaded image with the plain shaded image and the unshaded image using an image synthesis method includes obtaining a plain shaded image, obtaining a unshaded image, and obtaining a composite shaded image based on the plain shaded image and the unshaded image.
[0014] Optionally, synthesizing the composite shaded image with the simple shaded image and the unshaded image using the image synthesis method further includes transforming the simple shaded image and obtaining a composite shaded image based on the transformed simple shaded image and the unshaded image, wherein pixel values of non-shaded areas in the transformed simple shaded image are collectively set to a fixed numerical value a, and pixel values of shaded areas are numerical values between 0 and a, where a is a positive integer.
[0015] Optionally, the initial neural network further includes a module for making a type judgment on the sample data, and when it is determined that the sample data input to the initial neural network is a real image, the labeled data is a shadow-removed image collected in a real scene, and an internal parameter of the second-stage network is adjusted based on a difference between the shadow-removed image output from the initial neural network and the shadow-removed image as the labeled data; when it is determined that the sample data input to the initial neural network is a synthetic shadow image, the labeled data includes a shadow-free image and a simple shadow image collected in a real scene, and an internal parameter of the first-stage network is adjusted based on a difference between the shadow region mask image and the simple shadow image; and an internal parameter of the second-stage network is adjusted based on a difference between the shadow-removed image and the shadow-free image output from the initial neural network.
[0016] Optionally, when training the initial neural network using the sample data, the loss function includes at least one of pixel loss, feature loss, structural similarity loss, adversarial loss, shadow edge loss, and shadow intensity loss.
[0017] Optionally, the pixel loss includes a pixel blocking loss, where if the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is greater than a predetermined threshold, a loss of two pixels is calculated, and if the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is less than or equal to a predetermined threshold, the difference between the two pixels is ignored.
[0018] Optionally, the shadow luminance loss increases the difference between the luminance of an area corresponding to a shadow area in the shadow-removed image output from the neural network and the luminance of the shadow area in the input processed image to a value greater than 0, thereby improving the luminance of the area corresponding to the shadow area in the shadow-removed image.
[0019] Optionally, when the loss function includes a shadow edge loss, the image processing method includes performing a dilation operation on the shadow region mask image to obtain a dilated image, performing an erosion operation on the shadow region mask image to obtain an erosion image, and obtaining a difference set between the dilated image and the erosion image as a boundary region between shadow and non-shadow, and smoothing the difference set using TVLoss.
[0020] According to another aspect of the embodiment of the present application, there is further provided an image processing device including: an image collecting unit for acquiring a processed image including a shadow area; and a processing unit for receiving the processed image and processing the processed image using a trained neural network to obtain a shadow-removed image, wherein the neural network includes a first-stage network and a second-stage network cascaded in two stages, wherein the first-stage network receives the processed image and outputs a shadow area mask image, and the second-stage network simultaneously receives the processed image and the shadow area mask image, and outputs the shadow-removed image.
[0021] Optionally, the first stage network includes a first feature extraction module including a first encoder and extracting features of the processed image layer by layer to obtain a first set of feature data, and a shadow area estimation module connected to the output of the first feature extraction module, including a first decoder, estimating a shadow area based on the first set of feature data, and outputting a shadow area mask image.
[0022] Optionally, the second stage network includes a second encoder, a second feature extraction module connected to the output of the first stage network to receive the processed image and the shadow area mask image output from the first stage network to obtain a second set of feature data, and a result image output module connected to the output of the second feature extraction module, including a second decoder, to output a shadow-removed image based on the second set of feature data.
[0023] According to yet another aspect of an embodiment of the present application, there is further provided a storage medium storing a program which, when executed, controls an apparatus in which the storage medium is located so as to execute any of the image processing methods described above.
[0024] According to yet another aspect of an embodiment of the present application, there is further provided an electronic device including a processor and a memory storing executable instructions of the processor, the processor being configured to perform any of the image processing methods described above by executing the executable instructions.
[0025] The present application proposes a fast, effective shadow removal method that is applicable to mobile terminals such as mobile phones. By capturing the characteristics of the physical phenomenon of shadow and synthesizing training materials with a strong sense of realism, and combining various loss functions and effective network structures and modules for training, effective shadow removal is realized. In response to the characteristic that images captured by mobile terminals such as mobile phones have high resolution, the present application adopts downsampling technology and network pruning technology, so that fast processing speeds can still be achieved for high-resolution images. [Brief description of the drawings]
[0026] The drawings described herein are used to provide a further understanding of the present application, constitute a part of the present application, and the exemplary embodiments and the description thereof are intended to interpret the present application and are not intended to unduly limit the present application. [Figure 1] FIG. 1 is a flow chart of a selective image processing method according to an embodiment of the present application. [Diagram 2] FIG. 2 is a structural diagram of an optional neural network according to an embodiment of the present application. [Diagram 3] FIG. 3 is a flow chart of selective neural network training according to an embodiment of the present application. [Figure 4] FIG. 4 is a flow chart of a selective image synthesis method according to an embodiment of the present application. [Diagram 5] 5(a) and 5(b) are diagrams for comparing the effect of realizing shadow removal using the image processing method according to the embodiment of the present application. [Figure 6] FIG. 6 is a structural block diagram of an optional image processing apparatus according to an embodiment of the present application. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0027] In order to make those skilled in the art better understand the technical solution of the present application, the technical solution in the embodiments of the present application will be described below clearly and completely with reference to the drawings in the embodiments of the present application, and it is clear that the described embodiments are only some embodiments of the present application, and are not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor should fall within the scope of protection of the present application.
[0028] In addition, the terms "first", "second", etc. in the specification, claims and drawings are used to distinguish between similar objects and not necessarily to describe a particular order or priority. It should be understood that the orders used in this manner may be interchanged where appropriate, such that the embodiments of the present invention described herein may be practiced in orders other than those shown or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive "including", e.g., a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to the steps or units expressly listed, but may include other steps or units not expressly listed or inherent to the process, method, product or apparatus.
[0029] The following describes a flow chart of an optional image processing method according to an embodiment of the present application. It should be noted that the steps shown in the flow chart of the drawings may be performed in a computer system as a set of computer-executable instructions, and that although a logical order is shown in the flow chart, in some cases the steps shown or described may be performed in a different order than described here.
[0030] 1, which is a flow chart of a selective image processing method according to an embodiment of the present application. As shown in FIG. 1, the image processing method includes the following steps:
[0031] S100: A processed image including a shaded area is obtained. S102: Input the processed image into the trained neural network to obtain a shadow-removed image, where the neural network includes a first-stage network and a second-stage network connected in two stages in cascade, the first-stage network receives the processed image and outputs a shadow region mask image, and the second-stage network simultaneously receives the processed image and the shadow region mask image and outputs a shadow-removed image.
[0032] The image processing method described above allows accurate shadow region boundaries to be obtained, and the resulting shadow-removed image has a smooth transition between shadows and non-shadows.
[0033] In one alternative embodiment, as shown in FIG. 2, the neural network includes a first stage network 20 and a second stage network 22 connected in two cascade stages, the first stage network includes a first feature extraction module 200 and a shadow region estimation module 202, and the second stage network includes a second feature extraction module 204 and a result image output module 206. Here, the first feature extraction module 200 includes a first encoder, extracts features of the processed image layer by layer to obtain a first set of feature data; the shadow region estimation module 202 is connected to the output of the first feature extraction module 200, includes a first decoder, estimates a shadow region based on the first set of feature data, and outputs a shadow region mask image; the second feature extraction module 204 includes a second encoder, is connected to the output of the first-stage network, receives the processed image and receives the shadow region mask image output from the first-stage network to obtain a second set of feature data; the result image output module 206 is connected to the output of the second feature extraction module 204, includes a second decoder, and outputs a shadow-removed image based on the second set of feature data. The two-stage cascaded neural network can enhance the shadow removal effect. In one optional embodiment, the first-stage network and the second-stage network have the same structure except for the number of input channels being different, and may be constructed based on the classical partition network UNet, for example.
[0034] The output of each layer of the two encoders is respectively spliced to the output of the corresponding layer in the two decoders along the channel axis by the cross-layer connection. A multi-scale pyramid pooling module is added to the cross-layer connection between the encoder and the decoder. The multi-scale pyramid pooling module includes a plurality of pooling layers with different kernel sizes, a convolutional layer and an interpolation upsampling layer, which first extracts features of different scales by the pooling layer, then extracts low-level and / or high-level features by the convolutional layer, and further adjusts the output of the corresponding layer in the encoder and the decoder to the same size by the interpolation upsampling layer, and finally splices into one feature along the channel axis. Since the degree and area of the influence of shading vary greatly from image to image, the judgment of the shading region should not only refer to the local texture features, but also consider the global semantic information. The multi-scale pyramid pooling module can enhance the generalization of the network by fusing the features of different scales, and achieve a good effect of the network in shading images of different areas and degrees.
[0035] To improve the execution speed of the model on the device, the model may be pruned and the convolutional layers in the encoder may be replaced with grouped convolutions, so that each convolution kernel only convolves one channel, thereby reducing the computational complexity of the model and improving processing speed.
[0036] In order to better suppress the covariance drift and enhance the network's fitting ability to the data, an instance regularization layer is added after the convolutional layers of the encoder and decoder to regularize the features, thereby improving the shadow removal effect.
[0037] When the image to be processed has a high image resolution or a large amount of data, if the image to be processed is directly fed into a trained neural network, it may cause a video memory overflow or the processing time may be too long, which may affect the user experience. To solve this problem, a general interpolation scaling algorithm may be adopted, but it is prone to image information loss, and the generated image cannot be perfectly enlarged to the original image.
[0038] Considering the characteristic that the shadow area does not usually have significant gradient information, in one optional embodiment, an image pyramid algorithm is used to first downsample the processed image, and during downsampling, the gradient information of each image layer is retained to form a Laplacian pyramid, and then the image layer with the smallest pyramid size is fed into a trained neural network to obtain an output image, and finally the output image is reconstructed using the Laplacian pyramid. Since the gradient information of the shadow area is weak, even if part of the gradient information of the processed image is restored in the reconstruction process, the shadow removal effect is not affected. By using the gradient information of each image layer retained during downsampling to perform image reconstruction, the shadow can be removed without affecting the image resolution. By introducing downsampling and image reconstruction, on the one hand, the image processing speed is guaranteed, and on the other hand, the quality before and after image processing is not affected, which is advantageous for processing high-resolution images on devices with low computing power such as mobile phones.
[0039] As shown in FIG. 3, in order to obtain a trained neural network, the image processing method further includes the following steps:
[0040] S300: Build the initial neural network. S302: Train an initial neural network using sample data to obtain a trained neural network, where the sample data includes real-world images and synthetic shaded images, and the synthetic shaded images are synthesized with plain shaded images and unshaded images.
[0041] In the images that users often take, the types of shadows are very rich, so if we distinguish from the edges of the shadows, they include clear and sharp shadow edges taken when the distance from the light source to the background is close, and blurred shadow edges with gradual transition taken when the distance from the light source to the background is far. In addition, when the light source has different colors (for example, warm light with reddish yellow, cool light with bluish or sunlight), the shadows also have different colors. Therefore, considering these characteristics, in the entire image processing method, the sample data for training the initial neural network plays the most important role, and there are mainly two ways to obtain the sample data: real scene collection and image synthesis.
[0042] In the method of adopting real scene collection, the collector selects the corresponding lighting environment and shooting object according to the type of scene (e.g., different lighting scenes such as warm light, cool light, sunlight, etc.), fixes the shooting device such as a mobile phone or a camera on a tripod, adjusts the appropriate lighting direction and focal length, and uses the palm, mobile phone or other common object as an obstruction to block the light, forms a shadow on the shooting object and shoots it to obtain a shadow image, and then removes the obstruction and shoots again to obtain a shadow-free background image, thereby obtaining paired sample data.
[0043] However, in real scene collection, it is usually difficult to ensure high quality of sample data. On the one hand, due to changes in light caused by occlusion, background images and shaded images will have brightness and color differences in non-shaded areas, and it is difficult for the shaded image to match the background image perfectly. On the other hand, due to changes in light or focus, noise will be generated in the shaded image and background image, both of which have a significant impact on network training.
[0044] Alternatively, image synthesis methods may be used to generate realistic synthetic shaded images for training the neural network.
[0045] In one alternative embodiment, the image synthesis method includes the following steps. S400: A plain shaded image is obtained.
[0046] In one optional embodiment, in a preset lighting environment, a data collector lays a blank sheet of paper flat on a table, blocks the light with a palm, a mobile phone or other common objects, and leaves a simple shaded image S on the blank sheet, where all or part of the area of the simple shaded image S is a shaded area.
[0047] When acquiring a simple shaded image, the non-shaded areas on the white paper may not be displayed as a simple white color, so the boundary between the non-shaded areas and the shaded areas is not clear enough. Therefore, in another optional embodiment, the simple shaded image may be further transformed, for example, S'=min(a,S / mean(S)*a), where a is a positive integer. By the above transformation, the pixel values of the non-shaded areas in the transformed simple shaded image can be collectively set to one fixed value a (e.g., 255), and the pixel values of the shaded areas become values between 0 and a, so that the simple shaded image has a relatively clear boundary between the non-shaded areas and the shaded areas.
[0048] S402: A shadow-free image is obtained. In one alternative embodiment, the data collector takes shadow-free images B of each type of target in the same lighting environment.
[0049] S404: A synthetic shaded image is obtained based on the simple shaded image and the no-shading image. In one alternative embodiment, the plain shaded image S (or the transformed plain shaded image S') and the unshaded image B are multiplied pixel by pixel to obtain a composite shaded image.
[0050] This image synthesis method takes into account the attenuation of shadows with respect to light rays, and can effectively process shadows with gradual edge transitions, resulting in a strong sense of realism.
[0051] The sample data is mixed data including a real-life image and a synthetic shaded image, and the initial neural network further includes a module for making a type judgment on the sample data. Therefore, when it is determined that the sample data input to the initial neural network is a real-life image, the labeled data (Ground Truth, GT) is a shaded-out image collected in a real scene, and since the shaded area mask image of the real-life image cannot be adjusted, the internal parameters of the second-stage network 22 are adjusted based on the difference between the shaded-out image output from the initial neural network and the shaded-out image as the labeled data GT. When it is determined that the sample data input to the initial neural network is a synthetic shaded image, the labeled data (Ground Truth, GT) includes a shaded-out image and a simple shaded image collected in a real scene, and the internal parameters of the first-stage network 20 are adjusted based on the difference between the shaded area mask image and the simple shaded image. The internal parameters of the second-stage network 22 can be adjusted based on the difference between the shaded-out image output from the initial neural network and the shaded-out image as the labeled data. By using the mixed data as sample data for training, accurate masks can be obtained for shadows with gradual transitions, the quality of mask segmentation can be guaranteed, and the effect of shadow removal can be improved.
[0052] In one optional embodiment, the method for acquiring sample data may further include performing one or more processes on the acquired sample data, such as random flip, rotation, color temperature adjustment, channel swap, adding random noise, etc., so as to make the sample data richer and increase the robustness of the network.
[0053] In one optional embodiment, when supervising training the initial neural network, the loss function includes at least one of pixel loss, feature loss, structural similarity loss, and adversarial loss.
[0054] The pixel loss function is a function that measures the similarity of two images from the pixel plane of the image, and mainly includes image pixel value loss and gradient loss. In this embodiment, it mainly refers to the weighted sum of the mean square error of the compared pixel values of the output image and the label image from the initial neural network and the L1 norm error of the gradient of the two images. The pixel loss monitors the training process from the pixel plane so that the pixel values of each pixel of the output image and the label image from the initial neural network are as close as possible. In order to guide the initial neural network to pay attention to the difference between the shadow layer and the background layer in the shadow area rather than the noise of the whole image, in one optional embodiment, a pixel blocking loss may be introduced to block the pixel loss, that is, the loss of the two pixels is calculated only if the absolute difference between the two pixels is greater than a predetermined threshold, and the difference between the two pixels is ignored otherwise. After adding the pixel blocking loss, the network can be guided to pay attention to the shadow area to suppress the noise of the image, which not only improves the effect of shadow removal, but also greatly speeds up the convergence speed of the network.
[0055] The feature loss is mainly the weighted sum of the L1 norm error of the corresponding features of the input image and the label image to the initial neural network. In one optional embodiment, the VGG19 network pre-trained on the ImageNet dataset is used as a feature extractor, and the output image and the label image from the initial neural network are fed into the feature extractor respectively to obtain the features of each layer of VGG19, and then the L1 norm error of the corresponding features of the input image and the label image is calculated and weighted summed. The features of each layer of VGG19 are not sensitive to image details and noise and have good semantic properties, so even if there are defects such as noise and deviation in the input image and output image, the feature loss can still accurately generate effective shadow region differences, which complements the lack of sensitivity of pixel loss to noise and has good stability.
[0056] The structural similarity loss function is a function that measures the similarity between two images based on the global features of the images. In this embodiment, it mainly refers to the global brightness and contrast difference of the output image and the label image from the initial neural network. By adding this loss function, the color shift of the network output can be effectively suppressed and the overall quality of the image can be improved.
[0057] The adversarial loss is mainly the loss value of the output result by the classifier and the true type of the output image. When the difference between the output image from the initial neural network and the label image becomes smaller in the later stage of training, the effects of pixel loss, feature loss, and structural similarity loss gradually become smaller, and the convergence of the network becomes slower. At this time, in order to support the training of the network, one classifier network is trained synchronously. First, the output image and the label image from the initial neural network are sent to the classifier, and the classifier determines whether the output image is a label image or not, and calculates a loss based on the output result by the classifier and the true type of the output image to update the parameters of the classifier. Then, the classification result of the classifier for the output image is taken as the loss of the degree of reality of the output image, and the parameters of the classifier are updated with the loss. When the classifier cannot distinguish the output image from the initial neural network and the label image, it indicates that the training is completed. The adversarial loss can effectively eliminate the side effects of the image caused by the network processing (such as the color mismatch problem of the shaded and non-shaded areas and the shade residual problem, etc.) and improve the degree of reality of the output image from the network.
[0058] Threshold cutoff loss. Due to the influence of lighting, paired data collected in real scenes may have subtle brightness differences and color changes even in non-shadow areas, but these differences are within the user's tolerance range and do not need to be processed. Therefore, in order to prevent the network from paying attention to these global subtle differences during training, the method introduces threshold cutoff loss, i.e., only if the difference between the network output and the GT is larger than a certain threshold, the difference is collectively counted in the gradient of the global loss calculation parameter, otherwise the loss is considered to be 0. The loss function allows for subtle differences between the network output and the GT, and moves the center of gravity of the network learning to the area with large differences, thereby effectively improving the network's ability to remove relatively clear shadows.
[0059] Shadow edge loss. First, a dilation process is performed on the shadow region mask image to obtain a dilated image, then an erosion process is performed on the shadow region mask image to obtain an eroded image, and then the difference set between the dilated image and the eroded image is obtained as the border region between shadow and non-shadow, and then smoothed using TVLoss to effectively transition between shadow and non-shadow regions.
[0060] The shadow luminance loss increases the difference between the luminance of the area corresponding to the shadow area in the shadow-removed image output from the neural network and the luminance of the shadow area in the input processed image to a value greater than 0, thereby improving the luminance of the area corresponding to the shadow area in the shadow-removed image.
[0061] In one alternative embodiment, the background layer output module of the initial neural network uses a weighted sum of all the above losses as the total loss, and uses a Wassertein generative adversarial network as the adversarial loss.
[0062] The network structure extracts global and local features of the input image to improve the degree of shadow removal while protecting non-shadow areas from causing side effects.
[0063] 5(a) and 5(b) are comparative diagrams of the processing effects achieved using the image processing method of an embodiment of the present application, where FIG. 5(a) is a processed image including shadows, and FIG. 5(b) is a shadow-removed image after processing by the image processing method. As can be seen from the comparison of the two images, the image processing method of the present application can effectively remove shadows without causing significant side effects in the background layer.
[0064] The neural network structure and loss function employed in the embodiments of the present application may be applied to application scenes such as shadow removal, rain removal, and fog removal, and are mainly used to process high-resolution images taken by mobile terminals such as mobile phones, but are also similarly applicable to processing images of various resolutions on the PC side or other embedded devices.
[0065] According to another aspect of an embodiment of the present application, there is further provided an electronic device including a processor and a memory storing executable instructions for the processor, the processor being configured to perform any of the above image processing methods by executing the executable instructions.
[0066] According to another aspect of an embodiment of the present application, there is further provided a storage medium storing a program which, when executed, controls an apparatus in which the storage medium is located to perform any one of the image processing methods described above.
[0067] According to another aspect of the embodiment of the present application, an image processing device is further provided. Referring to Fig. 6, a structural block diagram of an optional image processing device according to the embodiment of the present application is provided. As shown in Fig. 6, the image processing device 60 includes an image collecting unit 600 and a processing unit 602.
[0068] Each unit included in the image processing device 60 will be described in detail below. The image acquisition unit 600 acquires a processed image that includes a shadow area.
[0069] The processing unit 602 receives a processed image and processes the processed image using a trained neural network to obtain a shadow-removed image, where the neural network includes a first-stage network and a second-stage network connected in two cascades, and the processed image and the output image from the first-stage network are input to the second-stage network simultaneously.
[0070] In one alternative embodiment, the structure of the neural network is as shown in FIG. 2 and the associated description herein, and will not be repeated here.
[0071] The numbers of the above embodiments of the present application are merely for description purposes and do not indicate superiority or inferiority of the embodiments.
[0072] In the above embodiments of the present application, the descriptions of each embodiment have different emphasis, and for parts that are not described in detail in one embodiment, reference may be made to the relevant descriptions of other embodiments.
[0073] It should be understood that in some embodiments of the present application, the disclosed technical contents can be realized in other forms. Here, the above-described device embodiments are merely illustrative, for example, the division of the units may be a division of logical functions, and there may be other division methods in actual implementation, for example, multiple units or components may be combined or integrated into other systems, or some features may be omitted or not implemented. In addition, the couplings or direct couplings or communication connections between the disclosed or discussed mutually may be indirect couplings or communication connections via some interfaces, units or modules, and may be electrical or other forms.
[0074] The units described as separate components may or may not be physically separated, and the components listed as units may or may not be physical units, i.e., they may be located in one place or distributed among multiple units, some or all of which may be selected according to actual needs to achieve the purpose of the technical solution of this embodiment.
[0075] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, each unit may exist physically alone, or two or more units may be integrated into one unit. The above-mentioned integrated units may be realized in the form of hardware or software functional units.
[0076] When the integrated unit is realized in the form of a software functional unit and sold or used as an independent product, it may be stored in one computer-readable storage medium. Based on this understanding, the essential or conventional technical part of the technical solution of the present application, or all or part of the technical solution, may be embodied in the form of a software product, and the computer software product is stored in one storage medium containing some instructions for making a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes various media that can store program codes, such as a USB disk, a read-only memory (ROM), a random access memory (RAM), a removable hard disk, a magnetic disk, or an optical disk.
[0077] The above are only preferred embodiments of the present application, and it should be pointed out that those skilled in the art may make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be regarded as within the protection scope of the present application.
[0078] This application claims priority to Chinese Patent Application No. 202111210502.3, filed on October 18, 2021, the contents of which are incorporated herein by reference as part of this application.
Claims
1. Obtaining a processed image including a shaded region; inputting the processed image into a trained neural network to obtain a shadow-removed image; the neural network includes a first-stage network and a second-stage network connected in two cascade stages, the first-stage network receiving the processed image and outputting a shadow region mask image, and the second-stage network simultaneously receiving the processed image and the shadow region mask image and outputting the shadow-removed image.
2. The first stage network includes: a first feature extraction module including a first encoder for extracting features of the processed image layer by layer to obtain a first set of feature data; and a shadow area estimation module, connected to an output of the first feature extraction module, comprising a first decoder, for estimating shadow areas based on the first set of feature data and outputting a shadow area mask image.
3. The second stage network is a second feature extraction module including a second encoder, connected to the output of the first-stage network to receive the processed image and the shadow area mask image output from the first-stage network to obtain a second set of feature data; and a resultant image output module connected to an output of the second feature extraction module, the resultant image output module including a second decoder and configured to output the shadow removed image based on the second set of feature data.
4. 4. The image processing method according to claim 2, further comprising: splicing an output of each layer of the first decoder or the second decoder to an output of a corresponding layer in the first encoder or the second encoder along a channel axis by a cross-layer connection; and adding a multi-scale pyramid pooling module that fuses features of different scales to the cross-layer connection between the first decoder or the second decoder and the first encoder or the second encoder.
5. After obtaining a processed image including a shaded area, the image processing method includes: downsampling the processed image using an image pyramid algorithm, preserving gradient information of each image layer during downsampling to form a Laplacian pyramid; feeding the minimum size image layer into the trained neural network to obtain an output image; and 2. The image processing method of claim 1, further comprising: performing a low-to-high resolution reconstruction of the output image using a Laplacian pyramid to obtain the shadow-removed image.
6. Constructing an initial neural network; training the initial neural network with sample data to obtain the trained neural network; 2. The image processing method according to claim 1, wherein the sample data includes an actual image and a synthetic shaded image, and the synthetic shaded image is synthesized from a simple shaded image and a non-shaded image using an image synthesis method.
7. compositing the composite shaded image from the plain shaded image and the unshaded image using an image composition method, Obtaining a plain shaded image; Obtaining a shadow-free image; and 2. The method of claim 1, further comprising: obtaining the synthetic shaded image based on the simple shaded image and the shade-free image.
8. compositing the composite shaded image from the plain shaded image and the unshaded image using an image composition method, transforming the simple shaded image and obtaining the composite shaded image based on the transformed simple shaded image and the shade-free image; 8. The image processing method according to claim 7, wherein pixel values of non-shaded areas in the converted simple shaded image are collectively set to a single fixed numerical value a, and pixel values of shaded areas are numerical values between 0 and a, where a is a positive integer.
9. The initial neural network further includes a module for performing a classification judgment on the sample data; if it is determined that the sample data input to the initial neural network is a real-life image, the labeled data is a shadow-removed image collected in a real scene, and an internal parameter of the second-stage network is adjusted based on a difference between the shadow-removed image output from the initial neural network and the shadow-removed image as the labeled data; 8. The image processing method according to claim 7, wherein, when it is determined that the sample data input to the initial neural network is a synthetic shadow image, the labeled data includes the shadow-free image and the simple shadow image collected in a real scene, and parameters within a first-stage network are adjusted based on a difference between the shadow region mask image and the simple shadow image, and parameters within a second-stage network are adjusted based on a difference between the shadow-removed image output from the initial neural network and the shadow-free image.
10. 7. The image processing method of claim 6, wherein when training the initial neural network with sample data, a loss function includes at least one of pixel loss, feature loss, structural similarity loss, adversarial loss, shadow edge loss, and shadow intensity loss.
11. The pixel loss includes pixel blocking loss; If the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is greater than a predetermined threshold, a loss of the two pixels is calculated; 11. The image processing method of claim 10, further comprising ignoring a difference between two corresponding pixels in the output image from the initial neural network and the label image if the absolute difference between the two pixels is less than or equal to the predetermined threshold.
12. 11. The image processing method according to claim 10, wherein the shadow luminance loss is achieved by making a difference between a luminance of an area corresponding to the shadow area in the shadow-removed image output from the neural network and a luminance of the shadow area in the input processed image greater than 0, thereby improving the luminance of the area corresponding to the shadow area in the shadow-removed image.
13. When the loss function includes the shadow edge loss, the image processing method further comprises: performing a dilation process on the shadow region mask image to obtain a dilated image; performing an erosion process on the shadow region mask image to obtain an erosion image; and The image processing method according to claim 10, further comprising obtaining a difference set between the dilated image and the eroded image as a boundary region between a shadow and a non-shadow, and smoothing the difference set using TVLoss.
14. an image acquisition unit for acquiring a processed image including a shadow region; a processing unit for receiving the processed image and processing the processed image using the trained neural network to obtain a shadow removed image; the neural network includes a first-stage network and a second-stage network connected in two cascade stages, the first-stage network receiving the processed image and outputting a shadow region mask image, and the second-stage network simultaneously receiving the processed image and the shadow region mask image and outputting the shadow-removed image.
15. The first stage network includes: a first feature extraction module including a first encoder for extracting features of the processed image layer by layer to obtain a first set of feature data; a shadow region estimation module, connected to an output of the first feature extraction module, comprising a first decoder, for estimating a shadow region based on the first set of feature data and outputting a shadow region mask image; The image processing apparatus according to claim 14 , further comprising:
16. The second stage network is a second feature extraction module including a second encoder, connected to the output of the first-stage network to receive the processed image and the shadow area mask image output from the first-stage network to obtain a second set of feature data; a result image output module coupled to an output of the second feature extraction module, the result image output module including a second decoder, the result image output module outputting a shadow removed image based on the second set of feature data; The image processing apparatus according to claim 14 , further comprising:
17. A storage medium storing a program, the program controlling a device in which the storage medium is located so as to execute an image processing method according to any one of claims 1 to 13, when the program is executed.
18. An electronic device, A processor; a memory for storing executable instructions for said processor; 14. An electronic device, wherein the processor is configured to perform the image processing method of any one of claims 1 to 13 by executing the executable instructions.
Citation Information
Patent Citations
High-resolution image shadow region recovery and reconstruction method based on deep learning
CN113178010A
Shadow region processing method and device, computer readable medium and electronic equipment
CN113205530A
Photographing game device, image processing method and image processing program
JP2016146623A
Image processing device, image processing method, and program
JP2019070872A
Image Enhancement Using Generative Adversarial Networks
JP2021521993A
