Image processing method, image processing apparatus, storage medium, and electronic device

Abstract

The present application discloses an image processing method, device, storage medium, and electronic device, in which the image processing method includes obtaining a processed image including a shadow area, and inputting the processed image into a trained neural network to obtain a shadow-removed image, the neural network including a first-stage network and a second-stage network connected in two stages in cascade, the first-stage network receiving the processed image and outputting a shadow area mask image, and the second-stage network receiving the processed image and the shadow area mask image simultaneously and outputting the shadow-removed image. The present application can solve the technical problem that the prior art is prone to causing side effects in the background layer of the image while removing the shadow area, and has high requirements for the hardware platform.

Description

Technical Field

[0001] This application relates to image processing technology, and specifically to an image processing method, apparatus, storage medium, and electronic device.

Background Art

[0002] When people take pictures of documents with a mobile phone, there are always shadows remaining on the document due to the shielding of the hand or the mobile phone against light and the shielding of other objects in the environment against light, which affects the visual experience of the captured image. By processing the captured image with computer vision processing technology, the shadow can be removed, the content of the text or picture behind the shadow can be restored, and the quality of the image can be effectively improved. Therefore, document shadow removal is an important technology with significant meaning, which can greatly improve the quality of the captured image and has broad market prospects.

[0003] Effectively removing the shadow layer without causing significant side effects on the background layer, having a fast execution speed, and having acceptable hardware configuration requirements are the basic needs and main challenges for applying the shadow removal method to a mobile phone. The current shadow removal methods either cannot completely remove the shadow, lose the information of the background layer, or have a slow execution speed, all of which are disadvantageous for general users to use.

[0004] Conventional deshading methods utilize a neural network including a global positioning module, an appearance modeling module, and a semantic modeling module. The global positioning module detects shaded areas and obtains their positional features; the appearance modeling module learns the features of unshaded areas so that the network output matches labeled data (Ground Truth, GT) in unshaded areas; and the semantic modeling module reconstructs the original content behind the shade. However, this method does not directly output the deshaded background image, but rather outputs the ratio of the shaded image to the background image, and further requires dividing the shaded image and the network output pixel by pixel to obtain the background image. This introduces a larger computational load and can affect computational stability due to the problem of division by zero in the division.

[0005] Therefore, there is a need to propose an image processing technique that can effectively remove shadows without causing significant side effects on the background layer, while also having fast execution speed and acceptable hardware configuration requirements. [Overview of the project] [Problems that the invention aims to solve]

[0006] The embodiments of this application provide an image processing method, apparatus, storage medium, and electronic device that solve the technical problems in the prior art, such as the tendency to cause side effects on the background layer of an image while removing shaded areas, and the high demands on the hardware platform. [Means for solving the problem]

[0007] According to one embodiment of the present invention, an image processing method is provided which includes acquiring an image to be processed that includes a shaded area, and inputting the image to be processed into a trained neural network to obtain a shadow-removed image, wherein the neural network includes a first-stage network and a second-stage network connected in cascaded order, the first-stage network receives the image to be processed and outputs a shadow-area mask image, and the second-stage network simultaneously receives the image to be processed and the shadow-area mask image and outputs a shadow-removed image.

[0008] Selectively, the first-stage network includes a first feature extraction module which includes a first encoder and extracts features from the image to be processed layer by layer to obtain a first set of feature data, and a shaded area estimation module connected to the output of the first feature extraction module which includes a first decoder and estimates shaded areas based on the first set of feature data to output a shaded area mask image.

[0009] Selectively, the second-stage network includes a second encoder, connected to the output of the first-stage network, which includes a second feature extraction module that receives the image to be processed and the shaded area mask image output from the first-stage network to obtain a second set of feature data, and a result image output module connected to the output of the second feature extraction module, which includes a second decoder and outputs a deshaded image based on the second set of feature data.

[0010] Selectively, the outputs of each layer of the first decoder or the second decoder are spliced ​​along the channel axis to the outputs of the corresponding layers in the first encoder or the second encoder via a cross-layer connection, and a multiscale pyramid pooling module is added to the cross-layer connection between the first decoder or the second decoder and the first encoder or the second encoder to fuse features of different scales.

[0011] The image processing method further includes selectively acquiring an image to be processed that includes shaded areas, downsampling the image to be processed using an image pyramid algorithm, forming a Laplacian pyramid by retaining gradient information of each image layer during downsampling, feeding the smallest size image layer to a trained neural network to obtain an output image, and reconstructing the output image from low resolution to high resolution using the Laplacian pyramid to obtain a deshaded image.

[0012] Selectively, the above image processing method further includes constructing an initial neural network, training the initial neural network using sample data, and obtaining a trained neural network, wherein the sample data includes real-world images and composite shaded images, and the composite shaded images are synthesized from simple shaded images and unshaded images using an image synthesis method.

[0013] Selectively combining a simple shaded image and a non-shaded image with a composite shaded image using an image synthesis method includes obtaining a simple shaded image, obtaining a non-shaded image, and obtaining a composite shaded image based on the simple shaded image and the non-shaded image.

[0014] Selectively combining a composite shaded image with a simple shaded image and a non-shaded image using an image synthesis method further includes transforming the simple shaded image and obtaining a composite shaded image based on the transformed simple shaded image and the non-shaded image, wherein the pixel values ​​of the non-shaded areas in the transformed simple shaded image are collectively set to a single fixed value a, and the pixel values ​​of the shaded areas are values ​​between 0 and a, where a is a positive integer.

[0015] Selectively, the initial neural network further includes a module that performs classification on sample data. If the initial neural network determines that the sample data input is a real-world image, the labeled data is a de-shading image collected from a real-world scene. Based on the difference between the de-shading image output from the initial neural network and the de-shading image as labeled data, the internal parameters of the second-stage network are adjusted. If the initial neural network determines that the sample data input is a composite shaded image, the labeled data includes both a non-shaded image and a simple shaded image collected from a real-world scene. Based on the difference between the shaded area mask image and the simple shaded image, the internal parameters of the first-stage network are adjusted. Based on the difference between the de-shading image output from the initial neural network and the non-shaded image, the internal parameters of the second-stage network are adjusted.

[0016] When selectively training an initial neural network using sample data, the loss function includes at least one of the following: pixel loss, feature loss, structural similarity loss, adversarial loss, shaded edge loss, and shaded luminance loss.

[0017] Selectively, the pixel loss includes pixel cutoff loss. If the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is greater than a predetermined threshold, the loss between the two pixels is calculated. If the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is less than or equal to the predetermined threshold, the difference between the two pixels is ignored.

[0018] Selectively, the shadow luminance loss increases the luminance of the shadowed regions in the shadow-removed image by making the difference between the luminance of the shadowed regions in the shadow-removed image and the luminance of the shadowed regions in the input image greater than zero.

[0019] Selectively, when the loss function includes shaded edge loss, the above image processing method includes: performing a dilation process on the shaded region mask image to obtain a dilated image; performing an erosion process on the shaded region mask image to obtain an erosion image; and obtaining the set of differences between the dilated image and the erosion image as the boundary region between shaded and unshaded areas, and smoothing it using TVLoss.

[0020] According to another embodiment of the present invention, an image processing apparatus is provided, comprising: an image acquisition unit that acquires an image to be processed including a shaded area; and a processing unit that receives the image to be processed and processes the image using a trained neural network to obtain a shadow-removed image, wherein the neural network includes a first-stage network and a second-stage network connected in cascaded order, the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network simultaneously receives the image to be processed and the shaded area mask image and outputs a shadow-removed image.

[0021] Selectively, the first-stage network includes a first feature extraction module which includes a first encoder and extracts features from the image to be processed layer by layer to obtain a first set of feature data, and a shaded area estimation module connected to the output of the first feature extraction module which includes a first decoder and estimates shaded areas based on the first set of feature data to output a shaded area mask image.

[0022] Selectively, the second-stage network includes a second encoder, connected to the output of the first-stage network, which includes a second feature extraction module that receives the image to be processed and the shaded area mask image output from the first-stage network to obtain a second set of feature data, and a result image output module connected to the output of the second feature extraction module, which includes a second decoder and outputs a deshaded image based on the second set of feature data.

[0023] According to yet another aspect of the embodiment of the present application, there is provided a storage medium storing a program, and when the program is executed, the storage medium controls a device in which the storage medium is located to execute the image processing method described in any of the above.

[0024] According to yet another aspect of the embodiment of the present application, there is provided an electronic device including a processor and a memory storing instructions executable by the processor, and the processor is configured to execute the image processing method described in any of the above by executing the executable instructions.

[0025] The present application proposes a shadow removal method that is fast, has good effects, and is applicable to mobile terminals such as mobile phones. By capturing the characteristics of the physical phenomenon of shadows and synthesizing training materials with a strong sense of reality, and combining various loss functions, effective network structures, and modules for training, good shadow removal effects are realized. For the characteristic that the resolution of images captured by mobile terminals such as mobile phones is high, the present application adopts downsampling technology and network pruning technology to still achieve a fast processing speed in high-resolution images.

Brief Description of the Drawings

[0026] The drawings described herein are used to provide a further understanding of the present application, form a part of the present application, and the exemplary embodiments and descriptions thereof of the present application are for interpreting the present application and do not unduly limit the present application. [Figure 1] FIG. 1 is a flowchart of a selective image processing method according to an embodiment of the present application. [Figure 2] FIG. 2 is a structural diagram of a selective neural network according to an embodiment of the present application. [Figure 3] FIG. 3 is a flowchart of a selective neural network training according to an embodiment of the present application. [Figure 4] FIG. 4 is a flowchart of a selective image synthesis method according to an embodiment of the present application. [Figure 5] Figures 5(a) and 5(b) are comparative diagrams showing the effects of achieving shadow removal using the image processing method of the embodiment of the present application. [Figure 6] Figure 6 is a structural block diagram of a selective image processing apparatus according to an embodiment of the present invention. [Modes for carrying out the invention]

[0027] Hereinafter, in order for those skilled in the art to better understand the technical concept of the present application, the technical concept in the embodiments of the present application will be clearly and completely described with reference to the drawings of the embodiments, and it will be clear that the embodiments described are only a part of the embodiments of the present application, not all of them. All other embodiments that a person skilled in the art could obtain without creative work based on the embodiments of the present application should fall within the scope of protection of the present application.

[0028] Furthermore, terms such as “first,” “second,” etc., in the specification, claims, and drawings of this application are intended to distinguish similar objects and are not necessarily intended to describe a specific order or priority. It should be understood that the order used in this manner is interchangeable where appropriate so that the embodiments of this application described herein may be carried out in an order other than that illustrated or described herein. Also, the terms “includes” and “has” and any variations thereof are intended to cover non-exclusive “includes,” for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to the steps or units explicitly listed, but may include other steps or units not explicitly listed or that are specific to those processes, methods, products, or apparatus.

[0029] The following describes a flowchart of a selective image processing method according to an embodiment of the present invention. The steps shown in the flowchart may be performed on a computer system, such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, the steps shown or described may be performed in a different order in some cases.

[0030] Referring to Figure 1, this is a flowchart of a selective image processing method according to an embodiment of the present application. As shown in Figure 1, the image processing method includes the following steps.

[0031] S100: Acquires the image to be processed, including shaded areas. S102: The image to be processed is input to a trained neural network to obtain a de-shading image. Here, the neural network includes a first-stage network and a second-stage network connected in cascaded order. The first-stage network receives the image to be processed and outputs a shaded area mask image, while the second-stage network simultaneously receives the image to be processed and the shaded area mask image and outputs a de-shading image.

[0032] The above image processing method allows for obtaining accurate shaded area boundaries, and the resulting deshaded image can smoothly transition between shaded and unshaded areas.

[0033] In one selective embodiment, as shown in Figure 2, the neural network includes a first-stage network 20 and a second-stage network 22 connected in cascades, the first-stage network including a first feature extraction module 200 and a shaded region estimation module 202, and the second-stage network including a second feature extraction module 204 and a result image output module 206. Here, the first feature extraction module 200 includes a first encoder and extracts features from the image to be processed layer by layer to obtain a first set of feature data; the shaded area estimation module 202 is connected to the output of the first feature extraction module 200 and includes a first decoder and estimates the shaded area based on the first set of feature data and outputs a shaded area mask image; the second feature extraction module 204 includes a second encoder and is connected to the output of the first-stage network and receives the image to be processed and the shaded area mask image output from the first-stage network to obtain a second set of feature data; and the result image output module 206 is connected to the output of the second feature extraction module 204 and includes a second decoder and outputs a deshading image based on the second set of feature data. The two-stage cascaded neural network can enhance the deshading effect. In one selective embodiment, the first-stage network and the second-stage network have the same structure except for the number of input channels, and may be built, for example, based on a classic partitioned network UNet.

[0034] The outputs of each layer of two encoders are spliced ​​along the channel axis to the outputs of the corresponding layers in the two decoders via a cross-layer connection. A multiscale pyramid pooling module is added to the cross-layer connection between the encoders and decoders. The multiscale pyramid pooling module includes multiple pooling layers, convolutional layers, and interpolation upsampling layers of different kernel sizes. First, the pooling layers extract features of different scales, then the convolutional layers extract low-level and / or high-level features, and finally the interpolation upsampling layers adjust the outputs of the corresponding layers in the encoders and decoders to the same size, and finally splice them into a single feature along the channel axis. Since the degree and area of ​​shading vary greatly from image to image, the determination of shaded areas should consider not only local texture features but also global semantic information. By fusing features of different scales, the multiscale pyramid pooling module enhances the generalizability of the network and can achieve good network performance in shaded images of different areas and degrees.

[0035] To improve the execution speed of the model on the device, the model may be pruned and the convolutional layers in the encoder may be replaced with grouped convolutions so that each convolutional kernel convolves only one channel, thereby reducing the computational complexity of the model and improving processing speed.

[0036] To better suppress covariance drift and enhance the network's ability to fit data, the deshading effect is improved by adding an instance regularization layer after the convolutional layers of the encoder and decoder to regularize the features.

[0037] When the image resolution of the image to be processed is high or the data volume is large, directly feeding the image to a trained neural network can cause video memory overflow or excessively long processing times, potentially impacting the user experience. To solve this problem, general interpolation scaling algorithms can be employed, but these are prone to image information loss and cannot perfectly enlarge the generated image back to the original.

[0038] Considering that shaded areas typically lack significant gradient information, in one selective embodiment, an image pyramid algorithm may be used to first downsample the image to be processed, retain the gradient information of each image layer during downsampling to form a Laplacian pyramid, then send the image layer with the smallest pyramid size to a trained neural network to obtain the output image, and finally reconstruct the output image using the Laplacian pyramid. Because the gradient information of shaded areas is weak, restoring some of the gradient information of the image to be processed during the reconstruction process does not affect the shadow removal effect. By performing image reconstruction using the gradient information of each image layer retained during downsampling, shadows can be removed without affecting the image resolution. By introducing downsampling and image reconstruction, image processing speed is guaranteed on the one hand, and the quality before and after image processing is not affected on the other hand, making it advantageous for processing high-resolution images on devices with low processing power such as mobile phones.

[0039] As shown in Figure 3, the image processing method further includes the following steps to obtain a trained neural network.

[0040] S300: Build the initial neural network. S302: An initial neural network is trained using sample data to obtain the trained neural network. Here, the sample data includes real-world images and composite shaded images, and the composite shaded images are created by combining simple shaded images and unshaded images.

[0041] In images commonly taken by users, the variety of shadows is extremely rich. Distinguishing them by shadow edges reveals two types: sharp, clear shadow edges when the light source is close to the background, and blurred, smoothly transitioning shadow edges when the light source is far away. Furthermore, when the light source appears in different colors (e.g., warm light with a reddish-yellow tint, cool light with a bluish tint, or sunlight), the shadows also appear in different colors. Therefore, considering these characteristics, sample data for training the initial neural network is of paramount importance in the overall image processing method, and there are two main methods for obtaining sample data: real-world scene collection and image synthesis.

[0042] In the method employing real-world sampling, the collector selects the appropriate lighting environment and subject according to the type of scene (e.g., different lighting scenes such as warm light, cool light, and sunlight), fixes a photographic device such as a mobile phone or camera on a tripod, adjusts the appropriate lighting direction and focal length, uses a palm, mobile phone, or other common object as a shield to block light, creates a shadow on the subject, and takes a photograph to obtain a shadowed image. After that, the shield is removed and the photograph is taken again to obtain a background image without shadows, and in this way, pairs of sample data are obtained.

[0043] However, real-world sampling usually makes it difficult to guarantee high quality sample data. On the one hand, changes in light due to occlusion cause differences in brightness and color in the unshaded areas of the background and shaded images, and the shaded images are difficult to perfectly match to the background images. On the other hand, changes in light or focus introduce noise into the shaded and background images, and both of these have a significant impact on network training.

[0044] Alternatively, a realistic composite shading image can be generated using an image synthesis method and used for training a neural network.

[0045] In one selective embodiment, the image synthesis method includes the following steps: S400: Acquires a simple shaded image.

[0046] In one selective embodiment, a data collector lays a blank sheet of paper flat on a table in a pre-set lighting environment, shields it from light with their palm, mobile phone, or other common object, and leaves a simple shaded image S on the blank sheet of paper, where all or part of the simple shaded image S is a shaded area.

[0047] When acquiring a simple shaded image, the unshaded areas on a blank surface may not be displayed as simple white, resulting in an unclear boundary between the unshaded and shaded areas. Therefore, in another selective embodiment, the simple shaded image may be further transformed, for example, by S' = min(a, S / mean(S) * a), where a is a positive integer. This transformation allows the pixel values ​​of the unshaded areas in the transformed simple shaded image to be collectively set to a single fixed value a (e.g., 255), and the pixel values ​​of the shaded areas become numbers between 0 and a, resulting in a relatively clear boundary between the unshaded and shaded areas in the simple shaded image.

[0048] S402: Obtain an image without shadows. In one selective embodiment, the data collector captures shadowless images B of each type of subject under the same lighting conditions as described above.

[0049] S404: A composite shaded image is obtained based on a simple shaded image and a non-shaded image. In one selective embodiment, a simple shaded image S (or a converted simple shaded image S') and an unshaded image B are multiplied pixel by pixel to obtain a composite shaded image.

[0050] This image synthesis method takes into account the attenuation effect of light on shadows, allowing for smooth processing of shadows with gentle edge transitions, resulting in a highly realistic appearance.

[0051] The sample data is a mixed data set containing real-world images and composite shaded images. The initial neural network further includes a module that performs classification on the sample data. If the initial neural network determines that the sample data input is a real-world image, the labeled data (Ground Truth, GT) is a de-shading image collected from a real-world scene. Since the shaded area mask image of the real-world image cannot be adjusted, the internal parameters of the second-stage network 22 are adjusted based on the difference between the de-shading image output from the initial neural network and the de-shading image as labeled data GT. If the initial neural network determines that the sample data input is a composite shaded image, the labeled data (Ground Truth, GT) includes unshaded images and simple shaded images collected from a real-world scene. The internal parameters of the first-stage network 20 are adjusted based on the difference between the shaded area mask image and the simple shaded image. The internal parameters of the second-stage network 22 can be adjusted based on the difference between the de-shading image output from the initial neural network and the unshaded image as labeled data. By training with mixed data as sample data, it is possible to obtain accurate masks for shading with smooth transitions, ensure the quality of mask division, and improve the effectiveness of shading removal.

[0052] In one selective embodiment, the method for acquiring sample data may further include performing one or more operations on the acquired sample data, such as random flipping, rotation, color temperature adjustment, channel swapping, and adding random noise, in order to enrich the sample data and increase the robustness of the network.

[0053] In one selective embodiment, when training an initial neural network, the loss function includes at least one of pixel loss, feature loss, structural similarity loss, and adversarial loss.

[0054] The pixel loss function is a function that measures the similarity between two images from the pixel plane of the image, and mainly consists of image pixel value loss and gradient loss. In this embodiment, it mainly refers to the weighted sum of the mean squared error of the comparison pixel values ​​of the output image and label image from the initial neural network and the L1 norm error of the gradients of the two images. The pixel loss monitors the training process from the pixel plane so that the pixel values ​​of each pixel in the output image and label image from the initial neural network are as close as possible. In one selective embodiment, pixel cutoff loss may be introduced to cut off the pixel loss in order to guide the initial neural network to focus on the difference between the shaded layer and the background layer in the shaded region rather than the noise of the entire image. That is, the loss of two pixels is calculated only if the absolute difference between the two pixels is greater than a predetermined threshold, and otherwise the difference between the two pixels is ignored. After adding pixel cutoff loss, the network can be guided to focus on the shaded region to suppress image noise, improving the effectiveness of shadow removal and significantly increasing the network's convergence speed.

[0055] Feature loss is primarily the weighted sum of the L1 norm errors of corresponding features in the input image and label image to the initial neural network. In one selective embodiment, a VGG19 network pre-trained on the ImageNet dataset is used as a feature extractor. The output image and label image from the initial neural network are fed into the feature extractor, respectively. Features from each layer of the VGG19 are obtained, and then the L1 norm errors of corresponding features in the input image and label image are calculated and weighted together. Since the features of each layer of the VGG19 are not sensitive to image detail or noise and have good semantic properties, even if defects such as noise or misalignment are present in the input and output images, the feature loss can still accurately generate effective differences in shaded areas, compensating for the lack of sensitivity of pixel loss to noise and providing good stability.

[0056] The structural similarity loss function is a function that measures the similarity between two images based on the global features of the images. In this embodiment, it mainly refers to the global luminance and contrast difference between the output image from the initial neural network and the label image. By adding this loss function, color shift in the network output can be effectively suppressed, and the overall image quality can be improved.

[0057] Adversarial loss is primarily the loss value of the true type of the output result and the output image by the classifier. In the later stages of training, as the difference between the output image from the initial neural network and the label image decreases, the effects of pixel loss, feature loss, and structural similarity loss gradually decrease, and the network convergence slows down. At this point, to support network training, one classifier network is trained synchronously. First, the output image and label image from the initial neural network are fed to the classifier, which determines whether the output image is the label image or not. Based on the classifier's output result and the true type of the output image, the loss is calculated and the classifier's parameters are updated. Then, the classifier's classification result for the output image is used as the loss of the degree of realism of the output image, and the classifier's parameters are updated with this loss. If the classifier cannot distinguish between the output image from the initial neural network and the label image, it indicates that training is complete. Adversarial loss can effectively remove image side effects caused by network processing (e.g., color mismatch problems in shaded and unshaded areas, and residual shading problems), and improve the degree of realism of the output image from the network.

[0058] Threshold cutoff loss. Due to lighting effects, paired data collected from real-world scenes may have subtle differences in brightness or color changes even in unshaded areas. However, these differences are within an acceptable range for the user and do not need to be processed. Therefore, to prevent the network from focusing on these small global differences during training, the method introduces threshold cutoff loss, meaning that only if the difference between the network output and GT is greater than a predetermined threshold, the difference is aggregated into the gradient of the overall loss calculation parameter; otherwise, the loss is considered to be 0. The loss function tolerates small differences between the network output and GT, and effectively improves the network's ability to remove relatively clear shading by shifting the center of gravity of network learning to areas with larger differences.

[0059] Shaded edge loss. First, the shaded area mask image is subjected to an expansion process to obtain an expanded image. Next, the shaded area mask image is subjected to an erosion process to obtain an erosion image. Then, the set of differences between the expanded image and the erosion image is obtained as the boundary region between shaded and unshaded areas. By smoothing this using TVLoss, an effective transition between shaded and unshaded areas can be achieved.

[0060] Shadow luminance loss increases the luminance of the shadowed areas in the shadow-removed image by making the difference between the luminance of the shadowed areas in the shadow-removed image output from the neural network and the luminance of the shadowed areas in the input image greater than 0.

[0061] In one selective embodiment, the background layer output module of the initial neural network uses the weighted sum of all the above losses as the total loss, and the Wassertein generative adversarial network as the adversarial loss.

[0062] This network structure improves the degree of shadow removal by extracting global and local features from the input image, while also protecting against side effects in unshaded areas.

[0063] Figures 5(a) and 5(b) are comparative diagrams of the processing effects achieved using the image processing method of the embodiment of the present invention. Figure 5(a) is the image to be processed, including shading, and Figure 5(b) is the image with shading removed after processing by the image processing method. As can be seen from the comparison of the two images, the image processing method of the present invention can effectively remove shading without causing significant side effects to the background layer.

[0064] The neural network structure and loss function employed in the embodiment of this application may be applied to application scenarios such as shadow removal, rain removal, and fog removal, and are mainly used for processing high-resolution images captured by mobile terminals such as mobile phones, but can also be applied to processing images of various resolutions on a PC or other embedded device.

[0065] According to another embodiment of the present invention, the present invention further provides an electronic device comprising a processor and a memory for storing executable instructions for the processor, wherein the processor is configured to perform any of the above-described image processing methods by executing the executable instructions.

[0066] According to another embodiment of the present invention, a storage medium storing a program is provided, which controls a device on which the storage medium is located so as to execute any of the above-described image processing methods when the program is executed.

[0067] Another embodiment of the present invention further provides an image processing apparatus. Referring to Figure 6, a structural block diagram of a selective image processing apparatus according to an embodiment of the present invention is shown. As shown in Figure 6, the image processing apparatus 60 includes an image acquisition unit 600 and a processing unit 602.

[0068] The following describes in detail each unit included in the image processing device 60. The image acquisition unit 600 acquires the image to be processed, including shaded areas.

[0069] The processing unit 602 receives the image to be processed and processes it using a trained neural network to obtain a de-shading image. Here, the neural network includes a first-stage network and a second-stage network connected in cascaded order, and the image to be processed and the output image from the first-stage network are simultaneously input to the second-stage network.

[0070] In one selective embodiment, the structure of the neural network is as shown in Figure 2 and the relevant description herein, and will not be repeated here.

[0071] The numbering of the above embodiments in this application is for descriptive purposes only and does not indicate any superiority or inferiority among the embodiments.

[0072] In the embodiments described above, each embodiment has a different emphasis, and for parts not explained in detail in one embodiment, you can refer to the relevant descriptions in other embodiments.

[0073] It should be understood that the technical details presented in some embodiments of this application can be realized in other forms. Herein, the embodiments of the apparatus described above are merely illustrative, and for example, the division of the units may be a division of logical function, there may be other division methods in actual implementation, for example, multiple units or components may be combined or integrated into other systems, or some features may be omitted or not performed. Furthermore, the coupling, direct coupling or communication connection between the listed or considered components may be an indirect coupling or communication connection via several interfaces, units or modules, and may be in an electrical or other form.

[0074] The unit described as a separating member may or may not be physically separated, and the member displayed as a unit may or may not be a physical unit; that is, it may be located in one place or distributed among multiple units. Depending on the actual needs, some or all of these units may be selected to realize the objective of the technical proposal of this embodiment.

[0075] Furthermore, each functional unit in each embodiment of the present application may be integrated into a single processing unit, each unit may exist physically independently, or two or more units may be integrated into a single unit. The integrated unit may be implemented in hardware form or in the form of a software functional unit.

[0076] If the aforementioned integrated unit is implemented in the form of a software function unit and sold or used as an independent product, it may be stored on a single computer-readable storage medium. Based on this understanding, the essential or prior art contributions of the present invention, or all or part of the present invention, may be embodied in the form of a software product, which is stored on a single storage medium containing some instructions for causing a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage mediums include various media capable of storing program code, such as USB disks, read-only memory (ROM), random access memory (RAM), removable hard disks, magnetic disks, or optical disks.

[0077] The foregoing are merely preferred embodiments of the present application, and it should be noted that those skilled in the art can make some improvements and modifications without departing from the principles of the present application, and these improvements and modifications should also be considered within the scope of protection of the present application.

[0078] This application claims priority to Chinese Patent Application No. 202111210502.3, filed on 18 October 2021, and incorporates the contents of this Chinese Patent Application as part of this application.

Claims

1. To construct an initial neural network, The initial neural network is trained using sample data, and the trained neural network is obtained. To obtain the image to be processed, which includes shaded areas, This includes inputting the image to be processed into the trained neural network to obtain a shadow-removed image, The initial neural network and the trained neural network include a first-stage network and a second-stage network connected in cascaded order, wherein the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network simultaneously receives the image to be processed and the shaded area mask image and outputs a shaded area removal image, in an image processing method, The aforementioned sample data includes real-life images and composite shaded images, and the composite shaded image is created by combining a simple shaded image and a non-shaded image using an image synthesis method. The initial neural network further includes a module that performs classification on sample data, and this module If the initial neural network determines that the sample data input is a real-world image, the labeled data is a de-shading image collected from a real scene, and the parameters within the second-stage network are adjusted based on the difference between the de-shading image output from the initial neural network and the de-shading image as labeled data. If the sample data input to the initial neural network is determined to be a composite shaded image, the labeled data includes the unshaded image and the simple shaded image collected from a real scene, and the parameters inside the first-stage network are adjusted based on the difference between the shaded area mask image and the simple shaded image, and the parameters inside the second-stage network are adjusted based on the difference between the deshaded image output from the initial neural network and the unshaded image.

2. The aforementioned first-stage network is A first feature extraction module, which includes a first encoder, extracts features of the image to be processed layer by layer to obtain a first set of feature data, The image processing method according to claim 1, further comprising: a shaded region estimation module connected to the output of the first feature extraction module, including a first decoder, which estimates shaded regions based on the first set of feature data and outputs a shaded region mask image.

3. The aforementioned second-stage network is A second feature extraction module, which includes a second encoder, is connected to the output of the first-stage network, receives the image to be processed, and receives the shaded area mask image output from the first-stage network to obtain a second set of feature data, The image processing method according to claim 1, further comprising: a result image output module connected to the output of the second feature extraction module, including a second decoder, which outputs the de-shading image based on the second set of feature data.

4. If the first stage network includes a first encoder and a first decoder, the output of each layer of the first decoder is spliced ​​along the channel axis to the output of the corresponding layer in the first encoder via a cross-layer connection, and a multiscale pyramid pooling module that fuses features of different scales is added to the cross-layer connection between the first decoder and the first encoder. The image processing method according to claim 3, characterized in that, if the second-stage network includes a second encoder and a second decoder, the output of each layer of the second decoder is spliced ​​along the channel axis to the output of the corresponding layer in the second encoder by cross-layer connection, and a multiscale pyramid pooling module that fuses features of different scales is added to the cross-layer connection between the second decoder and the second encoder.

5. After acquiring the image to be processed, which includes shaded areas, the image processing method is as follows: The image to be processed is downsampled using an image pyramid algorithm, and the gradient information of each image layer is retained during downsampling to form a Laplacian pyramid. The process involves feeding the smallest possible image layer to the trained neural network to obtain an output image, and The image processing method according to claim 1, further comprising reconstructing the output image from low resolution to high resolution using a Laplacian pyramid to obtain the shadow-removed image.

6. Using an image synthesis method to synthesize the composite shaded image from a simple shaded image and a non-shaded image is, Obtaining a simple shaded image, To obtain an image without shading, and The image processing method according to claim 1, characterized in that it includes obtaining the composite shaded image based on the simple shaded image and the unshaded image.

7. Using an image synthesis method to synthesize the composite shaded image from a simple shaded image and a non-shaded image is, The method further includes converting the simple shaded image and obtaining the composite shaded image based on the converted simple shaded image and the unshaded image, The image processing method according to claim 6, characterized in that the pixel values ​​of the non-shaded areas in the converted simple shaded image are all set to a single fixed value a, the pixel values ​​of the shaded areas are values ​​between 0 and a, and a is a positive integer.

8. The image processing method according to claim 1, characterized in that when training the initial neural network using sample data, the loss function includes at least one of pixel loss, feature loss, structural similarity loss, adversarial loss, shaded edge loss, and shaded brightness loss.

9. The aforementioned pixel loss includes pixel cutoff loss, If the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is greater than a predetermined threshold, the loss of the two pixels is calculated. The image processing method according to claim 8, characterized in that if the absolute difference between two corresponding pixels in the output image from the initial neural network and the label image is less than or equal to the predetermined threshold, the difference between the two pixels is ignored.

10. The image processing method according to claim 8, characterized in that the shadow brightness loss makes the difference between the brightness of the region corresponding to the shadowed area in the shadow removal image output from the neural network and the brightness of the shadowed area in the input image to be processed greater than 0, thereby improving the brightness of the region corresponding to the shadowed area in the shadow removal image.

11. If the loss function includes the shaded edge loss, the image processing method is The aforementioned shaded area mask image is subjected to an expansion process to obtain an expanded image, The process involves performing an erosion process on the aforementioned shaded area mask image to obtain an erosion image, and The image processing method according to claim 8, characterized by comprising obtaining the set of differences between the dilated image and the erosion image as the boundary region between shaded and unshaded areas, and smoothing it using TVLoss.

12. An image acquisition unit that acquires the image to be processed, including shaded areas, The system includes a processing unit that trains an initial neural network using sample data, obtains the trained neural network, receives an image to be processed, and processes the image using the trained neural network to obtain a shadow-removed image. The initial neural network and the trained neural network include a first-stage network and a second-stage network connected in cascaded order, wherein the first-stage network receives the image to be processed and outputs a shaded area mask image, and the second-stage network simultaneously receives the image to be processed and the shaded area mask image and outputs a deshading image, in an image processing apparatus. The aforementioned sample data includes real-life images and composite shaded images, and the composite shaded image is created by combining a simple shaded image and a non-shaded image using an image synthesis method. The initial neural network further includes a module that performs classification on sample data, and this module If the initial neural network determines that the sample data input is a real-world image, the labeled data is a de-shading image collected from a real scene, and the parameters within the second-stage network are adjusted based on the difference between the de-shading image output from the initial neural network and the de-shading image as labeled data. If the sample data input to the initial neural network is determined to be a composite shaded image, the labeled data includes the unshaded image and the simple shaded image collected from the actual scene. Based on the difference between the shaded area mask image and the simple shaded image, the parameters inside the first-stage network are adjusted, and based on the difference between the deshaded image output from the initial neural network and the unshaded image, the parameters inside the second-stage network are adjusted.

13. The aforementioned first-stage network is A first feature extraction module, which includes a first encoder, extracts features of the image to be processed layer by layer to obtain a first set of feature data, A shaded region estimation module connected to the output of the first feature extraction module, including a first decoder, which estimates shaded regions based on the first set of feature data and outputs a shaded region mask image, The image processing apparatus according to claim 12, characterized by including the following:

14. The aforementioned second-stage network is A second feature extraction module, which includes a second encoder, is connected to the output of the first-stage network, receives the image to be processed, and receives the shaded area mask image output from the first-stage network to obtain a second set of feature data, A result image output module connected to the output of the second feature extraction module, including a second decoder, which outputs a deshading image based on the second set of feature data, The image processing apparatus according to claim 12, characterized by including the following:

15. A storage medium that stores a program, characterized in that when the program is executed, it controls the device on which the storage medium is located so as to execute the image processing method described in any one of claims 1 to 11.

16. It is an electronic device, Processor and Includes a memory for storing executable commands of the processor, The electronic device is characterized in that the processor is configured to perform the image processing method according to any one of claims 1 to 11 by executing the executable commands.

Images