Image processing method and device, storage medium and electronic device

Abstract

The application discloses an image processing method and device, a storage medium and an electronic device. The image processing method comprises the following steps: acquiring a to-be-processed image containing a first region; inputting the to-be-processed image into a trained neural network to obtain a result image in which the first region is removed, wherein the neural network contains a double-branch network. The application can solve the technical problem that the prior art is prone to causing side effects on an image background layer while eliminating an interference region and has high requirements on a hardware platform.

Description

Technical Field

[0001] This invention relates to image processing technology, and more specifically, to an image processing method, apparatus, storage medium, and electronic device. Background Technology

[0002] It is common for photographic images to contain distracting areas such as reflections, shadows, water stains, and obstructions. These distracting areas usually have a negative impact on the aesthetics and quality of the image.

[0003] One existing fast single image reflection suppression algorithm based on convex optimization (Fast Single Image Reflection Suppression via Convex Optimization) is a method that eliminates reflections by solving a convex optimization problem based on prior knowledge. It obtains the convex function to be solved using the prior knowledge that "the reflection image is a linear superposition of the background layer and the reflection layer," and then uses convex optimization to find the optimal solution, returning the optimal solution as the reflection suppression result. However, this algorithm ignores the complexity and diversity of reflections in real-world scenes, and is only effective in simple reflection scenarios. When dealing with scenes with complex reflections and diverse textures, it not only fails to accurately eliminate reflections but also incorrectly eliminates textures in other non-reflective areas. Furthermore, this algorithm requires highly complex cosine transforms and inverse cosine transforms, resulting in significant time consumption and poor usability on ordinary mobile devices.

[0004] Another existing deep learning-based reflection removal method trains a convolutional neural network by inputting reflection image samples and using background image samples as supervision. The training process is supervised by various loss functions, resulting in a reflection removal network model. This model is then used to process the reflection image to obtain a de-reflected result. However, this algorithm uses a very large network model, including a complex VGG19 pre-trained head and 13 residual network blocks, resulting in very slow inference speed and high device performance requirements, making it unsuitable for deployment on mobile devices. Furthermore, during training, this algorithm only uses the background image as supervision, ignoring the reflection layer component, thus failing to fully allow the network to learn the features of the reflection layer, leading to poor performance.

[0005] Therefore, it is necessary to propose an image processing technique that can effectively eliminate interference areas such as reflections, shadows, water stains, and obstructions without causing significant side effects on the background layer, while also having a fast running speed and acceptable hardware configuration requirements. Summary of the Invention

[0006] The present invention provides an image processing method, apparatus, storage medium, and electronic device to at least solve the technical problems in the prior art that it is easy to cause side effects on the image background layer while eliminating interference areas and that it has high requirements for hardware platforms.

[0007] According to one aspect of the present invention, an image processing method is provided, comprising: acquiring an image to be processed containing a first region; inputting the image to be processed into a trained neural network to obtain a result image with the first region removed, wherein the neural network comprises a dual-branch network.

[0008] Optionally, the neural network includes: a feature extraction module, comprising an encoder, for extracting features of the image to be processed layer by layer to obtain feature data; a first image output module, connected to the output of the feature extraction module, comprising a first decoder, for outputting a first image based on the feature data; a second image output module, connected to the output of the feature extraction module, comprising a second decoder, for outputting a second image based on the feature data; and a result image output module, connected to the first image output module and / or the second image output module, for outputting a result image of removing the first region based on the first image and / or the second image.

[0009] Optionally, the first area is at least one of the following types: reflective area, shadow area, water stain area, or occluded area.

[0010] Optionally, the first image is a background image, and the second image is at least one of the following types: reflective image, shadow image, water stain image, or occlusion image.

[0011] Optionally, the outputs of each layer of the first decoder and the second decoder are spliced ​​along the channel axis with the output of the corresponding layer of the encoder through cross-layer connections. A multi-scale pyramid pooling module is added to the cross-layer connections between the encoder and the first decoder and the second decoder. The multi-scale pyramid pooling module fuses features of different scales.

[0012] Optionally, a multi-scale feature selection module is added to the cross-layer connection between the encoder and the first decoder and the second decoder. The multi-scale feature selection module dynamically selects and adjusts the features and proportions of each scale.

[0013] Optionally, the convolutional layers in the encoder are grouped convolutions.

[0014] Optionally, the convolution operator in the neural network is gated convolution.

[0015] Optionally, after obtaining the image to be processed containing the first region, the image processing method further includes: downsampling the image to be processed using an image pyramid algorithm, and saving the gradient information of each layer to form a Laplacian pyramid while downsampling; feeding the smallest layer into a trained neural network to obtain an output image; and using the Laplacian pyramid to reconstruct the output image from low resolution to high resolution to obtain a result image with the first region removed.

[0016] Optionally, the above image processing method further includes: constructing an initial neural network; training the initial neural network using sample data to obtain the trained neural network, wherein the sample data includes a first sample image and a second sample image obtained using an image synthesis method.

[0017] Optionally, obtaining the second sample image using an image synthesis method includes: acquiring an original image and a reference image; determining a first region on the original image; generating a mask image of the same size as the original image, and feathering the boundary of the second region corresponding to the first region to obtain a mask image with a blurred effect; applying Gaussian blur to the reference image to obtain a reference image with a blurred effect; and synthesizing the second sample image based on the original image, the mask image with the blurred effect, and the reference image with the blurred effect.

[0018] Optionally, synthesizing the second sample image based on the original image, the mask image of the blurred effect, and the reference image of the blurred effect includes: normalizing the mask image of the blurred effect to obtain a normalized mask image; multiplying the pixels in the normalized mask image element-wise with the reference image of the blurred effect in the second region corresponding to the first region to obtain an intermediate image; and linearly superimposing the original image and the intermediate image to synthesize the second sample image.

[0019] Optionally, when training the initial neural network using sample data, the loss function includes at least one of the following: pixel loss, feature loss, structural similarity loss, and adversarial loss.

[0020] Optionally, the pixel loss includes pixel truncation loss. When the absolute difference between two corresponding pixels in the output image of the initial neural network and the label image is greater than a given threshold, the loss of the two pixels is calculated; when the absolute difference between two corresponding pixels in the output image of the initial neural network and the label image is not greater than the given threshold, the difference of the two pixels is ignored.

[0021] Optionally, the above image processing method uses a VGG19 network as the feature extractor of the initial neural network to calculate the feature loss.

[0022] According to another aspect of the present invention, an image processing apparatus is provided, comprising: an image acquisition unit for acquiring an image to be processed containing a first region; and a processing unit for receiving the image to be processed and processing the image to be processed using a trained neural network to obtain a result image with the first region removed, wherein the neural network comprises a dual-branch network.

[0023] Optionally, the neural network includes: a feature extraction module, comprising an encoder, for extracting features of the image to be processed layer by layer to obtain feature data; a first image output module, connected to the output of the feature extraction module, comprising a first decoder, for outputting a first image based on the feature data; a second image output module, connected to the output of the feature extraction module, comprising a second decoder, for outputting a second image based on the feature data; and a result image output module, connected to the first image output module and / or the second image output module, for outputting a result image of removing the first region based on the first image and / or the second image.

[0024] Optionally, the outputs of each layer of the first decoder and the second decoder are spliced ​​along the channel axis with the output of the corresponding layer of the encoder through cross-layer connections. A multi-scale pyramid pooling module is added to the cross-layer connections between the encoder and the first decoder and the second decoder, and the multi-scale pyramid pooling module fuses features of different scales.

[0025] According to another aspect of the present invention, a storage medium is also provided, including a stored program, wherein the program controls the device where the storage medium is located to execute the image processing method described in any one of the above embodiments when the program is running.

[0026] According to another aspect of the present invention, an electronic device is also provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method described in any one of the preceding embodiments by executing the executable instructions.

[0027] This invention proposes a fast and effective image processing method that can capture the characteristics of interference areas such as reflective areas, shadow areas, water stains, and occlusion areas. By synthesizing highly realistic sample images as training materials, and combining various loss functions and effective network structures for training, it can effectively eliminate interference areas, improve image quality and aesthetics, and achieve fast processing speed when processing high-resolution images by employing downsampling and network pruning techniques. This method is suitable for smart terminals with limited computing power, such as mobile phones. Attached Figure Description

[0028] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings: Figure 1 This is a flowchart of an optional image processing method according to an embodiment of the present invention; Figure 2 This is a structural diagram of an optional neural network according to an embodiment of the present invention; Figure 3 This is a flowchart of an optional training neural network according to an embodiment of the present invention; Figure 4 This is a flowchart of an optional image synthesis method according to an embodiment of the present invention; Figure 5 This is a structural block diagram of an optional image processing apparatus according to an embodiment of the present invention. Detailed Implementation

[0029] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0030] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such orders can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0031] The following is a flowchart illustrating an optional image processing method according to an embodiment of the present invention. It should be noted that the steps shown in the flowchart can be executed in a computer system such as a set of computer-executable instructions, and although a logical order is shown in the flowchart, in some cases the steps shown or described may be executed in a different order than that shown here.

[0032] refer to Figure 1 This is a flowchart of an optional image processing method according to an embodiment of the present invention. Figure 1 As shown, the image processing method includes the following steps: S100, Obtain the image to be processed containing the first region; S102, the image to be processed is input into a trained neural network to obtain a result image with the first region removed, wherein the neural network includes a two-branch network.

[0033] In one alternative embodiment, the first region can be a reflective region, a shadow region, a water stain region, etc.

[0034] In one alternative embodiment, such as Figure 2 As shown, the neural network is a two-branch network, including a feature extraction module, a first image output module, a second image output module, and a result image output module.

[0035] The feature extraction module, which includes an encoder, is used to extract features from the image to be processed layer by layer to obtain feature data. A first image output module is connected to the output of the feature extraction module and includes a first decoder for outputting a first image based on feature data. The second image output module is connected to the output of the feature extraction module and includes a second decoder for outputting a second image based on feature data. The result image output module is connected to the first image output module and / or the second image output module, and is used to output the result image of removing the first region based on the first image and / or the second image.

[0036] The first image is the background image, and the second image is a reflection image, a shadow image, a water stain image, an occlusion image, etc.

[0037] In applications such as removing reflections, shadows, and water stains from images, the result image output module can directly discard the second image output by the second image output module and directly use the first image output by the first image output module as the result image after removing interference factors such as reflections, shadows, water stains, and occlusions. The processed result image can be used to improve the accuracy of subsequent image processing, such as image segmentation, recognition, and classification.

[0038] The first image output module and the second image output module constitute a two-branch structure that takes the feature data output by the feature extraction module as input. By simultaneously supervising the training of the two branches of the first image output module and the second image output module, compared with the single-output network structure, the neural network in this embodiment can better learn the differences between the first image and the second image, extract the first region more accurately, and obtain the result image with the first region removed.

[0039] The outputs of each layer in the first and second decoders are concatenated along the channel axis with the outputs of the corresponding layers in the encoder via cross-layer connections. A multi-scale pyramid pooling module is added to the cross-layer connections between the encoder and the first and second decoders. This module includes multiple pooling layers, convolutional layers, and interpolation upsampling layers with different kernel sizes. First, pooling layers extract features at different scales. Then, convolutional layers extract low-level and / or high-level features. Next, interpolation upsampling layers adjust the outputs of the corresponding layers in the encoder, first decoder, and second decoder to the same size. Finally, these are concatenated along the channel axis to form a single feature. Since the influence and area of ​​the first region on the image vary greatly across different images, the determination of the first region must consider both local texture features and global semantic information. The multi-scale pyramid pooling module fuses features at different scales, enhancing the network's generalization ability and enabling it to achieve better results when processing images with first regions of varying sizes and degrees.

[0040] In another optional embodiment, a multi-scale feature selection module is added to the cross-layer connections between the encoder and the first and second decoders. This module dynamically selects and adjusts features and their proportions at each scale, further enhancing the model's adaptability to different reflective areas. The multi-scale feature learning module consists of four parallel convolutional layers with different dilation coefficients and a feature fusion layer. The dilation coefficients of the convolutional layers are doubled sequentially, equivalent to expanding the receptive field by 2, 4, and 8 times respectively. Features from the four different receptive fields are merged along the channel axis, and the feature fusion layer dynamically adjusts the selection weights of each set of features. By dynamically fusing and selecting features at different scales in a parametrically learnable manner, the model's processing performance for reflective maps of different areas and degrees is further enhanced.

[0041] To improve the model's running speed on the device, the model can be pruned by replacing the convolutional layers in the encoder with grouped convolutions, where each convolutional kernel convolves only one channel, thereby reducing the model's computational load and improving processing speed.

[0042] To improve the model's generalization ability and deglare reduction performance, the convolution operator was changed to gated convolution. Ordinary convolution treats every pixel as valid input, which is unsuitable for reflective scenes. Pixels in the reflective and background layers need to be distinguished, and different feature selection weights are used during convolution computation. Gated convolution provides an online-learnable, dynamic feature selection mechanism for each location in each feature layer. This learned mechanism distinguishes between the reflective and background layers and dynamically adjusts the feature selection weights for each layer based on the intensity of the reflection. Through dynamic learning and feature selection of the reflective and background layers, the model can better distinguish between them and adapt to different reflection intensities, further improving deglare reduction and generalization capabilities.

[0043] When the image to be processed has a high resolution or a large amount of data, directly feeding the image to be processed into the trained neural network can cause memory overflow or excessive processing time, affecting the user experience. To solve this problem, conventional interpolation scaling algorithms can be used, but this can easily lead to the loss of image information, making it impossible to perfectly enlarge the generated image to the original image.

[0044] Considering that the first region typically lacks significant gradient information, in an optional embodiment, an image pyramid algorithm can be used to first downsample the image to be processed, while simultaneously preserving the gradient information of each layer to form a Laplacian pyramid. The smallest layer is then fed into a trained neural network to obtain the output image. Finally, the Laplacian pyramid is used to reconstruct the output image from low resolution to high resolution, resulting in the image with the first region removed. Since the gradient information in the first region is very weak, even if the reconstruction process restores some gradient information from the image to be processed, it will not affect the effectiveness of removing the first region. By utilizing the gradient information of each layer preserved during downsampling for image reconstruction, reflections can be eliminated without affecting image resolution. By introducing downsampling and image reconstruction, the speed of image processing is guaranteed, and the quality before and after processing is not affected, which is beneficial for processing high-resolution images on devices with limited computing power, such as mobile phones.

[0045] like Figure 3 As shown, in order to obtain the trained neural network, the image processing method further includes: S300: Construct the initial neural network; S302: Train the initial neural network using sample data to obtain a trained neural network, wherein the sample data includes a first sample image and a second sample image obtained using an image synthesis method.

[0046] In one optional embodiment, the first sample image is a background sample image, and the second sample image is a reflective sample image, a shadow sample image, a water stain sample image, an occlusion sample image, etc.

[0047] The sample data used to train the initial neural network plays a crucial role in the entire image processing method. There are two main methods for obtaining sample data: real-scene acquisition and image synthesis.

[0048] For example, for an image to be processed that contains reflective areas, the sample data used to train the initial neural network includes background sample images and reflective sample images.

[0049] In the real-scene data collection method, the data collectors select an outdoor or indoor environment with good lighting, choose a suitable reflection angle, fix the mobile phone or camera with a tripod, place a piece of transparent glass in front of the lens, adjust the appropriate angle to create reflection, and then take a picture to obtain a reflection sample image; then remove the glass, take another picture, and obtain the corresponding background sample image.

[0050] However, real-world data acquisition often fails to guarantee high-quality sample data. On the one hand, due to the movement of objects such as pedestrians and vehicles in the scene, as well as the refraction of light through glass, it is difficult for reflective sample images to be perfectly aligned with background sample images. On the other hand, due to changes in lighting or camera focus, noise is generated in both reflective and background sample images. All of these factors can significantly impact the training of the network.

[0051] To address this, image synthesis methods can be used to generate more realistic reflective sample images.

[0052] like Figure 4 As shown, in an optional embodiment, the method of synthesizing a second sample image using images includes: S400: Acquire the original image and reference image; The original image and the reference image can be any two images acquired by a camera device, or any two images in the dataset. The original image and the reference image are not required to correspond.

[0053] S402: Determine the first region on the original image; In an optional embodiment, determining the first region on the original image includes: randomly selecting a number of points on the original image and connecting them to form a connected region, and using the connected region as the first region; S404: Generate a mask image of the same size as the original image, and feather the boundary of the second region corresponding to the first region in the mask image to obtain a mask image with a blurred effect; In an optional embodiment, the pixel value in the region corresponding to the first region in the mask image is 255, and the pixel value in other regions is 0. The feathering process for the boundary of the region corresponding to the first region can be performed using methods such as Gaussian blur.

[0054] S406: Apply Gaussian blur to the reference image to obtain a blurred reference image; S408: Synthesize a second sample image based on the original image, the mask image with the blurred effect, and the reference image with the blurred effect.

[0055] In an optional embodiment, synthesizing the second sample image based on the original image, the mask image of the blurred effect, and the reference image of the blurred effect includes: Normalize the mask image with the blurred effect to obtain a normalized mask image; In the second region corresponding to the first region, the pixels in the normalized mask image are multiplied element-wise with the reference image of the blurred effect to obtain an intermediate image; The original image and the intermediate image are linearly superimposed to synthesize a second sample image.

[0056] The second sample image synthesized through the above steps S400-S408 takes into account the characteristics of the first region, namely that the first region may not occupy the entire second sample image, but only a certain area of ​​the second sample image. Using this synthesized second sample image as sample data can guide the network to identify the first region and the regions outside the first region, eliminating the side effects of the network on the regions outside the first region due to misidentification. The randomly generated connected regions and their boundary feathering operations make the boundary of the first region random and blurred, improving the network's ability to identify the first region.

[0057] In an optional embodiment, the method for acquiring sample data may further include performing one or more processing steps on the acquired sample data, such as random flipping, rotation, channel swapping, or adding random noise, to enrich the sample data and increase the robustness of the network.

[0058] In an alternative embodiment, when training an initial neural network using sample data, the loss function includes at least one of the following: pixel loss, feature loss, structural similarity loss, and adversarial loss.

[0059] The pixel loss function measures the similarity between two images at the pixel level, primarily consisting of pixel value loss and gradient loss. In this embodiment, it mainly refers to the weighted sum of the mean square error of pixel values ​​and the L1 norm error of the gradients of the two images compared to the output image and label image of the initial neural network. Pixel loss supervises the training process at the pixel level, ensuring that the pixel values ​​of each pixel in the output image and label image of the initial neural network are as close as possible. To guide the initial neural network to focus on the differences between the first sample image and the second sample image in the first region rather than the noise of the entire image, in an optional embodiment, a pixel truncation loss can be introduced. This truncates the pixel loss, meaning that the loss of two pixels is only calculated when the absolute difference between two corresponding pixels in the output image and label image is greater than a given threshold; otherwise, the difference between the two pixels is ignored. Adding the pixel truncation loss guides the network to focus on the first region, suppressing image noise. This not only enhances the effect of removing noise from the first region but also significantly accelerates the convergence speed of the network.

[0060] The feature loss function measures the similarity between two images based on their semantic features. A common method is to feed the images into a pre-trained network and extract features at different levels, then calculate the similarity of these features. In this embodiment, the feature loss mainly refers to the weighted sum of the L1 norm errors of the corresponding features of the input image and the label image in the initial neural network. In an optional embodiment, a VGG19 network pre-trained on the ImageNet dataset is used as the feature extractor. The output image and the label image of the initial neural network are fed into this feature extractor to obtain the features of each layer of the VGG19 network. Then, the L1 norm errors of the corresponding features of the input image and the label image are calculated and summed with weights. The features of each layer of the VGG19 network are not sensitive to image details and noise, and have good semantic properties. Therefore, even if the input and output images have defects such as noise or misalignment, the feature loss can still accurately generate effective differences in reflective areas, making up for the lack of sensitivity to noise in pixel loss and exhibiting good stability.

[0061] The structural similarity loss function measures the similarity between two images based on their global features. In this embodiment, it mainly refers to the global difference in brightness and contrast between the output image and the label image of the initial neural network. Adding this loss function can effectively suppress color casts in the network output and improve the overall image quality.

[0062] Adversarial loss primarily refers to the loss value between the discriminator's output and the true category of the output image. In the later stages of training, when the difference between the initial neural network's output image and the label image becomes smaller, the effects of pixel loss, feature loss, and structural similarity loss gradually diminish, and network convergence slows down. At this point, a discriminator network is trained simultaneously to assist the network's training. First, the output image and label image of the initial neural network are fed into the discriminator. The discriminator determines whether the output image is the label image, calculates the adversarial loss based on the discriminator's output and the true category of the output image, and updates the discriminator parameters. When the discriminator can no longer distinguish between the initial neural network's output image and the label image, training ends. Adversarial loss effectively eliminates image side effects caused by network processing, improving the realism of the network's output image.

[0063] In an optional embodiment, the first image output module of the initial neural network uses the weighted sum of the above-mentioned pixel loss, feature loss and structural similarity loss as the total loss, and adopts Wassertein generative adversarial network as adversarial loss; the second image output module of the initial neural network uses the difference between the grayscale image of the input image and the grayscale image of the label image as the label image, and uses pixel loss as the total loss.

[0064] In summary, the neural network structure and loss function used in the embodiments of the present invention can be applied to application scenarios such as removing reflections, removing shadows, removing rain, removing fog, and removing fences. It is mainly used to process high-resolution images taken by mobile phones, but it is also applicable to processing images of various resolutions on PCs or other embedded devices.

[0065] The image processing method provided by this invention can effectively eliminate reflective areas without producing significant side effects on the background image.

[0066] The image processing method provided by this invention can effectively eliminate shadow areas without producing significant side effects on the background image.

[0067] Those skilled in the art will understand that by replacing the training data, similarly good results can be achieved in removing shadows, water stains (e.g., rain and fog), and obstructions (e.g., fences). In the field of intelligent driving, the ability to process images acquired by in-vehicle cameras in a timely manner and obtain clear, high-quality images free of interference factors provides a guarantee for assisted driving functions.

[0068] According to another aspect of the present invention, an electronic device is also provided, including: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to perform the image processing method of any of the above-described embodiments by executing the executable instructions.

[0069] According to another aspect of the present invention, a storage medium is also provided, the storage medium including a stored program, wherein, when the program is running, it controls the device where the storage medium is located to execute the image processing method described above.

[0070] According to another aspect of the present invention, an image processing apparatus is also provided. (See reference) Figure 5 This is a structural block diagram of an optional image processing apparatus according to an embodiment of the present invention. Figure 5 As shown, the image processing device 70 includes an image acquisition unit 700 and a processing unit 702.

[0071] The various units included in the image processing apparatus 70 will now be described in detail.

[0072] Image acquisition unit 700 is used to acquire an image to be processed containing a first region.

[0073] The processing unit 702 is used to receive the image to be processed and to process the image using a trained neural network to obtain a result image with the first region removed, wherein the neural network includes a two-branch network.

[0074] In one alternative embodiment, the first region can be a reflective region, a shadow region, a water stain region, an obstruction, etc.

[0075] In one alternative embodiment, the neural network is structured as follows: Figure 2 As shown in the figure and the relevant descriptions in this article, they will not be repeated here.

[0076] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0077] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0078] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0079] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0080] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0081] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0082] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image processing method, comprising: Obtain an image to be processed containing a first region, wherein the first region is at least one of the following types: reflective region, shadow region, water stain region, or occluded region; The image to be processed is input into a trained neural network to obtain a result image with the first region removed, wherein the neural network includes a two-branch network; The neural network includes: The feature extraction module includes an encoder, which is used to extract features of the image to be processed layer by layer to obtain feature data; A first image output module is connected to the output of the feature extraction module and includes a first decoder for outputting a first image based on the feature data, wherein the first image is a background image. The second image output module is connected to the output of the feature extraction module and includes a second decoder for outputting a second image based on the feature data. The second image is at least one of the following types: reflection image, shadow image, water stain image, and occlusion image. The result image output module is connected to the first image output module and the second image output module, and is used to output a result image of removing the first region based on the first image and the second image; The method also includes: Construct the initial neural network; The initial neural network is trained using sample data to obtain the trained neural network. The sample data includes a first sample image and a second sample image obtained using an image synthesis method; The second sample image was obtained using image synthesis methods, including: Obtain the original image and the reference image; Determining the first region on the original image includes: randomly selecting several points on the original image and connecting them to form a connected region, and using the connected region as the first region; Generate a mask image of the same size as the original image, and feather the boundary of the second region corresponding to the first region to obtain a mask image with a blurred effect; Gaussian blur is applied to the reference image to obtain a blurred reference image; The process of synthesizing the second sample image based on the original image, the mask image of the blurred effect, and the reference image of the blurred effect includes: normalizing the mask image of the blurred effect to obtain a normalized mask image; multiplying the pixels in the normalized mask image element-wise with the reference image of the blurred effect in the second region corresponding to the first region to obtain an intermediate image; and linearly superimposing the original image and the intermediate image to synthesize the second sample image.

2. The image processing method according to claim 1, characterized in that, The outputs of each layer of the first decoder and the second decoder are spliced ​​along the channel axis with the output of the corresponding layer of the encoder through cross-layer connections. A multi-scale pyramid pooling module is added to the cross-layer connections between the encoder and the first decoder and the second decoder. The multi-scale pyramid pooling module fuses features of different scales.

3. The image processing method according to claim 2, characterized in that, A multi-scale feature selection module is added to the cross-layer connection between the encoder and the first decoder and the second decoder. The multi-scale feature selection module dynamically selects and adjusts the features and proportions of each scale.

4. The image processing method according to claim 1, characterized in that, The convolutional layers in the encoder are grouped convolutions.

5. The image processing method according to claim 1, characterized in that, The convolution operator in the neural network is gated convolution.

6. The image processing method according to claim 1, characterized in that, After acquiring the image to be processed containing the first region, the image processing method further includes: The image to be processed is downsampled using an image pyramid algorithm, and the gradient information of each layer is saved during downsampling to form a Laplacian pyramid; The smallest layer is fed into a trained neural network to obtain the output image; The output image is reconstructed from low resolution to high resolution using the Laplacian pyramid to obtain a result image with the first region removed.

7. The image processing method according to claim 1, characterized in that, When training the initial neural network using sample data, the loss function includes at least one of the following: pixel loss, feature loss, structural similarity loss, and adversarial loss.

8. The image processing method according to claim 7, characterized in that, The pixel loss includes pixel truncation loss. When the absolute difference between two corresponding pixels in the output image of the initial neural network and the label image is greater than a given threshold, the loss of the two pixels is calculated; when the absolute difference between two corresponding pixels in the output image of the initial neural network and the label image is not greater than the given threshold, the difference of the two pixels is ignored.

9. The image processing method according to claim 7, characterized in that, The VGG19 network was used as the feature extractor of the initial neural network to calculate the feature loss.

10. An image processing apparatus, comprising: An image acquisition unit is used to acquire an image to be processed containing a first region, wherein the first region is at least one of the following types: reflective region, shadow region, water stain region, or occluded region. A processing unit is configured to receive an image to be processed and process the image using a trained neural network to obtain a result image with the first region removed, wherein the neural network includes a two-branch network. A training unit is used to construct an initial neural network; the initial neural network is trained using sample data to obtain the trained neural network, wherein the sample data includes a first sample image and a second sample image obtained using an image synthesis method; the neural network includes: The feature extraction module includes an encoder, which is used to extract features of the image to be processed layer by layer to obtain feature data; A first image output module is connected to the output of the feature extraction module and includes a first decoder for outputting a first image based on the feature data, wherein the first image is a background image. The second image output module is connected to the output of the feature extraction module and includes a second decoder for outputting a second image based on the feature data. The second image is at least one of the following types: reflection image, shadow image, water stain image, and occlusion image. The result image output module is connected to the first image output module and the second image output module, and is used to output a result image of removing the first region based on the first image and the second image; The second sample image was obtained using image synthesis methods, including: Obtain the original image and the reference image; Determining the first region on the original image includes: randomly selecting several points on the original image and connecting them to form a connected region, and using the connected region as the first region; Generate a mask image of the same size as the original image, and feather the boundary of the second region corresponding to the first region to obtain a mask image with a blurred effect; Gaussian blur is applied to the reference image to obtain a blurred reference image; The process of synthesizing the second sample image based on the original image, the mask image of the blurred effect, and the reference image of the blurred effect includes: normalizing the mask image of the blurred effect to obtain a normalized mask image; multiplying the pixels in the normalized mask image element-wise with the reference image of the blurred effect in the second region corresponding to the first region to obtain an intermediate image; and linearly superimposing the original image and the intermediate image to synthesize the second sample image.

11. The image processing apparatus according to claim 10, characterized in that, The outputs of each layer of the first decoder and the second decoder are spliced ​​along the channel axis with the output of the corresponding layer of the encoder through cross-layer connections. A multi-scale pyramid pooling module is added to the cross-layer connections between the encoder and the first decoder and the second decoder. The multi-scale pyramid pooling module fuses features of different scales.

12. A storage medium, characterized in that, The storage medium includes a stored program, wherein, when the program is executed, it controls the device where the storage medium is located to perform the image processing method according to any one of claims 1 to 9.

13. An electronic device, characterized in that, include: processor; as well as Memory for storing the executable instructions of the processor; The processor is configured to perform the image processing method of any one of claims 1 to 9 by executing the executable instructions.

Images