Image processing method and device, electronic equipment and computer readable storage medium

Abstract

The invention discloses an image processing method and device, electronic equipment and a computer readable storage medium, and the method comprises the steps: carrying out the mirror reflection detection of a to-be-processed reflection image, and determining a mirror reflection region in the to-be-processed reflection image; performing content restoration on the mirror surface reflection area to obtain a mirror surface restoration image of the to-be-processed reflection image; performing reflection elimination on the to-be-processed reflection image to obtain a reflection elimination image of the to-be-processed reflection image; and fusing the mirror surface repair image and the reflection elimination image to obtain a result image of the to-be-processed reflection image. According to the invention, image reflection can be eliminated, content missing caused by mirror reflection can be repaired, and the image quality of the reflection image can be improved.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method, device, electronic device and computer-readable storage medium. Background Art

[0002] At present, users usually use electronic devices with shooting functions (such as mobile phones, tablet computers, etc.) to shoot images to record what is happening around them and what they see anytime and anywhere. Affected by the shooting capabilities of the electronic devices themselves and the shooting environment, the images shot by the electronic devices will inevitably have various degradation problems. For example, when shooting a portrait, if the ambient light is strong, the light reflected by the human body will affect the imaging effect, resulting in reflections in the image. In addition, if the subject is wearing glasses or other objects with a high reflection coefficient, mirror reflections are very likely to occur. Not only will the image after imaging have reflections, but the content of the mirror reflection area will also be lost. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, electronic device, and computer-readable storage medium, which can improve the image quality of reflective images.

[0004] In a first aspect, the image processing method provided by the present application comprises:

[0005] Performing specular reflection detection on the reflective image to be processed, and determining the specular reflection area in the reflective image to be processed;

[0006] Performing content repair on the mirror reflection area to obtain a mirror repair image of the reflective image to be processed;

[0007] Performing reflection elimination on the reflective image to be processed to obtain a reflection eliminated image of the reflective image to be processed;

[0008] The mirror repair image and the reflection removal image are fused to obtain a result image of the reflection image to be processed.

[0009] In a second aspect, the image processing device provided by the present application includes:

[0010] A reflection detection module, used for performing specular reflection detection on the reflective image to be processed, and determining the specular reflection area in the reflective image to be processed;

[0011] A content repair module is used to perform content repair on the mirror reflection area to obtain a mirror repair image of the reflective image to be processed;

[0012] A reflection elimination module is used to eliminate reflections from the reflective image to be processed, and obtain a reflection eliminated image of the reflective image to be processed;

[0013] The image fusion module is used to fuse the mirror repair image and the reflection elimination image to obtain a result image of the reflection image to be processed.

[0014] In a third aspect, the electronic device provided in the present application includes a memory and a processor, the memory stores a computer program, and the processor is used to run the computer program in the memory to implement the steps in the image processing method provided in the present application.

[0015] In a fourth aspect, the computer-readable storage medium provided in the present application stores a computer program, which is suitable for loading by a processor to implement the steps in the image processing method provided in the present application.

[0016] In the present application, two types of processing are performed on the reflective image to be processed. One is to perform mirror reflection detection on the reflective image to be processed, determine the area with mirror reflection in the reflective image to be processed, perform targeted content repair on the missing content in the area caused by mirror reflection, restore the missing image content as much as possible, and obtain a mirror repaired image of the reflective image to be processed. The second is to perform reflection elimination on the entire reflective image to be processed, eliminate the reflection in the image, and obtain a reflection eliminated image; finally, the mirror repaired image and the reflection eliminated image are fused to finally obtain a result image that both eliminates the reflection and restores the missing image content, thereby achieving the purpose of improving the image quality of the reflective image. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solution in this embodiment, the drawings required for use in the description of the embodiment will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 is a schematic diagram of a scene of an image processing system provided in an embodiment of the present application;

[0019] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;

[0020] Figure 3 is a first structural example diagram of the image processing model in this embodiment;

[0021] Figure 4 is a second structural schematic diagram of the image processing model in this embodiment;

[0022] Figure 5 yes Figure 4 Schematic diagram of the detailed structure of the content repair branch;

[0023] Figure 6 yes Figure 5 Schematic diagram of the detailed structure of the memory module;

[0024] Figure 7 is a third structural schematic diagram of the image processing model in this embodiment;

[0025] Figure 8 yes Figure 7 Schematic diagram of the refined structure of the reflection elimination branch;

[0026] Fig. 9 is a fourth structural schematic diagram of the image processing model in this embodiment;

[0027] Fig.10 is another flowchart of the image processing method provided by an embodiment of the present application;

[0028] Fig.11 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0029] Fig.12 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0030] It should be noted that the principles of the present application are illustrated by implementing them in an appropriate computing environment. The following description is based on the illustrated specific embodiments of the present application and should not be considered as limiting other specific embodiments of the present application that are not described in detail herein.

[0031] In the following description of the present application, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it can be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0032] In the following description of the present application, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0033] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0034] In order to improve image quality, the present application provides an image processing method, an image processing device, an electronic device, and a computer-readable storage medium. The image processing method can be executed by the image processing device, or by an electronic device integrated with the image processing device.

[0035] The following will be combined with the drawings in this embodiment to clearly and completely describe the technical solution in this embodiment. Obviously, the described embodiment is only a part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0036] Please refer to Figure 1 , the present application also provides an image processing system, such as Figure 1 As shown, the image processing system includes an electronic device 100, and the electronic device 100 is integrated with the image processing device provided by the present application. For example, the electronic device 100 can obtain an image that needs to optimize the image quality, which is recorded as a reflective image to be processed; then, the reflective image to be processed is subjected to mirror reflection detection, and the area in the reflective image to be processed where mirror reflection exists is determined, which is recorded as a mirror reflection area; then, the determined mirror reflection area is subjected to content repair, and the reflective image to be processed after content repair is recorded as a mirror repair image; in addition, the reflective image to be processed is also subjected to reflection elimination, and the reflective image to be processed after reflection elimination is recorded as a reflection eliminated image; finally, the mirror repair image and the reflection eliminated image are fused, and the fused image is used as the result image of the reflective image to be processed.

[0037] Among them, the electronic device 100 can be any device equipped with a processor and has processing capabilities, such as a smart phone, a tablet computer, a PDA, a laptop computer, a virtual reality device, an augmented reality device or a mixed reality device, etc., a mobile electronic device with a processor, or a desktop computer, a television, a server, an industrial equipment, etc., a fixed electronic device with a processor.

[0038] In addition, if Figure 1 As shown, the image processing system may further include a data warehouse 200 for storing original data, intermediate data and result data in the image processing process. For example, the electronic device 100 stores the acquired reflective image to be processed (original data), the determined mirror reflection area, the mirror repaired image obtained by content repair, the reflection eliminated image (intermediate data) obtained by reflection elimination, and the result image (result data) of the reflective image to be processed obtained by final fusion into the data warehouse 200.

[0039] It should be noted that Figure 1The scene diagram of the image processing system shown is only an example. The image processing system and scene described in the embodiment of the present application are intended to more clearly illustrate the technical solution of the embodiment of the present application, and do not constitute a limitation on the technical solution provided in the embodiment of the present application. Ordinary technicians in this field can know that with the evolution of the image processing system and the emergence of new business scenarios, the technical solution provided in the embodiment of the present application is also applicable to similar technical problems.

[0040] It should be noted that the serial numbers of the following embodiments are not intended to limit the preferred order of the embodiments.

[0041] Please refer to Figure 2 , Figure 2 is a flow chart of the image processing method provided in the embodiment of the present application, such as Figure 2 As shown, the process of the image processing method provided by this application is as follows:

[0042] In 110 , specular reflection detection is performed on the reflective image to be processed to determine a specular reflection area in the reflective image to be processed.

[0043] In this embodiment, the electronic device first obtains an image that needs to be processed, which is recorded as a reflective image to be processed. There is no specific limitation on the method of obtaining the reflective image to be processed. For example, when the electronic device itself is equipped with an image acquisition component, it can use its own image acquisition component to collect images of the collection object, and use the collected images as the reflective image to be processed; in addition, the electronic device can also obtain the reflective image to be processed from other electronic devices equipped with image acquisition components, and can also obtain pre-collected object images from the server, and the object images are uploaded to the server after other electronic devices equipped with image acquisition components collect the collected images.

[0044] After obtaining the reflective image to be processed, the electronic device first performs internal reflection detection on the reflective image to be processed according to the configured mirror reflection detection strategy, and determines the area where the mirror reflection exists in the reflective image to be processed, which is recorded as the mirror reflection area. There is no specific restriction on the configuration of the mirror reflection detection strategy here. For example, an artificial intelligence-based mirror reflection detection strategy can be configured, and a non-artificial intelligence-based mirror detection strategy can also be configured.

[0045] Exemplarily, brightness statistics can be performed on the reflective image to be processed to obtain the brightness statistics result of the reflective image to be processed, and the brightness threshold can be determined based on the brightness statistics result, and the connected area in the reflective image to be processed whose brightness value is greater than or equal to the brightness threshold can be determined as the mirror reflection area; or a neural network model for performing mirror reflection detection on images can be pre-trained, and mirror reflection detection can be performed on the reflective image to be processed through the neural network model, and the mirror reflection area of ​​the reflective image to be processed can be determined accordingly.

[0046] In 120 , content restoration is performed on the mirror reflection area to obtain a mirror restoration image of the reflection image to be processed.

[0047] As described above, after determining the mirror reflective area of ​​the reflective image to be processed, the electronic device performs content repair on the mirror reflective area in the processed image according to the configured content repair strategy, and accordingly records the reflective image to be processed after the content repair as the mirror repair image. There is no specific restriction on the configuration of the content repair strategy here. For example, an artificial intelligence-based content repair strategy can be configured, or a non-artificial intelligence-based content repair strategy can be configured.

[0048] Exemplarily, a frequency domain selective reconstruction algorithm can be used to implement content restoration of the mirror reflective area; a fast matching algorithm based on image gradient can also be used to implement content restoration of the mirror reflective area; a neural network model for image content restoration can also be pre-trained, and the content restoration of the mirror reflective area of ​​the reflective image to be processed can be performed through the neural network model.

[0049] In 130 , the reflection-eliminating process is performed on the reflection-eliminating image to be processed to obtain a reflection-eliminating image of the reflection-eliminating image to be processed.

[0050] In this embodiment, in addition to performing targeted content repair on the mirror reflective area in the reflective image to be processed, a universal reflection elimination is also performed on the reflective image to be processed. The electronic device eliminates the reflection of the reflective image to be processed according to the configured reflection elimination strategy, and accordingly records the reflective image to be processed after the reflection elimination as the reflection eliminated image. There is no specific restriction on the configuration of the reflection elimination strategy here. For example, an artificial intelligence-based reflection elimination strategy can be configured, and a non-artificial intelligence-based reflection elimination strategy can also be configured.

[0051] Exemplarily, a dark channel-based reflection elimination algorithm can be used to eliminate reflections from the reflective image to be processed; a neural network model for eliminating reflections from images can also be pre-trained, and reflections can be eliminated from the reflective image to be processed through the neural network model.

[0052] It should be noted that the execution order of step 130 is not affected by the size of the sequence number, and it can be executed before 110, or after 120, or it can be executed synchronously during the execution process of 110 / 120.

[0053] In 140 , the mirror repair image and the reflection elimination image are fused to obtain a result image of the reflection image to be processed.

[0054] As mentioned above, after performing content repair on the mirror reflective area in the reflective image to be processed and eliminating the reflection of the entire reflective image to be processed, the electronic device further fuses the mirror repaired and reflection eliminated images to obtain a fused image that combines the advantages of both, and uses the fused image as the result image of the reflective image to be processed.

[0055] For example, the image content corresponding to the mirror reflection area in the mirror repair image can be taken and image stitching can be performed with the image content outside the corresponding mirror reflection area in the reflection elimination image to achieve the fusion of the mirror repair image and the reflection elimination image.

[0056] As can be seen from the above, in the present embodiment, two types of processing are performed on the reflective image to be processed. One is to perform mirror reflection detection on the reflective image to be processed, determine the area with mirror reflection in the reflective image to be processed, perform targeted content repair on the missing content in the area caused by mirror reflection, restore the missing image content as much as possible, and obtain a mirror repaired image of the reflective image to be processed. The second is to perform reflection elimination on the entire reflective image to be processed, eliminate the reflection in the image, and obtain a reflection eliminated image. Finally, the mirror repaired image and the reflection eliminated image are fused to finally obtain a result image that has both eliminated the reflection and restored the missing image content, thereby achieving the purpose of improving the image quality of the reflective image.

[0057] Optionally, in one embodiment, a mirror reflection detection method based on artificial intelligence is provided to realize the mirror reflection detection of the reflective image to be processed. It should be noted that artificial intelligence (AI) is a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that the machines have the functions of perception, reasoning and decision-making.

[0058] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes machine learning (ML) technology, among which deep learning (DL) is a new research direction in machine learning. It is introduced into machine learning to make it closer to the original goal, namely artificial intelligence. At present, deep learning is mainly used in fields such as machine vision and natural language processing.

[0059] Deep learning is the process of learning the inherent laws and representation levels of sample data. The information obtained in the learning process is of great help in interpreting data such as text, images, and sound. Using deep learning technology and corresponding training sets, network models that implement different functions can be trained. For example, a gender classification model for gender classification can be trained based on a training set, and an image optimization model for image optimization can be trained based on another training set. Accordingly, in this embodiment, an image processing model is pre-trained. Please refer to Figure 3 The image processing model includes a feature extraction network and a specular reflection detection branch, which form an encoder-decoder architecture, wherein the feature extraction network is configured as an encoder to perform feature extraction on an input image to obtain a feature image of the input image, and the specular reflection detection branch is configured as a decoder to perform specular reflection detection on the feature image extracted by the feature extraction network to obtain a regional mask image for indicating a specular reflection area in the input image.

[0060] In this embodiment, performing specular reflection detection on the reflective image to be processed and determining the specular reflection area in the reflective image to be processed includes:

[0061] Get the image processing model;

[0062] Inputting the reflective image to be processed into a feature extraction network for feature extraction to obtain a feature image of the reflective image to be processed;

[0063] The feature image is input into the specular reflection detection branch for specular reflection detection to obtain a regional mask image, which is used to indicate the specular reflection area in the reflection image to be processed.

[0064] Among them, the image processing model can be deployed locally on the electronic device after training is completed, or it can be deployed on a cloud server after training is completed. The electronic device first obtains the image processing model, inputs the reflective image to be processed into the feature extraction network of the image processing model for feature extraction, and obtains a feature map of the reflective image to be processed. There is no restriction on the network structure of the feature extraction network here, and it can be configured by those skilled in the art according to actual needs. For example, network structures such as VGGNet, ResNet, DenseNet and DarkNet can be used.

[0065] As described above, after extracting the feature image of the reflective image to be processed through the feature extraction network, the extracted feature image is further input into the mirror reflection detection branch for mirror reflection detection, and a mask image for indicating the mirror reflection area in the reflective image to be processed is obtained, which is recorded as the regional mask image. The network structure of the reflection detection branch is not limited here, and can be configured by those skilled in the art according to actual needs. For example, it can be obtained by stacking multiple deconvolution layers.

[0066] It should be noted that the mask image is a binary image composed of 0 and 1. Depending on actual needs, 0 can be used to indicate the desired pixel, or 1 can be used to indicate the desired pixel. For example, in this embodiment, through the training of the image processing model, the regional mask image is configured to use 1-valued pixels to indicate the pixels in the mirror reflection area, and 0-valued pixels to indicate the pixels in the non-mirror reflection area. For example, please continue to refer to Figure 3 , the reflective image to be processed is a portrait image of a person wearing glasses. Due to the specular reflection of the glasses lenses under the irradiation of light, the eye content of the portrait is missing. At the same time, the area outside the glasses lenses has a weaker reflection than the glasses lens area because the reflection coefficient is lower than the reflection coefficient of the glasses lenses. The portrait image is input into the feature extraction network for feature extraction, and the feature image output by the feature extraction network is further input into the specular reflection detection branch. The feature image is mapped into a mask image indicating the specular reflection area in the portrait image. The white area composed of 1 in the mask image indicates the specular reflection area of ​​the portrait image, that is, the area where the eye content of the portrait is missing.

[0067] Optionally, in one embodiment, please refer to Figure 4 The image processing model further includes a content restoration branch, which shares a feature extraction network with the mirror reflection detection branch. The content restoration branch also forms an encoder-decoder architecture with the feature extraction network. The content restoration branch is configured as a decoder to perform content restoration on the input image to restore texture details that may be missing in the input image. Accordingly, in this embodiment, content restoration is performed on the mirror reflection area to obtain a mirror restoration image of the reflective image to be processed, including:

[0068] According to the regional mask image, the content of the mirror reflection area of ​​the reflective image to be processed is repaired through the content repair branch to obtain a mirror repair image of the reflective image to be processed.

[0069] For example, please continue to refer to Figure 4 The reflective image to be processed is a portrait image of a person wearing glasses. Due to the specular reflection of the glasses lenses under the irradiation of light, the eye content of the portrait is missing. At the same time, the area outside the glasses lenses has a weaker reflection than the glasses lens area because the reflection coefficient is lower than the reflection coefficient of the glasses lenses. The portrait image and the regional mask image indicating the specular reflection area are input into the content repair branch, and the content repair branch is guided to repair the missing eye content in the portrait image, so as to obtain a specular repair image with restored eye content, but the reflection of the area outside the specular reflection area is not eliminated.

[0070] Optionally, in one embodiment, the reflective image to be processed includes an object image of a target object wearing glasses, see Figure 5 , the content repair branch includes a multiplication module, a memory module, a first cascade module and a content repair module. Among them, the multiplication module is configured to perform multiplication operation on the two input images to obtain a multiplication result image; the memory module is configured to memorize the semantic feature vector of the lens area when the sample object wears glasses but no mirror reflection occurs; the first cascade module is configured to cascade the two input images, that is, splice them in the channel dimension; the content repair module is configured to perform content repair on the input image and restore the texture details that may be missing in the input image. Among them, the sample object and the target object are the same type of objects, which can be human objects or animal objects that have the ability to wear glasses.

[0071] According to the regional mask image, the content of the mirror reflection area of ​​the reflective image to be processed is repaired through the content repair branch to obtain the mirror repair image of the reflective image to be processed, including:

[0072] Obtaining an inverted image of the regional mask image, and inputting the inverted image and the feature image into a multiplication module for multiplication operation to obtain a multiplication result image;

[0073] Reading a target feature vector similar to a feature vector in the multiplication result image from the memory module, and obtaining a target feature image composed of the target feature vector;

[0074] Inputting the multiplication result image and the target feature image into the first cascade module for cascade operation to obtain a cascade image;

[0075] The cascaded image is input into the content restoration module for content restoration to obtain a mirror restoration image of the reflective image to be processed.

[0076] In this embodiment, the regional mask image is first inverted, and the inverted regional mask image is recorded as an inverted image. In the inverted image, 0 indicates pixels in the mirror reflection area, and 1 indicates pixels in the non-mirror reflection area.

[0077] The content fix branch also includes the scaling module ( Figure 5 (not shown) is used to scale the inverted image to the same size as the feature image. After that, the inverted image and the feature image of the same size are input into the multiplication module for multiplication to obtain the multiplication result image E∈R HxWxC , where H indicates the height of the multiplication result image, W indicates the width of the multiplication result image, and C indicates the number of channels of the multiplication result image. It should be noted that since the 0-valued pixels in the inverted image indicate the pixels of the mirror reflection area in the reflective image to be processed, and the 1-valued pixels indicate the pixels of the non-mirror reflection area in the reflective image to be processed, by multiplying the feature image with the inverted image, the multiplication result image obtained will retain the feature vector corresponding to the non-mirror reflection area in the feature image, thereby eliminating the interference of the mirror reflection area.

[0078] Please refer to Figure 6 The memory module includes an update unit, a read unit, and a memory unit. The memory size of the memory unit is initialized to C×M, where M represents the number of semantic feature vectors of each channel memorized by the memory unit. The read unit is used to read the semantic feature vector memorized in the memory unit, and the update unit is used to update the semantic feature vector memorized in the memory unit.

[0079] When reading a target feature vector similar to the feature vector in the multiplication result image from the memory module, m j ∈R 1×1×C The jth semantic feature vector representing the memory unit, e i ∈R 1×1×C Representing the i-th feature vector in the multiplication result image E, in each channel, first calculate the similarity ω between each feature vector in the multiplication result image and each semantic feature vector memorized by the memory unit according to formula (1): i,j :

[0080]

[0081] Similarityω i,j Indicates the similarity between the i-th feature vector of each channel in the multiplication result image E and each semantic feature vector of the corresponding channel memorized by the memory unit, m j′ Represents the semantic feature vectors except the j-th semantic feature vector.

[0082] Then, according to the calculated similarity, the semantic feature vector closest to the i-th feature vector is obtained as the most similar feature vector currently read.

[0083]

[0084] That is, in each channel, for the i-th feature vector in the multiplication result image, the similarity between the feature vector and the semantic feature vector of the corresponding channel memorized by the memory unit is used as the weight, and all the semantic feature vectors of the channel are fused into the target semantic feature vector corresponding to the i-th feature vector in a weighted manner.

[0085] As above, by performing a read operation on all feature vectors of each channel in the multiplication result image E, that is, searching for target feature vectors similar to all feature vectors of each channel of the multiplication result image E in the memory unit, the target feature image similar to the multiplication result image is formed by these read target feature vectors.

[0086] After that, the multiplication result image and the target feature image read by the memory module are further input into the first cascade module for cascade operation (Concat), that is, the multiplication result image and the target feature image are spliced ​​in the channel dimension to obtain the cascade image E′∈R HxWx2C In this way, by cascading the multiplication result image and the target feature image, the semantic feature vector of the mirror reflection area when no mirror reflection occurs can be provided to the content restoration module as auxiliary information, thereby further improving the image quality of the mirror restoration image of the reflective image to be processed restored by the content restoration module.

[0087] Among them, the content repair module can be obtained by stacking multiple layers of deconvolution layers. The cascade image is input into the content repair module, and the content repair is achieved by performing deconvolution operations on the cascade image layer by layer to obtain a mirror repair image of the reflective image to be processed.

[0088] It should be noted that in this embodiment, the update unit in the memory module is disabled, that is, in the inference stage of the image processing model, only the memory module is read to reduce the amount of calculation. In other embodiments, the update unit in the memory module is enabled in the training stage, and the semantic feature vector memorized in the memory unit is updated by the update unit, thereby enhancing the learning ability of the memory module to provide more accurate auxiliary information for content repair.

[0089] Optionally, in one embodiment, please refer to Figure 7The image processing model further includes a reflection elimination branch. The reflection elimination branch shares a feature extraction network with the mirror reflection detection branch and the content restoration branch. The reflection elimination branch and the feature extraction network also form an encoder-decoder architecture. The reflection elimination branch is configured as a decoder to eliminate reflections in the input image. In this embodiment, reflection elimination is performed on the reflective image to be processed to obtain a reflection eliminated image of the reflective image to be processed, including:

[0090] The reflection image to be processed is input into the reflection elimination branch to eliminate reflections, thereby obtaining a reflection eliminated image of the reflection image to be processed.

[0091] For example, please continue to refer to Figure 7 The reflective image to be processed is a portrait image of a person wearing glasses. Due to the specular reflection of the glasses lenses under the irradiation of light, the eye content of the portrait is missing. At the same time, the area outside the glasses lenses has a weaker reflection than the glasses lens area because the reflection coefficient is lower than the reflection coefficient of the glasses lenses. The portrait image is input into the reflection elimination branch for reflection elimination, and a reflection elimination image is obtained in which the reflection outside the specular reflection area is eliminated, but the missing eye content in the specular reflection area is not repaired.

[0092] Optionally, in one embodiment, please refer to Figure 8 The reflection elimination branch includes a reflection elimination module and an image optimization module. The reflection image to be processed is input into the reflection elimination branch for reflection elimination to obtain a reflection elimination image of the reflection image to be processed, including:

[0093] Input the feature image into the reflection elimination module to eliminate the reflection, and obtain an initial reflection elimination image;

[0094] The initial de-glare image is input into the image optimization module for image optimization to obtain a de-glare image.

[0095] In this embodiment, the reflection removal module and the feature extraction network form an encoder-decoder architecture. In order to improve the reflection removal capability of the reflection removal branch, an image optimization module is added after the reflection removal module and the feature extraction network form the encoder-decoder architecture. The reflection removal module can be formed by stacking multiple deconvolution layers.

[0096] The image optimization module includes H image optimization units, H second cascade modules and a common convolution layer. The image optimization unit includes a gated convolution layer, an ablation convolution layer and a channel attention layer connected in sequence. The initial reflection-removed image is input into the image optimization module for image optimization to obtain a reflection-removed image, including:

[0097] Inputting the initial reflection-removed image into the first image optimization unit for image optimization to obtain the first optimized initial reflection-removed image;

[0098] Inputting the first optimized initial de-glare image and the initial de-glare image into the first second cascade module to obtain a first initial de-glare cascade image, and inputting the first initial de-glare cascade image into the second image optimization unit for image optimization to obtain a second optimized initial de-glare image;

[0099] Input the h-1th optimized initial reflection elimination image and the h-2th initial reflection elimination cascade image into the h-1th second cascade module to obtain the h-1th initial reflection elimination cascade image, and input the h-1th initial reflection elimination cascade image into the hth image optimization unit for image optimization to obtain the hth optimized initial reflection elimination image, h∈[3,H];

[0100] The Hth optimized initial reflection-removed image and the H-1th initial reflection-removed cascade image are input into the Hth second cascade module to obtain the Hth initial reflection-removed cascade image, and the Hth initial reflection-removed cascade image is input into the common convolution layer for convolution processing to obtain the reflection-removed image.

[0101] Among them, for each image optimization unit, the gated convolution layer is used to assign different weight values ​​to each pixel of the input image to avoid the influence of reflective pixels; the ablation convolution layer is used to increase the receptive field of the network; the channel attention mechanism introduces global information under different channels into the input image by calculating the mean, variance and maximum value under each channel of the input image, which is conducive to eliminating reflections.

[0102] As described above, by optimizing the initial reflection elimination image obtained by performing reflection elimination on the reflection elimination module image unit by image unit, the reflection elimination effect can be enhanced.

[0103] Optionally, in one embodiment, please refer to Fig. 9 , the image processing model also includes a fusion network, which fuses the mirror repair image and the reflection removal image to obtain a result image of the reflection image to be processed, including:

[0104] According to the regional mask image, the mirror repair image and the reflection elimination image are fused through a fusion network to obtain the result image of the reflection image to be processed.

[0105] For example, please continue to refer to Fig. 9The reflective image to be processed is a portrait image of a person wearing glasses. Due to the specular reflection of the glasses lenses under the irradiation of light, the eye content of the portrait is missing. At the same time, the area outside the glasses lenses has a weaker reflection than the area of ​​the glasses lenses because the reflection coefficient is lower than the reflection coefficient of the glasses lenses. The regional mask image indicating the specular reflection area is obtained through the specular reflection detection branch. The specular repair image in which the missing eye content is repaired but the reflection in the area outside the specular reflection area is not eliminated is obtained through the content repair branch. The reflection elimination branch obtains a reflection elimination image in which the reflection in the area outside the specular reflection area is eliminated but the missing eye content is not repaired. Finally, according to the regional mask image, the specular repair image and the reflection elimination image are fused through the fusion network to obtain a result image that not only repairs the missing eye content but also eliminates the reflection outside the specular reflection area.

[0106] Optionally, in one embodiment, the fusion network includes a weight mapping module and a fusion module, and the weight mapping module is configured to take the detection result of the mirror reflection detection branch, the repair result of the content repair branch, and the elimination result of the reflection elimination branch as input, and predict a pixel-by-pixel fusion weight map according to a pre-set reflection intensity threshold. The fusion weight map records the weight value of each pixel position depending on the elimination result, wherein the pixel position outside the mirror reflection area corresponds to a high weight, and the pixel position within the mirror reflection area corresponds to a low weight. The fusion module is configured to adaptively fuse the elimination result and the repair result according to the fusion weight map. In this embodiment, according to the regional mask image, the mirror repair image and the reflection elimination image are fused through the fusion network to obtain a result image of the reflection image to be processed, including:

[0107] The regional mask image, the mirror repair image and the reflection elimination image are input into the weight mapping layer for weight mapping to obtain a fusion weight map, which is used to indicate the fusion weight of the mirror repair image and the reflection elimination image;

[0108] According to the fusion weight map, the mirror repair image and the reflection elimination image are fused through a fusion module to obtain a result image of the reflection image to be processed.

[0109] In this embodiment, the weight mapping layer can be obtained by connecting a cascade sublayer, multiple convolution layers, and a Sigmoid function layer. The cascade sublayer is configured in the channel dimension to perform a cascade operation (Concat) on the regional mask image, the mirror repair image, and the reflection elimination image, and then the fusion weight map is obtained through the convolution operation of multiple convolution layers and the post-mapping of the Sigmoid function layer.

[0110] In other embodiments, the fusion weight map may also record the weight value of each pixel position depending on the restoration result, but the pixel position outside the specular reflection area corresponds to a low weight, and the pixel position within the specular reflection area corresponds to a high weight.

[0111] Optionally, in one embodiment, the fusion module includes a partial convolution layer, and according to the fusion weight map, the mirror repair image and the reflection elimination image are fused through the fusion module to obtain a result image of the reflection image to be processed, including:

[0112] According to the fusion weight map, the mirror repair image and the reflection removal image are partially convolved through a partial convolution layer to obtain a result image of the reflection image to be processed.

[0113] In order to fully utilize the obtained fusion weight map to fuse the mirror repair image and the reflection removal image, this embodiment uses a partial convolution method to achieve image fusion.

[0114] The above fusion process can be expressed as:

[0115]

[0116] I cat =[I e ⊙Map,I i ⊙(D-Map)](4)

[0117] Among them, I out represents the result image, W represents the weight parameter of the partial convolution layer, b represents the bias of the partial convolution layer, * represents the convolution operation, ⊙ represents the element-by-element multiplication operation, It represents the average weight value of the pixel position in the 3x3 neighborhood of the pixel position to be fused in the fusion weight map. e Represents the reflection-removed image, I i represents the content repair image, D represents the region mask image, Map represents the fusion weight map, and I cat It represents a cascade image of the second multiplication result image and the third multiplication result image. The second multiplication result image is obtained by multiplying the reflection elimination image with the fusion weight map, and the third multiplication result image is obtained by multiplying the content restoration image and the difference image between the region mask image and the fusion weight map.

[0118] Optionally, in one embodiment, a training scheme for an image processing model is provided as follows:

[0119] Acquire a sample reflection image with a mirror reflection area and a sample reference image without reflection, wherein the sample reflection image and the sample reference image are acquired according to the same posture of the sample object wearing glasses;

[0120] Acquire a marked area mask image for indicating a specular reflection area in a sample reflection image;

[0121] Input the sample reflection image into the feature extraction network of the image processing model, obtain the sample area mask image output by the mirror reflection detection branch, obtain the sample mirror restoration image output by the content restoration branch, obtain the sample reflection removal image output by the reflection removal branch, obtain the sample result image output by the fusion network, and obtain the sample fusion weight map output by the weight mapping layer in the fusion network;

[0122] Obtaining the mirror reflection detection loss according to the sample area mask image and the annotated area mask image;

[0123] Obtaining content restoration loss according to the sample mirror restoration image and the sample reference image;

[0124] Obtaining the reflection elimination loss according to the sample reflection elimination image and the sample reference image;

[0125] Obtain image fusion loss according to the sample result image and the sample reference image;

[0126] Obtain the weight mapping loss according to the grayscale difference between the sample fusion weight map, the sample reflection image and the sample reference image;

[0127] According to the mirror reflection detection loss, content restoration loss, reflection removal loss, image fusion loss and weight mapping loss, the network parameters of the image processing model are updated until the preset update stop conditions are met.

[0128] It should be noted that, in this embodiment, there is no specific restriction on the types of the above-mentioned mirror reflection detection loss, content restoration loss, reflection elimination loss, image fusion loss, and weight mapping loss, which can be selected by those skilled in the art according to actual needs. In addition, the preset update stop condition can also be configured by those skilled in the art according to actual needs. For example, the preset update stop condition can be configured as the number of updates of the network parameters of the image processing model reaches a preset number, or the preset update stop condition can be configured as the convergence of the image processing model, etc.

[0129] Exemplarily, in this embodiment, the mirror reflection detection loss is configured to adopt focus loss, and the content restoration loss, reflection elimination loss, and image fusion loss are configured to adopt perception loss and reconstruction loss.

[0130] Furthermore, the weight mapping loss is expressed as:

[0131]

[0132] Among them, l weightrepresents the weight mapping loss, R represents the grayscale difference image between the sample reflection image and the sample reference image (that is, the image obtained by subtracting the grayscale image of the sample reflection image from the grayscale image of the sample reference image), Map′ represents the sample fusion weight map, and t represents the grayscale threshold, which can be determined by the following formula:

[0133] t=0.5max(I in -I gt )(6)

[0134] Among them, I in Represents the sample reflection image, I gt represents a sample reference image.

[0135] In addition, the preset update stop condition is configured as: the number of updates of the network parameters of the image processing model reaches 40 times.

[0136] In addition, the present embodiment also updates the semantic feature vector memorized by the memory module during the training process. The sample reflection image is input into the feature extraction network to obtain a feature extraction result which is recorded as a sample feature image. For a semantic feature vector to be updated, the feature vector with the maximum correlation value in the sample feature image is selected to update the semantic feature vector.

[0137] First, each semantic feature vector m memorized by the memory module is retrieved according to the following formula j The most relevant feature vector in the sample feature image

[0138] k i = argmax i ω i′,j (7)

[0139] Since there may be multiple most relevant query vectors retrieved, we can use (Formula 4) to convert m j The most relevant query vectors are used to update the feature item m j :

[0140]

[0141] Among them, f() represents L2 regularization, I() represents the indicator function, N′ represents the size of the sample feature image, and e i′ represents the i′th feature vector in the sample feature image, ω i′,j It represents the similarity between the i′th feature vector in the sample feature image and the jth semantic feature vector memorized by the memory module.

[0142] Optionally, in one embodiment, the network parameters of the image processing model are updated according to the mirror reflection detection loss, the content restoration loss, the reflection elimination loss, the image fusion loss and the weight mapping loss until a preset update stop condition is met, further comprising:

[0143] Get the horizontally flipped image of the sample reference image;

[0144] The symmetric mapping relationship between the sample reference image and the horizontally flipped image is obtained through the symmetric learning model;

[0145] Obtain symmetric loss based on the sample reflection image, symmetric mapping relationship and horizontal flip image;

[0146] According to the mirror reflection detection loss, content restoration loss, reflection removal loss, image fusion loss and weight mapping loss, the network parameters of the image processing model are updated until the preset update stop conditions are met, including:

[0147] According to the mirror reflection detection loss, content restoration loss, reflection removal loss, image fusion loss, weight mapping loss and symmetry loss, the network parameters of the image processing model are updated until the preset update stop condition is met.

[0148] Considering that human faces have natural symmetry, especially the eyes, this embodiment uses this prior information to further improve the image processing quality.

[0149] It is understandable that not all faces are symmetrical in the image, and it is difficult to directly establish a symmetrical mapping relationship on the image. Based on this, this embodiment pre-trains a symmetric learning model to learn the symmetrical mapping relationship between the input eye image and its horizontally flipped image. Among them, the symmetric learning model adopts an encoder-decoder architecture, and the model structure can be set by those skilled in the art according to actual needs.

[0150] For example, given an eye image P and its corresponding horizontally flipped image P′, the symmetry learning model predicts a symmetric mapping relationship Ω=(Ω x ,Ω y ), that is, for each pixel (p, q) in the eye image P, it can be mapped to the corresponding pixel in its corresponding horizontal flipped image P′ through this symmetric mapping relationship Since E′ is the horizontal flip image of E, P(p,q) and They are the corresponding pixels on different eyes. The eye symmetry mapping relationship learned by the symmetry learning model represents the mapping relationship between each pixel in the input eye image and the corresponding pixel in its horizontally flipped image.

[0151] In order to better train the symmetry learning model proposed in this embodiment, the above sample reference images are reused to implement the training of the symmetry learning model. gt and its corresponding horizontal flipped image I′ gt The 68 facial key points are then extracted from the sample reference image I gt Select 22 eye-related key points from the 68 facial key points From the horizontally flipped image I′ gt Select 22 eye-related key points from the 68 facial key points As the eye benchmark, sparse eye key points are used to constrain the symmetric mapping relationship Ω in a semi-supervised manner. should be approximately equal to Therefore, the benchmark loss for aligning faces provided in this embodiment is expressed as:

[0152]

[0153] In addition, since using only 22 sparse eye key points may cause the learned eye mapping relationship to be non-smooth, the total variation loss is also used in this embodiment to further constrain its smoothness. The total variation loss is expressed as:

[0154]

[0155] In the formula, Δ x and Δ y are the gradients of Ω along the x-axis and y-axis respectively. In the symmetric learning model, the baseline loss l lm The detected facial key points are used to align the face, and the total variational loss constrains the smoothness of the learned symmetric mapping relationship in the gradient domain in the x-axis and y-axis directions. Therefore, the total loss of training the symmetric learning model is expressed as:

[0156] l flow =λ lm l lm +λ TV l TV (11)

[0157] Among them, l flow represents the total loss of training the symmetric learning model, λ lm Denotes the baseline loss l lm The weight, λ TV Represents λ TV The weight, λ TV and λ lm The value can be determined by technicians in this field according to actual needs.

[0158] In addition, the preset update stop condition of the symmetric learning model can also be configured by technical personnel in this field according to actual needs. For example, the preset update stop condition can be configured as the number of updates of the network parameters of the symmetric learning model reaches a preset number, or the preset update stop condition can be configured as the convergence of the symmetric learning model, etc.

[0159] According to the image processing method provided in the above embodiments, the following is further described in detail by taking the reflective image to be processed as an object image of a target object wearing glasses and the image processing device integrated in an electronic device as an example.

[0160] Please refer to Fig.10 , the process of the image processing method can also be as follows:

[0161] In 210 , the electronic device obtains a reflective object image of a target object wearing glasses.

[0162] In 220, the electronic device obtains an image processing model, which includes a feature extraction network, a mirror reflection detection branch, a content restoration branch, a reflection elimination branch, and a fusion network.

[0163] In 230, the electronic device inputs the reflective object image into a feature extraction network for feature extraction to obtain a feature image of the reflective object image.

[0164] In 240 , the electronic device inputs the feature image of the reflective object image into the specular reflection detection branch to perform specular reflection detection, and obtains a specular mask image for indicating the specular reflection area in the reflective object image.

[0165] In 250, the electronic device performs content restoration on the reflective object image through a content restoration branch according to the mirror restoration image to obtain a content restoration image of the reflective object image.

[0166] In 260 , the electronic device performs reflection elimination on the reflective object image through the reflection elimination branch to obtain a reflection eliminated image of the reflective object image.

[0167] In 270, the electronic device fuses the content restoration image and the reflection removal image of the reflective object image through a fusion network according to the mirror mask image to obtain a result image of the reflective object image.

[0168] It should be noted that for the technical details of the above steps, please refer to the relevant description in the above embodiments, which will not be repeated here.

[0169] In order to better implement the image processing method provided in the embodiment of the present application, the embodiment of the present application also provides an image processing device based on the above image processing method. The meanings of the terms are the same as those in the above image processing method. For specific implementation details, please refer to the description in the above method embodiment.

[0170] Please refer to Fig.11 , Fig.11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device may include a reflection detection module 310, a content repair module 320, a reflection elimination module 330 and an image fusion module 340, wherein:

[0171] A reflection detection module 310, configured to perform specular reflection detection on the reflective image to be processed, and determine a specular reflection area in the reflective image to be processed;

[0172] The content repair module 320 is used to perform content repair on the mirror reflection area to obtain a mirror repair image of the reflective image to be processed;

[0173] A reflection elimination module 330 is used to eliminate reflections from the reflection image to be processed to obtain a reflection eliminated image of the reflection image to be processed;

[0174] The image fusion module 340 is used to fuse the mirror repair image and the reflection elimination image to obtain a result image of the reflection image to be processed.

[0175] Optionally, in one embodiment, the reflection detection module 310 is used to obtain an image processing model, the image processing model includes a feature extraction network and a mirror reflection detection branch; and input the reflection image to be processed into the feature extraction network for feature extraction to obtain a feature image of the reflection image to be processed; and input the feature image into the mirror reflection detection branch for mirror reflection detection to obtain a regional mask image, and the regional mask image is used to indicate the mirror reflection area in the reflection image to be processed.

[0176] Optionally, in one embodiment, the image processing model further includes a content repair branch, and the content repair module 320 is used to perform content repair on the mirror reflection area according to the regional mask image through the content repair branch to obtain a mirror repair image of the reflection image to be processed.

[0177] Optionally, in one embodiment, the reflective image to be processed includes an object image of a target object wearing glasses, and the content repair branch includes a multiplication module, a memory module, a first cascade module and a content repair module, the memory module is used to memorize the semantic feature vector of the lens area when the sample object wears glasses but no mirror reflection occurs, the content repair module 320 is used to obtain an inverted image of the regional mask image, and input the inverted image and the feature image into the multiplication module for multiplication operation to obtain a multiplication result image; and read a target feature vector similar to the feature vector in the multiplication result image from the memory module to obtain a target feature image composed of the target feature vector; and input the multiplication result image and the target feature image into the first cascade module for cascade operation to obtain a cascade image; and input the cascade image into the content repair module for content repair to obtain a mirror repaired image.

[0178] Optionally, in one embodiment, the image processing model further includes a reflection elimination branch, and the reflection elimination module 330 is used to input the reflection image to be processed into the reflection elimination branch to eliminate reflections, so as to obtain a reflection eliminated image of the reflection image to be processed.

[0179] Optionally, in one embodiment, the reflection elimination branch includes a reflection elimination module and an image optimization module, and the reflection elimination module 330 is used to input the feature image into the reflection elimination module for reflection elimination to obtain an initial reflection elimination image; and input the initial reflection elimination image into the image optimization module for image optimization to obtain a reflection elimination image.

[0180] Optionally, in one embodiment, the image optimization module includes H image optimization units, H second cascade modules and a common convolution layer, the image optimization unit includes a gated convolution layer, an ablation convolution layer and a channel attention layer connected in sequence, and the reflection elimination module 330 is used to input the initial reflection elimination image into the first image optimization unit for image optimization to obtain the first optimized initial reflection elimination image; and input the first optimized initial reflection elimination image and the initial reflection elimination image into the first second cascade module to obtain the first initial reflection elimination cascade image, and input the first initial reflection elimination cascade image into the second image optimization unit for image optimization to obtain the second optimized initial reflection elimination image. light-eliminated image; and inputting the h-1th optimized initial reflection-eliminated image and the h-2th initial reflection-eliminated cascade image into the h-1th second cascade module to obtain the h-1th initial reflection-eliminated cascade image, and inputting the h-1th initial reflection-eliminated cascade image into the h-th image optimization unit for image optimization to obtain the hth optimized initial reflection-eliminated image, h∈[3,H]; and inputting the Hth optimized initial reflection-eliminated image and the H-1th initial reflection-eliminated cascade image into the Hth second cascade module to obtain the Hth initial reflection-eliminated cascade image, and inputting the Nth initial reflection-eliminated cascade image into the ordinary convolution layer for convolution processing to obtain the reflection-eliminated image.

[0181] Optionally, in one embodiment, the image processing model further includes a fusion network, and the image fusion module 340 is used to fuse the mirror repair image and the reflection elimination image through the fusion network according to the regional mask image to obtain a result image of the reflection image to be processed.

[0182] Optionally, in one embodiment, the fusion network includes a weight mapping module and a fusion module, and the image fusion module 340 is used to input the regional mask image, the mirror repair image and the reflection elimination image into the weight mapping layer for weight mapping to obtain a fusion weight map, and the fusion weight map is used to indicate the fusion weights of the mirror repair image and the reflection elimination image; and according to the fusion weight map, the mirror repair image and the reflection elimination image are fused through the fusion module to obtain a result image of the reflection image to be processed.

[0183] Optionally, in one embodiment, the fusion module includes a partial convolution layer, and the image fusion module 340 is used to perform partial convolution on the mirror repair image and the reflection removal image through the partial convolution layer according to the fusion weight map to obtain a result image of the reflection image to be processed.

[0184] Optionally, in one embodiment, the image processing device provided by the present application also includes a model training module, which is used to obtain a sample reflection image with a mirror reflection area and a sample reference image without reflection, the sample reflection image and the sample reference image are obtained according to the same posture of the sample object wearing glasses; obtain a labeled area mask image for indicating the mirror reflection area in the sample reflection image; input the sample reflection image into the feature extraction network of the image processing model, obtain the sample area mask image output by the mirror reflection detection branch, obtain the sample mirror restoration image output by the content restoration branch, obtain the sample reflection elimination image output by the reflection elimination branch, obtain the sample result image output by the fusion network, and obtain the fusion. The sample fusion weight map output by the weight mapping layer in the network is combined; according to the sample area mask image and the annotated area mask image, the mirror reflection detection loss is obtained; according to the sample mirror restoration image and the sample reference image, the content restoration loss is obtained; according to the sample reflection elimination image and the sample reference image, the reflection elimination loss is obtained; according to the sample result image and the sample reference image, the image fusion loss is obtained; according to the sample fusion weight map, the sample reflection image and the grayscale difference image of the sample reference image, the weight mapping loss is obtained; and according to the mirror reflection detection loss, the content restoration loss, the reflection elimination loss, the image fusion loss and the weight mapping loss, the network parameters of the image processing model are updated until the preset update stop condition is met.

[0185] Optionally, in one embodiment, the model training module is also used to obtain a horizontally flipped image of the sample reference image; obtain the symmetric mapping relationship between the sample reference image and the horizontally flipped image through a symmetric learning model; obtain the symmetric loss based on the sample reflection image, the symmetric mapping relationship and the horizontally flipped image; and update the network parameters of the image processing model based on the mirror reflection detection loss, content restoration loss, reflection elimination loss, image fusion loss, weight mapping loss and symmetric loss until a preset update stop condition is met.

[0186] The specific implementation of each of the above units can be found in the previous embodiments, which will not be described in detail here.

[0187] An embodiment of the present application also provides an electronic device, including a memory and a processor, wherein the processor is used to execute the steps of the image processing method provided in this embodiment by calling a computer program stored in the memory.

[0188] Please refer to Fig.12 , Fig.12 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application.

[0189] The electronic device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will appreciate that Fig.12 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0190] The processor 101 is the control center of the electronic device, which uses various interfaces and lines to connect various parts of the entire electronic device, and executes various functions of the electronic device and processes data by running or executing software programs and / or modules stored in the memory 102, and calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 101.

[0191] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 102 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0192] The electronic device also includes a power supply 103 for supplying power to each component. Optionally, the power supply 103 can be logically connected to the processor 101 through a power management system, so as to manage charging, discharging, and power consumption through the power management system. The power supply 103 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators, and other arbitrary components.

[0193] The electronic device may further include an input unit 104, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0194] Although not shown, the electronic device may also include a display unit, an image acquisition component, etc., which will not be described in detail here. Specifically in this embodiment, the processor 101 in the electronic device will load the executable code corresponding to one or more computer programs into the memory 102 according to the following instructions, and the processor 101 will execute the steps in the image processing method provided in this application, such as:

[0195] Performing specular reflection detection on the reflective image to be processed, and determining the specular reflection area in the reflective image to be processed;

[0196] Performing content repair on the mirror reflection area to obtain a mirror repair image of the reflective image to be processed;

[0197] Performing reflection elimination on the reflective image to be processed to obtain a reflection eliminated image of the reflective image to be processed;

[0198] The mirror repair image and the reflection removal image are fused to obtain a result image of the reflection image to be processed.

[0199] It should be noted that the electronic device provided in the embodiment of the present application and the image processing method in the above embodiment belong to the same concept, and its specific implementation process is detailed in the above related embodiments and will not be repeated here.

[0200] The present application also provides a computer-readable storage medium on which a computer program is stored. When the computer program stored therein is executed on a processor of an electronic device provided in an embodiment of the present application, the processor of the electronic device executes the steps in the image processing method provided in the present application. The storage medium may be a magnetic disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0201] The above is a detailed introduction to an image processing method, device, electronic device and computer-readable storage medium provided by the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea; at the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

[0202] It should be noted that when the above embodiments of the present application are applied to specific products or technologies, the relevant data of the user is involved, and the user's permission or consent is required, and the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

Claims

1. An image processing method, characterized in that: include: Performing specular reflection detection on the reflective image to be processed to determine the specular reflection area in the reflective image to be processed; Performing content repair on the mirror reflection area to obtain a mirror repaired image of the reflective image to be processed; Performing reflection elimination on the reflective image to be processed to obtain a reflection eliminated image of the reflective image to be processed; The mirror repair image and the reflection elimination image are fused to obtain a result image of the reflection image to be processed.

2. The image processing method according to claim 1, characterized in that: The step of performing specular reflection detection on the reflective image to be processed and determining the specular reflection area in the reflective image to be processed comprises: Acquire an image processing model, wherein the image processing model includes a feature extraction network and a specular reflection detection branch; Inputting the reflective image to be processed into the feature extraction network to perform feature extraction, thereby obtaining a feature image of the reflective image to be processed; The feature image is input into the specular reflection detection branch to perform specular reflection detection, and a regional mask image is obtained, wherein the regional mask image is used to indicate the specular reflection area in the reflection image to be processed.

3. The image processing method according to claim 2, characterized in that: The image processing model further includes a content repair branch, wherein the content repair of the mirror reflection area is performed to obtain a mirror repair image of the reflective image to be processed, including: According to the regional mask image, the content of the mirror reflection area is repaired by the content repair branch to obtain a mirror repair image of the reflective image to be processed.

4. The image processing method according to claim 3, characterized in that: The reflective image to be processed includes an object image of a target object wearing glasses, the content repair branch includes a multiplication module, a memory module, a first cascade module and a content repair module, the memory module is used to memorize the semantic feature vector of the lens area when the sample object wears glasses but no mirror reflection occurs, and the content repair of the mirror reflection area is performed by the content repair branch according to the regional mask image to obtain a mirror repair image of the reflective image to be processed, including: Acquire an inverted image of the regional mask image, and input the inverted image and the feature image into the multiplication module for multiplication operation to obtain a multiplication result image; Reading a target feature vector similar to a feature vector in the multiplication result image from the memory module to obtain a target feature image composed of the target feature vector; Inputting the multiplication result image and the target feature image into the first cascade module for cascading operation to obtain a cascade image; The cascade image is input into the content repair module for content repair to obtain the mirror repair image.

5. The image processing method according to claim 4, characterized in that: The image processing model further includes a reflection elimination branch, wherein the reflection elimination is performed on the reflective image to be processed to obtain a reflection eliminated image of the reflective image to be processed, including: The reflection image to be processed is input into the reflection elimination branch to eliminate reflections, so as to obtain a reflection eliminated image of the reflection image to be processed.

6. The image processing method according to claim 5, characterized in that: The reflection elimination branch includes a reflection elimination module and an image optimization module. The step of inputting the reflection image to be processed into the reflection elimination branch for reflection elimination to obtain a reflection elimination image of the reflection image to be processed includes: Inputting the characteristic image into the reflection elimination module to eliminate reflections, and obtaining an initial reflection elimination image; The initial de-glare image is input into the image optimization module for image optimization to obtain the de-glare image.

7. The image processing method according to claim 6, characterized in that: The image optimization module includes H image optimization units, H second cascade modules and a common convolution layer, the image optimization unit includes a gated convolution layer, an ablation convolution layer and a channel attention layer connected in sequence, the initial de-glare image is input into the image optimization module for image optimization to obtain the de-glare image, including: Inputting the initial reflection-removed image into a first image optimization unit for image optimization to obtain a first optimized initial reflection-removed image; Inputting the first optimized initial de-glare image and the initial de-glare image into the first second cascade module to obtain a first initial de-glare cascade image, and inputting the first initial de-glare cascade image into the second image optimization unit for image optimization to obtain a second optimized initial de-glare image; Input the h-1th optimized initial reflection elimination image and the i-2th initial reflection elimination cascade image into the h-1th second cascade module to obtain the i-1th initial reflection elimination cascade image, and input the h-1th initial reflection elimination cascade image into the i-th image optimization unit for image optimization to obtain the hth optimized initial reflection elimination image, h∈[3,H]; The Hth optimized initial reflection-removed image and the H-1th initial reflection-removed cascade image are input into the Hth second cascade module to obtain the Hth initial reflection-removed cascade image, and the Hth initial reflection-removed cascade image is input into the common convolution layer for convolution processing to obtain the reflection-removed image.

8. The image processing method according to claim 5, characterized in that: The image processing model further includes a fusion network, and the fusion of the mirror repair image and the reflection elimination image to obtain a result image of the reflection image to be processed includes: According to the regional mask image, the mirror repair image and the reflection elimination image are fused through a fusion network to obtain a result image of the reflection image to be processed.

9. The image processing method according to claim 8, characterized in that: The fusion network includes a weight mapping module and a fusion module. According to the regional mask image, the mirror repair image and the reflection elimination image are fused through the fusion network to obtain a result image of the reflection image to be processed, including: Inputting the regional mask image, the mirror repair image and the reflection elimination image into the weight mapping layer for weight mapping to obtain a fusion weight map, wherein the fusion weight map is used to indicate the fusion weight of the mirror repair image and the reflection elimination image; According to the fusion weight map, the mirror repair image and the reflection elimination image are fused through the fusion module to obtain a result image of the reflection image to be processed.

10. The image processing method according to claim 8, characterized in that: The fusion module includes a partial convolution layer, and according to the fusion weight map, the mirror repair image and the reflection elimination image are fused through the fusion module to obtain a result image of the reflection image to be processed, including: According to the fusion weight map, the mirror restoration image and the reflection elimination image are partially convolved through the partial convolution layer to obtain a result image of the reflection image to be processed.

11. The image processing method according to claim 9, characterized in that: Before performing specular reflection detection on the reflective image to be processed and determining the specular reflection area in the reflective image to be processed, the method further includes: Acquire a sample reflection image with a specular reflection area and a sample reference image without reflection, wherein the sample reflection image and the sample reference image are acquired according to the same posture of the sample object wearing glasses; Acquire a marked area mask image for indicating a specular reflection area in the sample reflection image; Input the sample reflection image into the feature extraction network of the image processing model, obtain the sample area mask image output by the mirror reflection detection branch, obtain the sample mirror restoration image output by the content restoration branch, obtain the sample reflection elimination image output by the reflection elimination branch, obtain the sample result image output by the fusion network, and obtain the sample fusion weight map output by the weight mapping layer in the fusion network; Acquire the mirror reflection detection loss according to the sample area mask image and the marked area mask image; Obtaining content restoration loss according to the sample mirror restoration image and the sample reference image; Obtaining a reflection elimination loss according to the sample reflection elimination image and the sample reference image; Obtaining image fusion loss according to the sample result image and the sample reference image; Obtaining a weight mapping loss according to the sample fusion weight map, the sample reflection image and the grayscale difference image of the sample reference image; The network parameters of the image processing model are updated according to the mirror reflection detection loss, the content restoration loss, the reflection elimination loss, the image fusion loss and the weight mapping loss until a preset update stop condition is met.

12. The image processing method according to claim 11, characterized in that: The updating of the network parameters of the image processing model according to the mirror reflection detection loss, the content restoration loss, the reflection elimination loss, the image fusion loss and the weight mapping loss until a preset update stop condition is met also includes: Obtain a horizontally flipped image of the sample reference image; Acquire a symmetric mapping relationship between the sample reference image and the horizontally flipped image through a symmetric learning model; Acquire a symmetry loss according to the sample reflected image, the symmetry mapping relationship and the horizontal flipped image; The updating of the network parameters of the image processing model according to the mirror reflection detection loss, the content restoration loss, the reflection elimination loss, the image fusion loss and the weight mapping loss until a preset update stop condition is met includes: According to the mirror reflection detection loss, the content restoration loss, the reflection elimination loss, the image fusion loss, the weight mapping loss and the symmetry loss, the network parameters of the image processing model are updated until a preset update stop condition is met.

13. An image processing device, characterized in that: include: A reflection detection module, used for performing specular reflection detection on the reflective image to be processed, and determining the specular reflection area in the reflective image to be processed; A content repair module, used for performing content repair on the mirror reflection area to obtain a mirror repair image of the reflective image to be processed; A reflection elimination module, used for eliminating reflections from the reflective image to be processed to obtain a reflection eliminated image of the reflective image to be processed; The image fusion module is used to fuse the mirror repair image and the reflection elimination image to obtain a result image of the reflection image to be processed.

14. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and the processor is used to run the computer program in the memory to execute the steps in the image processing method according to any one of claims 1 to 12.

15. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the image processing method according to any one of claims 1 to 12.

Images