Image highlight removal method, electronic device and storage medium

Abstract

The embodiments of the present disclosure provide an image highlight removal method, an electronic device and a storage medium. The method comprises: performing highlight detection on a highlight image to obtain a predicted highlight mask image, wherein the predicted highlight mask image is used for indicating image features of a highlight region in the highlight image; performing highlight removal on the highlight image to obtain a highlight-removed image; and fusing the predicted highlight mask image and the highlight-removed image to obtain a diffuse reflection image corresponding to the highlight image.

Description

Highlight removal method of image, electronic device and storage medium

[0001] The present disclosure claims priority to Chinese Patent Application No. 202410644854.7, filed on May 22, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to the technical field of computer vision, and particularly relates to a highlight removal method of image, an electronic device and a storage medium. BACKGROUND

[0003] Mirror highlight is a common physical phenomenon, which usually appears in the form of a light spot on the surface of a smooth object, which changes the color of the object surface, destroys the outline of the object, and blocks the texture of the object surface. The existence of highlights causes many disturbances to computer vision and image processing tasks such as object detection, image segmentation, edge detection, precision measurement and face recognition, and reduces the overall performance of the vision system. SUMMARY

[0004] In one aspect, the present disclosure provides a highlight removal method of image. The method comprises: performing highlight detection on a highlight image to obtain a predicted highlight mask image, the predicted highlight mask image being used to indicate image features of a highlight region in the highlight image; performing highlight removal on the highlight image to obtain a highlight removal image; and fusing the predicted highlight mask image and the highlight removal image to obtain a diffuse reflection image corresponding to the highlight image.

[0005] In another aspect, the present disclosure provides a highlight removal device of image. The device comprises: a processing module.

[0006] The processing module is configured to perform highlight detection on a highlight image to obtain a predicted highlight mask image, the predicted highlight mask image being used to indicate image features of a highlight region in the highlight image. The processing module is further configured to perform highlight removal on the highlight image to obtain a highlight removal image. The processing module is further configured to fuse the predicted highlight mask image and the highlight removal image to obtain a diffuse reflection image corresponding to the highlight image.

[0007] In yet another aspect, the present disclosure provides an electronic device. The electronic device comprises: a memory and a processor. The memory and the processor are coupled. The memory is configured to store a computer program. The processor implements the highlight removal method of image of any of the above embodiments when executing the computer program.

[0008] In yet another aspect, the present disclosure provides a computer-readable storage medium having computer program instructions stored thereon, the computer program instructions being executed by a processor to implement the highlight removal method of image of any of the above embodiments.

[0009] In yet another aspect, the embodiments of the present disclosure provide a computer program product comprising computer program instructions, which, when executed, implement the high-light removal method of the image according to any of the above embodiments. BRIEF DESCRIPTION OF DRAWINGS

[0010] In order to more clearly illustrate the technical solutions in the present disclosure, the drawings needed to be used in some embodiments of the present disclosure will be briefly introduced as follows. Obviously, the drawings described in the following description are only some of the drawings of the present disclosure, and other drawings can also be obtained by those of ordinary skill in the art based on these drawings.

[0011] FIG. 1 is a system architecture diagram of a high-light processing model according to some embodiments of the present disclosure;

[0012] FIG. 2 is a flowchart of a high-light removal method of an image according to some embodiments of the present disclosure;

[0013] FIG. 3 is a network structure diagram of a high-light detection module according to some embodiments of the present disclosure;

[0014] FIG. 4 is a network structure diagram of a U-net attention conversion block according to some embodiments of the present disclosure;

[0015] FIG. 5 is an example diagram of generating a diffuse reflection image from a high-light image according to some embodiments of the present disclosure;

[0016] FIG. 6 is a training flowchart of a high-light processing model according to some embodiments of the present disclosure;

[0017] FIG. 7 is a structural diagram of a high-light removal device of an image according to some embodiments of the present disclosure;

[0018] FIG. 8 is a structural diagram of another high-light removal device of an image according to some embodiments of the present disclosure. DETAILED DESCRIPTION

[0019] The technical solutions in the embodiments of the present disclosure will be described clearly and completely below with reference to the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only some of the embodiments of the present disclosure, rather than all the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those of ordinary skill in the art without creative work fall within the scope of protection of the present disclosure.

[0020] It should be noted that in the present disclosure, the expressions such as “exemplarily” or “for example” are used to represent an example, illustration, or description. Any embodiment or design scheme described as “exemplarily” or “for example” in the present disclosure should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the expressions such as “exemplarily” or “for example” are intended to present relevant concepts in a detailed manner.

[0021] Hereinafter, the terms “first” and “second” are used only for descriptive purposes, and should not be construed as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with “first” and “second” can explicitly or implicitly include one or more of the features.

[0022] In the description of the present disclosure, “ / ” means “or” unless otherwise specified. For example, A / B can mean A or B. “And / or” in this document only describes the relationship between the associated objects, which means that there can be three relationships. For example, A and / or B can mean only A, A and B, and only B. In addition, “at least one” means one or more, and “multiple” means two or more.

[0023] Specular highlight is a common physical phenomenon that usually appears in the form of a light spot on the surface of a smooth object, which changes the color of the object surface, destroys the outline of the object, and blocks the texture of the object surface. The existence of highlights causes many disturbances to computer vision and image processing tasks such as object detection, image segmentation, edge detection, precision measurement, and face recognition, and reduces the overall performance of the vision system.

[0024] Highlight detection and removal has been an important research topic in the field of computer vision and image for a long time, which plays an indispensable role in other vision research and is widely used in daily life. In recent decades, many methods have been proposed to solve the problem of specular highlight detection and removal. Although great progress has been made in highlight detection or removal of a single image, few works have considered both problems together. At present, existing solutions can be roughly divided into two categories: model-based methods and deep learning-based methods.

[0025] Model-based methods are usually some traditional techniques, such as optimization-based methods, clustering, and filtering, etc. The disadvantage of these methods is that they lack semantic information of images, so they cannot perform semantic removal for highlights and pure white in complex real-world scenes.

[0026] Most of the recent methods are mainly based on deep learning techniques. Although they have made substantial progress, there are still some common problems. For example, first, most of the methods use synthetic data sets, resulting in poor high light removal effects in real scenes. Second, these methods are not natural in the transition between high light and non-high light areas, and are prone to artifacts, making the high light removal results appear unrealistic.

[0027] Therefore, how to improve the realism of the high light removal effect in the image has become a technical problem to be solved.

[0028] To solve the above technical problems, the present embodiment provides a high light removal method for an image, which is a single image high light detection and removal method based on mask guidance, especially a mirror high light removal method from a source domain to a target domain, mainly solving the problem of combining single image high light detection and removal, that is, the problems of poor real scene effect, unnatural transition between high light and non-high light areas, and easy to appear artifacts in the high light detection and removal algorithm.

[0029] The single image high light detection and removal method based on mask guidance can play a role in many application fields and application scenarios. For example, this method can play a role in the following application fields.

[0030] Computer vision and image processing: In the field of computer vision, the existence of high light may interfere with image analysis tasks such as object detection, image segmentation, and feature extraction. Through high light detection and removal, the performance and accuracy of computer vision systems can be improved.

[0031] Photography and image editing: In the field of photography and image editing, high light is usually caused by lens reflection or strong light sources. However, high light can cause problems such as loss of detail, exposure imbalance, and color distortion. Through the high light detection and removal method, the high light area in the image can be repaired or improved, and better visual effects can be obtained.

[0032] Computer graphics and rendering: High light is often a challenge in simulating real light, as it involves complex physical processes such as light reflection and refraction. Through high light detection and removal, more realistic and realistic images and animations can be generated.

[0033] Video analysis and processing: High light can interfere with tasks such as target tracking, behavior recognition, and video encoding. Through high light detection and removal, the robustness and accuracy of video analysis and processing algorithms can be improved.

[0034] Medical imaging and biological imaging: The presence of highlights can interfere with the interpretation and analysis of images, affecting the accuracy of diagnosis and research results. Through highlight detection and removal, the quality of medical images and biological images can be improved to assist doctors and researchers in making accurate judgments.

[0035] In the context of image enhancement, the highlight removal method for images provided by the embodiments of the present disclosure can be used for image enhancement to improve image quality. For example, in smartphones or digital cameras, due to the limitations of lighting conditions or photography techniques, there may be overexposed highlight areas in the image. Through the removal of highlights, the impact of highlights can be reduced, making the image clearer and more natural.

[0036] In the context of video post-production, the highlight removal method for images provided by the embodiments of the present disclosure can improve the balance and consistency of the picture by detecting and removing highlights in the video, making the entire video look more professional and attractive.

[0037] In the context of medical image analysis, highlights can interfere with the analysis and diagnosis of medical images. For example, in X-ray, magnetic resonance imaging (MRI) or computed tomography (CT) scan images, highlights may hide important anatomical structures or lesion features. The highlight removal method for images provided by the embodiments of the present disclosure can improve the visualization and analysis capabilities of medical images by reducing the impact of highlights, assisting doctors in making accurate diagnoses.

[0038] In the context of virtual reality and augmented reality, the presence of highlights can reduce the realism of virtual or augmented content. The highlight removal method for images provided by the embodiments of the present disclosure can improve the realism and consistency of virtual or augmented content by reducing the interference of highlights.

[0039] It should be noted that with the continuous development of future technology, it is expected that more application fields and application scenarios will adopt this method.

[0040] The high light removal method for an image provided by the embodiment of the present disclosure is built on a mask-guided high light detection and removal combined framework, which is composed of an encoder, a decoder, a high light detection module and a high light removal module. Two sets of masks and red green blue (RGB) images are obtained through feature sharing of the high light detection module and the high light removal module. The two sets of images focus on the global features and the local features of the high light image respectively. The final image obtained through fusion of the global features and the local features can effectively reflect the global features and the local features, so that the high light removal effect appears more realistic. That is, based on the basic structure of the encoder-decoder, through methods such as feature extraction, attention mechanism and feature fusion, the features of the mask are captured, the semantic and conceptual information in the image can be better obtained, and thus the adaptability and accuracy of the high light removal task are improved. Compared with the traditional model-based and deep learning-based methods, the global features and the local features of the input image can be better captured, so that the high light removal effect is more accurate and realistic.

[0041] In this way, by fusing the global features focused on by the high light detection and the local features focused on by the high light removal, the diffuse reflection image obtained through processing of the high light image can effectively reflect the global features and the local features, and the high light removal effect appears more realistic, thereby improving the authenticity of the high light removal effect in the image.

[0042] Exemplarily, as shown in FIG. 1, FIG. 1 is a system architecture diagram of a high light processing model according to an embodiment of the present disclosure. The high light processing model 100 is deployed in a high light removal device of an image, and includes an encoder 101, a decoder 102, a high light detection module 103, a high light removal module 104 and a fusion module 105.

[0043] The encoder 101 uses EfficientNet as a feature extraction backbone network to convert the input image into a series of feature maps, and outputs feature maps of different scales to the decoder 102. EfficientNet is a kind of efficient and scalable convolutional neural network (CNN) architecture.

[0044] The decoder 102 gradually restores the multiple feature maps from the encoder 101 to the original size through upsampling, and outputs the restored feature maps to the high light detection module 103 and the high light removal module 104.

[0045] The high light detection module 103 performs upsampling and feature fusion on the feature maps from the decoder 102 by deepening the network layers and adding residual blocks (ResidualBlock), to generate a predicted mask (i.e., a predicted high light mask image), and then extract more feature information.

[0046] In the high light removal module 104, a convolutional block attention module (CBAM) is added based on the U-Net to dynamically learn the importance of each channel and spatial position in the input feature map, and then adjust the brightness of the input image and the predicted mask to obtain a high light removal image sequence. The U-Net is a CNN architecture for image segmentation.

[0047] The fusion module 105 performs image fusion on the high light mask image output by the high light detection module 103 and the high light removal image output by the high light removal module 104 to obtain a diffuse reflection image.

[0048] For example, the high light detection module 103 can map the predicted high light mask to the input feature map, output a feature map with a predicted high light mask, and input the feature map to the high light removal module 104, thereby realizing feature sharing of high light detection and high light removal, ensuring the mapping relationship between the high light detection result and the high light removal result, and improving the accuracy of the diffuse reflection image after fusion.

[0049] In summary, the main advantage of the entire system architecture is that the attention mechanism of CBAM and the brightness adjustment unit are used to help the model learn the input information better. By using loss functions such as structural similarity index measure (SSIM) and smooth L1 loss, the effects of high light detection and removal network are continuously adjusted to improve the adaptability and accuracy of complex scenes and high light removal tasks, so that the image is more realistic and natural, while retaining the details and structural information.

[0050] It should be noted that the high light removal device for images is not limited by the embodiments of the present application. That is, the high light removal device for images can be a server, or the high light removal device for images can be a terminal.

[0051] The terminal can be a device with wireless transceiving function. The terminal can be a mobile phone, a tablet computer (Pad), a computer with wireless transceiving function, a virtual reality (VR) terminal, an augmented reality (AR) terminal, a wireless terminal in industrial control, a wireless terminal in self driving, a wireless terminal in remote medical, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The embodiments of the present disclosure do not limit the application scenarios. The terminal can also be referred to as a user, a user equipment (UE), an access terminal, a UE unit, a UE station, a mobile station, a mobile station, a remote station, a remote terminal, a mobile device, a UE terminal, a wireless communication device, a UE agent or a UE apparatus, etc. The embodiments of the present disclosure do not limit this.

[0052] The server can be a single physical server, or can also be a server cluster composed of multiple servers. Alternatively, the server cluster can also be a distributed cluster. Alternatively, the server can be a cloud server. The embodiments of the present disclosure do not limit the implementation of the server.

[0053] It should be noted that FIG. 1 is only an exemplary framework diagram, the number of modules (or network layers) included in FIG. 1, the names of various modules are not limited, and the system architecture can also include other modules such as a fully connected layer in addition to the modules shown in FIG. 1.

[0054] The embodiments of the present disclosure do not limit the application scenarios. The system architecture and business scenarios described in the embodiments of the present disclosure are used to more clearly illustrate the technical solutions of the embodiments of the present disclosure, and do not constitute a limitation on the technical solutions provided by the embodiments of the present disclosure. Those skilled in the art can know that with the evolution of network architecture and the appearance of new business scenarios, the technical solutions provided by the embodiments of the present disclosure are also applicable to similar technical problems.

[0055] FIG. 2 shows a flow diagram of a highlight removal method of an image. As shown in FIG. 2, the method includes S201-S203.

[0056] S201, detecting highlights of a highlight image to obtain a predicted highlight mask image.

[0057] The predicted highlight mask image is used to indicate the image features of the highlight region in the highlight image.

[0058] In the embodiments of the present disclosure, the terminal can obtain a plurality of first feature maps of the highlight image at different scales, and determine a second feature map and a plurality of first highlight mask images according to the plurality of first feature maps. Then, in the process of performing highlight detection on the highlight image to obtain the predicted highlight mask image, the terminal can perform highlight detection on the second feature map to determine a second highlight mask image, and fuse the second highlight mask image with the plurality of first highlight mask images to determine the predicted highlight mask image. The second feature map is formed by fusing the plurality of first feature maps, each first highlight mask image corresponds to a first feature map, the first highlight mask image is used to indicate the image features of the highlight region in the corresponding first feature map, and the second highlight mask image is used to indicate the image features of the highlight region in the second feature map.

[0059] That is, by referring to the feature maps at different scales, the highlight mask used to indicate the highlight region in the highlight image is comprehensively predicted. In this way, the accuracy of the predicted highlight mask can be improved.

[0060] As an implementation manner, the terminal is deployed with a highlight processing model, and the highlight processing model can include a highlight detection module. In the process of performing highlight detection on the highlight image to obtain the predicted highlight mask image, the terminal can perform highlight detection on the highlight image based on the highlight detection module in the highlight processing model to obtain the predicted highlight mask image.

[0061] The highlight detection module performs feature extraction on the RGB image by deepening the network layers and the processing of the ResidualBlock based on the neural network to generate the predicted highlight mask image. That is, the input of the highlight detection module is the feature map F∈R C×H×W , and the output is the feature map with the highlight mask.

[0062] Exemplarily, as shown in FIG. 3, a network structure of the highlight detection module is shown, which includes a Conv2d (a network layer performing convolution operation on two-dimensional data), a BatchNorm2d (a network layer of batch normalization), and a ResidualBlock. The highlight detection module can use a convolution layer (i.e., Conv2d) for feature extraction, together with a batch normalization layer and a Sigmoid activation function. Then, the highlight detection module performs feature extraction through two ResidualBlock modules, each of which includes two convolution layers and a residual connection, and each residual block is processed using a batch normalization layer and a LeakyReLU activation function. After that, the highlight detection module uses a convolution layer for processing, and outputs a feature map with a size of M∈R 1×H×Wa predicted highlight mask, and the output of the convolutional layer is nonlinearly mapped by a Sigmoid activation function.

[0063] For example, the highlight processing model can further include an encoder and a decoder. The terminal can acquire the plurality of first feature maps based on the encoder in the highlight processing model, and determine the second feature map and the plurality of first highlight mask images based on the decoder in the highlight processing model.

[0064] The encoder extracts features based on a pre-trained network of EfficientNet, in order to capture image information of different scales, an image feature pyramid is constructed on the output of the encoder, a series of image features with different scales are obtained by performing feature extraction on different levels of the encoder and performing scale transformation or sampling operations, and then the input image is converted into a series of feature maps to obtain multi-level encoder features (i.e., the plurality of first feature maps).

[0065] Exemplarily, an input image I ∈ R C×H×W where C represents the number of channels, H represents the height of I, and W represents the width of I, the encoder can use a plurality of convolutional layers to extract features from the feature map x, i.e., x = Convolution(x, W) + b, where W represents the convolution kernel weight, and b represents the bias term. Next, the encoder uses a pooling layer to reduce the spatial size of the feature map while preserving the main features, usually in conjunction with a batch normalization layer. Then, the encoder solves the gradient vanishing problem in the deep network by introducing a residual connection, and the residual block is composed of three convolution kernels, which contains a skip connection that directly adds the input of the previous layer to the output of the subsequent layer, so that the network can learn the residual mapping, and the number of feature channels can be increased at different levels of the network to increase the expression ability of the feature encoder. Then, in order to capture image information of different scales, an image feature pyramid is constructed on the output of the feature encoder. By performing feature extraction on different levels of the encoder and performing scale transformation or sampling operations, a series of image features with different scales are obtained.

[0066] That is, the encoder converts the input image into a feature pyramid through a series of convolution and pooling operations, where each layer contains feature information of different scales and abstraction levels. For example, the encoder can increase the dimension by using a 3x3 convolutional layer, which is usually used with a batch normalization layer and an activation function. Next, the encoder is processed by a plurality of kxk convolutional layers, and then the dimension is reduced by a 3x3 convolutional layer, global average pooling and two fully connected layers to obtain multi-level features.

[0067] The decoder is used to up-sample and feature fusion of the feature map through the decoder, and generate highlight mask according to the characteristics of different mask layers, and capture more low-level features. That is, for the feature map information x received from the encoder, up-sampling (such as interpolation method, deconvolution) is performed to restore the scale of the feature map to the same as the original image; secondly, the feature fusion is performed between the feature map with restored scale and the corresponding feature map in the encoder through the jump connection, that is, x = Concatenate(x encoder , x decoder ); then, a series of convolution operations and nonlinear activation functions are used to process the fused feature map to extract and represent the features of the highlight area; finally, the Sigmoid activation function is used to map the feature map to the range of [0, 1], and two highlight masks are generated from the third layer from the bottom and the last layer. The highlight mask of the third layer from the bottom captures more low-level features, which contains more detailed information (i.e., basic and local features in the image, such as edge information, texture, etc.); the highlight mask of the last layer can capture more high-level features, i.e., the features of the whole object or scene, such as the shape between objects, the overall structure, etc.

[0068] It should be noted that the feature map information x is the feature map information of a layer in the feature pyramid. The features of the highlight area are the purpose of the convolution layer and the nonlinear activation function processing. Through the above operations, the representation information of the highlight area can be learned from the feature map, and the position and boundary of the highlight area are determined through the highlight detection module.

[0069] It can be understood that the highlight processing model is introduced to realize highlight detection and highlight removal, so as to improve the accuracy of highlight detection and highlight removal through the training of the highlight processing model.

[0070] It should be noted that for the above feature fusion process, reference can be made to the existing introduction of feature fusion through feature splicing, weighted summation, attention mechanism or other complex combination strategies in the deep learning framework, which will not be repeated here.

[0071] S202, performing highlight removal on the highlight image to obtain a highlight removed image.

[0072] In the embodiments of the present application, the terminal can perform highlight removal on the second feature map to obtain a highlight removed image.

[0073] That is, through the feature sharing of highlight detection and highlight removal, the mapping relationship between the highlight detection result and the highlight removal result can be ensured to improve the accuracy of the diffuse reflection image after the fusion of the two.

[0074] As an implementation manner, the high-light processing model can further include a high-light removal module. In the process of performing high-light removal on the high-light image to obtain a high-light removal image, the terminal can perform high-light removal on the high-light image based on the high-light removal module in the high-light processing model to obtain the high-light removal image.

[0075] The high-light removal module can extract image features through a U-net optimization module (U-net optimization module) inside the high-light removal module, perform feature extraction and depth enhancement operations on the obtained feature map through a convolution layer and a batch normalization layer of a U-Net ConvBlock (convolution block), and improve the basic module of the U-Net. Through the CBAM attention mechanism, the importance of the channel and the spatial position is dynamically learned, so that the foreground and the background are better distinguished, and the representation ability of the feature map is improved. In addition, the high-light removal module can introduce a convolutional gated recurrent unit cell (ConvGRUCell) to perform multiple recursive calculations on the feature map, and the output layer includes a convolution layer and a hyperbolic tangent (Tanh) activation function, which is used to convert the output of the ConvGRUCell into an image after brightness adjustment.

[0076] Exemplarily, as shown in FIG. 4, a network structure of a U-net attention ConvBlock in the high-light removal module is shown, which includes a Conv2d, a BatchNorm2d, and a channel attention module (CAM) and a spatial attention module (SAM) of a convolutional block attention module (CBAM). The U-net attention ConvBlock module extracts image features, and uses the CBAM attention mechanism to learn the importance of the channel and the spatial position of the feature map F∈R C×H×W After the global maximum pooling and the global average pooling processing, a channel attention map M c ∈R C×1×1 and a spatial attention map M s ∈R 1×H×W. Then, the U-net attention ConvBlock module sends them into a multilayer perceptron (MLP) respectively. The MLP mainly learns the nonlinear relationship between the features in CBAM, and sends the two attention maps into the MLP to weight the channel dimension and the spatial dimension, so as to improve the representation ability of the features and the diversity of the extracted features, and facilitate the subsequent information fusion. Then, the U-net attention ConvBlock module adds the features output by the MLP, and then performs a Sigmoid activation operation (σ is the Sigmoid operation) to generate the final channel attention map M c (As shown in Formula One). M c (F) = σ (MLP (AvgPool (F)) + MLP (MaxPool (F))) Formula One.

[0077] And the U-net attention ConvBlock module performs an element-wise multiplication operation on M c and the input feature map F, which represents element-level multiplication, and the middle adopts a broadcast mechanism for dimension transformation and matching to generate the input feature F' required by the spatial attention module (as shown in Formula Two).

[0078] Then, the U-net attention ConvBlock module processes F' based on global maximum pooling and global average pooling to obtain two R 1×H×W feature maps, and combines them based on the channel, and then reduces the dimension to 1 channel through a 7x7 convolution layer, and generates a spatial attention map M s (As shown in Formula Three). M s (F') = σ (f 7×7 ([AvgPool (F') ) ; (MaxPool (F')) ]) Formula Three.

[0079] Finally, the U-net attention ConvBlock module multiplies M s and the input feature F' to generate the final feature F'' (as shown in Formula Four).

[0080] The enhanced feature F'' will be used as the input of the subsequent network layer, while retaining the key information, suppressing noise and irrelevant information;

[0081] Secondly, the U-net attention ConvBlock module introduces ConvGRUCell to perform multiple recursive computations on the feature map. The output layer contains a convolutional layer and a Tanh activation function to convert the ConvGRUCell output into a brightness-adjusted image. ConvGRUCell is a gated recurrent neural network that processes input features temporally and spatially through convolutional operations, allowing the model to more effectively capture the temporal and spatial features in the input sequence. Firstly, convolutional operations are applied to the input value x in the reset and update gates, respectively. t The output h from the previous time step t-1 After multiplying the weights, W and U are the corresponding weight values, which, after activation by the Sigmoid function, yield z. t and r t Two threshold values ​​(as shown in Formulas 5 and 6). z t =σ(W z x t +U z h t-1 Formula 5. t =σ(W t x t +U t h t-1 Formula 6.

[0082] Furthermore, convolution operations are used to compute candidate hidden state values. (As shown in Formula 7).

[0083] By updating the door and To calculate the new hidden state h t (As shown in Formula 8).

[0084] In each iteration, the model's output is used as the input for the next iteration to gradually remove highlights. Finally, a series of RGB images with highlights removed are obtained by using convolutional layers and the Tanh activation function in the output layer.

[0085] S203. The predicted specular mask image and the specular removal image are fused to obtain the diffuse reflection image corresponding to the specular image.

[0086] In this embodiment, the specular processing model may further include a fusion module. The terminal can perform image fusion between the predicted specular mask and the specular removal image based on the fusion module in the specular processing model to obtain the diffuse reflection image corresponding to the specular image.

[0087] It should be noted that for the above image fusion process, reference can be made to the existing introduction of image fusion based on pixel-level fusion, feature-level fusion, decision-level fusion and other fusion strategies, which will not be repeated here.

[0088] The process of obtaining the diffuse reflection image corresponding to the highlight image will be introduced below with examples. As shown in FIG. 5, the highlight image with a size of 200*200*3 is input into the encoder to obtain the feature maps at different scales (i.e., *16, *32, *64, *128, *256). Then the decoder performs 3*3 convolution processing on the feature map of *256, and sequentially performs feature fusion with the feature maps of other scales to restore the feature map of *16. Then, the restored feature map is processed by highlight detection to obtain a highlight mask image, and at the same time, the restored feature map is processed by highlight removal and fused with the highlight mask image to obtain the diffuse reflection image corresponding to the highlight image.

[0089] It can be understood that by fusing the global features focused on by the highlight detection with the local features focused on by the highlight removal, the diffuse reflection image obtained by processing the highlight image can effectively reflect the global features and local features, and the highlight removal effect will be more realistic, thereby improving the authenticity of the highlight removal effect in the image.

[0090] In some embodiments, the highlight processing model can be obtained by training in the following manner: the terminal trains an initial highlight processing model based on a training set until the loss function of the initial highlight processing model converges to obtain the highlight processing model. The loss function of the highlight processing model includes a mask loss corresponding to the highlight detection module and an image loss corresponding to the highlight removal module, the mask loss is used to indicate the error between the predicted highlight mask and the real highlight mask, and the image loss is used to indicate the similarity index and the smoothness loss between the highlight removal image and the original image.

[0091] It should be noted that the terminal can update the model parameters adaptively.

[0092] That is, by training the highlight processing model including highlight detection and highlight removal, the adaptability and accuracy of the complex scene and highlight removal task can be improved.

[0093] Exemplarily, as shown in FIG. 6, it shows the training process of the highlight processing model. The highlight image training set is input into the encoder and the decoder in sequence to obtain the feature map and the highlight mask at different layers. Then, the feature map is processed by the highlight detection to obtain the feature map with the predicted highlight mask, and the predicted mask is fused with the highlight mask at different layers to obtain the final highlight mask image. Then, the final highlight mask image and the feature map output by the highlight detection are processed by the highlight removal to obtain the output result of the highlight processing model.

[0094] The commonly used index for image similarity evaluation is the structural similarity index measure (SSIM). For the original image x and the generated highlight-removed image y, the loss between them is shown in Equation Nine. ssim = 1 - SSIM(x, y) Equation Nine.

[0095] As an implementation of the structural similarity theory, SSIM(x, y) defines the structural information from the perspective of image composition as independent of brightness and contrast, reflects the properties of the structure of objects in the scene, and models distortion as a combination of brightness, contrast, and structure. The larger the value of the structural similarity index, the higher the similarity of the two pictures (as shown in Equation Ten). SSIM(x, y) = [l(x, y)] α [c(x, y)] β [s(x, y)] γ Equation Ten.

[0096] μ x , μ y , σ x , σ y are the mean and standard deviation of x and y, respectively. σ x σ y is the covariance of x and y, and α > 0, β > 0, γ > 0, C1, C2, C3 are constants. l(x, y) compares the brightness of x and y (as shown in Equation Eleven), c(x, y) compares the contrast of x and y (as shown in Equation Twelve), and s(x, y) compares the structure of x and y (as shown in Equation Thirteen).

[0097] The original image and the generated image are smoothed using a smoothing function to reduce the highlight part of the image. The loss function uses Smooth L1 Loss, and m is the numerical difference between the predicted box and the real box, as shown in Equation Fourteen.

[0098] By comparing the original image and the generated image using SSIM, and comparing the smoothed images using SSIM, the highlight-removed image loss L rgb is obtained (as shown in Equation Fifteen). rgb = λ1·smooth l1 + λ2·L ssim Equation Fifteen.

[0099] where λ1, λ2 are loss proportion weight values, L rgb can be used as the objective function of the optimization algorithm to improve the generated results in the highlight removal task.

[0100] The segmentation mask loss is shown in Equation 16.

[0101] m i 、 respectively represent the predicted highlight mask and the real image mask.

[0102] The final overall loss function is shown in Equation 17. total = β1·L mask + β2·L rgb Equation 17.

[0103] where β1 and β2 are corresponding weight parameters, and the two parameters are adjusted through continuous iteration until the optimal effect is achieved.

[0104] It can be understood that the CBAM attention mechanism and the brightness adjustment unit are used to help the model better learn the input information, and the SSIM, Smooth L1 Loss, and other loss functions are used to continuously adjust the effect of the highlight detection and removal network, so as to better capture the global features and local features of the input image, make the highlight removal effect more accurate and realistic, and improve the adaptability and accuracy of the complex scene and highlight removal task.

[0105] It should be noted that the training data used by the existing highlight removal is all synthetic data, and the synthetic data is affected by the generation algorithm and parameters, and may not be able to completely capture the complex changes and noise in the real world, so that the generalization ability of the model in the real scene is affected, thereby affecting the accuracy of the model.

[0106] In the embodiments of the present disclosure, the training set includes: a first image of a real scene, a second image obtained by removing highlights from the first image, a third image used to indicate the highlight intensity of the first image, and a fourth image used to indicate the highlight area in the first image.

[0107] As an implementation manner, the terminal can collect a multi-illumination picture sequence: images are obtained from a natural scene, and input images under multiple-view illumination (a) are obtained through ambient light distortion, and initial real data are obtained: diffuse reflection images (b) and specular highlight images (c) are obtained by using a traditional highlight removal method, then high-quality picture blocks (d) are screened: the quality is judged according to the specular highlight elimination effect, and high-quality diffuse reflection images (e) and corresponding specular highlight images (f) are obtained, and finally desaturation and binarization are performed: the desaturation operation is performed on (f), the color information of (f) is removed to obtain (g), and (g) and (e) are added and fused to obtain (h), that is, (g) + (e) = (h). In addition, the binarization operation is performed on (g) to obtain a specular highlight mask (i), wherein the value "1" represents a specular highlight area, and "0" represents a non-specular highlight area; finally, (d), (h), (f) and (i) obtained are combined to form a data set (SHIQ).

[0108] Exemplarily, first, a multi-illumination picture sequence is collected: images under multiple natural scenes are photographed by fixing the brightness of a light source, and image sequences a1, a2, …, aNunder multiple views are obtained through ambient light distortion. n wherein n is the length of the image sequence;

[0109] Secondly, initial real images are obtained: diffuse reflection images (b) and specular highlight images (c) are obtained by processing a1, a2, …, aNusing a robust principal component analysis method. n

[0110] Then, high-quality picture blocks (d) are screened: since the robust principal component analysis may produce an undesirable result for a complex illumination and material picture sequence, the quality of the picture blocks is judged according to the specular highlight elimination effect. Each picture needs to be cropped with a block size of k x k and a step length of l to obtain diffuse reflection images (e) and specular highlight images (f) with effective highlight removal.

[0111] Next, desaturation and binarization operations are performed: assuming that the specular highlight is white, the desaturation operation is performed on (f) to remove the color information of the layer to obtain an image (g), then (g) and (e) are added and fused to obtain an image (h), that is, (g) + (e) = (h). In addition, the binarization operation is performed on (g) to obtain a Mask image (i), wherein the value "1" represents a specular highlight area, and "0" represents a non-specular highlight area.

[0112] Finally, (d), (h), (f) and (i) are combined to form a required highlight detection and removal training set (SHIO).

[0113] ​That is, by grouping annotation on real data sets, and model processing to get two groups of masks and RGB images, the two groups of images are fused, and the final image can effectively focus on the global and local features of the highlight image, so that the highlight removal effect is more realistic.

[0114] It can be understood that, in order to realize the above functions, the image highlight removal device comprises a hardware structure and / or a software module corresponding to each function. Those skilled in the art should easily realize that, in combination with the algorithm steps of each example described in the embodiments of the present disclosure, the present disclosure can be realized in the form of hardware or a combination of hardware and computer software. Whether a certain function is executed in the form of hardware or computer software driven hardware depends on the specific application and design constraints of the technical solution. Professional technicians can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present disclosure.

[0115] The embodiments of the present disclosure can divide the functions of the image highlight removal device according to the above-mentioned method embodiments, for example, each function module can be divided according to each function, or two or more functions can be integrated into one function module. The integrated module can be realized in the form of hardware or software. It should be noted that the division of modules in the embodiments of the present disclosure is illustrative, and is only a logical function division. In actual implementation, another division method can be used. The following will be described taking the example of dividing each function module according to each function.

[0116] FIG. 7 is a structural schematic diagram of an image highlight removal device according to an embodiment of the present disclosure. The image highlight removal device can execute the image highlight removal method provided by the above-mentioned method embodiments S201-S203. As shown in FIG. 7, the image highlight removal device 700 comprises a processing module 701.

[0117] The processing module 701 is configured to perform highlight detection on the highlight image to obtain a predicted highlight mask image, the predicted highlight mask image being used to indicate the image features of the highlight region in the highlight image. The processing module 701 is further configured to perform highlight removal on the highlight image to obtain a highlight removal image. The processing module 701 is further configured to fuse the predicted highlight mask image and the highlight removal image to obtain a diffuse reflection image corresponding to the highlight image.

[0118] In some embodiments, the highlight removal device 700 for an image further comprises an acquisition module 702. The acquisition module 702 is configured to acquire a plurality of first feature maps of the highlight image at different scales. The processing module 701 is further configured to determine a second feature map fused from the plurality of first feature maps and a plurality of first highlight mask images according to the plurality of first feature maps, each first highlight mask image corresponding to a first feature map and being used to indicate image features of a highlight region in the corresponding first feature map. The processing module 701 is configured to perform highlight detection on the second feature map to determine a second highlight mask image used to indicate image features of a highlight region in the second feature map. The processing module 701 is further configured to fuse the second highlight mask image with the plurality of first highlight mask images to determine a predicted highlight mask image.

[0119] In some embodiments, the processing module 701 is configured to perform highlight removal on the second feature map to obtain a highlight removal image.

[0120] In some embodiments, the processing module 701 is configured to perform highlight detection on the highlight image based on a highlight detection module in the highlight processing model to obtain the predicted highlight mask image. The processing module 701 is configured to perform highlight removal on the highlight image based on a highlight removal module in the highlight processing model to obtain the highlight removal image. The processing module 701 is configured to fuse the predicted highlight mask image with the highlight removal image based on a fusion module in the highlight processing model to obtain the diffuse reflection image corresponding to the highlight image.

[0121] In some embodiments, the processing module 701 is further configured to train an initial highlight processing model based on the training set until a loss function of the initial highlight processing model converges to obtain the highlight processing model, the loss function of the highlight processing model including a mask loss corresponding to the highlight detection module and an image loss corresponding to the highlight removal module.

[0122] In some embodiments, the mask loss is used to indicate an error between the predicted highlight mask and a real highlight mask, and the image loss is used to indicate a similarity index and a smooth loss between the highlight removal image and the original image.

[0123] In some embodiments, the training set comprises a first image of a real scene, a second image obtained by performing highlight removal on the first image, a third image used to indicate a highlight intensity of the first image, and a fourth image used to indicate a highlight region in the first image.

[0124] In the case of implementing the functions of the above-mentioned integrated modules in the form of hardware, the disclosure embodiments provide another structure of the image highlight removal device involved in the above-mentioned embodiments. As shown in FIG. 8, the image highlight removal device 800 includes a processor 802, a bus 804. In some embodiments, the image highlight removal device can further include a memory 801; in some embodiments, the image highlight removal device can further include a communication interface 803.

[0125] The processor 802 can implement or execute the various exemplary logical blocks, modules and circuits described in combination with the disclosure embodiments. The processor 802 can be a central processing unit, a general purpose processor, a digital signal processor, an application specific integrated circuit, a field programmable gate array or other programmable logic device, a transistor logic device, a hardware component or any combination thereof, which can implement or execute the various exemplary logical blocks, modules and circuits described in combination with the disclosure embodiments. The processor 802 can also be a combination of implementing computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0126] The communication interface 803 is used to connect with other devices through a communication network. The communication network can be an Ethernet, a wireless access network, a wireless local area network (WLAN), etc.

[0127] The memory 801 can be a read-only memory (ROM) or other type of static storage device that can store static information and instructions, a random access memory (RAM) or other type of dynamic storage device that can store information and instructions, an electrically erasable programmable read-only memory (EEPROM), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program codes in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0128] As an implementation manner, the memory 801 can exist independently of the processor 802, and the memory 801 can be connected with the processor 802 through the bus 804, for storing instructions or program codes. When the processor 802 invokes and executes the instructions or program codes stored in the memory 801, the image highlight removal method provided by the disclosure embodiments can be implemented.

[0129] In another implementation manner, the memory 801 can also be integrated with the processor 802.

[0130] Bus 804, which can be an extended industry standard architecture (EISA) bus, a peripheral component interconnect (PCI) bus, or another type of bus, connects the components in computer system 800. Bus 804 can be implemented with one or more buses, and can be implemented as a mixture of buses having different bus architectures.

[0131] Some embodiments of the present disclosure provide a computer readable storage medium (for example, a non-transitory computer readable storage medium) having stored computer program instructions, which, when executed on a computer, cause the computer to perform the high-light removal method of an image according to any one of the above embodiments.

[0132] Exemplarily, the above computer readable storage medium can include, but is not limited to, a magnetic storage device (for example, a hard disk, a floppy disk, or a magnetic tape, etc.), an optical disc (for example, a compact disk (CD), a digital versatile disk (DVD), etc.), a smart card and a flash memory device (for example, an erasable programmable read-only memory (EPROM), a card, a stick or a key drive, etc.). The various computer readable storage media described in the present disclosure can represent one or more devices and / or other machine readable storage media for storing information. The term "machine readable storage medium" can include, but is not limited to, a wireless channel and various other media capable of storing, containing, and / or carrying instructions and / or data.

[0133] The embodiments of the present disclosure provide a computer program product containing instructions, which, when executed on a computer, cause the computer to perform the high-light removal method of an image according to any one of the above embodiments.

[0134] The above description is merely exemplary of the present disclosure, but the protection scope of the present disclosure is not limited thereto. Any change or substitution within the technical scope disclosed in the present disclosure should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be subject to the protection scope of the claims.

Claims

1. A method of highlight removal of an image, wherein, The method comprises: high light detection is performed on the high light image to obtain a predicted high light mask image, the predicted high light mask image being used to indicate image features of a high light region in the high light image; high light removal is performed on the high light image to obtain a high light removed image; the predicted high light mask image and the high light removed image are fused to obtain a diffuse reflection image corresponding to the high light image.

2. The method of claim 1, wherein, The method further comprises: a plurality of first feature maps of the high light image at different scales are obtained; a second feature map and a plurality of first high light mask images are determined according to the plurality of first feature maps, the second feature map being fused from the plurality of first feature maps, each first high light mask image corresponding to a first feature map, the first high light mask image being used to indicate image features of a high light region in the corresponding first feature map; high light detection is performed on the high light image to obtain the predicted high light mask image, comprising: high light detection is performed on the second feature map to determine a second high light mask image, the second high light mask image being used to indicate image features of a high light region in the second feature map; the second high light mask image and the plurality of first high light mask images are fused to determine the predicted high light mask image.

3. The method of claim 2, wherein, High light removal is performed on the high light image to obtain the high light removed image, comprising: high light removal is performed on the second feature map to obtain the high light removed image.

4. The method of claim 1, wherein, High light detection is performed on the high light image to obtain the predicted high light mask image, comprising: high light detection is performed on the high light image based on a high light detection module in a high light processing model to obtain the predicted high light mask image; High light removal is performed on the high light image to obtain the high light removed image, comprising: high light removal is performed on the high light image based on a high light removal module in the high light processing model to obtain the high light removed image; The predicted high light mask image and the high light removed image are fused to obtain the diffuse reflection image corresponding to the high light image, comprising: The predicted high light mask image and the high light removed image are fused based on a fusion module in the high light processing model to obtain the diffuse reflection image corresponding to the high light image.

5. The method of claim 4, wherein, The high light processing model is trained in the following manner: An initial high light processing model is trained based on a training set until a loss function of the initial high light processing model converges, to obtain the high light processing model, the loss function of the high light processing model comprising a mask loss corresponding to the high light detection module and an image loss corresponding to the high light removal module.

6. The method of claim 5, wherein, The mask loss is used to indicate an error between a predicted high light mask and a real high light mask, and the image loss is used to indicate a similarity index and a smoothing loss between a high light removed image and an original image.

7. The method of claim 5, wherein, The training set comprises: a first image of a real scene, a second image obtained after high light removal of the first image, a third image used to indicate a high light intensity of the first image, and a fourth image used to indicate a high light region in the first image.

8. An electronic device, comprising: The electronic device includes a memory and a processor; the memory and the processor are coupled; the memory is configured to store instructions executable by the processor; the processor executes the instructions to perform the method according to any one of claims 1-7.

9. A computer readable storage medium, wherein, The computer readable storage medium stores computer instructions, and when the computer instructions run on a computer, the computer executes the method according to any one of claims 1-7.

10. A computer program product, wherein, The computer program product includes computer program instructions, and when the computer program instructions are executed, the method according to any one of claims 1-7 is implemented.

Images