Image processing method and device

Abstract

The present application provides an image processing method and device, the device comprising: a first reflection elimination module, used to perform reflection elimination processing on an input image to obtain a first image; a reflection judgment module, used to obtain a judgment result of whether the input image has reflection according to the first image and the input image; a second reflection elimination module, used to obtain an output image according to the judgment result of the reflection judgment module. The image processing device provided in the embodiment of the present application can improve the reflection elimination effect of an image.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing method and device. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making. Research in the field of artificial intelligence includes robotics, natural language processing, computer vision, decision-making and reasoning, human-computer interaction, recommendation and search, basic AI theory, etc.

[0003] With the rapid development of artificial intelligence technology, neural networks (for example, deep neural networks) have made great achievements in the processing and analysis of various media signals such as images, videos, and voices in recent years. In daily photo shooting, there may be reflections in the images taken. In order to meet the user's demand for intelligent optimization of photos, AI algorithms can be used to remove reflections from images to reduce the impact of reflections on the picture effect.

[0004] Existing reflection removal technology is mainly implemented through generative adversarial networks (GAN). When using this technology, the original image with reflection is input into the trained reflection removal generative network, and the network inference generates a reflection-free image. However, the GAN neural network algorithm only has reflection removal means, but no reflection recognition means. After the reflection removal of the reflection-free image, the image quality will deteriorate. Summary of the invention

[0005] The present application provides an image processing method and device, which can improve the reflection elimination effect of an image.

[0006] In a first aspect, an image processing device is provided, comprising: a first reflection elimination module, used to perform reflection elimination processing on an input image to obtain a first image; a reflection judgment module, used to obtain a judgment result of whether the input image has reflection based on the first image and the input image; and a second reflection elimination module, used to obtain an output image based on the judgment result.

[0007] In the above technical solution, the image processing device can determine whether the input image has reflection based on the result of preliminary reflection elimination, and then perform reflection elimination processing on the original image that is judged to have reflection, thereby realizing automatic recognition and reflection elimination of reflective images.

[0008] Optionally, the device may further include a preprocessing module, configured to preprocess the original image to obtain an input image, wherein the size of the original image is larger than the size of the input image.

[0009] Optionally, when the size of the original image meets the input size requirement of the first reflection elimination module, the original image can be used as the input image.

[0010] In the above technical solution, the original image is preprocessed to realize the reflection recognition and elimination of the original image with a larger size.

[0011] In one implementation, the second reflection elimination module is specifically configured to obtain the original image as the output image when the judgment result is that the input image has no reflection.

[0012] When the judgment result is that the input image has no reflection, the original image can be directly output without performing any unnecessary processing on the original image, thereby avoiding the situation where the image quality is deteriorated after the reflection of the reflection-free image is eliminated.

[0013] In one implementation, when the judgment result is that the input image has reflection and the input image is the original image, the second reflection elimination module is specifically used to generate a reflection layer thumbnail according to the first image and the input image, and obtain the output image according to the reflection layer thumbnail and the original image.

[0014] In one implementation, when the judgment result is that the input image has reflection, and the input image is an image preprocessed from the original image, the second reflection elimination module is specifically used to generate a reflection layer thumbnail based on the first image and the input image, upsample the reflection layer thumbnail to obtain a reflection layer, and obtain an output image based on the reflection layer and the original image.

[0015] In the above technical solution, the reflective layer is sampled to obtain a reflective layer, and then the output image is obtained based on the original image and the reflective layer. Finally, an output image with the same size and resolution as the original image and without reflection can be obtained, thereby improving the quality of the output image.

[0016] Optionally, the reflection judgment module includes a reflection judgment model, which is a neural network with the first image and the input image as input, and the output of the reflection judgment model represents the confidence that the input image has reflection, or the output is a judgment result.

[0017] In one implementation, the reflection judgment model is a neural network based on a Markov discriminator and is trained based on a first sample set, wherein the first sample set includes: original image samples, image samples after the original image samples have been reflection eliminated, and labels indicating whether the original image samples have reflections.

[0018] Optionally, the reflective layer judgment module includes a reflective layer intensity judgment unit, which is used to generate a reflective layer thumbnail based on the input image and the first image; the reflective layer intensity judgment unit is also used to calculate the grayscale values ​​of multiple pixel points included in the reflective layer thumbnail, and determine the judgment result based on the grayscale values ​​of the multiple pixel points.

[0019] In one implementation, the light-reflecting layer intensity judgment unit determines the judgment result according to weighted results of at least one or more of the following information: the maximum value of the grayscale values ​​of multiple pixels, the intensity of at least one pixel whose grayscale value is greater than a first threshold, and the area of ​​at least one pixel whose grayscale value is greater than a second threshold; wherein the intensity of at least one pixel whose grayscale value is greater than the first threshold is the sum of the grayscale values ​​of the pixels whose grayscale values ​​are greater than the first threshold, or the intensity of at least one pixel whose grayscale value is greater than the first threshold is the average value of the grayscale values ​​of the pixels whose grayscale values ​​are greater than the first threshold; and the area of ​​at least one pixel whose grayscale value is greater than the second threshold is the number of pixels whose grayscale value is greater than the second threshold.

[0020] Optionally, the reflection judgment module includes a reflection judgment model, a reflection layer intensity judgment unit and a joint judgment unit, the reflection judgment model is a neural network with the first image and the input image as input, and the output of the reflection judgment model is a first judgment value indicating whether the input image has reflection; the reflection layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and outputs a second judgment value indicating whether the input image has reflection based on the gray value; the joint judgment unit determines the judgment result based on the first judgment value and the second judgment value.

[0021] In the above technical solution, it is possible to determine whether the original image has reflections through a variety of models or modules, thereby improving the accuracy of the determination result.

[0022] In a second aspect, an image processing method is provided, including: performing reflection elimination processing on an input image to obtain a first image; judging whether the input image has reflection based on the input image and the first image; and obtaining an output image based on whether the input image has reflection.

[0023] In the above technical solution, the image processing device can determine whether the input image has reflection based on the result of preliminary reflection elimination, and then perform reflection elimination processing on the original image that is judged to have reflection, thereby realizing automatic recognition and reflection elimination of reflective images.

[0024] In one implementation, the original image may be preprocessed to obtain an input image, where the size of the original image is larger than the size of the input image.

[0025] In the above technical solution, the original image is preprocessed to realize the reflection recognition and elimination of the original image with a larger size.

[0026] Optionally, according to whether the input image has reflection, obtaining the output image includes: when the input image has no reflection, the output image is the original image.

[0027] When the judgment result is that the input image has no reflection, the original image can be directly output without performing any unnecessary processing on the original image, thereby avoiding the situation where the image quality is deteriorated after the reflection of the reflection-free image is eliminated.

[0028] Optionally, according to whether the input image has reflection, obtaining the output image includes: when the input image has reflection and the input image is an original image, the method also includes: generating a reflective layer thumbnail according to the input image and the first image; and obtaining the output image according to the reflective layer thumbnail and the original image.

[0029] Optionally, based on whether the input image has reflection, obtaining the output image includes: when the input image has reflection, and the input image is an image preprocessed from the original image, the method also includes: generating a reflective layer thumbnail based on the input image and the first image; upsampling the reflective layer thumbnail to obtain a reflective layer; and obtaining the output image based on the reflective layer and the original image.

[0030] In the above technical solution, the reflective layer is sampled to obtain a reflective layer, and then the output image is obtained based on the original image and the reflective layer. Finally, an output image with the same size and resolution as the original image and without reflection can be obtained, thereby improving the quality of the output image.

[0031] Optionally, the above-mentioned judging whether the input image has reflection based on the input image and the first image includes: judging whether the input image has reflection based on a reflection judgment model; the reflection judgment model is a neural network with the first image and the input image as input, and the output of the reflection judgment model represents the confidence that the input image has reflection, or the output is the judgment result of whether the input image has reflection.

[0032] In one implementation, the reflection judgment model is a neural network based on a Markov discriminator and is trained based on a first sample set, wherein the first sample set includes: original image samples, image samples after the original image samples have been reflection eliminated, and labels indicating whether the original image samples have reflections.

[0033] Optionally, the above-mentioned judging whether the input image has reflection based on the input image and the first image includes: judging whether the input image has reflection based on a reflecting layer intensity judging unit; the reflecting layer intensity judging unit is used to generate a reflecting layer thumbnail based on the input image and the first image; the reflecting layer intensity judging unit is also used to calculate the grayscale values ​​of multiple pixel points included in the reflecting layer thumbnail, and determine the judgment result based on the grayscale values ​​of the multiple pixel points.

[0034] In one implementation, the reflective layer judgment module determines the judgment result according to weighted results of at least one or more of the following information: the maximum value of the grayscale values ​​of multiple pixels, the intensity of at least one pixel whose grayscale value is greater than a first threshold, and the area of ​​at least one pixel whose grayscale value is greater than a second threshold; wherein the intensity of at least one pixel whose grayscale value is greater than the first threshold is the sum of the grayscale values ​​of the pixels whose grayscale value is greater than the first threshold, or the intensity of at least one pixel whose grayscale value is greater than the first threshold is the average value of the grayscale values ​​of the pixels whose grayscale value is greater than the first threshold; and the area of ​​at least one pixel whose grayscale value is greater than the second threshold is the number of pixels whose grayscale value is greater than the second threshold.

[0035] Optionally, the above-mentioned judging whether the input image has reflection based on the input image and the first image includes: judging whether the input image has reflection based on a reflection judgment model, a reflection layer intensity judgment unit and a joint judgment unit; the reflection judgment model is a neural network with the first image and the input image as input, and the output of the reflection judgment model is a first judgment value indicating whether the input image has reflection; the reflection layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and outputs a second judgment value indicating whether the input image has reflection based on the gray value; the joint judgment unit determines the judgment result based on the first judgment value and the second judgment value.

[0036] In the above technical solution, it is possible to determine whether the original image has reflections through a variety of models or modules, thereby improving the accuracy of the determination result.

[0037] According to a third aspect, a computer-readable medium is provided, wherein the computer-readable medium stores a program code for execution by a device, wherein the program code includes a method for executing the image processing method in the second aspect or any one of the implementations of the second aspect.

[0038] In a fourth aspect, a computer program product is provided, comprising: a computer program code, which, when executed on a computer, enables the computer to execute the image processing method in the second aspect or any one of the implementations of the second aspect.

[0039] In a fifth aspect, a chip is provided, comprising a processor and a data interface, wherein the processor reads instructions stored in a memory through the data interface to execute the image processing method in the above-mentioned second aspect or any one of the implementations of the second aspect.

[0040] Optionally, as an implementation method, the chip may also include a memory, in which instructions are stored, and the processor is used to execute the instructions stored in the memory. When the instructions are executed, the processor is used to execute the image processing method in the second aspect or any one of the implementation methods of the second aspect.

[0041] In a sixth aspect, a device is provided, comprising: a processor and a memory, the memory being used to store the computer program code, and when the computer program code runs on the processor, the device executes the image processing method in the second aspect or any one of the implementations of the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] Figure 1 It is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application;

[0043] Figure 2 It is a schematic diagram of an application scenario provided by an embodiment of the present application;

[0044] Figure 3 is a schematic diagram of another application scenario provided by an embodiment of the present application;

[0045] Figure 4 It is a schematic diagram of the system architecture provided by the embodiment of the present application;

[0046] Figure 5 It is a schematic diagram of the chip hardware structure provided in the embodiment of the present application;

[0047] Figure 6 is a schematic diagram of the software architecture of the reflection detection and removal system provided in an embodiment of the present application;

[0048] Figure 7 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0049] Figure 8 It is a structural schematic diagram of a reflection judgment model provided in an embodiment of the present application;

[0050] Fig. 9 is a flowchart of another image processing method provided in an embodiment of the present application;

[0051] Fig.10 It is a structural schematic diagram of a reflection intensity judgment model provided in an embodiment of the present application;

[0052] Fig.11 is a flowchart of another image processing method provided in an embodiment of the present application;

[0053] Fig.12 is a schematic diagram of the hardware structure of an image processing device provided in an embodiment of the present application;

[0054] Fig.13 is a schematic diagram of the hardware structure of another image processing device provided in an embodiment of the present application;

[0055] Fig.14 It is a schematic diagram of the hardware structure of another image processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0056] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application. It can be understood that the described embodiments are only part of the embodiments of the present application, rather than all of the embodiments.

[0057] Figure 1 It is a schematic diagram of an application scenario of the image processing method provided in an embodiment of the present application.

[0058] like Figure 1 As shown, the image processing method implemented in the present application can be applied to smart terminals. For example, the original image captured by the camera in the smart device can be processed to obtain a high-definition large image after the original image is removed from the reflection in the picture, thereby improving the quality of image display. The above-mentioned smart terminal can be mobile or fixed. For example, the smart terminal can be a mobile phone with image processing function, a tablet personal computer (tablet personal computer, TPC), a media player, a smart TV, a laptop computer (laptop computer, LC), a personal digital assistant (personal digital assistant, PDA), a personal computer (personal computer, PC), a camera, a camcorder, a smart watch, augmented reality (augmented reality, AR) / virtual reality (virtual reality, VR), a wearable device (wearable device, WD) or an autonomous driving vehicle, etc., and the embodiments of the present application are not limited to this.

[0059] The following is an example of a specific application scenario of the embodiment of the present application.

[0060] Application scenario 1: Smart terminal photography

[0061] In one embodiment, Figure 2As shown, the image processing method of the embodiment of the present application can be applied to the shooting of smart terminal devices (for example, mobile phones). When taking pictures of objects through the camera of a smart terminal, in scenes such as museums and shopping malls, there may be reflective areas in the pictures of glass and windows. Through the image method of the embodiment of the present application, that is, the method for removing reflections, an output image with high picture quality can be obtained by removing the reflective areas of the original image.

[0062] Exemplarily, when a smart terminal takes real-time photos using the image processing method of the embodiment of the present application, the original image obtained by taking the photos can be downsampled to convert the original image into a smaller image; the processed small image can be subjected to reflection elimination processing, and the original small image and the small image after reflection elimination can be compared to determine whether there is a reflective area in the original image; if there is a reflective area in the original image, the reflective layer small image obtained by subtracting the original small image from the small image after reflection elimination can be upsampled, and the upsampled reflective layer small image can be stripped from the original image to obtain a high-quality image with reflection eliminated, and the output image can be displayed on the screen of the smart terminal.

[0063] Exemplarily, the image processing method of the embodiment of the present application can be used to downsample the original image obtained by taking real-time photos on a smart terminal, and process the original image into a smaller image; the processed small image is subjected to reflection elimination processing, and the original small image and the small image after reflection elimination are compared to determine whether there is a reflective area in the original image; if there is a reflective area in the original image, the reflective layer small image obtained by subtracting the original small image from the small image after reflection elimination is upsampled, and the upsampled reflective layer small image is stripped from the original image to obtain a high-quality image with reflection eliminated, and the output image is saved in the photo album of the smart terminal.

[0064] Application scenario 2: Autonomous driving

[0065] In one embodiment, Figure 3 As shown, the image processing method of the embodiment of the present application can be applied to the field of autonomous driving. For example, it can be applied to the navigation system of an autonomous driving vehicle. Through the image processing method in the present application, the autonomous driving vehicle can process the image with glass reflective areas in the acquired picture during the navigation process of road driving, eliminate the reflective areas in the image, and obtain a high-quality non-reflective image, thereby achieving the safety of the autonomous driving vehicle.

[0066] Illustratively, the present application provides an image processing method, including: downsampling the original image obtained by shooting to process the original image into a smaller image; performing reflection elimination processing on the processed small image, comparing the original small image and the small image after reflection elimination to determine whether there is a reflective area in the original image; if there is a reflective area in the original image, upsampling the reflective layer small image obtained by subtracting the original small image from the small image after reflection elimination, and stripping the upsampled reflective layer small image from the original image to obtain a high-quality road image with reflection eliminated.

[0067] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and concepts of the neural networks that may be involved in the embodiments of the present application are first introduced below.

[0068] (1) Neural Network

[0069] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0070]

[0071] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer, and the activation function can be a sigmoid function. A neural network is a network formed by connecting multiple single neural units mentioned above, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0072] (2) Deep Neural Networks

[0073] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural network inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally speaking, the first layer is the input layer, the last layer is the output layer, and the layers in between are all hidden layers. The layers are fully connected, that is, any neuron in the i-th layer must be connected to any neuron in the i+1-th layer.

[0074] Although DNN looks complicated, the work of each layer is not complicated. In simple terms, it can be expressed as the following linear relationship: in, is the input vector, is the output vector, is the offset vector, W is the weight matrix (also called coefficient), and α() is the activation function. Each layer is just an input vector After such a simple operation, the output vector Since DNN has many layers, the coefficient W and the offset vector The number of these parameters is also relatively large. The definitions of these parameters in DNN are as follows: Take the coefficient W as an example: Assume that in a three-layer DNN, the linear coefficient from the 4th neuron in the second layer to the 2nd neuron in the third layer is defined as The superscript 3 represents the layer number of the coefficient W, while the subscripts correspond to the third layer index 2 of the output and the second layer index 4 of the input.

[0075] In summary, the coefficients from the kth neuron in the L-1th layer to the jth neuron in the Lth layer are defined as

[0076] It should be noted that the input layer does not have a W parameter. In a deep neural network, more hidden layers allow the network to better describe complex situations in the real world. Theoretically, the more parameters a model has, the higher its complexity and the greater its "capacity", which means it can complete more complex learning tasks. Training a deep neural network is the process of learning the weight matrix, and its ultimate goal is to obtain the weight matrix of all layers of the trained deep neural network (the weight matrix formed by many layers of vector W).

[0077] (3) Convolutional Neural Network

[0078] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. Convolutional neural network contains a feature extractor consisting of a convolution layer and a subsampling layer, which can be regarded as a filter. Convolutional layer refers to the neuron layer in the convolutional neural network that performs convolution processing on the input signal. In the convolutional layer of the convolutional neural network, a neuron can only be connected to some neurons in the adjacent layers. A convolutional layer usually contains several feature planes, each of which can be composed of some rectangularly arranged neural units. The neural units in the same feature plane share weights, and the shared weights here are convolution kernels. Shared weights can be understood as the way to extract image information is independent of position. The convolution kernel can be initialized in the form of a matrix of random size, and the convolution kernel can obtain reasonable weights through learning during the training process of the convolutional neural network. In addition, the direct benefit of shared weights is to reduce the connection between the layers of the convolutional neural network, while reducing the risk of overfitting.

[0079] (4) Generative Adversarial Networks

[0080] Generative Adversarial Networks (GAN) is a deep learning model. The model includes at least two modules: one is the generative model, and the other is the discriminative model. These two modules learn from each other through competition, thereby producing better output. Both the generative model and the discriminative model can be neural networks, specifically deep neural networks or convolutional neural networks. The basic principle of GAN is as follows: Taking the GAN that generates images as an example, suppose there are two networks, G (Generator) and D (Discriminator), where G is a network that generates images. It receives a random noise z and generates images through this noise, denoted as G(z); D is a discriminative network that is used to determine whether an image is "real". Its input parameter is x, x represents an image, and the output D(x) represents the probability that x is a real image. If it is 1, it means that it is 100% a real image, and if it is 0, it means that it cannot be a real image. In the process of training the generative adversarial network, the goal of the generative network G is to generate realistic images as much as possible to deceive the discriminative network D, while the goal of the discriminative network D is to distinguish the images generated by G from the real images as much as possible. In this way, G and D constitute a dynamic "game" process, which is also the "adversary" in the "generative adversarial network". The final result of the game is that, under ideal conditions, G can generate an image G(z) that is "real" enough, while D has difficulty determining whether the image generated by G is real, that is, D(G(z)) = 0.5. In this way, an excellent generative model G is obtained, which can be used to generate images.

[0081] (5) Loss Function

[0082] In the process of training a deep neural network, because we hope that the output of the deep neural network is as close as possible to the value we really want to predict, we can compare the predicted value of the current network with the target value we really want, and then update the weight vector of each layer of the neural network according to the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the deep neural network). For example, if the predicted value of the network is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the deep neural network can predict the target value we really want or a value very close to the target value we really want. Therefore, it is necessary to pre-define "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the deep neural network becomes a process of minimizing this loss as much as possible.

[0083] Figure 4 This is a system architecture 100 provided in an embodiment of the present application.

[0084] exist Figure 4 In the embodiment, the data acquisition device 160 is used to acquire training data. For the image processing method of the present application, the image processing model can be further trained by the training data, that is, the training data acquired by the data acquisition device 160 can be a training image.

[0085] Exemplarily, in the embodiment of the present application, the training data for training the image processing model may include the original image and the sample image corresponding to the original image. The training data may also include a label, such as a label indicating whether the original image has reflection.

[0086] For example, the original image may refer to an image with a reflective area; the sample image may refer to an output image after image reflection elimination processing, for example, it may refer to an image with a reflective area partially or completely eliminated relative to the original image.

[0087] After the training data is collected, the data collection device 160 stores the training data in the database 130, and the training device 120 obtains the target model / rule 101 (i.e., the image processing model in the embodiment of the present application) through training based on the training data maintained in the database 130. The training device 120 inputs the training data into the image processing model until the difference between the predicted image and the sample image output by the training image processing model meets a preset condition (for example, the difference between the predicted image and the sample image is less than a certain threshold, or the difference between the predicted image and the sample image remains unchanged or no longer decreases), thereby completing the training of the target model / rule 101.

[0088] Exemplarily, the image processing model used to execute the image processing method in the embodiment of the present application can achieve end-to-end training. For example, the image processing model can achieve end-to-end training by inputting an original image and an image after eliminating the reflective area corresponding to the input original image (for example, a true value image).

[0089] In the embodiment provided in the present application, the target model / rule 101 is obtained by training an image processing model, that is, the image processing model may be a model for judging and eliminating reflections. It should be noted that in actual applications, the training data maintained in the database 130 may not necessarily come from the data acquisition device 160, but may also be received from other devices.

[0090] It should also be noted that the training device 120 does not necessarily train the target model / rule 101 entirely based on the training data maintained by the database 130, and it is also possible to obtain training data from the cloud or other places for model training, and the above description should not be used as a limitation on the embodiments of the present application. It should also be noted that at least part of the training data maintained in the database 130 can also be used in the process of executing the device 110 to process the processing to be processed.

[0091] The target model / rule 101 trained by the training device 120 can be applied to different systems or devices, such as Figure 4 The execution device 110 shown may be a terminal, such as a mobile phone terminal, a tablet computer, a laptop computer, AR / VR, a vehicle-mounted terminal, etc., or may be a server or a cloud, etc.

[0092] exist Figure 4 In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. A user may input data to the I / O interface 112 through a client device 140. The input data may include an image to be processed input by the client device in the embodiment of the present application.

[0093] Preprocessing module 113 and preprocessing module 114 are used to perform preprocessing according to the input data (such as the image to be processed) received by I / O interface 112. In the embodiment of the present application, preprocessing module 113 and preprocessing module 114 may be omitted (or only one of the preprocessing modules may be present), and the computing module 111 may be used directly to process the input data.

[0094] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculations and other related processing, the execution device 110 can call the data, code, etc. in the data storage system 150 for corresponding processing, and can also store the data, instructions, etc. obtained from the corresponding processing into the data storage system 150.

[0095] Finally, the I / O interface 112 returns the processing result, such as the image of the de-glare area of ​​the original image to be processed, to the client device 140, so as to provide it to the user.

[0096] It is worth noting that the training device 120 can generate corresponding target models / rules 101 based on different training data for different goals or different tasks. The corresponding target models / rules 101 can be used to achieve the above goals or complete the above tasks, thereby providing users with the desired results.

[0097] exist Figure 4 In the case shown in FIG. 1 , in one case, the user can manually provide input data, and the manual provision can be operated through the interface provided by the I / O interface 112 .

[0098] In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 needs to obtain the user's authorization to automatically send the input data, the user can set the corresponding permissions in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data input to the I / O interface 112 and the output results of the I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible to collect without going through the client device 140, but the I / O interface 112 directly stores the input data input to the I / O interface 112 and the output results of the I / O interface 112 as new sample data in the database 130.

[0099] It is worth noting that Figure 4This is only a schematic diagram of a system architecture provided by an embodiment of the present application, and the positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 4 In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110.

[0100] like Figure 4 As shown, the target model / rule 101 is obtained by training according to the training device 120. The target model / rule 101 can be an image processing model in the embodiment of the present application. Specifically, the image processing model provided in the embodiment of the present application can be a deep neural network, a convolutional neural network, or a deep convolutional neural network, etc.

[0101] Figure 5 The hardware structure of a chip provided in an embodiment of the present application includes a neural-network processing unit 300 (NPU). The chip can be arranged in Figure 4 The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 4 The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule 101.

[0102] The NPU 300 is mounted on the main central processing unit (CPU) as a coprocessor, and the main CPU assigns tasks. The core part of the NPU 300 is the operation circuit 303, and the controller 304 controls the operation circuit 303 to extract data from the memory (weight memory or input memory) and perform operations.

[0103] In some implementations, the operation circuit 303 includes multiple processing units (PEs) inside. In some implementations, the operation circuit 303 is a two-dimensional systolic array; the operation circuit 303 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 303 is a general-purpose matrix processor.

[0104] For example, assume there is an input matrix A, a weight matrix B, and an output matrix C; the operation circuit 303 takes the corresponding data of matrix B from the weight memory 302 and caches it on each PE in the operation circuit 303; the operation circuit 303 takes the matrix A data from the input memory 301 and performs matrix operations with the matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 308 (accumulator).

[0105] The vector calculation unit 307 can further process the output of the operation circuit 303, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector calculation unit 307 can be used for network calculations of non-convolutional / non-FC layers in a neural network, such as pooling, batch normalization, local response normalization, etc.

[0106] In some implementations, the vector calculation unit 307 can store the processed output vector to the unified memory 306. For example, the vector calculation unit 307 can apply a nonlinear function to the output of the operation circuit 303, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 307 generates a normalized value, a merged value, or both.

[0107] In some implementations, the vector of processed outputs can be used as activation input to the computational circuit 303, for example for use in subsequent layers in a neural network.

[0108] The unified memory 306 is used to store input data and output data. The weight data is directly stored in the input data in the external memory into the input memory 301 and / or the unified memory 306 through the storage unit access controller 305 (direct memory access controller, DMAC), the weight data in the external memory is stored in the weight memory 302, and the data in the unified memory 306 is stored in the external memory.

[0109] The bus interface unit 310 (BIU) is used to implement the interaction between the main CPU, DMAC and instruction fetch memory 309 through the bus.

[0110] The instruction fetch buffer 309 connected to the controller 304 is used to store instructions used by the controller 304; the controller 304 is used to call the instructions cached in the instruction fetch buffer 309 to realize the working process of controlling the computing accelerator.

[0111] Generally, the unified memory 306, the input memory 301, the weight memory 302 and the instruction fetch memory 309 are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDR SDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0112] Introduced above Figure 4 The execution device 110 in the embodiment of the present application can execute each step of the image processing method. Figure 5 The chip shown can also be used to execute each step of the image processing method of the embodiment of the present application.

[0113] The image processing method provided in the embodiment of the present application can be deployed on the computing nodes of related equipment, and the reflection detection and removal method in the embodiment of the present application can be effectively implemented through software algorithms. Figure 6 The system architecture of the embodiment of the present application is described in detail.

[0114] like Figure 6 As shown, the image processing method of the embodiment of the present application can be implemented by an image reflection detection and removal system software architecture 500, wherein the reflection detection and removal system software architecture may include: an image pre-processing module 510, a network reasoning module 520, an image post-processing module 530 and a software processing module 540. Each module is described in detail below.

[0115] The image preprocessing module 510 is used to preprocess the original image, for example, to process the original high-definition large image into a small image of a fixed size. The input of the image preprocessing module 510 is the original image, and the output is a small image of a fixed size, which can be used as the input image of the network reasoning module 520, and the size of the input image is the input size of the neural network model of the network reasoning module 520. Since the original high-definition large image often has a higher resolution, such as 4K resolution, which exceeds the input size of the image reflection detection and removal neural network model, it is necessary to preprocess the original image so that the processed image can adapt to the input of the image reflection detection and removal neural network model. Exemplarily, the original image can be preprocessed by downsampling, mean reduction, or normalization. Exemplarily, the input image obtained after preprocessing is a small image of 896×896 pixels. It should be understood that the size of the input image obtained after preprocessing can also be other sizes that meet the input requirements of the image reflection detection and removal neural network model, and the embodiment of the present application is not limited to this.

[0116] The image preprocessing module 510 is an optional module. In some embodiments, the size of the original image meets the input size requirement of the neural network model of the network reasoning module 520. In this case, there is no need to preprocess the original image, and the input image to the network reasoning module 520 is the original image.

[0117] The network inference module 520 includes a reflection elimination model, which is used to perform preliminary reflection elimination processing on the input image. The reflection elimination model inputs an original image, or an input image obtained by image preprocessing the original image, and outputs an image after reflection elimination of the input image (for example, the first image described below).

[0118] In some embodiments, the network reasoning module 520 may further include a reflection judgment model for judging whether the input image has reflection. The input of the reflection judgment module is the first image and the input image, and the output is a preliminary reflection judgment result. For example, the judgment result is: the input image has reflection; or the input image has no reflection.

[0119] Some reflection elimination models have restrictions on the size of the input image. In other words, such reflection elimination models can only process images with lower resolutions, such as 896×896 pixels. In this case, the original image of the high-definition large image (e.g., 4K resolution) can be preprocessed into a small image and input into the reflection model by the image preprocessing module 510. This processing scheme of the present application enables the reflection problem of high-definition large images to be processed even when a low-resolution reflection elimination model is used.

[0120] The image post-processing module 530 is used to determine whether the input image has reflections, process the original image according to the determination result, and output an image with the reflections eliminated.

[0121] In some embodiments, the network reasoning module 520 includes a reflection judgment model, and the image post-processing module 530 is used to determine whether the input image has reflection according to the judgment result of the reflection judgment model. When it is determined that the input image has reflection, the original image is processed and an image with the reflection eliminated is output; otherwise, the original image is output.

[0122] In some embodiments, the image post-processing module 530 includes a reflective layer strength judgment unit, the input of which is the first image and the input image, and the output is a judgment value for judging whether the input image has reflection. The image post-processing module 530 is also used to determine whether the input image has reflection according to the judgment result of the reflective layer strength judgment unit. When it is determined that the input image has reflection, the original image is processed and an image with the reflection eliminated is output; otherwise, the original image is output.

[0123] In some embodiments, the network reasoning module 520 includes a reflection judgment model, and the image post-processing module 530 includes a reflection layer strength judgment unit and a joint judgment unit. The image post-processing module 530 is used to judge whether the input image has reflection according to the judgment result of the reflection judgment model and the judgment result of the reflection layer strength judgment unit. When it is determined that the input image has reflection, the original image is processed and an image with the reflection eliminated is output; otherwise, the original image is output.

[0124] In some embodiments, the input image is an original image. In this case, when the result of the network inference module 520 is that the input image has reflections, the image post-processing module 530 is used to subtract the first image from the input image to obtain a small image of the reflection layer, and remove the small image of the reflection layer from the original image to obtain an output image. The output image is an image obtained after the reflections are eliminated from the original image.

[0125] When the result of the network reasoning module 520 is that the input image has no reflection, the image post-processing module 530 does not need to process the original image. In this case, the original image is output as the output image of the image post-processing module 530.

[0126] In some other embodiments, the input image is an image obtained after the original image has been preprocessed as described above.

[0127] In this case, when the result of the network inference module 520 is that the input image has reflections, the image post-processing module 530 is used to subtract the first image from the input image to obtain a small image of the reflection layer. The image post-processing module 530 is also used to upsample the small image of the reflection layer to obtain a reflection layer, wherein the size of the reflection layer is the size of the original image. The image post-processing module 530 strips the reflection layer from the original image to obtain an output image after the original image eliminates the reflections.

[0128] When the result of the network reasoning module 520 is that the input image has no reflection, the image post-processing module 530 does not need to process the original image. In this case, the output image is the original image.

[0129] It should be understood that the image determined by the network reasoning module 520 as non-reflective may be an image without any reflection, or an image with very low reflection components that does not affect the image quality (e.g., an image with a reflection confidence value below a certain threshold value obtained by reflection judgment). Performing traditional reflection elimination processing on such an image may reduce the image quality. The embodiment of the present application can directly use these original images as output images to improve the reflection elimination effect.

[0130] The software processing module 540 is used to receive the output image of the image post-processing module 530 and present the output image to the user. Specifically, the software processing module 540 mainly plays a scheduling role, controls the device to obtain images from the gallery or camera, controls the operation of the image pre-processing module 510, the network reasoning module 520, and the image post-processing module 530, and interacts with the user to recommend the reflection result and the non-reflection image to the user.

[0131] In some embodiments, the image pre-processing module 510, the image post-processing module 530 and the software processing module 540 may be composed of Figure 5 The main CPU of the chip shown in FIG. 5 is implemented; the network reasoning module 520 can be implemented by Figure 5 The neural network processor 300 of the chip shown is implemented.

[0132] The reflection elimination solution of the present application does not simply eliminate all images, but further determines whether there is reflection in the original image through the image with preliminary reflection elimination, and then provides the final output image according to the judgment result, which can improve the reflection processing effect of the image.

[0133] For example, an image that is judged to be non-reflective or has a low degree of reflection may not need to be processed to eliminate reflections, so as to avoid introducing more processing noise and reducing image quality.

[0134] On the other hand, if the reflection elimination model directly processes large high-definition images, the network complexity of the model and the power consumption of the device will increase significantly, which is not suitable for some small devices, such as mobile terminals. The reflection elimination model of the embodiment of the present application can maintain a relatively small network complexity, and through processing such as upsampling of the network inference module, the reflection elimination effect of large high-definition images (such as images taken by the camera of a mobile terminal) is still relatively good.

[0135] Combine the following Figures 7 to 11 The image processing method according to the embodiment of the present application is described in detail.

[0136] Figure 7 is a schematic diagram of an image processing method 700 according to an embodiment of the present application. The method may be Figure 6 The method can also be performed by a device capable of image processing, for example, Figure 4 The execution device 110 in the method 700 is executed. The steps in the method 700 are described in detail below.

[0137] S710, preprocessing the original image to obtain a first image that meets the input size of the reflection elimination model. Step S710 can be performed by Figure 6The image preprocessing module 510 in the software architecture shown is implemented. Specifically, the original image often has a larger size or a higher resolution, such as 4K resolution. The size of such an original image exceeds the input size of the reflection elimination model, so the original image needs to be preprocessed.

[0138] Exemplarily, the original image can be processed by downsampling to generate an input image from the high-definition original image. The input image can be a small image of a fixed size, and the size of the input image meets the input size requirements of the reflection removal model. For example, the size of the input image obtained after preprocessing is 896×896 pixels. It should be understood that the size of the input image obtained after preprocessing can also be other sizes that meet the input requirements of the reflection removal model, and the embodiments of the present application are not limited to this.

[0139] In some embodiments, the size of the original image meets the input size requirement of the reflection elimination model, so the image preprocessing operation in step S710 may not be performed. In this case, the original image may be directly used as the input image to perform the following steps.

[0140] S720, input the input image into the reflection elimination model to obtain a first image after reflection elimination. Step S720 can be performed by Figure 6 The network inference module 520 in the software architecture shown is implemented. In step 720, the reflection elimination model can be a generative network obtained by generative adversarial networks (GAN) or other neural networks, which is not limited in the embodiment of the present application.

[0141] S730, input the first image and the input image into the reflection judgment model to obtain a judgment result of the reflection judgment model (for example, the first judgment value described below). Step S730 can be performed by Figure 6 The network inference module 520 in the software architecture shown is implemented. In the embodiment of the present application, the reflection judgment neural network model can be a binary classification neural network model, which is used to judge whether the original image is a reflection image.

[0142] Exemplarily, in some embodiments, the reflection judgment model is a binary classification neural network based on a Markov discriminator. Figure 8 Schematic diagram of a reflection judgment model according to an embodiment of the present application.

[0143] like Figure 8As shown, the reflection judgment model is composed of multiple convolutional layers, and the convolutional layers can expand the receptive field of each neuron. The output of the Markov discriminator is a matrix, and each output in the matrix represents a receptive field of the input image, corresponding to a patch of the input image. The output of the last convolutional layer of the reflection judgment model is the prediction result of multiple patches. The input of the reflection judgment model is the first image and the input image, and the output is the first judgment value. For example, the first judgment value is that the input image has reflection, or the first judgment value is that the input image has no reflection.

[0144] In some embodiments, a fully connected layer may be added after the last convolutional layer to perform nonlinear mapping on the predicted results of multiple patches to obtain a reflection confidence coefficient. The reflection confidence coefficient may also be called reflection confidence, reflection coefficient, etc., and is used to indicate the possibility that the image has a reflection component, or the possibility that the image does not have a reflection component. In the following embodiments, the reflection confidence coefficient is used as an example to indicate the possibility that the image has a reflection component.

[0145] In some embodiments, the first judgment value can be determined according to the value of the reflection confidence coefficient. For example, when the value of the reflection confidence coefficient is greater than a specific threshold, the first judgment value is 1, indicating that the input image has reflection; otherwise, the first judgment value is 0, indicating that the input image has no reflection. Alternatively, when the value of the confidence coefficient is less than a specific threshold, the first judgment value is 1, indicating that the input image has reflection; otherwise, the first judgment value is 0, indicating that the input image has no reflection. The specific value of the first judgment value and the corresponding relationship between whether the image has reflection or not are not limited to the above examples, and various reasonable settings can be made as needed.

[0146] In some other embodiments, the value of the reflection confidence coefficient may be directly used as the first judgment value. In this case, the subsequent modules or units of the reflection judgment model further determine whether there is reflection in the image according to the first judgment value.

[0147] In some embodiments, the reflection judgment model can be trained in the following manner:

[0148] Pairs of original image samples and the results of removing reflections from the original image samples, as well as the label of whether the original image is a reflective image, are input into the reflection judgment model for training. For example, if the original image is a reflective image, the label is 1; if the original image is a non-reflective image, the label is 0. The reflection judgment model is essentially a binary classification network used to distinguish the authenticity of samples. Cross entropy can be used to judge the similarity of the distribution, and the loss function is:

[0149]

[0150] Where N is the number of samples;

[0151] x i represents the i-th pair of samples, i.e., the paired original image and the result of removing reflection from the image;

[0152] y i Represents x i The label of the positive class is 1 and the negative class is 0. For example, as described above, if the original image has reflection, the label can be 1, and if the original image has no reflection, the label can be 0;

[0153] D(x i ) represents the sample x i The predicted result is the probability of the above “positive class”.

[0154] When the value of the loss function is less than the threshold, the reflection judgment model training is considered complete.

[0155] S740, according to the first judgment value, remove the reflection of the original image. Step S740 can be performed by Figure 6 The image post-processing module 530 in the software architecture shown is implemented. In step S740, the input is the original image, the first image, the input image and the first judgment value obtained in step 730; the output is the image after the original image is de-reflected. Specifically, de-reflection processing of the original image according to the de-reflection judgment result (first judgment value) can be divided into the following cases:

[0156] Case 1: The input image has reflections and is the original image.

[0157] In this case, the input image and the first image may be subtracted to obtain a small image of the reflective layer. In some embodiments, the result of subtracting the first image and the input image may be filtered to obtain a small image of the reflective layer.

[0158] For example, the reflective layer thumbnail can be obtained by the following calculation method:

[0159] M blend -M low_trans =M low_ref

[0160] Among them, M blend is the input image, M low_trans is the first image, M low_ref This is a small picture of the reflective layer.

[0161] For example, the input image and the first image are both three-channel RGB images, and the sizes of the input image and the first image are both 896×896 pixels. Then M blend and M low_transThe matrix is ​​896×896×3. Each element of the matrix represents the brightness value of each pixel of the image in the three color channels of red (red, R), green (green, G), and blue (blue, B).

[0162] In this case, the input image is subtracted from the first image to obtain M low_ref It is also 896×896×3, where each element in the matrix represents the brightness value of each pixel of the reflective layer thumbnail in the RGB three color channels.

[0163] Since the size of the input image is the size of the original image, the reflective layer thumbnail is also the size of the original image. The reflective layer thumbnail is peeled off from the original image to obtain an output image, which is the image of the original image after removing the reflection.

[0164] For example, the image after removing the reflection from the original image can be obtained by the following calculation method:

[0165] M orig -M ref =M trans

[0166] Among them, M orig is the original image, M ref For the reflective layer, M trans The result after removing the reflective layer from the original image.

[0167] In the embodiment described in case 1, since the size of the input image is the size of the original image, the reflective layer thumbnail is also the size of the original image, so the reflective layer M ref With reflective layer thumbnail M low_ref In this embodiment, M orig and M ref Both are 896×896×3 matrices. Therefore, the original image is subtracted from the reflective layer to obtain the image M after removing the reflective layer. trans It is also a 896×896×3 matrix, where each element in the matrix represents the brightness value of each pixel in the original image after the reflection is eliminated in the three color channels of RGB.

[0168] Case 2: The input image has reflections and is a preprocessed version of the original image.

[0169] In this case, the input image and the first image may be subtracted to obtain a small image of the reflective layer. In some embodiments, the result of subtracting the first image and the input image may be filtered to obtain a small image of the reflective layer.

[0170] For example, the reflective layer thumbnail can be obtained by the following calculation method:

[0171] M blend -M low_trans =M low_ref

[0172] Among them, M blend is the input image, M low_trans is the first image, M low_ref This is a small picture of the reflective layer.

[0173] For example, the input image and the first image are both three-channel RGB images, and the sizes of the input image and the first image are both 896×896 pixels. Then M blend and M low_trans The matrices are all 896×896×3. Each element of the matrix represents the brightness value of each pixel in the RGB color channels.

[0174] In this case, the input image is subtracted from the first image to obtain M low_ref It is also 896×896×3, where each element in the matrix represents the brightness value of each pixel of the reflective layer thumbnail in the RGB three color channels.

[0175] Since the input image is an image obtained by preprocessing the original image, the size of the reflective layer thumbnail is smaller than the size of the original image. In this case, the reflective layer thumbnail can be upsampled to obtain a reflective layer, and the size of the reflective layer is the size of the original image. In some embodiments, the first reflective layer thumbnail can be upsampled by interpolation. The reflective layer is stripped from the original image to obtain an output image, which is the image of the original image after the reflection is eliminated.

[0176] For example, the image after removing the reflection from the original image can be obtained by the following calculation method:

[0177] M orig -M ref =M trans

[0178] Among them, M orig is the original image, M ref For the reflective layer, M trans The result after removing the reflective layer from the original image.

[0179] In the embodiment described in Case 2, the input image is an image obtained by preprocessing the original image, so the size of the reflective layer thumbnail is smaller than the size of the original image.

[0180] For example, the size of the original image is 4K (3000×4000 pixels), M origis a 3000×4000×3 matrix. In this case, the reflective layer image M low_ref By upsampling by difference, we can get a reflective layer M with the same size as the original image. ref , M ref It is also a 3000×4000×3 matrix, where each element in the matrix represents the brightness value of each pixel of the reflective layer in the three RGB color channels.

[0181] The original image M orig Subtract M from the reflective layer ref The original image obtained after removing the reflection is M trans It is also a 3000×4000×3 matrix, where each element in the matrix represents the brightness value of each pixel in the original image after removing the reflection in the three RGB color channels.

[0182] Case 3: The input image has no reflection.

[0183] In this case, since the input image has no reflection, there is no need to process the original image. Therefore, the output image is the original image.

[0184] According to the reflection layer intensity judgment model, it is possible to judge whether there is reflection in the image, so as to select a suitable reflection processing method according to the judgment result, thereby improving the reflection processing effect of the image.

[0185] The image processing method provided in the embodiment of the present application can, on the one hand, automatically identify whether the original image has reflections, and perform reflection elimination processing on the original image with reflections. On the other hand, when the original image size is large, the image processing method in the embodiment of the present application first downsamples the original image to adapt to the input scale of the neural network, then upsamples the small reflective layer image, and peels off the upsampled reflective layer from the original image, thereby eliminating reflections in high-resolution images.

[0186] In addition, the image processing method of the embodiment of the present application realizes the functions of identifying and eliminating reflections through two lightweight neural networks, namely a reflection judgment model and a reflection elimination model, thereby overcoming the problem that the reflection elimination network required for high-definition images is too large.

[0187] Fig. 9 is a schematic diagram of an image processing method 900 according to another embodiment of the present application. The method can be Figure 6 The method can also be performed by a device capable of image processing, for example, Figure 4 The execution device 110 in the method 900 is executed. The steps in the method 900 are described in detail below.

[0188] S910, preprocessing the original image to obtain an input image that meets the input size of the reflection elimination model. Step S910 can be performed by Figure 6 The image pre-processing module 510 in the software architecture shown is implemented.

[0189] S920, input the input image into the reflection elimination model to obtain a first image after the reflection elimination. Step S920 can be performed by Figure 6 The network reasoning module 520 is implemented in the software architecture shown.

[0190] In method 900, steps S910 to S920 are Figure 7 Steps S710 to S720 in the method 700 are similar and will not be described in detail.

[0191] S930: The first image and the input image are input into a reflective layer strength judgment unit to obtain a judgment result (eg, a second judgment value described below) of the reflective layer strength judgment unit. Step S930 may be performed by Figure 6 The image post-processing module 530 in the software architecture shown is implemented. In some embodiments, it can be determined whether the input image has reflections by the grayscale difference between the first image and the input image.

[0192] Fig.10 It is a schematic diagram of the judgment method of the reflective layer intensity judgment unit.

[0193] like Fig.10 As shown, the input of the light-reflecting layer intensity judgment unit is the first image and the input image, and the output is a second judgment value: the input image has light reflection, or the input image has no light reflection.

[0194] The light-reflecting layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and the gray value can represent the brightness of the image.

[0195] Exemplarily, the size of the input image and the first image is 896×896 pixels. As described in step S740 of the above method 700, the input image and the first image are subtracted to obtain a 896×896×3 matrix representing the reflective layer thumbnail. Each element of the matrix represents the brightness value of each pixel of the reflective layer thumbnail in the three RGB color channels.

[0196] In some embodiments, the gray value can be calculated according to the following formula:

[0197] Gray(i,j)=0.299×R(i,j)+0.587×G(i,j)+0.144×B(i,j)

[0198] Among them, Gray(i, j) represents the gray value of each pixel of the reflective layer thumbnail;

[0199] R(i, j) represents the absolute value of the red channel brightness of each pixel in the reflective layer image. R(i, j) = |M low_ref (i, j, 0)|;

[0200] G(i, j) represents the absolute value of the green channel brightness value of each pixel in the reflective layer thumbnail. G(i, j) = |M low_ref (i, j, 1)|;

[0201] B(i, j) represents the absolute value of the blue channel brightness value of each pixel in the reflective layer thumbnail. B(i, j) = |M low_ref (i, j, 2)|.

[0202] It should be understood that the embodiments of the present application can also calculate the grayscale value according to other methods, such as taking the average of the brightness values ​​of the three RGB color channels of each pixel as the grayscale value of each pixel, etc. The embodiments of the present application are not limited to this.

[0203] In some embodiments, the second judgment value can be determined based on the weighted results of at least one or more of the maximum grayscale value of each pixel in the reflective layer thumbnail, the intensity of the pixel whose grayscale value is greater than the threshold TH1, and the area of ​​the pixel whose grayscale value is greater than the threshold TH2.

[0204] Among them, the intensity of pixels whose grayscale values ​​are greater than the threshold TH1 can be the sum of the grayscale values ​​of pixels whose grayscale values ​​are greater than the threshold TH1, or it can be the average of the grayscale values ​​of pixels whose grayscale values ​​are greater than the threshold TH1, or it can be other numerical values ​​used to characterize the intensity, which is not limited in the embodiments of the present application.

[0205] The area of ​​pixels whose grayscale values ​​are greater than the threshold TH2 represents the number of pixels whose grayscale values ​​are greater than the threshold TH2.

[0206] Exemplarily, it can be determined that the input image has reflections based on the fact that the intensity of pixels with a grayscale greater than the threshold TH1 is greater than the threshold TH3.

[0207] Exemplarily, it can be determined that the input image has reflections based on the fact that a weighted result of the maximum grayscale value and the area of ​​pixels with grayscales greater than the threshold TH2 is greater than the threshold TH3.

[0208] Exemplarily, the input image may be determined to have reflections based on the weighted result of the maximum value of the grayscale image, the intensity of pixels with grayscale greater than the threshold TH1, and the area of ​​pixels with grayscale greater than the threshold TH2 being greater than the threshold TH3.

[0209] S940, performing a light-eliminating process on the original image according to the second judgment value. Step S940 may be performed by Figure 6The image post-processing module 530 in the software architecture shown is implemented.

[0210] For example, assuming that the second judgment value is 1, it indicates that the input image has reflection; otherwise, the second judgment value is 0, indicating that the input image has no reflection. The specific value of the second judgment value and the corresponding relationship between the image having reflection or not are not limited to the above examples, and various reasonable settings can be made as needed.

[0211] Specifically, the de-reflection processing of the original image according to the reflection judgment result (second judgment value) can be divided into the following cases:

[0212] Case 1: The input image has reflections and is the original image.

[0213] In this case, the first image can be subtracted from the input image to obtain a small image of the reflective layer. In some embodiments, the result of subtracting the first image from the input image can also be filtered to obtain a small image of the reflective layer. Since the size of the input image is the size of the original image, the small image of the reflective layer is also the size of the original image. The small image of the reflective layer is stripped from the original image to obtain an output image, which is an image of the original image after the reflection is eliminated.

[0214] In this case, the calculation method of the reflective layer thumbnail and the original image with the reflection eliminated is similar to the calculation method described in Case 1 in step S740 of the above method 700, and will not be repeated here.

[0215] Case 2: The input image has reflections and is a preprocessed version of the original image.

[0216] In this case, the first image may be subtracted from the input image to obtain a small reflective layer image. In some embodiments, the result of the subtraction between the first image and the input image may be filtered to obtain a small reflective layer image. Since the input image is an image obtained by preprocessing the original image, the size of the small reflective layer image is smaller than the size of the original image. In this case, the small reflective layer image may be upsampled to obtain a reflective layer image, the size of which is the size of the original image. In some embodiments, the first reflective layer image may be upsampled by interpolation. The reflective layer image may be stripped from the original image to obtain an output image, which is an image of the original image after the reflection is eliminated.

[0217] In this case, the calculation method of the reflective layer thumbnail, the reflective layer, and the original image with the reflection eliminated is similar to the calculation method described in Case 2 in step S740 of the above method 700, and will not be repeated here.

[0218] Case 3: Input image has no reflection.

[0219] In this case, since the input image has no reflection, there is no need to process the original image. Therefore, the output image is the original image.

[0220] According to the reflective layer intensity judgment unit, it can be judged whether there is reflection in the image, so that a suitable reflection processing method can be selected according to the judgment result, which can improve the reflection processing effect of the image.

[0221] The image processing method provided in the embodiment of the present application can, on the one hand, automatically identify whether the original image has reflections, and perform reflection elimination processing on the original image with reflections. On the other hand, when the original image size is large, the image processing method in the embodiment of the present application first downsamples the original image to adapt to the input scale of the neural network, then upsamples the small reflective layer image, and peels off the upsampled reflective layer from the original image, thereby eliminating reflections in high-resolution images.

[0222] In addition, the image processing method of the embodiment of the present application realizes the functions of identifying and eliminating reflections through a reflective layer strength judgment unit and a lightweight reflection elimination model, thereby overcoming the problem of an overly large reflection elimination network required for high-definition images.

[0223] Fig.11 is a schematic diagram of an image processing method 1100 according to another embodiment of the present application. The method can be Figure 6 The method can also be performed by a device capable of image processing, for example, Figure 4 The execution device 110 in the method 1100 is executed. The steps in the method 1100 are described in detail below.

[0224] S1110, preprocessing the original image to obtain an input image that meets the input size of the reflection elimination model. Step S1110 can be performed by Figure 6 The image pre-processing module 510 in the software architecture shown is implemented.

[0225] S1120, input the input image into the reflection elimination model to obtain a first image after the reflection elimination. Step S1120 can be performed by Figure 6 The network reasoning module 520 is implemented in the software architecture shown.

[0226] In method 1100, steps S1110 to S1120 are Figure 7 Steps S710 to S720 in the method 700 are similar and will not be described in detail.

[0227] S1130, the first image and the input image are input into the reflection judgment model to obtain a first judgment value. Step S1130 can be performed by Figure 6 The network reasoning module 520 is implemented in the software architecture shown.

[0228] In method 1100, step S1130 and Figure 7 Step S730 in the method 700 is similar and will not be described in detail.

[0229] S1140, the first image and the input image are input into the reflective layer intensity judgment unit to obtain a second judgment value. Step S1140 can be performed by Figure 6 The image post-processing module 530 in the software architecture shown is implemented.

[0230] In method 1100, step S1140 and Fig. 9 Step S930 in the method 900 is similar and will not be described in detail.

[0231] S1150, performing a joint judgment based on the first judgment value and the second judgment value to obtain a reflection judgment result. Step S1150 can be performed by Figure 6 The image post-processing module 530 in the software architecture shown is implemented.

[0232] In some embodiments, for example, the first judgment value may indicate whether the input image has reflection. At this time, when both the first judgment value and the second judgment value indicate reflection, the reflection judgment result is reflection; when at least one of the first judgment value and the second judgment value indicates no reflection, the reflection judgment result is no reflection. Or as another implementation, when at least one of the first judgment value and the second judgment value indicates reflection, the reflection judgment result is reflection; when both the first judgment value and the second judgment value indicate no reflection, the reflection judgment result is no reflection.

[0233] In other embodiments, for example, the first judgment value may indicate the confidence of whether the input image has reflection (i.e., the above-mentioned reflection confidence or reflection confidence coefficient). At this time, when the first judgment value is greater than or equal to a specific threshold and the second judgment value indicates that there is reflection, the reflection judgment result output by step S1150 is reflection; conversely, when the first judgment value is less than the specific threshold and / or the second judgment value indicates that there is no reflection, the reflection judgment result output by step S1150 is no reflection. Or as another implementation, when the first judgment value is greater than or equal to a specific threshold and / or the second judgment value indicates that there is reflection, the reflection judgment result output by step S1150 is reflection; conversely, when the first judgment value is less than the specific threshold and the second judgment value indicates that there is no reflection, the reflection judgment result output by step S1150 is no reflection.

[0234] Alternatively, the first judgment value may be further refined in intervals, for example, when the first judgment value is greater than the threshold value A1 and the second judgment value indicates no reflection, the reflection judgment result is reflection; when the first judgment value is less than the threshold value A1 and greater than the threshold value A2, and the second judgment value indicates reflection, the reflection judgment result is reflection; in other cases, the reflection judgment result is no reflection, where A1 is greater than A2. A suitable joint judgment algorithm may be designed as needed, and these changes are within the scope of the embodiments of the present application.

[0235] S1160, based on the reflection judgment result, the original image is subjected to a reflection elimination process. Step S1160 may be performed by Figure 6 The image post-processing module 530 in the software architecture shown is implemented.

[0236] In method 1100, step S1140 and Figure 7 Step S740 in the method 700 is similar and will not be described in detail.

[0237] The use of multiple judgment models and units for joint judgment can more accurately judge the reflection of an image than a single model, thereby improving the reflection processing effect of the image. It should be understood that if there are other suitable judgment models and / or judgment units, they can also be combined into the reflection judgment module or joint judgment mode of the present application.

[0238] Fig.12 Schematic diagram of the structure of the image processing device of the embodiment of the present application. Fig.12 As shown, the image processing device 1200 of the embodiment of the present application includes a first reflection elimination module 1210 , a reflection judgment module 1220 and a second reflection elimination module 1230 .

[0239] The first reflection elimination module 1210 is used to perform reflection elimination processing on the input image to obtain a first image. In some embodiments, the first reflection elimination module 1210 can be used to implement Figure 7 Step S720 in the method 700 shown, or Fig. 9 Step S920 in method 900 shown, or Fig.11 The function of step S1120 in the method 1100 is shown. The specific functions and beneficial effects of the first reflection elimination module 1210 can be found in the description of the above method, and will not be repeated here for the sake of brevity.

[0240] The reflection judgment module 1220 is used to judge whether the input image has reflection according to the first image and the input image.

[0241] In some embodiments, the reflection judgment module 1220 may include a reflection judgment model for implementing Figure 7The function of step S730 in method 700 is shown.

[0242] In some other embodiments, the reflective layer determination module 1220 may include a reflective layer intensity determination model for implementing Fig. 9 The function of step S930 in method 900 is shown.

[0243] In some other embodiments, the reflection judgment module 1220 may include a reflection judgment model, a reflection layer intensity judgment model and a joint judgment unit, for implementing Fig.11 The functions of steps S1130 to S1150 in the method 1100 are shown.

[0244] The specific functions and beneficial effects of the reflection judgment module 1220 can be found in the description of the above method, and for the sake of brevity, they will not be repeated here.

[0245] The second reflection elimination module 1230 is used to obtain an output image according to the judgment result of the reflection judgment module. In some embodiments, the second reflection elimination module is used to implement Figure 7 Step S740 in method 700, or Fig. 9 Step S940 in method 900 shown, or Fig.11 The function of step S1160 in the method 1100 is shown. The specific functions and beneficial effects of the second reflection elimination module 1230 can be found in the description of the above method, and will not be repeated here for the sake of brevity.

[0246] In some embodiments, the image processing device may further include a preprocessing module 1240. The preprocessing module 1240 is used to preprocess the original image to generate an input image that meets the input size of the first reflective output module 1210. In some embodiments, the preprocessing module 1240 is used to implement Figure 7 Step S710 in the method 700 shown, or Fig. 9 Step S910 in method 900 shown, or Fig.11 The function of step S1100 in the method 1100 is shown. The specific functions and beneficial effects of the preprocessing module 1240 can be found in the description of the above method, and will not be repeated here for the sake of brevity.

[0247] It should be understood that in some embodiments, Fig.12 The image processing device shown can be composed of Figure 5 The chip shown is implemented as shown. Figure 5 The neural network processor 300 in the chip shown can implement Fig.12 Functions of the reflection judgment model in the first reflection elimination module 1210 and the reflection judgment module 1220 of the image processing device shown; Figure 5The main CPU in the chip shown can realize the functions of the reflective layer intensity judgment unit, the joint judgment unit and the second reflective elimination module 1230 in the reflective judgment module 1220. In other embodiments, Figure 5 The main CPU in the chip shown can also implement the functions of the pre-processing module 1240.

[0248] In some embodiments, Fig.12 The image processing device shown can realize Figure 6 The first reflection elimination module 1210 can implement Figure 6 The reflection elimination model in the network reasoning module 520 of the software architecture shown in the figure can realize the reflection judgment model in the reflection judgment module 1220. Figure 6 The functions of the reflection judgment model in the network reasoning module of the software architecture shown in the figure; the reflection layer intensity judgment unit, the joint judgment unit and the second reflection elimination module 1230 in the reflection judgment module 1220 can realize Figure 6 The functions of the image post-processing module 530 in the software architecture shown in FIG. 12 are shown in FIG. 12. In other embodiments, the pre-processing module 1240 may implement Figure 6 The software architecture shows the functionality of the image preprocessing module.

[0249] In some embodiments, Fig.12 The image processing device shown can be deployed in Figure 4 The execution device 110 shown in the figure implements the functions of each module.

[0250] Fig.13 It should be understood that the device of the embodiment of the present application may be one or more chips, or may be included in a terminal or other device.

[0251] like Fig.13 As shown, the processor system 1300 of the embodiment of the present application includes a central processing unit (CPU) 1310, a neural network processor (neutral processing unit, NPU) 1320 and an image signal processor (image signal processing, ISP) 1330.

[0252] In the above method, Figure 6The image pre-processing module 510, the image post-processing module 530 and the software processing module 540 of the software architecture shown can be applied to the central processing unit 1310, or implemented by the central processing unit 1310. The central processing unit 1310 can also be used to run general operating system software, and control the neural network processor 1320 and the image signal processor 1330 under the action of the general operating system software. In the above method, Figure 6 The network reasoning module 520 of the software architecture shown can be applied to the neural network processor 1320, or implemented by the neural network processor 1320.

[0253] The image signal processor 1330 is used to generate an image. The image signal processor can match image sensors of different manufacturers to process the image data output by the front-end image sensor and generate a corresponding image signal according to the image data, such as the original image mentioned above. In some embodiments, Fig.12 The image processing device shown can be composed of Fig.13 The processor system shown is implemented. The central processor 1310 can implement the functions of the reflective layer intensity judgment unit and / or the joint judgment unit in the reflective judgment module 1220 and the second reflective elimination module 1230; in other embodiments, the central processor 1310 can also implement the functions of the preprocessing module 1240.

[0254] Among them, the neural network processor 1320 can implement Fig.12 The functions of the reflection judgment model in the first reflection elimination module 1210 and the reflection judgment module 1220 in the image processing device shown.

[0255] It should be understood that Fig.13 The processor system shown may include one or more Figure 5 The chip shown, Figure 5 The main CPU in the chip shown can implement Fig.13 The functions of the CPU 1310 of the processor system shown; Figure 5 The neural network processor 300 in the chip shown can implement Fig.13 The functionality of the neural network processor 1320 in the processor system shown.

[0256] It should be understood that Fig.13 The processor system shown may also be deployed in Figure 4 The execution device 110 is shown, and this embodiment of the present application does not limit this.

[0257] Fig.14 Schematic diagram of the hardware structure of another image processing device provided in an embodiment of the present application. Fig.14The image processing device 1400 shown includes a memory 1410, a processor 1420, a communication interface 1430, and a bus 1440. The memory 1410, the processor 1420, and the communication interface 1430 are connected to each other through the bus 1440.

[0258] The memory 1410 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1410 may store a program. When the program stored in the memory 1410 is executed by the processor 1420, the processor 1420 is used to execute each step of the image processing method of the embodiment of the present application, for example, to execute Figures 7 to 11 The steps shown.

[0259] It should be understood that the image processing device shown in the embodiment of the present application can be a server, for example, it can be a server in the cloud, or it can be a chip configured in a server in the cloud; or, the image processing device shown in the embodiment of the present application can be a smart terminal, or it can be a chip configured in a smart terminal.

[0260] The image processing method disclosed in the above embodiment of the present application can be applied to the processor 1420, or implemented by the processor 1420. The processor 1420 may be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above image processing method can be completed by an integrated logic circuit of hardware in the processor 1420 or an instruction in the form of software. For example, the processor 1420 may include Figure 5 The NPU chip is shown.

[0261] The above-mentioned processor 1420 can be a central processing unit (CPU), a graphics processing unit (GPU), a general processor, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The various methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor to be executed, or the hardware and software modules in the decoding processor can be combined to be executed. The software module can be located in a mature storage medium in the field such as a random access memory (RAM), a flash memory, a read-only memory (ROM), a programmable read-only memory or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1410, and the processor 1420 reads the instructions in the memory 1410, and completes the implementation of the present application in combination with its hardware. Figure 6 The software architecture of the reflection detection and elimination system shown in the figure or Fig.12 The functions required to be performed by the modules included in the image processing device shown, or the functions required to perform the method embodiments of the present application Figures 7 to 11 The individual steps of the image processing method shown.

[0262] The communication interface 1430 uses a transceiver device such as but not limited to a transceiver to implement communication between the apparatus 1400 and other devices or a communication network.

[0263] The bus 1440 may include a path for transmitting information between various components of the image processing device 1400 (eg, the memory 1410 , the processor 1420 , and the communication interface 1430 ).

[0264] It should be noted that although the above-mentioned image processing device 1400 only shows a memory, a processor, and a communication interface, in the specific implementation process, those skilled in the art should understand that the image processing device 1400 may also include other devices necessary for normal operation. At the same time, those skilled in the art should understand that according to specific needs, the above-mentioned image processing device 1400 may also include hardware devices for implementing other additional functions. In addition, those skilled in the art should understand that the above-mentioned image processing device 1400 may also only include the devices necessary for implementing the embodiments of the present application, and does not necessarily include Fig.14 All devices shown in .

[0265] An embodiment of the present application also provides a computer-readable medium, which stores a computer program (also referred to as code or instruction) that, when executed on a computer, enables the computer to execute a method in any of the above method embodiments.

[0266] An embodiment of the present application also provides a chip system, including a memory and a processor, wherein the memory is used to store a computer program, and the processor is used to call and run the computer program from the memory, so that an image processing device equipped with the chip system executes a method in any of the above method embodiments.

[0267] Among them, the chip system may include an input circuit or interface for sending information or data, and an output circuit or interface for receiving information or data.

[0268] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a storage medium or transmitted from one storage medium to another storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The storage medium may be any available medium that a computer can access or a data storage device such as a server or a data center that includes one or more available media integrated. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a high-density digital video disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).

[0269] In this application, "at least two" or "multiple" means two or more. The term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone, where A and B can be singular or plural. The character " / " generally indicates that the associated objects are in an "or" relationship.

[0270] The predefined in the present application may be understood as defined, predefined, stored, pre-stored, pre-negotiated, pre-configured, solidified, or pre-burned.

[0271] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the sequence number of the above-mentioned processes does not mean the order of execution, and the execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application.

[0272] The terms "component", "module", "system", etc. used in this specification are used to represent computer-related entities, hardware, firmware, a combination of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program and / or a computer. By way of illustration, both applications running on a computing device and a computing device can be components. One or more components may reside in a process and / or an execution thread, and a component may be located on a computer and / or distributed between two or more computers. In addition, these components may be executed from various computer-readable media having various data structures stored thereon. Components may, for example, communicate through local and / or remote processes according to signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system and / or a network, such as the Internet interacting with other systems through signals).

[0273] Those of ordinary skill in the art will appreciate that, in conjunction with the various method steps described in the embodiments disclosed herein, they can be implemented with electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the steps of the various embodiments have been generally described in terms of function in the above description. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those of ordinary skill in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0274] Those of ordinary skill in the art will appreciate that the various illustrative logical blocks and steps described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or in a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel may use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0275] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0276] In the several embodiments provided in the present application, it can be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0277] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0278] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0279] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing device, characterized in that: include: A first reflection elimination module, used for performing reflection elimination processing on an input image to obtain a first image; a reflection judgment module, used for obtaining a judgment result of whether the input image has reflection according to the first image and the input image; A second reflection elimination module, used for acquiring an output image according to the judgment result; Among them, when the judgment result is that the input image has reflection, the second reflection elimination module is specifically used to generate a reflection layer thumbnail according to the first image and the input image, and obtain the output image according to the reflection layer thumbnail and the original image.

2. The device according to claim 1, characterized in that The device also includes: The preprocessing module is used to preprocess the original image to obtain the input image, wherein the size of the original image is larger than the size of the input image.

3. The device according to claim 1, characterized in that The second reflection elimination module is specifically configured to obtain the original image as the output image when the judgment result is that the input image has no reflection.

4. The device according to claim 2, characterized in that The second reflection elimination module is specifically used to generate the reflection layer thumbnail according to the first image and the input image when the judgment result is that the input image has reflection, upsample the reflection layer thumbnail to obtain the reflection layer, and obtain the output image according to the reflection layer and the original image.

5. The device according to any one of claims 1 to 4, characterized in that The reflection judgment module includes a reflection judgment model, which is a neural network with the first image and the input image as input, and the output of the reflection judgment model represents the confidence that the input image has reflection, or the output is the judgment result.

6. The device according to claim 5, characterized in that The reflection judgment model is a neural network based on a Markov discriminator and is obtained by training based on a first sample set, wherein the first sample set includes: original image samples, image samples of the original image samples after reflection elimination, and labels of whether the original image samples have reflections.

7. The device according to any one of claims 1 to 4, characterized in that The reflective layer judgment module includes a reflective layer intensity judgment unit, which is used to generate a reflective layer thumbnail according to the input image and the first image; the reflective layer intensity judgment unit is also used to calculate the grayscale values ​​of multiple pixel points included in the reflective layer thumbnail, and determine the judgment result according to the grayscale values ​​of the multiple pixel points.

8. The device according to claim 7, characterized in that The light-reflecting layer strength judgment unit determines the judgment result according to a weighted result of at least one or more of the following information: the maximum value among the grayscale values ​​of the plurality of pixels, the intensity of at least one pixel whose grayscale value is greater than a first threshold, and the area of ​​at least one pixel whose grayscale value is greater than a second threshold; The intensity of at least one pixel whose grayscale value is greater than the first threshold is the sum of the grayscale values ​​of the pixels whose grayscale values ​​are greater than the first threshold, or, The intensity of at least one pixel whose grayscale value is greater than the first threshold is the average value of the grayscale values ​​of the pixels whose grayscale values ​​are greater than the first threshold; The area of ​​at least one pixel whose grayscale value is greater than the second threshold is the number of pixels whose grayscale value is greater than the second threshold.

9. The device according to any one of claims 1 to 4, characterized in that The reflection judgment module includes a reflection judgment model, a reflection layer intensity judgment unit and a joint judgment unit. The reflection judgment model is a neural network with the first image and the input image as input, and the output of the reflection judgment model is a first judgment value indicating whether the input image has reflection; The light-reflecting layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and outputs a second judgment value indicating whether the input image is reflective according to the gray value; The joint judgment unit determines the judgment result according to the first judgment value and the second judgment value.

10. An image processing method, characterized in that: include: Performing a reflection elimination process on the input image to obtain a first image; Determining whether the input image has reflection according to the input image and the first image; Obtaining an output image according to whether the input image has reflection; Wherein, obtaining the output image according to whether the input image has reflection includes: When the input image has reflections, generating a reflection layer thumbnail according to the input image and the first image; The output image is obtained according to the reflective layer thumbnail and the original image.

11. The method according to claim 10, characterized in that The method further comprises: The original image is preprocessed to obtain the input image, and the size of the original image is larger than the size of the input image.

12. The method according to claim 10, characterized in that The acquiring the output image according to whether the input image has reflection includes: when the input image has no reflection, acquiring the original image as the output image.

13. The method according to claim 11, characterized in that The obtaining of the output image according to the reflective layer thumbnail and the original image comprises: Upsampling the reflective layer thumbnail to obtain a reflective layer; The output image is obtained according to the reflective layer and the original image.

14. The method according to any one of claims 10 to 13, characterized in that The determining, according to the input image and the first image, whether the input image has reflection comprises: Determining whether the input image has reflection according to a reflection judgment model; The reflection judgment model is a neural network that takes the first image and the input image as input, and the output of the reflection judgment model represents the confidence that the input image has reflection, or the output is a judgment result of whether the input image has reflection.

15. The method according to claim 14, characterized in that The reflection judgment model is a neural network based on a Markov discriminator and is obtained by training based on a first sample set, wherein the first sample set includes: original image samples, image samples of the original image samples after reflection elimination, and labels of whether the original image samples have reflections.

16. The method according to any one of claims 10 to 13, characterized in that The determining, according to the input image and the first image, whether the input image has reflection comprises: Determining whether the input image has reflection according to the reflection layer intensity determination unit; The reflective layer intensity judgment unit is used to calculate a reflective layer thumbnail based on the input image and the first image; the reflective layer intensity judgment unit is also used to calculate the grayscale values ​​of multiple pixel points included in the reflective layer thumbnail, and determine the judgment result based on the grayscale values ​​of the multiple pixel points.

17. The method according to claim 16, characterized in that The reflective layer determination unit determines the determination result according to a weighted result of at least one or more of the following information: the maximum value among the grayscale values ​​of the plurality of pixels, the intensity of at least one pixel whose grayscale value is greater than a first threshold, and the area of ​​at least one pixel whose grayscale value is greater than a second threshold; The intensity of at least one pixel whose grayscale value is greater than the first threshold is the sum of the grayscale values ​​of the pixels whose grayscale values ​​are greater than the first threshold, or, The intensity of at least one pixel whose grayscale value is greater than the first threshold is the average value of the grayscale values ​​of the pixels whose grayscale values ​​are greater than the first threshold; The area of ​​at least one pixel whose grayscale value is greater than the second threshold is the number of pixels whose grayscale value is greater than the second threshold.

18. The method according to any one of claims 10 to 13, characterized in that The determining, according to the input image and the first image, whether the input image has reflection comprises: Determine whether the input image has reflection according to the reflection judgment model, the reflection layer intensity judgment unit and the joint judgment unit; The reflection judgment model is a neural network with the first image and the input image as input, and the output of the reflection judgment model is a first judgment value indicating whether the input image has reflection; The light-reflecting layer intensity judgment unit calculates the gray value of the difference between the input image and the first image, and outputs a second judgment value indicating whether the input image is reflective according to the gray value; The joint judgment unit determines a judgment result according to the first judgment value and the second judgment value.

Images