Method for removing specular reflection from an image, method and apparatus for training an image specular reflection removal model

Abstract

A method for removing specular reflection from an image, a method and device for training an image specular reflection removal model, which relate to the field of image processing; the method includes displaying a first interface, the first interface including a first control; detecting a first operation on the first control; in response to the first operation, obtaining a first image and a second image, the first image being an image of a shooting scene obtained by a first camera module, the second image being an image of the shooting scene obtained by a second camera module, the shooting scene including a first shooting object, a transparent glass, and a second shooting object, the first shooting object and the second shooting object being located on both sides of the transparent glass; turning on an infrared flash and obtaining a third image, the third image being an image of the shooting object obtained by the first camera module; obtaining a fourth image according to the second image and the third image; inputting the first image and the fourth image into an image specular reflection removal model to obtain a fifth image; based on the technical solution of the present application, the effect of removing specular reflection from an image can be improved.

Description

Technical Field

[0001] This application relates to the field of image processing, and specifically, to a method for removing specular reflection from an image, a method for training an image specular reflection removal model, and an apparatus therefor. Background Art

[0002] With the continuous progress of technology, the shooting function of terminal devices is becoming more and more powerful, and users' requirements for image quality are getting higher and higher. However, when shooting, users will encounter some scenes with specular reflection. For example, when a user obtains an image of a shooting object through a transparent glass, since the shooting object is photographed through the transparent glass, the specular reflection light and the transmitted light existing in the transparent glass are mixed together, resulting in the masking of some detailed information of the shooting object. Therefore, a specular reflection area will appear in the image.

[0003] Currently, an algorithm for removing specular reflection based on a single-frame image is usually sampled. For example, a large number of non-specular reflection images and aliased images (for example, composite images including images with specular reflection areas) are used as sample data to train an image specular reflection removal model. Since the shooting scenes of the obtained sample data are limited, the generalization ability of the trained image specular reflection removal model is poor, that is, when the trained image specular reflection removal model processes an image of a shooting scene not included in the sample data, the effect of removing specular reflection from the image is poor.

[0004] Therefore, how to remove specular reflection in an image and improve the effect of removing specular reflection from the image has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides a method for removing specular reflection from an image, a method for training an image specular reflection removal model, and an apparatus therefor, which can remove specular reflection in an image and improve the effect of removing specular reflection from the image.

[0006] In a first aspect, a method for removing specular reflection from an image is provided, which is applied to an electronic device. The electronic device includes a first camera module, a second camera module, and an infrared flash. The spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is visible light and near-infrared light. The method includes:

[0007] Display a first interface, where the first interface includes a first control;

[0008] Detect a first operation on the first control;

[0009] In response to the first operation, a first image and a second image are acquired. The first image is an image of a shooting scene acquired by the first camera module, and the second image is an image of the shooting scene acquired by the second camera module. The shooting scene includes a first shooting object, a transparent glass, and a second shooting object. The first shooting object and the second shooting object are located on both sides of the transparent glass;

[0010] The infrared flash lamp is turned on, and a third image is acquired. The third image is an image of the shooting scene acquired by the first camera module;

[0011] A fourth image is obtained according to the second image and the third image. The fourth image is used to represent the reflection information of the second shooting object on the first transmitted light. The first transmitted light is obtained by the near-infrared light passing through the transparent glass;

[0012] The first image and the fourth image are input into an image anti-reflection model to obtain a fifth image;

[0013] Wherein, the image anti-reflection model includes a first sub-model and a second sub-model. The first sub-model is used to separate the reflection information of the first shooting object on the visible light; the second sub-model is used to separate the reflection information of the second shooting object on the second transmitted light. The second transmitted light is obtained by the visible light passing through the transparent glass.

[0014] Optionally, in the embodiments of the present application, the first sub-model and the second sub-model included in the image anti-reflection model may refer to a neural network model; for example, the first sub-model and the second sub-model may be a convolutional neural network or other neural network structures, and the present application does not make any limitation thereto.

[0015] It should be noted that, in the embodiments of the present application, the second image acquired by the second camera module may refer to a Raw image; the Raw image acquired by the second camera module refers to a single-channel image; the Raw image acquired by the second camera module can be used to represent the intensity information of photons superimposed together; for example, the Raw image acquired by the second camera module may be a single-channel grayscale image.

[0016] It should be understood that in the shooting scenario involved in the embodiments of the present application, there are a first shooting object (front object), a transparent glass, and a second shooting object (target object). The front object and the target object are located on both sides of the transparent glass; the terminal device and the front object are on the same side of the transparent glass; the front object may refer to the front scene. When the position of the transparent glass is replaced with a mirror, any object appearing in the mirror can be regarded as the front scene. For example, the front object may refer to an object placed in front of the transparent glass, or the sky in the shooting scenario, or the user in the shooting scenario, etc.; since the user shoots the target object through the transparent glass, the specular reflection light and the transmitted light existing in the transparent glass will be mixed together, resulting in the masking of some detailed information of the target object. Therefore, there will be a reflective area in the image of the target object obtained; the purpose of image anti-reflection is to remove the information of the front object included in the image of the target object.

[0017] In the embodiments of the present application, images of the shooting scenario are obtained through the first camera module and the second camera module; since the spectral range received by the first camera module is the visible light region (for example, 400nm - 700nm), and the spectral range received by the second camera module is the visible light region and the near-infrared light region (for example, 400nm - 1000nm); therefore, the spectral information of the two camera modules can be complementary, and more information of the shooting scenario can be obtained; the image anti-reflection effect can be improved by the image anti-reflection method provided in the embodiments of the present application, and the clarity of the anti-reflection image can be improved. In addition, an infrared flash is included in the electronic device in the embodiments of the present application. Since the on state and off state of the infrared flash are invisible to the user; therefore, the image anti-reflection method of the embodiments of the present application can be executed without the user's perception, and the specular reflection in the image can be removed, improving the image anti-reflection effect.

[0018] In combination with the first aspect, in some implementation manners of the first aspect, it further includes:

[0019] Determine the classification information of the shooting scenario, where the classification information is used to indicate the scenario classification to which the shooting scenario belongs;

[0020] Determine that there is specular reflection in the shooting scenario according to the classification information.

[0021] Optionally, the classification information may include but is not limited to: green plant scenario, portrait scenario, indoor scenario, outdoor scenario, etc.

[0022] Exemplarily, a binary classification neural network can be learned through a large number of sample data, and the binary classification neural network can identify whether there is specular reflection in the current shooting scenario; for example, it can identify whether the transparent glass is included in the shooting scenario.

[0023] In an embodiment of the present application, the electronic device can automatically identify whether there is specular reflection in the current shooting scene; when it is determined that there is specular reflection in the shooting scene, the method for removing specular reflection from an image provided in the embodiment of the present application is executed.

[0024] In combination with the first aspect, in some implementation manners of the first aspect, it further includes:

[0025] A second operation is detected, and the second operation is used to indicate that there is specular reflection in the shooting scene.

[0026] In an embodiment of the present application, the user can identify whether there is specular reflection in the current shooting scene; when there is specular reflection in the shooting scene, the user can turn on the anti-reflection mode of the electronic device; after the electronic device recognizes that the user has turned on the anti-reflection mode, the method for removing specular reflection in the image provided in the embodiment of the present application is executed.

[0027] In combination with the first aspect, in some implementation manners of the first aspect, obtaining the fourth image according to the first image and the third image includes:

[0028] Subtracting the third image from the second image to obtain a sixth image;

[0029] Performing registration processing on the sixth image with the first image as a reference to obtain the fourth image.

[0030] It should be noted that since the first camera module and the second camera module are respectively arranged at different positions in the electronic device, there is a certain baseline distance between the first camera module and the second camera module, that is, there is a certain parallax between the image collected by the first camera module and the image collected by the second camera module, and registration processing needs to be performed on the images collected by both.

[0031] It should be understood that the second image and the third image are two consecutive frames of images, and the two frames of images are respectively collected with the infrared flash off and on, and the pixels between the two frames of images are basically aligned; by subtracting the pixels of the two frames of images, an image irradiated only by the infrared flash can be obtained.

[0032] In combination with the first aspect, in some implementation manners of the first aspect, the first interface refers to the main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to the control corresponding to the camera application.

[0033] In a possible implementation manner, the first operation refers to an operation of clicking on the camera application.

[0034] In combination with the first aspect, in certain implementations of the first aspect, the first interface refers to a video recording interface, and the first control refers to a control for indicating video recording.

[0035] In a possible implementation, the first operation refers to an operation of clicking on the control for indicating video recording.

[0036] In combination with the first aspect, in certain implementations of the first aspect, the first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

[0037] In a possible implementation, the first operation refers to an operation of clicking on the control for indicating a video call.

[0038] The above takes the first operation as a click operation as an example for illustration; the first operation may also include a voice indication operation, or other operations for instructing the electronic device to take a photo or make a video call; the above is for illustration purposes only and does not limit this application in any way.

[0039] In a second aspect, a method for training an image anti-reflection model is provided. The image anti-reflection model includes a first sub-model and a second sub-model, and the method includes:

[0040] Obtain training data, where the training data includes a first sample image, a second sample image, a third sample image, and a fourth sample image. The first sample image is an image of a sample shooting scene obtained by a first camera module; the sample shooting scene includes a first sample object, a transparent glass, and a second sample object, and the first sample object and the second sample object are located on both sides of the transparent glass; the second sample image is an image obtained by a second camera module for representing the reflection information of the second sample object on the first transmitted light, and the first transmitted light is obtained by near-infrared light passing through the transparent glass; the spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is visible light and near-infrared light; the third sample image refers to an image of the first sample object obtained by the first camera module when the transparent glass is blocked; the fourth sample image refers to an image of the second sample object obtained by the first camera module;

[0041] Use the first sample image and the second sample image as input data, and use the third sample image as the first target data to train the first sub-model, obtaining a trained first sub-model. The first sub-model is used to separate the reflection information of the first sample object on the visible light;

[0042] Taking the output data of the trained first sub-model and the first sample image as input data, and using the fourth sample image as the second target data to train the second sub-model, to obtain the trained second sub-model, where the second sub-model is used to separate the reflection information of the second sample object from the second transmitted light, and the second transmitted light is obtained by passing visible light through the transparent glass.

[0043] It should be understood that the shooting scenario involved in the embodiments of the present application includes a first sample object (front object), a transparent glass, and a second sample object (target object), where the front object and the target object are located on both sides of the transparent glass; the terminal device and the front object are on the same side of the transparent glass; the front object can refer to the front scene, that is, when the position of the transparent glass is replaced with a mirror, any object that appears in the mirror can be regarded as the front scene. For example, the front object can refer to an object placed in front of the transparent glass, or the sky in the shooting scene, or the user in the shooting scene, etc.; since the user shoots the target object through the transparent glass, the specular reflection light and the transmitted light of the transparent glass will be mixed together, resulting in some detailed information of the target object being covered, so there will be a reflective area in the obtained image of the target object; the purpose of the image anti-reflection model is to remove the information of the front object included in the image of the target object.

[0044] It should be understood that in the embodiments of the present application, the image anti-reflection model may include a first sub-model and a second sub-model. The first sub-model and the second sub-model may be two cascaded models in the image anti-reflection model, that is, the input data is first processed by the first sub-model; the output data of the first sub-model can be used as the input data of the second sub-model; the entire network training of the image anti-reflection model can adopt a segmented training method. For example, the first sub-model is trained first; after the first sub-model converges, the weights of the first sub-model can be fixed, and then the second sub-model is trained.

[0045] In the embodiments of the present application, the image anti-reflection model includes a first sub-model and a second sub-model. When training the first sub-model, the first sample image and the second sample image are used as input data, and the second sample image only includes the optical path information related to the second sample object (target object); by using the information of the target object as prior information to train the image anti-reflection model, the generalization ability of the image anti-reflection model can be improved.

[0046] Optionally, in the embodiments of the present application, the first sub-model and the second sub-model included in the image anti-reflection model may refer to neural network models; for example, the first sub-model and the second sub-model may be convolutional neural networks or other neural network structures, and the present application does not make any limitations in this regard.

[0047] In combination with the second aspect, in some implementations of the second aspect, the obtaining of the training data includes:

[0048] When the infrared flash is turned on, obtain a fifth sample image of the sample shooting scene through the second camera module;

[0049] When the infrared flash is turned off, obtain a sixth sample image of the sample shooting scene through the second camera module;

[0050] Obtain the second sample image according to the fifth sample image and the sixth sample image.

[0051] In combination with the second aspect, in some implementations of the second aspect, the obtaining of the second sample image according to the fifth sample image and the sixth sample image includes:

[0052] Subtract the fifth sample image from the sixth sample image to obtain a seventh sample image;

[0053] Register the seventh sample image with reference to the first sample image to obtain the second sample image.

[0054] It should be noted that since the first camera module and the second camera module are respectively arranged at different positions in the electronic device, there is a certain baseline distance between the first camera module and the second camera module, that is, there is a certain parallax between the images collected by the first camera module and the images collected by the second camera module, and the images collected by both need to be registered.

[0055] It should be understood that the fifth sample image and the sixth sample image are two consecutive frames of images. The two frames of images are respectively collected under the condition of turning off the infrared flash and turning on the infrared flash, and the pixels between the two frames of images are basically aligned; by subtracting the pixels of the two frames of images, an image of only the second sample object irradiated by the infrared flash can be obtained.

[0056] In combination with the second aspect, in some implementations of the second aspect, the parameters of the first sub-model are obtained by iteration through backpropagation according to the difference between the prediction data output by the first sub-model and the first target data.

[0057] In combination with the second aspect, in some implementations of the second aspect, the parameters of the second sub-model are obtained by iteration through backpropagation according to the difference between the prediction data output by the second sub-model and the second target data.

[0058] In a third aspect, an anti-reflection device for images is provided. The device includes one or more processors, a memory, a first camera module, a second camera module, and an infrared flash; the memory is coupled to the one or more processors, and the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the device to perform:

[0059] Display a first interface, the first interface includes a first control; detect a first operation on the first control; in response to the first operation, obtain a first image and a second image, the first image is an image of a shooting scene obtained by the first camera module, and the second image is an image of the shooting scene obtained by the second camera module. The shooting scene includes a first shooting object, a transparent glass, and a second shooting object, and the first shooting object and the second shooting object are located on both sides of the transparent glass; turn on the infrared flash and obtain a third image, the third image is an image of the shooting scene obtained by the first camera module; obtain a fourth image according to the second image and the third image, and the fourth image is used to represent the reflection information of the second shooting object on the first transmitted light, and the first transmitted light is obtained by the near-infrared light passing through the transparent glass;

[0060] Input the first image and the fourth image into an image anti-reflection model to obtain a fifth image;

[0061] Wherein, the image anti-reflection model includes a first sub-model and a second sub-model. The first sub-model is used to separate the reflection information of the first shooting object on the visible light; the second sub-model is used to separate the reflection information of the second shooting object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

[0062] In combination with the third aspect, in some implementation manners of the third aspect, the one or more processors call the computer instructions to cause the device to perform:

[0063] Determine classification information of the shooting scene, and the classification information is used to indicate the scene classification to which the shooting scene belongs;

[0064] Determine that there is specular reflection in the shooting scene according to the classification information.

[0065] In combination with the third aspect, in some implementation manners of the third aspect, the one or more processors call the computer instructions to cause the device to perform:

[0066] Detect a second operation, and the second operation is used to indicate that there is specular reflection in the shooting scene.

[0067] In combination with the third aspect, in some implementations of the third aspect, the one or more processors call the computer instructions to cause the device to perform:

[0068] Obtaining a fourth image according to the first image and the third image includes:

[0069] Subtracting the second image from the third image to obtain a sixth image;

[0070] Performing registration processing on the sixth image with the first image as a reference to obtain the fourth image.

[0071] In combination with the third aspect, in some implementations of the third aspect, the first interface refers to the main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to the control corresponding to the camera application.

[0072] In combination with the third aspect, in some implementations of the third aspect, the first interface refers to a video recording interface, and the first control refers to a control for indicating video recording.

[0073] In combination with the third aspect, in some implementations of the third aspect, the first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

[0074] In a fourth aspect, a training device for an image anti-reflection model is provided. The image anti-reflection model includes a first sub-model and a second sub-model. The training device includes one or more processors and a memory; the memory is coupled to the one or more processors, and the memory is used to store computer program code. The computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the training device to perform:

[0075] Obtain training data, where the training data includes a first sample image, a second sample image, a third sample image, and a fourth sample image. The first sample image is an image of a sample shooting scene obtained by a first camera module; the sample shooting scene includes a first sample object, a transparent glass, and a second sample object, and the first sample object and the second sample object are located on both sides of the transparent glass; the second sample image is an image obtained by a second camera module for representing the reflection information of the second sample object on the first transmitted light, and the first transmitted light is obtained by near-infrared light passing through the transparent glass; the spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is the visible light and near-infrared light; the third sample image is an image of the first sample object obtained by the first camera module when the transparent glass is blocked; the fourth sample image is an image of the second sample object obtained by the first camera module.

[0076] Use the first sample image and the second sample image as input data, and use the third sample image as the first target data to train a first sub-model, and obtain the trained first sub-model. The first sub-model is used to separate the reflection information of the first sample object on the visible light.

[0077] Use the output data of the trained first sub-model and the first sample image as input data, and use the fourth sample image as the second target data to train a second sub-model, and obtain the trained second sub-model. The second sub-model is used to separate the reflection information of the second sample object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

[0078] In combination with the fourth aspect, in some implementation manners of the fourth aspect, the one or more processors call the computer instructions to cause the training device to execute:

[0079] When the infrared flash is turned on, obtain a fifth sample image of the sample shooting scene through the second camera module;

[0080] When the infrared flash is turned off, obtain a sixth sample image of the sample shooting scene through the second camera module;

[0081] Obtain the second sample image according to the fifth sample image and the sixth sample image.

[0082] In combination with the fourth aspect, in some implementation manners of the fourth aspect, the one or more processors call the computer instructions to cause the training device to execute:

[0083] Subtract the sixth sample image from the fifth sample image to obtain a seventh sample image;

[0084] Register the seventh sample image with reference to the first sample image to obtain the second sample image.

[0085] Combined with the fourth aspect, in some implementation manners of the fourth aspect, the parameters of the first sub-model are obtained by iteration through backpropagation according to the difference between the prediction data output by the first sub-model and the first target data.

[0086] Combined with the fourth aspect, in some implementation manners of the fourth aspect, the parameters of the second sub-model are obtained by iteration through backpropagation according to the difference between the prediction data output by the second sub-model and the second target data.

[0087] In a fifth aspect, there is provided an image anti-reflection device, including a module / unit for executing any method in the first aspect or the first aspect.

[0088] In a sixth aspect, there is provided a training device for an image anti-reflection model, including a module / unit for executing any method in the second aspect or the second aspect.

[0089] In a seventh aspect, there is provided an image anti-reflection device, the device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the device to execute any method in the first aspect or the first aspect.

[0090] In an eighth aspect, there is provided a training device for an image anti-reflection model, the training device includes: one or more processors and a memory; the memory is coupled to the one or more processors, the memory is used to store computer program code, the computer program code includes computer instructions, and the one or more processors call the computer instructions to cause the training device to execute any training method in the second aspect or the second aspect.

[0091] In a ninth aspect, there is provided a chip system, the chip system is applied to an electronic device, the chip system includes one or more processors, and the processors are used to call computer instructions to cause the electronic device to execute any method in the first aspect or the second aspect.

[0092] In a tenth aspect, a computer-readable storage medium is provided. The computer-readable storage medium stores computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to execute any one of the methods in the first aspect or the second aspect.

[0093] In an eleventh aspect, a computer program product is provided. The computer program product includes: computer program code, and when the computer program code is run on an electronic device, the electronic device is caused to execute any one of the methods in the first aspect or the second aspect.

[0094] In an embodiment of the present application, images of a shooting scene are acquired by a first camera module and a second camera module; since the spectral range received by the first camera module is the visible light region (for example, 400nm - 700nm), and the spectral range received by the second camera module is the visible light region and the near-infrared light region (for example, 400nm - 1000nm); therefore, the spectral information of the two camera modules can be complementary, and more information about the shooting scene is acquired; the method for removing specular reflection of images provided by the embodiment of the present application can improve the effect of specular reflection removal of images and enhance the clarity of the de-specular images. In addition, the electronic device in the embodiment of the present application includes an infrared flash, and since the on state and off state of the infrared flash are invisible to the user; therefore, the method for removing specular reflection of images in the embodiment of the present application can be executed without the user's perception, and the specular reflection in the images can be removed, improving the effect of specular reflection removal of images. BRIEF DESCRIPTION OF THE DRAWINGS

[0095] Figure 1 is a schematic diagram of a hardware system of a device applicable to the present application;

[0096] Figure 2 is a schematic diagram of a software system of a device applicable to the present application;

[0097] Figure 3 is a schematic diagram of an application scenario applicable to an embodiment of the present application;

[0098] Figure 4 is a schematic diagram of the optical path of a shooting scene applicable to an embodiment of the present application;

[0099] Figure 5 is a schematic diagram of a method for training an image de-specular reflection model applicable to the present application;

[0100] Figure 6 is a schematic diagram of a method for removing specular reflection of images provided by an embodiment of the present application;

[0101] Figure 7 is a schematic diagram of the effect of the method for removing specular reflection of images provided by the present application;

[0102] Figure 8 It is a schematic diagram of a display interface of an electronic device provided by an embodiment of the present application;

[0103] Figure 9 It is a schematic diagram of a display interface of an electronic device provided by an embodiment of the present application;

[0104] Figure 10 It is a schematic diagram of a display interface of an electronic device provided by an embodiment of the present application;

[0105] Figure 11 It is a schematic structural diagram of a training device for an image anti-reflection model provided by an embodiment of the present application;

[0106] Figure 12 It is a schematic structural diagram of a device for image anti-reflection provided by an embodiment of the present application;

[0107] Figure 13 It is a schematic structural diagram of an electronic device provided by an embodiment of the present application. Detailed implementation manners

[0108] Since the embodiments of the present application involve a large number of applications of neural networks, for the convenience of understanding, the following first introduces the relevant terms and concepts that may be involved in the embodiments of the present application.

[0109] 1. Neural network

[0110] A neural network refers to a network formed by connecting multiple single neural units together, that is, the output of one neural unit can be the input of another neural unit; the input of each neural unit can be connected to the local receptive field of the previous layer to extract the features of the local receptive field, and the local receptive field can be a region composed of several neural units.

[0111] 2. Deep neural network

[0112] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. According to the position of different layers, the neural networks inside the DNN can be divided into three categories: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the middle layers are all hidden layers; the layers can be fully connected, that is, any neuron in the i-th layer can be connected to any neuron in the (i + 1)-th layer.

[0113] 3. Convolutional neural network

[0114] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network contains a feature extractor composed of convolutional layers and subsampling layers, and this feature extractor can be regarded as a filter; a convolutional layer refers to the layer of neurons in a convolutional neural network that performs convolutional processing on the input signal. In the convolutional layer of a convolutional neural network, a neuron can be connected only to some neighboring layer neurons. In a convolutional layer, there are usually several feature planes, and each feature plane can be composed of some rectangularly arranged neural units.

[0115] 4. Backpropagation Algorithm

[0116] A neural network can use the backpropagation (BP) algorithm to correct the magnitudes of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, forward propagating the input signal until the output will generate an error loss, and the initial neural network model parameters are updated by backpropagating the error loss information, thereby converging the error loss. The backpropagation algorithm is a backpropagation movement dominated by the error loss, used to obtain the optimal parameters of the neural network model; for example, the weight matrix.

[0117] 5. Specular Reflection

[0118] Specular reflection means that if the reflecting surface is relatively smooth, when parallel incident light hits this reflecting surface, it will still be reflected parallelly in one direction, and this kind of reflection belongs to specular reflection.

[0119] Exemplarily, glass reflection belongs to a case of specular reflection; for example, the object to be photographed is at the rear side far from the camera glass; for example, for photographing objects inside a glass window; or, a user takes a photo of an object outside the window through the glass, etc.

[0120] 6. All - pass Module

[0121] In the embodiments of the present application, an all - pass module refers to a camera module whose receivable spectral range is 400nm - 1000nm; compared with the main camera module of the terminal device (receiving a spectral range of 400nm - 700nm), the all - pass module has a wider receivable spectral range; the all - pass module can obtain the spectrum in the near - infrared band (for example, the spectral range is 780nm - 1000nm).

[0122] 7. Near - infrared Light (NIR)

[0123] Near-infrared light refers to the electromagnetic wave between visible light and mid-infrared light; near-infrared light can be divided into two regions: short-wave near-infrared (780nm - 1100nm) and long-wave near-infrared (1100nm - 2526nm).

[0124] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings.

[0125] Figure 1 Shows a hardware system of a device applicable to the present application.

[0126] The device 100 can be a mobile phone, smart screen, tablet computer, wearable electronic device, in-vehicle electronic device, augmented reality (AR) device, virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), projector, etc. The embodiments of the present application do not impose any restrictions on the specific type of the device 100.

[0127] The device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone interface 170D, a sensor module 180, a key 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0128] It should be noted that Figure 1 The shown structure does not constitute a specific limitation on the device 100. In other embodiments of the present application, the device 100 may include more or fewer components than Figure 1 the components shown, or the device 100 may includeFigure 1 A combination of some of the components shown, or, the apparatus 100 may include Figure 1 sub-components of some of the components shown. Figure 1 The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0129] The processor 110 may include one or more processing units. For example, the processor 110 may include at least one of the following processing units: an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a neural-network processing unit (NPU). Among them, different processing units may be independent devices or integrated devices.

[0130] The controller may generate operation control signals according to the instruction operation code and timing signals to complete the control of fetching and executing instructions.

[0131] A memory may also be provided in the processor 110 for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory may save the instructions or data that the processor 110 has just used or recycled. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0132] Exemplarily, in an embodiment of the present application, the processor 110 may execute: displaying a first interface, the first interface including a first control; detecting a first operation on the first control; in response to the first operation, obtaining a first image and a second image, the first image being an image of a shooting scene obtained by the first camera module, and the second image being an image of the shooting scene obtained by the second camera module, the shooting scene including a first shooting object, a transparent glass, and a second shooting object, the first shooting object and the second shooting object being located on both sides of the transparent glass; turning on the infrared flash lamp to obtain a third image, the third image being an image of the shooting scene obtained by the first camera module; obtaining a fourth image according to the second image and the third image, the fourth image being used to represent the reflection information of the second shooting object on the first transmitted light, the first transmitted light being obtained by the near-infrared light passing through the transparent glass; inputting the first image and the fourth image into an image anti-reflection model to obtain a fifth image; wherein, the image anti-reflection model includes a first sub-model and a second sub-model, the first sub-model being used to separate the reflection information of the first shooting object on the visible light; and the second sub-model being used to separate the reflection information of the second shooting object on the second transmitted light, the second transmitted light being obtained by the visible light passing through the transparent glass.

[0133] Exemplarily, in an embodiment of the present application, the processor 110 may execute: obtaining training data, where the training data includes a first sample image, a second sample image, a third sample image, and a fourth sample image. The first sample image is an image of a sample shooting scene obtained by a first camera module. The sample shooting scene includes a first sample object, a transparent glass, and a second sample object, and the first sample object and the second sample object are located on both sides of the transparent glass. The second sample image is an image obtained by a second camera module for representing the reflection information of the second sample object on the first transmitted light, and the first transmitted light is obtained by the near-infrared light passing through the transparent glass. The spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is visible light and near-infrared light. The third sample image is an image of the first sample object obtained by the first camera module when the transparent glass is blocked. The fourth sample image is an image of the second sample object obtained by the first camera module. Using the first sample image and the second sample image as input data, and using the third sample image as the first target data to train a first sub-model, to obtain a trained first sub-model, where the first sub-model is used to separate the reflection information of the first sample object on the visible light. Using the output data of the trained first sub-model and the first sample image as input data, and using the fourth sample image as the second target data to train a second sub-model, to obtain a trained second sub-model, where the second sub-model is used to separate the reflection information of the second sample object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

[0134] Figure 1 The connection relationships shown among the various modules are only illustrative and do not constitute a limitation on the connection relationships among the modules of the device 100. Optionally, the various modules of the device 100 may also adopt a combination of various connection methods in the above embodiments.

[0135] The wireless communication function of the device 100 may be implemented by devices such as an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, a modulation and demodulation processor, and a baseband processor.

[0136] The antenna 1 and the antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in the device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be multiplexed to improve the utilization rate of the antennas. For example: the antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0137] Device 100 can implement the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or change display information.

[0138] The display screen 194 can be used to display images or videos.

[0139] Device 100 can implement the shooting function through an ISP, a camera 193, a video codec, a GPU, a display screen 194, and an application processor, etc.

[0140] The ISP is used to process the data fed back by the camera 193. For example, when taking a photo, the shutter is opened, and light passes through the lens and is transmitted to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing and converts it into an image visible to the naked eye. The ISP can perform algorithm optimization on the noise, brightness, and color of the image. The ISP can also optimize parameters such as the exposure and color temperature of the shooting scene. In some embodiments, the ISP can be provided in the camera 193.

[0141] The camera 193 is used to capture static images or videos. An object generates an optical image through the lens and projects it onto the photosensitive element. The photosensitive element can be a charge coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal and then transmits the electrical signal to the ISP to convert it into a digital image signal. The ISP outputs the digital image signal to the DSP for processing. The DSP converts the digital image signal into a standard red green blue (RGB), YUV, etc. format image signal. In some embodiments, device 100 may include 1 or N cameras 193, where N is a positive integer greater than 1.

[0142] The digital signal processor is used to process digital signals. In addition to processing digital image signals, it can also process other digital signals. For example, when device 100 selects a frequency point, the digital signal processor is used to perform Fourier transform on the frequency point energy, etc.

[0143] Video codecs are used to compress or decompress digital videos. Device 100 may support one or more video codecs. In this way, device 100 can play or record videos in multiple encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, and MPEG4.

[0144] The gyroscope sensor 180B can be used to determine the motion posture of device 100. In some embodiments, the angular velocity of device 100 around three axes (i.e., the x-axis, y-axis, and z-axis) can be determined by the gyroscope sensor 180B. The gyroscope sensor 180B can be used for anti-shake during shooting. For example, when the shutter is pressed, the gyroscope sensor 180B detects the angle of jitter of device 100, calculates the distance that the lens module needs to compensate according to the angle, and makes the lens offset the jitter of device 100 through reverse movement to achieve anti-shake. The gyroscope sensor 180B can also be used in scenarios such as navigation and motion-sensing games.

[0145] The acceleration sensor 180E can detect the magnitude of the acceleration of device 100 in various directions (generally the x-axis, y-axis, and z-axis). When device 100 is stationary, the magnitude and direction of gravity can be detected. The acceleration sensor 180E can also be used to identify the posture of device 100 and serve as an input parameter for application programs such as horizontal and vertical screen switching and pedometers.

[0146] The distance sensor 180F is used to measure distance. Device 100 can measure distance through infrared or laser. In some embodiments, for example, in a shooting scenario, device 100 can use the distance sensor 180F to measure distance to achieve rapid focusing.

[0147] The ambient light sensor 180L is used to sense the ambient light brightness. Device 100 can adaptively adjust the brightness of the display screen 194 according to the sensed ambient light brightness. The ambient light sensor 180L can also be used to automatically adjust the white balance during photography. The ambient light sensor 180L can also cooperate with the proximity light sensor 180G to detect whether device 100 is in a pocket to prevent accidental touch.

[0148] The fingerprint sensor 180H is used to collect fingerprints. Device 100 can use the collected fingerprint characteristics to implement functions such as unlocking, accessing application locks, taking pictures, and answering incoming calls.

[0149] The touch sensor 180K, also known as a touch control device. The touch sensor 180K can be disposed on the display screen 194, and the touch sensor 180K and the display screen 194 together form a touch screen, which is also known as a touch panel. The touch sensor 180K is used to detect touch operations acting on or near it. The touch sensor 180K can transmit the detected touch operations to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In some other embodiments, the touch sensor 180K can also be disposed on the surface of the device 100 and at a different position from the display screen 194.

[0150] The hardware system of the device 100 has been described in detail above. Next, the software system of the image device 100 will be introduced.

[0151] Figure 2 It is a schematic diagram of the software system of the device provided by the embodiments of the present application.

[0152] As Figure 2 shown, the system architecture may include an application layer 210, an application framework layer 220, a hardware abstraction layer 230, a driver layer 240, and a hardware layer 250.

[0153] The application layer 210 may include application programs such as a camera application, a gallery, a calendar, a call, a map, a navigation, a WLAN, a Bluetooth, a music, a video, a short message, etc.

[0154] The application framework layer 220 provides application programming interfaces (APIs) and programming frameworks for the application programs in the application layer; the application framework layer may include some predefined functions.

[0155] For example, the application framework layer 220 may include a camera access interface; the camera access interface may include a camera management and a camera device. Among them, the camera management can be used to provide an access interface for managing the camera; the camera device can be used to provide an interface for accessing the camera.

[0156] The hardware abstraction layer 230 is used to abstract the hardware. For example, the hardware abstraction layer may include a camera abstraction layer and other hardware device abstraction layers; the camera hardware abstraction layer can call the algorithms in the camera algorithm library.

[0157] For example, the camera algorithm library may include software algorithms for image processing.

[0158] The driver layer 240 is used to provide drivers for different hardware devices. For example, the driver layer may include a camera device driver; a digital signal processor driver, a graphics processor driver, or a central processor driver.

[0159] The hardware layer 250 may include a camera device and other hardware devices.

[0160] For example, the hardware layer 250 includes a camera device, a digital signal processor, a graphics processor, or a central processing unit; Exemplarily, an image signal processor may be included in the camera device, and the image signal processor may be used for image processing.

[0161] Exemplarily, the training method of the image de-reflection model provided in the embodiments of the present application may be executed in a digital signal processor, a graphics processor, or a central processing unit; or in other arithmetic hardware of the electronic device.

[0162] Currently, the de-reflection algorithm based on a single image is usually sampled. For example, a large number of non-reflective images and aliased images (for example, synthetic images of non-reflective images and reflective images) are used as sample data to train the image de-reflection model; Since the shooting scenes of the sample data of the de-reflection algorithm based on a single image are limited; When the trained neural network processes shooting scenes not included in the sample data, the effect of image de-reflection is poor; In other words, the generalization ability of the image de-reflection model obtained by training with a single image is poor.

[0163] In view of this, the present application provides a method for image de-reflection and a training method for an image de-reflection model. In the embodiments of the present application, an electronic device includes a first camera module and a second camera module, and images of a shooting scene are obtained through the first camera module and the second camera module; Since the spectral range received by the first camera module is the visible light region (for example, 400nm - 700nm), and the spectral range received by the second camera module is the visible light region and the near-infrared light region (for example, 400nm - 1000nm); Therefore, the spectral information of the two camera modules can be complementary, and more information of the shooting scene can be obtained; The method for image de-reflection provided by the embodiments of the present application can improve the effect of image reflection and the clarity of the de-reflected image. In addition, an infrared flash is included in the electronic device in the embodiments of the present application. Since the on state and off state of the infrared flash are invisible to the user; Therefore, the method for image de-reflection in the embodiments of the present application can be executed without the user's perception, and the specular reflection in the image can be removed, improving the effect of image de-reflection.

[0164] In the embodiments of the present application, training data is obtained through a first camera module, a second camera module, and an infrared flash; when the infrared flash is turned on, a first sample Raw image is obtained; when the infrared flash is turned off, a second sample Raw image is obtained; by subtracting the first sample Raw image from the second sample Raw image, the optical path information related only to the dorsal object (target object) is obtained; by training the image de-reflection model using the information of the dorsal object as prior information, the generalization ability of the image de-reflection model can be improved.

[0165] The following Figure 3 illustrates an application scenario of the image de-reflection method provided in the embodiments of the present application.

[0166] Application scenario: photography field

[0167] The method for removing specular reflection in an image of the present application can be applied to the photography field or the video recording field; for example, when shooting through glass, Figure 3 in (a), the image de-reflection method of the present application is not adopted, and the preview image of the shooting object 260 is directly obtained by the main camera module of the terminal device; Figure 3 in (b), the preview image of the shooting object 260 is obtained by the image de-reflection method provided in the embodiments of the present application; Figure 3 Compared with the preview image shown in Figure 3 in (a), the preview image shown in (b) can effectively reduce the reflection spots in the image, etc.; therefore, by the image de-reflection method of the embodiments of the present application, the specular reflection in the image can be removed, and the effect of image de-reflection can be improved.

[0168] It should be understood that the above is an example illustration of the application scenario and does not impose any limitation on the application scenario of the present application.

[0169] The following Figures 4 to 10 describes in detail the image de-reflection method provided in the embodiments of the present application.

[0170] First, Figure 4 describes the optical path principle of image shooting through a transparent glass. Figure 4 shows an optical path schematic diagram of image shooting through a transparent glass.

[0171] It should be understood that as Figure 4The transparent glass shown refers to the glass that allows transmission of light; the back object refers to the target object to be photographed, which is located on the back side of the transparent glass; the front object can refer to the front scene. When the position of the transparent glass is replaced with a mirror, any object that appears in the mirror can be regarded as the front scene. For example, the front object can refer to an object placed in front of the transparent glass, or the sky in the shooting scene, or the user in the shooting scene, etc. Since the user photographs the back object through the transparent glass, the specular reflection light and the transmitted light existing in the transparent glass will be mixed together, resulting in the masking of some detailed information of the back object. Therefore, there will be a specular reflection area in the image of the back object obtained. Through the image de-reflection method in the embodiments of the present application, the specular reflection area in the image can be removed, and the de-reflection effect of the image can be improved.

[0172] Exemplarily, as Figure 4 shown in (a) of, the back object refers to the target object to be photographed, that is, the target object is on the back side of the transparent glass; when the infrared flash is turned on, the light in the shooting environment can include ambient light and the infrared flash. Since the ambient light is divergent, the ambient light can irradiate the front object and the back object. Since the flash has a specific direction, for example, the flash is irradiated towards the target object to be photographed, the infrared flash only irradiates the back object. When the infrared flash is turned on, as Figure 4 shown in (a) of, the optical path obtained in the all-pass module includes three parts, namely, the infrared light (NIR) reflected by the back object, the ambient light (RGB + NIR) reflected by the back object, and the ambient light (RGB + NIR) reflected by the front object.

[0173] It should be noted that the spectral range that the main camera module can receive is: 400nm to 700nm, and the main camera module usually captures RGB images. Since the spectral range that the all-pass module can receive is: 400nm to 1000nm, the above RGB + NIR can be used to represent the spectral range in the ambient light received by the all-pass module.

[0174] Exemplarily, as Figure 4 shown in (b) of, when the infrared flash is turned off, the optical path obtained in the all-pass module includes two parts, namely, the ambient light (RGB + NIR) reflected by the back object and the ambient light (RGB + NIR) reflected by the front object. By subtracting the optical path diagram shown in (b) of Figure 4 from the optical path diagram shown in (a) of Figure 4 the infrared light (NIR) reflected by the back object can be obtained.

[0175] It should be noted that Figure 4 the optical path diagram shown in (c) of includes only the optical path information related to the back object; that is, byFigure 4 The optical path diagram shown in (c) in [] can obtain information related only to the back object, i.e., the target shooting object.

[0176] For example, two images of the back object can be obtained by turning on and off an infrared flash and controlling the exposure conditions to be consistent; by subtracting the two frames of images, image information related only to the back object can be obtained; in the embodiments of the present application, by using the information of the back object as prior information to train the image de-reflection model, the generalization ability of the image de-reflection model is significantly improved. The following combines [] Figure 5 to describe in detail the training method of the image de-reflection model.

[0177] Figure 5 is a schematic diagram of the training method of the image de-reflection model provided by the embodiments of the present application.

[0178] It should be understood that [] Figure 5 the training method of the image de-reflection model shown in [] can be executed by a training device, and the training device can specifically be [] Figure 1 the device shown in [], or can also be executed by [] Figure 2 a digital signal processor, a graphics processor or a central processor shown in [] Figure 5 the training method shown in [] includes steps S301 to S310, and the following will describe steps S301 to S310 in detail.

[0179] It should be understood that in the embodiments of the present application, the shooting scene is as [] Figure 4 shown, and the shooting scene for obtaining training data includes a front object (an example of the first sample object), a transparent glass, and a back object (an example of the second sample object). The front object and the back object are located on both sides of the transparent glass; the electronic device and the front object are on the same side of the transparent glass; the front object can refer to a front scene, that is, when the position of the transparent glass is replaced with a mirror, any object that appears in the mirror can be regarded as a front scene. For example, the front object can refer to an object placed in front of the transparent glass, or the sky in the shooting scene, or the user in the shooting scene, etc.; since the user shoots the back object through the transparent glass, the specular reflection light and the transmitted light of the transparent glass will be mixed together, resulting in some detailed information of the back object being masked. Therefore, there will be a reflective area in the image of the back object obtained; the purpose of the image de-reflection model is to remove the information of the front object included in the image of the back object.

[0180] Step S301: Obtain a first sample Raw image (an example of the fifth sample image).

[0181] Among them, the first sample Raw image refers to the Raw image collected by the all-pass module when the infrared flash is turned on.

[0182] For example, as shown in (a) of Figure 4 The first sample Raw image refers to the Raw image collected by the all-pass module in the direction towards the glass when the infrared flash is on. The first sample Raw image can be expressed as

[0183] It should be noted that in the embodiments of the present application, the spectral range that the main camera module (an example of the first camera module) can receive is: 400nm to 700nm, and the main camera module usually collects RGB images; the spectral range that the all-pass module (an example of the second camera module) can receive is: 400nm to 1000nm.

[0184] Step S302, obtain a second sample Raw image (an example of the sixth sample image).

[0185] Among them, the second sample Raw image refers to the Raw image collected by the all-pass module when the infrared flash is off.

[0186] For example, as shown in (b) of Figure 4 The second sample Raw image refers to the Raw image collected by the all-pass module in the direction towards the glass when the infrared flash is off. The second sample Raw image can be expressed as

[0187] Exemplarily, when obtaining the first sample Raw image and the second sample Raw image, the exposure conditions of the electronic device can be controlled to be consistent.

[0188] It should be noted that in the embodiments of the present application, the Raw image collected by the all-pass module refers to a single-channel image; the Raw image collected by the all-pass module is used to represent the intensity information of photons superimposed together; for example, the Raw image collected by the all-pass module can be a single-channel grayscale image.

[0189] Step S303, obtain a first sample RGB image (an example of the first sample image).

[0190] Among them, the first sample RGB image is the RGB image collected by the main camera module, and the first sample RGB image can be expressed as I ambinent .

[0191] It should be understood that the images obtained in step S301, step S302, and step S303 are images of the same shooting object in the same scene.

[0192] It is also understood that the embodiments of the present application do not make any limitation on the sequence of steps S301 to S303.

[0193] Exemplarily, the images obtained in the above steps S301 to S303 are training data; when training the image de-reflection model, target data also needs to be obtained; among them, the image de-reflection model may include a first sub-model and a second sub-model, and the target data of the first sub-model is the first target data. The first target data may refer to the RGB image of the front object obtained by the main camera module under the condition of isolating the transmitted light of the glass. The first target data can be expressed as R ambinent ; for example, the RGB image of the front object taken by the main camera module and processed by the ISP after covering the glass with a black cloth. This RGB image can be regarded as an image of pure reflected light; the target data of the second sub-module is the second target data, and the second target data may refer to the RGB image of the back object (i.e., the target object) obtained by the main camera module when the glass is removed. The second target data can be expressed as T ambinent ; for example, the RGB image of the back object taken by the main camera module and processed by the ISP after removing the glass.

[0194] Step S304: Subtract the first sample Raw image from the second sample Raw image to obtain a third sample Raw image (an example of the seventh sample image).

[0195] Exemplarily, the first sample Raw image and the second sample Raw image can be two consecutive frames of images, and the pixels between them are basically aligned; the exposure conditions (for example, exposure time and iso) can be exactly the same; the first sample Raw image and the second sample Raw image can represent the original signals output by the sensor, that is, without any non-linear transformation.

[0196] It should be understood that the third sample Raw image can represent the Raw image of the back object only under the illumination of the infrared flash. This image only contains the information of the transmitted light of the back object to the infrared flash.

[0197] It should also be understood that since the Raw image collected by the all-pass module is used to represent the intensity information of photons superimposed together; for example, the Raw image collected by the all-pass module can be a single-channel grayscale image; therefore, the third Raw image refers to a grayscale image.

[0198] For example, the third Raw image can be obtained by subtracting the first sample Raw image from the second sample Raw image pixel by pixel; that is Among them, represents the first sample Raw image, that is, the Raw image collected by the all-pass module in the direction towards the glass when the infrared flash is turned on; Represents the second sample Raw image, that is, the Raw image collected by the all-pass module in the direction towards the glass with the infrared flash turned off; Represents the third sample Raw image, that is, the Raw image of the dorsal object under the illumination of only the infrared flash.

[0199] Step S305: Perform image processing on the third sample Raw image through ISP to obtain the first sample grayscale image, and the first sample grayscale image can be denoted as I flash-only .

[0200] Exemplarily, the above image processing may include but is not limited to:

[0201] Black level correction (BLC), lens shading correction (LSC), gamma processing, denoising processing, contrast enhancement processing, etc.

[0202] It should be understood that due to the different positions of the main camera module and the all-pass module in the terminal device, there is a certain baseline distance between the main camera module and the all-pass module; therefore, there is a certain parallax between the images collected by the main camera module and the images collected by the all-pass module, and it is necessary to register the images collected by both.

[0203] Step S306: Perform registration processing on the first sample grayscale image based on the first sample RGB image to obtain the registered second sample grayscale image (an example of the second sample image), and the second sample grayscale image can be denoted as

[0204] Exemplarily, by performing a certain homography transformation on the first sample grayscale image, the second sample grayscale image is generated; for example, the first sample grayscale image can be transformed through algorithms such as the shift algorithm and the surf algorithm to generate the second sample grayscale image aligned with the image output by the main camera module.

[0205] It should be understood that since the field strength of the infrared flash is relatively weak compared to the ambient light, there is significant noise in the second sample grayscale image; in addition, the second sample grayscale image cannot be directly presented to the user, so it is necessary to improve the image quality of the second sample grayscale image through an image de-reflection model.

[0206] Step S307: Extract features from the first sample RGB image and the second sample grayscale image; and perform a splicing operation (contact) on the extracted image features, and use the image features after the splicing operation as the input data of the first sub-model, and use the first target data as the target value of the first sub-model to train the first sub-model.

[0207] For example, the first target data (an example of the third sample image) refers to an RGB image of the front object obtained by the main camera module under the condition of isolating the transmitted light of the glass; for example, an RGB image of the front object taken by the main camera module after covering the glass with a black cloth and processed by the ISP, and this RGB image can be regarded as an image of pure reflected light.

[0208] It should be understood that the input data of the first sub-model is the first sample RGB image and the second sample grayscale image. The first sample RGB image includes the reflected light of the object in front of the glass plate and the information of the reflection of the transmitted light by the object on the back side of the glass plate (the target object). The second sample grayscale image includes the information of the reflection of the transmitted light of the infrared flash by the object on the back side (the target object). The first sub-model can learn to separate the information of the second sample grayscale image from the first sample RGB image through a large number of sample data, that is, learn to separate the information including only the reflected light of the object in front of the glass plate from the information including the reflected light of the object in front of the glass plate and the information of the reflection of the transmitted light by the object on the back side of the glass plate (the target object).

[0209] Step S308: Perform a splicing operation on the first sample RGB image and the second sample grayscale image to obtain a first image feature; use the first image feature as input data and input it into the first sub-model to obtain a first predicted image.

[0210] Exemplarily, according to the difference between the first predicted image and the first target data, the parameters of the first sub-model are iterated through the backpropagation algorithm until the parameters of the first sub-model converge, and the trained first sub-model is obtained.

[0211] Optionally, the first sub-model is a convolutional neural network; for example, the first sub-model is a U-shaped network structure.

[0212] Step S309: Perform a splicing operation on the first sample RGB image and the first predicted image to obtain a second image feature; use the second image feature as input data and input it into the second sub-model, and use the second target data as the target value of the second sub-model.

[0213] For example, the second target data (an example of the fourth sample image) refers to an RGB image of the object on the back side (i.e., the target object) obtained by the main camera module when the glass is removed; for example, an RGB image of the object on the back side taken by the main camera module after removing the glass and processed by the ISP.

[0214] Step S310: The second sub-model outputs a second predicted image.

[0215] It should be understood that the input data of the second sub-model is the first sample RGB image and the first predicted image. The first sample RGB image includes the reflected light of the object in front of the glass plate and the information of the reflected light of the transmitted light by the object (target object) on the back side of the glass plate. The first predicted image refers to the image of the reflected light of the object in front of the glass plate. The second sub-model can learn to separate the first predicted image from the first sample RGB image through a large number of sample data, that is, learn to separate the information that only includes the reflected light of the transmitted light by the object (target object) on the back side of the glass plate from the information that includes the reflected light of the object in front of the glass plate and the information of the reflected light of the transmitted light by the object (target object) on the back side of the glass plate.

[0216] Exemplarily, the parameters of the second sub-model are iterated according to the difference between the second predicted image and the second target data through the backpropagation algorithm until the parameters of the second sub-model converge, and the trained second sub-model is obtained.

[0217] Optionally, the second sub-model is a convolutional neural network; for example, the second sub-model is a U-shaped network structure.

[0218] It should be noted that the first sub-model and the second sub-model can refer to two sub-models in the image de-reflection model; the two sub-models are cascaded models, that is, the input data is first processed by the first sub-model; the output data of the first sub-model can be used as the input data of the second sub-model; the entire network training of the image de-reflection model can adopt a segmented training method. For example, the first sub-model is trained first; after the first sub-model converges, the weights of the first sub-model can be fixed, and then the second sub-model is trained.

[0219] In the embodiments of the present application, training data is obtained through the main camera module, the all-pass module, and the infrared flash; when the infrared flash is turned on, the first sample Raw image is obtained; when the infrared flash is turned off, the second sample Raw image is obtained; by subtracting the first sample Raw image from the second sample Raw image, the optical path information only related to the object on the back side (target object) is obtained; by using the information of the object on the back side as the prior information to train the image de-reflection model, the generalization ability of the image de-reflection model can be improved.

[0220] In the embodiments of the present application, through Figure 5 The image de-reflection model obtained by the training method shown can be applied to image de-reflection processing to remove the specular reflection in the image; the following combines Figure 6 The image de-reflection method provided by the embodiments of the present application is described in detail.

[0221] Implementation method 1

[0222] In one example, when the terminal device acquires the current frame image, it can automatically identify the scene classification information of the current frame image; and identify whether there is a glass mirror surface in the scene. If there is a glass mirror surface in the current frame image, the terminal device can automatically execute the image anti-reflection method provided in the embodiments of the present application.

[0223] Figure 6 It is a schematic diagram of the image anti-reflection method provided in the embodiments of the present application. Figure 6 The method shown can be executed by Figure 1 the device shown (for example, an electronic device), or by a chip configured in Figure 1 the device shown; Figure 6 The image anti-reflection method 400 shown includes steps S401 to S411, and the following will describe steps S401 to S411 in detail.

[0224] It should be noted that the shooting scene involved in the embodiments of the present application includes a front object (an example of the first shooting object), a transparent glass, and a target object (an example of the second shooting object). The front object and the target object are located on both sides of the transparent glass; the terminal device and the front object are on the same side of the transparent glass; the front object can refer to the front scenery. When replacing the position of the transparent glass with a mirror, any object appearing in the mirror can be regarded as the front scenery. For example, the front object can refer to an object placed in front of the transparent glass, or the sky in the shooting scene, or the user in the shooting scene, etc.; since the user shoots the target object through the transparent glass, the specular reflection light and the transmitted light existing in the transparent glass will be mixed together, resulting in the masking of some detailed information of the target object. Therefore, there will be a reflective area in the acquired image of the target object; the purpose of image anti-reflection is to remove the information of the front object included in the image of the target object.

[0225] Step S401, acquire the current frame image.

[0226] Optionally, the electronic device displays a camera interface; after detecting that the user clicks on the control indicating shooting, the electronic device acquires the current frame image of the shooting scene.

[0227] Step S402, acquire the scene classification information of the current frame image.

[0228] Exemplarily, the scene classification information may include but is not limited to: green plant scene, portrait scene, indoor scene, or outdoor scene, etc.

[0229] Optionally, a scene classification neural network can be trained with a large amount of sample data; the scene classification neural network can identify the scene classification information of the current frame image and output the scene label corresponding to the current frame image.

[0230] Step S403: Determine whether there is glass mirror reflection in the current scene. If there is glass mirror reflection, execute Step S404; if there is no glass mirror reflection, execute Step S411.

[0231] Exemplarily, a binary classification neural network can be learned through a large number of sample data, and the binary classification neural network can identify whether there is glass mirror reflection in the captured scene. For example, the sample data can include a first sample image and a second sample image, and the first sample image and the second sample image are images belonging to the same classification scene; the first sample image includes the image information of the second sample image and the information of the transparent glass.

[0232] Step S404: Obtain the RGB image captured by the main camera module (an example of the first image) and the first Raw image captured by the all-pass module (an example of the second image).

[0233] Wherein, the first Raw image refers to the Raw image captured by the all-pass module when the infrared flash of the electronic device is turned off.

[0234] Step S405: Turn on the infrared flash.

[0235] In the embodiments of the present application, the electronic device includes an infrared flash. Since the on state and off state of the infrared flash are invisible to the user, the method for removing image reflection in the embodiments of the present application can be executed without the user's awareness, which can remove the mirror reflection in the image and improve the effect of removing image reflection.

[0236] Step S406: Obtain the second Raw image captured by the all-pass module (an example of the third image).

[0237] Wherein, the second Raw image refers to the Raw image captured by the all-pass module when the infrared flash is turned on.

[0238] Step S407: Turn off the infrared flash.

[0239] Optionally, after executing Step S406, Step S408 can also be directly executed.

[0240] Step S408: Obtain a third Raw image (an example of the sixth image) according to the first Raw image and the second Raw image.

[0241] For example, the third Raw image is obtained by subtracting the pixels of the first Raw image and the second Raw image.

[0242] It should be understood that the first Raw image and the second Raw image are images of two consecutive frames, which are respectively collected with the infrared flash off and on, and the pixels between the two frames are basically aligned; by subtracting the pixels of the two frames, an image of the dorsal object illuminated only by the infrared flash can be obtained, such as Figure 4 shown in (c) of

[0243] It should also be understood that in the embodiments of the present application, the Raw image collected by the all-pass module refers to a single-channel image; the Raw image collected by the all-pass module is used to represent the intensity information of photons superimposed together; for example, the Raw image collected by the all-pass module can be a single-channel grayscale image.

[0244] Step S409: Perform image processing and registration processing on the third Raw image to obtain a second grayscale image (an example of the fourth image).

[0245] Exemplarily, the third Raw image can be processed by ISP to obtain a first grayscale image; the first grayscale image is registered with the first RGB image as a reference to obtain a second grayscale image.

[0246] Step S410: Input the first RGB image and the second grayscale image into the trained image de-reflection model to obtain a de-reflected RGB image.

[0247] Exemplarily, the image de-reflection model includes a first sub-model and a second sub-model; input the first RGB image and the second grayscale image into the image de-reflection model, and the first predicted RGB image can be obtained through the processing of the first sub-model, and the first predicted RGB image includes the reflection information of the front object to visible light; input the first predicted image and the first RGB image into the second sub-model to obtain a de-reflected RGB image; wherein, the training method of the image de-reflection model can be referred to Figure 5 as shown, and will not be elaborated here.

[0248] Step S411: Output the processed image; for example, when there is specular reflection in the image, output the de-reflected RGB image (an example of the fifth image).

[0249] In an embodiment of the present application, images of a shooting scene are obtained through a main camera module and a full-pass module; since the spectral range received by the main camera module is the visible light region (for example, 400nm to 700nm), and the spectral range received by the full-pass module is the visible light region and the near-infrared light region (for example, 400nm to 1000nm); therefore, the spectral information of the two camera modules can be complementary, and more information of the shooting scene can be obtained; the image anti-reflection method provided by the embodiment of the present application can improve the anti-reflection effect of the image and improve the clarity of the anti-reflection image.

[0250] Implementation method two

[0251] In one example, when shooting the current frame image, the user can identify whether there is specular reflection in the shooting scene; when there is specular reflection in the scene, the user can turn on the anti-reflection mode of the electronic device; after the electronic device recognizes that the user has turned on the anti-reflection mode, the method for removing specular reflection in the image according to the embodiment of the present application is executed.

[0252] For example, as Figure 6 shown, after the electronic device detects that the user has turned on the anti-reflection mode, it can execute steps S404 to S411; for specific descriptions, reference can be made to Figure 6 the description shown, which will not be elaborated here.

[0253] Figure 7 is a schematic diagram of the effect of the image anti-reflection method provided by the embodiment of the present application.

[0254] As Figure 7 shown, when there is specular reflection in the shooting scene, Figure 7 the image shown in (a) in Figure 7 is the image taken by the main camera module; Figure 7 the image in (b) in Figure 7 is the image processed by the image anti-reflection method according to the embodiment of the present application; compared with the image shown in (a) in

[0255]

[0256] Figure 8 Exemplarily, when there is glass in the shooting scene, the user can turn on the anti-reflection mode in the camera application of the electronic device, and then the electronic device can perform anti-reflection processing on the collected image through the image anti-reflection method provided by the embodiment of the present application, so as to output the processed image or video.Disclosed is a graphical user interface (GUI) of an electronic device.

[0257] As Figure 8 shown in (a) of [], the GUI can be the display interface of the camera application in the photo-taking mode. The display interface may include a shooting interface 510; the shooting interface 510 may include a viewfinder 511 and controls; for example, the viewfinder 511 may include a control 512 for indicating photo-taking and a control 513 for indicating settings; upon detecting the user's operation of clicking the control 513, a settings interface is displayed in response to the user's operation, as Figure 8 shown in (b) of []; the settings interface includes an anti-reflection mode 514, and it is detected that the user enables the anti-reflection mode; after the electronic device enables the anti-reflection mode, the method for image anti-reflection provided in the embodiments of the present application can be executed.

[0258] Optionally, as Figure 9 shown in (a) of [], the shooting interface 510 may include a control 513 for indicating settings; upon detecting the user's operation of clicking the control 513, a settings interface is displayed in response to the user's operation, as Figure 9 shown in (b) of []; the settings interface includes an artificial intelligence (AI) anti-reflection mode control 515, and it is detected that the user enables the AI anti-reflection mode space 515; after the user enables the AI anti-reflection mode, the user does not need to manually select to enable the anti-reflection mode; that is, the electronic device can automatically identify whether there is glass in the shooting scene. If there is glass in the shooting scene, the electronic device automatically enables the anti-reflection mode; after enabling the anti-reflection mode, the method for image anti-reflection provided in the embodiments of the present application can be executed.

[0259] Optionally, as Figure 10 shown, in the photo-taking mode, the shooting interface 510 may further include a control 516 for indicating to turn on / off the anti-reflection mode; after the electronic device detects the user's operation of clicking the control 516, the electronic device can enable the anti-reflection mode and execute the method for image anti-reflection provided in the embodiments of the present application.

[0260] In an embodiment of the present application, images of a shooting scene are acquired by a first camera module and a second camera module; since the spectral range received by the first camera module is the visible light region (for example, 400 nm to 1000 nm), and the spectral range received by the second camera module is the visible light region and the near-infrared light region (for example, 780 nm to 1000 nm); therefore, the spectral information of the two camera modules can be complementary, and more information of the shooting scene can be obtained; the image anti-reflection method provided by the embodiment of the present application can improve the anti-reflection effect of the image and the clarity of the anti-reflection image. In addition, in the embodiment of the present application, the electronic device includes an infrared flash, and since the on state and the off state of the infrared flash are invisible to the user; therefore, the image anti-reflection method of the embodiment of the present application can be executed without the user's perception, and the anti-reflection effect of the image can be improved.

[0261] It should be understood that the above examples are for helping those skilled in the art to understand the embodiments of the present application, rather than limiting the embodiments of the present application to the specific numerical values or specific scenarios illustrated. Those skilled in the art can clearly make various equivalent modifications or changes according to the above examples, and such modifications or changes also fall within the scope of the embodiments of the present application.

[0262] The above is combined with Figures 1 to 10 The training method of the image anti-reflection model and the image anti-reflection method provided by the embodiments of the present application are described in detail; the device embodiments of the present application will be described in detail below in combination with Figures 11 to 13 The device embodiments of the present application are described in detail. It should be understood that the devices in the embodiments of the present application can execute various methods of the foregoing embodiments of the present application, that is, the specific working processes of the following various products can refer to the corresponding processes in the foregoing method embodiments.

[0263] Figure 11 FIG. is a schematic structural diagram of a training device for an image anti-reflection model provided by an embodiment of the present application. The training device 700 includes an acquisition module 710 and a processing module 720.

[0264] Among them, the obtaining module 710 is used to obtain training data, where the training data includes a first sample image, a second sample image, a third sample image, and a fourth sample image. The first sample image is an image of a sample shooting scene obtained by a first camera module. The sample shooting scene includes a first sample object, a transparent glass, and a second sample object, and the first sample object and the second sample object are located on both sides of the transparent glass. The second sample image is an image obtained by a second camera module for representing the reflection information of the second sample object on the first transmitted light, and the first transmitted light is obtained by near-infrared light passing through the transparent glass. The spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is the visible light and near-infrared light. The third sample image refers to an image of the first sample object obtained by the first camera module when the transparent glass is blocked. The fourth sample image refers to an image of the second sample object obtained by the first camera module. The processing module 720 is used to use the first sample image and the second sample image as input data, and use the third sample image as the first target data to train a first sub-model to obtain a trained first sub-model. The first sub-model is used to separate the reflection information of the first sample object on the visible light. Using the output data of the trained first sub-model and the first sample image as input data, and using the fourth sample image as the second target data to train a second sub-model to obtain a trained second sub-model. The second sub-model is used to separate the reflection information of the second sample object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

[0265] Optionally, as an embodiment, the obtaining module 710 is specifically used for:

[0266] When the infrared flash is turned on, obtain a fifth sample image of the sample shooting scene through the second camera module;

[0267] When the infrared flash is turned off, obtain a sixth sample image of the sample shooting scene through the second camera module;

[0268] The processing module 720 is used for:

[0269] Obtain the second sample image according to the fifth sample image and the sixth sample image.

[0270] Optionally, as an embodiment, the processing module 720 is used for:

[0271] Subtract the fifth sample image from the sixth sample image to obtain a seventh sample image;

[0272] Register the seventh sample image based on the first sample image to obtain the second sample image.

[0273] Optionally, as an embodiment, the parameters of the first sub-model are iteratively obtained through backpropagation based on the difference between the prediction data output by the first sub-model and the first target data.

[0274] Optionally, as an embodiment, the parameters of the second sub-model are iteratively obtained through backpropagation based on the difference between the prediction data output by the second sub-model and the second target data.

[0275] It should be noted that the above training device 700 is embodied in the form of functional modules. The term "module" here can be implemented in software and / or hardware forms, and no specific limitation is made thereto.

[0276] For example, the "module" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a proprietary processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a merged logic circuit, and / or other suitable components that support the described functions.

[0277] Therefore, the units of the examples described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0278] Figure 12 It is a schematic structural diagram of an image anti-reflection device provided by an embodiment of the present application. The device 800 includes a display module 810 and a processing module 820; the device 800 further includes a first camera module, a second camera module, and an infrared flash. The spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is visible light and near-infrared light.

[0279] Among them, the display module 810 is used to display a first interface, and the first interface includes a first control; the processing module 820 is used to detect a first operation on the first control; in response to the first operation, a first image and a second image are obtained. The first image is an image of a shooting scene obtained by the first camera module, and the second image is an image of the shooting scene obtained by the second camera module. The shooting scene includes a first shooting object, a transparent glass, and a second shooting object, and the first shooting object and the second shooting object are located on both sides of the transparent glass; the infrared flash is turned on to obtain a third image, and the third image is an image of the shooting scene obtained by the first camera module; a fourth image is obtained according to the second image and the third image, and the fourth image is used to represent the reflection information of the second shooting object on the first transmitted light, and the first transmitted light is obtained by the near-infrared light passing through the transparent glass; the first image and the fourth image are input into an image anti-reflection model to obtain a fifth image; wherein, the image anti-reflection model includes a first sub-model and a second sub-model, and the first sub-model is used to separate the reflection information of the first shooting object on the visible light; the second sub-model is used to separate the reflection information of the second shooting object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

[0280] Optionally, as an embodiment, the processing module 820 is further used for:

[0281] Determine the classification information of the shooting scene, and the classification information is used to indicate the scene classification to which the shooting scene belongs;

[0282] Determine that there is specular reflection in the shooting scene according to the classification information.

[0283] Optionally, as an embodiment, the processing module 820 is further used for:

[0284] Detect a second operation, and the second operation is used to indicate that there is specular reflection in the shooting scene.

[0285] Optionally, as an embodiment, the processing module 820 is specifically used for:

[0286] Subtract the second image from the third image to obtain a sixth image;

[0287] Perform registration processing on the sixth image based on the first image to obtain the fourth image.

[0288] Optionally, as an embodiment, the first interface refers to the main screen interface of the electronic device, the main screen interface includes a camera application, and the first control refers to the control corresponding to the camera application.

[0289] Optionally, as an embodiment, the first interface refers to a video recording interface, and the first control refers to a control for indicating video recording.

[0290] Optionally, as an embodiment, the first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

[0291] It should be noted that the above device 800 is embodied in the form of functional modules. The term "module" here can be implemented in software and / or hardware, and no specific limitation is made thereto.

[0292] For example, a "module" can be a software program, a hardware circuit, or a combination of the two that implements the above functions. The hardware circuit may include an application specific integrated circuit (ASIC), an electronic circuit, a processor (such as a shared processor, a dedicated processor, or a group of processors, etc.) for executing one or more software or firmware programs, and a memory, a merged logic circuit, and / or other suitable components that support the described functions.

[0293] Therefore, the units of the examples described in the embodiments of the present application can be implemented by electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are executed in a hardware or software manner depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the present application.

[0294] Figure 13 The structural schematic diagram of an electronic device provided by the present application is shown. Figure 13 The dashed line in represents that the unit or the module is optional; the electronic device 900 can be used to implement the method described in the above method embodiments.

[0295] The electronic device 900 includes one or more processors 901, and the one or more processors 901 can support the electronic device 900 to implement the training method of the image anti-reflection model or the method of image anti-reflection in the method embodiments. The processor 901 can be a general-purpose processor or a dedicated processor. For example, the processor 901 can be a central processing unit (CPU), a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, such as discrete gates, transistor logic devices, or discrete hardware components.

[0296] The processor 901 can be used to control the electronic device 900, execute software programs, and process the data of software programs. The electronic device 900 may further include a communication unit 905 for implementing signal input (reception) and output (transmission).

[0297] For example, the electronic device 900 can be a chip, and the communication unit 905 can be the input and / or output circuit of the chip, or the communication unit 905 can be the communication interface of the chip, and the chip can be a component of a terminal device or other electronic devices.

[0298] Again, for example, the electronic device 900 can be a terminal device, and the communication unit 905 can be the transceiver of the terminal device, or the communication unit 905 can be the transceiver circuit of the terminal device.

[0299] The electronic device 900 may include one or more memories 902, on which there is a program 904. The program 904 can be run by the processor 901 to generate instructions 903, so that the processor 901 executes the training method or the method of image anti-reflection described in the above method embodiments according to the instructions 903.

[0300] Optionally, data may also be stored in the memory 902. Optionally, the processor 901 can also read the data stored in the memory 902. The data can be stored at the same storage address as the program 904, or the data can be stored at a different storage address from the program 904.

[0301] The processor 901 and the memory 902 can be set separately or integrated together. For example, they can be integrated on a system on chip (SOC) of a terminal device.

[0302] Exemplarily, the memory 902 can be used to store the relevant program 904 of the training method of the image de - reflection model provided in the embodiments of the present application. The processor 901 can be used to call the relevant program 904 of the training method of the image de - reflection model stored in the memory 902 when executing the training of the image de - reflection model, and execute the training method of the image de - reflection model in the embodiments of the present application. For example, obtain training data, where the training data includes a first sample image, a second sample image, a third sample image, and a fourth sample image. The first sample image is an image of a sample shooting scene obtained by a first camera module. The sample shooting scene includes a first sample object, a transparent glass, and a second sample object, and the first sample object and the second sample object are located on both sides of the transparent glass. The second sample image is an image obtained by a second camera module for representing the reflection information of the second sample object on the first transmitted light, and the first transmitted light is obtained by near - infrared light passing through the transparent glass. The spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is visible light and near - infrared light. The third sample image is an image of the first sample object obtained by the first camera module when the transparent glass is blocked. The fourth sample image is an image of the second sample object obtained by the first camera module. Using the first sample image and the second sample image as input data, and using the third sample image as the first target data to train a first sub - model, to obtain a trained first sub - model, where the first sub - model is used to separate the reflection information of the first sample object on the visible light. Using the output data of the trained first sub - model and the first sample image as input data, and using the fourth sample image as the second target data to train a second sub - model, to obtain a trained second sub - model, where the second sub - model is used to separate the reflection information of the second sample object on the second transmitted light, and the second transmitted light is obtained by visible light passing through the transparent glass.

[0303] Exemplarily, the memory 902 can be used to store the relevant program 904 of the method for image anti-reflection provided in the embodiments of the present application. The processor 901 can be used to call the relevant program 904 of the method for image anti-reflection stored in the memory 902 when performing image anti-reflection, and execute the method for image anti-reflection of the embodiments of the present application; for example, display a first interface, where the first interface includes a first control; detect a first operation on the first control; in response to the first operation, obtain a first image and a second image, where the first image is an image of a shooting scene obtained by the first camera module, and the second image is an image of the shooting scene obtained by the second camera module. The shooting scene includes a first shooting object, a transparent glass, and a second shooting object, and the first shooting object and the second shooting object are located on both sides of the transparent glass; turn on the infrared flash and obtain a third image, where the third image is an image of the shooting scene obtained by the first camera module; obtain a fourth image according to the second image and the third image, and the fourth image is used to represent the reflection information of the second shooting object on the first transmitted light, and the first transmitted light is obtained by the near-infrared light passing through the transparent glass; input the first image and the fourth image into an image anti-reflection model to obtain a fifth image; where the image anti-reflection model includes a first sub-model and a second sub-model, and the first sub-model is used to separate the reflection information of the first shooting object on the visible light; the second sub-model is used to separate the reflection information of the second shooting object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

[0304] The present application also provides a computer program product, which when executed by the processor 901 implements the training method or the method for image anti-reflection described in any method embodiment of the present application.

[0305] The computer program product can be stored in the memory 902, for example, it is the program 904. After processes such as preprocessing, compilation, assembly, and linking, the program 904 is finally converted into an executable target file that can be executed by the processor 901.

[0306] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by the computer, it implements the training method or the method for image anti-reflection described in any method embodiment of the present application; the computer program can be a high-level language program or an executable target program.

[0307] The computer-readable storage medium is, for example, the memory 902. The memory 902 can be a volatile memory or a non-volatile memory, or the memory 902 can include both a volatile memory and a non-volatile memory. Among them, the non-volatile memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory can be a random access memory (RAM), which is used as an external cache. By way of example but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchlink dynamic random access memory (SLDRAM), and direct rambus random access memory (DR RAM).

[0308] Those skilled in the art can clearly understand that, for the convenience and conciseness of description, the specific working processes and the resulting technical effects of the above-described devices and apparatuses can refer to the corresponding processes and technical effects in the foregoing method embodiments, and will not be described herein again.

[0309] In several embodiments provided in this application, the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, some features of the above-described method embodiments can be ignored or not executed. The above-described apparatus embodiments are merely illustrative. The division of units is only a logical functional division, and there can be other division methods in actual implementation. Multiple units or components can be combined or integrated into another system. In addition, the coupling between units or the coupling between components can be direct coupling or indirect coupling. The above couplings include electrical, mechanical, or other forms of connection.

[0310] It should be understood that in various embodiments of the present application, the magnitude of the serial numbers of the various processes does not imply the order of execution, and the order of execution of the various processes should be determined according to their functions and internal logics, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0311] In addition, the terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is merely a description of the associated relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist simultaneously, and B exists alone. In addition, the character " / " in this article generally represents an "or" relationship between the associated objects before and after.

[0312] In summary, the above are only the preferred embodiments of the technical solution of the present application, and are not used to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. A method for removing specular reflection from an image, characterized in that, Applied to an electronic device, the electronic device includes a first camera module, a second camera module, and an infrared flash. The spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is the visible light and near-infrared light. The method includes: Display a first interface, the first interface including a first control; Detect a first operation on the first control; In response to the first operation, obtain a first image and a second image. The first image is an image of a shooting scene obtained by the first camera module, and the second image is an image of the shooting scene obtained by the second camera module. The shooting scene includes a first shooting object, a transparent glass, and a second shooting object. The first shooting object and the second shooting object are located on both sides of the transparent glass; Turn on the infrared flash and obtain a third image, which is an image of the shooting scene obtained by the first camera module; Obtain a fourth image according to the second image and the third image. The fourth image is used to represent the reflection information of the second shooting object on the first transmitted light. The first transmitted light is obtained by the near-infrared light passing through the transparent glass; Input the first image and the fourth image into an image anti-reflection model to obtain a fifth image; Wherein, the image anti-reflection model includes a first sub-model and a second sub-model. The first sub-model is used to separate the reflection information of the first shooting object on the visible light; the second sub-model is used to separate the reflection information of the second shooting object on the second transmitted light. The second transmitted light is obtained by the visible light passing through the transparent glass.

2. The method according to claim 1, characterized in that It further includes: Determine the classification information of the shooting scene, and the classification information is used to indicate the scene classification to which the shooting scene belongs; Determine that there is specular reflection in the shooting scene according to the classification information.

3. The method according to claim 1, characterized in that, It further includes: Detect a second operation, and the second operation is used to indicate that there is specular reflection in the shooting scene.

4. The method according to any one of claims 1 to 3, characterized in that, The obtaining the fourth image according to the first image and the third image includes: Subtract the second image from the third image to obtain a sixth image; Register the sixth image with the first image as a reference to obtain the fourth image.

5. The method according to any one of claims 1 to 4, characterized in that, The first interface refers to the main screen interface of the electronic device, and the main screen interface includes a camera application. The first control refers to the control corresponding to the camera application.

6. The method according to any one of claims 1 to 4, characterized in that The first interface refers to a video recording interface, and the first control refers to a control for indicating video recording.

7. The method according to any one of claims 1 to 4, characterized in that, The first interface refers to a video call interface, and the first control refers to a control for indicating a video call.

8. A training method for an image anti-reflection model, characterized in that, The image anti-reflection model includes a first sub-model and a second sub-model, including: Obtain training data, where the training data includes a first sample image, a second sample image, a third sample image, and a fourth sample image. The first sample image is an image of a sample shooting scene obtained by a first camera module; the sample shooting scene includes a first sample object, a transparent glass, and a second sample object, and the first sample object and the second sample object are located on both sides of the transparent glass; the second sample image is an image obtained by a second camera module for representing the reflection information of the second sample object on the first transmitted light, and the first transmitted light is obtained by near-infrared light passing through the transparent glass; the spectrum received by the first camera module is visible light, and the spectrum received by the second camera module is the visible light and near-infrared light; the third sample image is an image of the first sample object obtained by the first camera module when the transparent glass is blocked; the fourth sample image is an image of the second sample object obtained by the first camera module. Use the first sample image and the second sample image as input data, and use the third sample image as the first target data to train the first sub-model, obtaining the trained first sub-model, where the first sub-model is used to separate the reflection information of the first sample object on the visible light. Use the output data of the trained first sub-model and the first sample image as input data, and use the fourth sample image as the second target data to train the second sub-model, obtaining the trained second sub-model, where the second sub-model is used to separate the reflection information of the second sample object on the second transmitted light, and the second transmitted light is obtained by the visible light passing through the transparent glass.

9. The training method according to claim 8, characterized in that The obtaining of the training data includes: When the infrared flash is turned on, obtain a fifth sample image of the sample shooting scene through the second camera module; When the infrared flash is turned off, obtain a sixth sample image of the sample shooting scene through the second camera module; Obtain the second sample image according to the fifth sample image and the sixth sample image.

10. The training method according to claim 9, characterized in that, The obtaining of the second sample image according to the fifth sample image and the sixth sample image includes: Subtract the sixth sample image from the fifth sample image to obtain a seventh sample image; Register the seventh sample image based on the first sample image to obtain the second sample image.

11. The training method according to any one of claims 8 to 10, characterized in that The parameters of the first sub-model are obtained by iteration through backpropagation according to the difference between the predicted data output by the first sub-model and the first target data.

12. The training method according to any one of claims 8 to 11, characterized in that, The parameters of the second sub-model are obtained by iteration through backpropagation according to the difference between the predicted data output by the second sub-model and the second target data.

13. An image anti-reflection device, characterized in that, Includes: One or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the device to perform the method according to any one of claims 1 to 7.

14. A training device for an image anti-reflection model, characterized in that, Comprising: One or more processors and a memory; The memory is coupled to the one or more processors, and the memory is configured to store computer program code, the computer program code including computer instructions, and the one or more processors call the computer instructions to cause the training device to perform the training method according to any one of claims 8 to 12.

15. A chip system, characterized in that, The chip system is applied to an electronic device, the chip system includes one or more processors, and the processors are configured to call computer instructions to cause the electronic device to perform the method according to any one of claims 1 to 7, or 8 to 12.

16. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7, or 8 to 12.

17. A computer program product, characterized in that, The computer program product includes computer program code, and when the computer program code is executed by a processor, the processor is caused to perform the method according to any one of claims 1 to 7, or 8 to 12.

Images