Reflection-removing model training method, reflection-removing method and device
By constructing the data of simulation and real reflection pairing, weak supervision training is used to use preset loss function to generate a de-reflective model, solving the problems of manual processing and insufficient data in the existing technology, and achieving efficient and accurate automatic removal of glasses' reflection sex.
Patent Information
- Application Number
- CN202411155948.4
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-22
- Publication Date
- 2024-11-22
AI Technical Summary
Prior art When processing pictures containing glasses reflections, manual manual smearing methods are inconvenient and easily lead to image defects, and deep learning is difficult to obtain sufficient model training data to remove glasses lens reflections in the image.
By constructing simulated reflective pairing data and real reflective pairing data, weak supervision training is used to generate a dereflective model, and the glasses reflection in the picture is automatically removed.
It improves the convenience and accuracy of the removal of the reflective treatment, avoids the defects introduced by manual smear, solves the problem of insufficient real reflective matching data, and realizes automatic removal of glasses.
Smart Images

Figure CN119006340A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of image processing technology, and in particular to a de-reflection model training method, a de-reflection method and a de-reflection device. Background Art
[0002] In daily life, pictures of glasses often reflect light from the glasses, which affects the beauty of the pictures. A common method to deal with this reflection phenomenon in glasses pictures is to manually smear the reflective areas in the pictures using image processing tools such as PS (Photoshop). However, the manual smearing method is not very convenient in actual operation and may also cause more defects in the pictures. Summary of the invention
[0003] Based on this, it is necessary to provide a de-reflection model training method, a de-reflection method and a device to address the above technical problems, which can remove the reflection of glasses in the picture.
[0004] In a first aspect, the present application provides a de-reflection model training method, the method comprising:
[0005] Based on the non-reflective glasses image and the simulated lens reflective image, simulated reflective pairing data is obtained;
[0006] Based on the real reflective glasses image and the real de-reflective glasses image, real reflective pairing data is obtained;
[0007] The simulated reflection pairing data and the real reflection pairing data are randomly input into the model to be trained, and a preset loss function is used for weak supervision training. The parameters of the model to be trained are adjusted based on the output value of the preset loss function to obtain a de-reflection model.
[0008] In one embodiment, obtaining simulated reflective pairing data based on a non-reflective glasses image and a simulated lens reflective image includes:
[0009] Taking the glasses area as the target, the non-reflective glasses image is segmented to obtain the lens mask;
[0010] Based on the simulated reflective area mask, the reflected image and the non-reflective glasses image are fused to obtain a simulated lens reflective image; the simulated reflective area mask is obtained by randomly segmenting the lens mask; the reflected image includes multiple pure color images of different colors;
[0011] The simulated reflection pairing data is obtained according to the non-reflective glasses image, the simulated lens reflection image and the simulated reflection area mask.
[0012] In one embodiment, the simulated reflection pairing data and the real reflection pairing data are randomly input into the model to be trained, and a preset loss function is used for weak supervision training, and the parameters of the model to be trained are adjusted based on the output value of the preset loss function, including:
[0013] Based on the input simulated reflection paired data and real reflection paired data, a predicted reflection area mask and a predicted de-reflected image output by the to-be-trained model are obtained;
[0014] The preset loss function is used to calculate the output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflected image, and the parameters of the to-be-trained model are adjusted based on the output value of the preset loss function.
[0015] In one embodiment, the preset loss function includes a first preset loss function and a second preset loss function, and the preset loss function is used to calculate the output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflected image, including:
[0016] In response to inputting the simulated reflective paired data into the model to be trained, using a first preset loss function, based on the predicted reflective area mask, the predicted de-reflective image, and the simulated reflective paired data, calculate an output value of the first preset loss function;
[0017] In response to inputting the real reflection paired data into the model to be trained, the output value of the second preset loss function is calculated based on the predicted reflective area mask and the predicted de-reflected image and the real reflection paired data using the second preset loss function.
[0018] In one embodiment, the first preset loss function is as follows:
[0019]
[0020] Among them, Loss1 is the output value of the first preset loss function, is the mean square error loss, L1 is the mean absolute error loss, To predict the reflective area mask, To predict the de-reflected image; A simulated reflective area mask corresponding to a simulated lens reflective image; For non-reflective glasses images, is the weight of the mean square error loss in the first preset loss function, and y is the weight of the mean absolute error loss in the first preset loss function.
[0021] In one embodiment, using the second preset loss function, based on the predicted reflective area mask, the predicted de-reflective image, and the real reflective paired data, calculating the output value of the second preset loss function includes:
[0022] Performing difference processing on the real reflective glasses image and the real de-reflective glasses image to obtain a difference image;
[0023] Threshold the difference image to obtain an approximate reflective area mask;
[0024] Based on the real reflection pairing data, the approximate reflection area mask, the predicted reflection area mask and the predicted de-reflection image, the output value of the preset loss function is calculated using the second preset loss function; the second preset loss function is as follows:
[0025]
[0026] Among them, Loss2 is the output value of the second preset loss function, is the mean square error loss, L1 is the mean absolute error loss, To predict the reflective area mask, To predict the de-reflected image; is the approximate reflective area mask corresponding to the real reflective glasses image; For the real de-reflective glasses image, is the weight of the mean square error loss in the second preset loss function, and y is the weight of the mean absolute error loss in the second preset loss function.
[0027] In a second aspect, the present application provides a method for removing reflections, the method comprising:
[0028] Get the image to be processed;
[0029] The reflective glasses image to be processed is input into any de-reflection model of the first aspect to perform de-reflection processing to obtain a de-reflection image.
[0030] In a third aspect, the present application provides a de-reflection model training device, the device comprising:
[0031] A simulated reflective pairing data acquisition module, used to obtain simulated reflective pairing data based on a non-reflective glasses image and a simulated lens reflective image;
[0032] A real reflective pairing data acquisition module, used to obtain real reflective pairing data based on the real reflective glasses image and the real de-reflective glasses image;
[0033] A training module is used to randomly input the simulated reflection pairing data and the real reflection pairing data into a model to be trained, and use a preset loss function to perform weakly supervised training, and adjust the parameters of the model to be trained based on the output value of the preset loss function to obtain a de-reflection model.
[0034] In a fourth aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which implements the steps of the method of any one of the first and second aspects when the computer program is executed by a processor.
[0035] In a fifth aspect, the present application provides an electronic device, including:
[0036] A memory for storing instructions executed by one or more processors of the electronic device;
[0037] And a processor, which is one of the processors of the electronic device, is used to execute instructions stored therein to implement any of the methods described in the first aspect and the second aspect.
[0038] The above-mentioned de-reflection model training method, de-reflection method, de-reflection model training device, readable storage medium, computer program product and electronic device input simulated reflection pairing data and real reflection pairing data into the model to be trained, use a preset loss function to perform weak supervision training, and adjust the parameters of the model to be trained based on the output value of the preset loss function, so as to obtain a de-reflection model. Through the above-mentioned method, an effective de-reflection model training framework is provided, and the obtained de-reflection model can automatically perform de-reflection processing on the input image to be processed, thereby improving the convenience and accuracy of de-reflection processing. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.
[0040] Figure 1 is a flowchart of a de-reflection model training method in one embodiment;
[0041] Figure 2 A schematic diagram of a process for obtaining simulated reflective pairing data in one embodiment;
[0042] Figure 3 A schematic diagram of a process for training a de-reflection model in one embodiment;
[0043] Figure 4 A schematic diagram of a process for obtaining an output value of a preset loss function in one embodiment;
[0044] Figure 5 A schematic diagram of a process for obtaining an output value of a second preset loss function in one embodiment;
[0045] Figure 6 is a schematic diagram of a de-reflection model training process in one embodiment;
[0046] Figure 7 is a flow chart of a de-reflection method in one embodiment;
[0047] Figure 8 is a structural block diagram of a de-reflection training model device in one embodiment;
[0048] Fig. 9 FIG. 4 is a diagram showing the internal structure of an electronic device in one embodiment. DETAILED DESCRIPTION
[0049] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0050] In one embodiment, Figure 1 As shown, a de-reflection model training method is provided. This embodiment uses the method applied to a server as an example for illustration. It can be understood that the method can also be applied to a terminal, and can also be applied to a system including a terminal and a server, and is implemented through the interaction between the terminal and the server. In this embodiment, the method includes the following steps:
[0051] Step S102, obtaining simulated reflective pairing data based on the non-reflective glasses image and the simulated lens reflective image.
[0052] The non-reflective glasses image may refer to an image that includes glasses but is not reflective, and the simulated lens reflection image may refer to an image that artificially adds the reflection features of glasses to the non-reflective glasses image. Based on the non-reflective glasses image and its corresponding artificially constructed simulated lens reflection image, they can be paired to obtain simulated reflection pairing data.
[0053] Specifically, based on the acquired non-reflective glasses images, glasses reflective features are artificially added to these non-reflective glasses images to obtain corresponding simulated lens reflective images. Based on the non-reflective glasses images and their corresponding simulated reflective images, a series of simulated reflective pairing data can be constructed.
[0054] Step S104, obtaining real reflection pairing data based on the real reflection glasses image and the real de-reflection glasses image.
[0055] Among them, the real reflective glasses image may refer to an image containing reflective glasses, and the real de-reflective glasses image may refer to the corresponding image after the de-reflective processing of the above-mentioned real reflective glasses image. Exemplarily, taking the camera shooting glasses image as an example, first shoot the real reflective glasses image containing reflective glasses, and then add (or enable) a polarizer to the camera to remove the image reflection, so as to obtain the real de-reflective glasses image corresponding to the above-mentioned real reflective glasses image. In the above manner, a set of real reflection pairing data including the real reflective glasses image and its corresponding real de-reflective glasses image can be obtained. It can be understood that due to external factors, the size or ratio of the above-mentioned real reflective glasses image (the image before shooting with the polarizer) and its corresponding real de-reflective glasses image (the image after shooting with the polarizer) may be different to a certain extent, and the ratio and size of the real reflective glasses image and its corresponding real de-reflective glasses image can be made consistent by manually adjusting the alignment.
[0056] Specifically, based on the acquired real reflective glasses image and its corresponding real de-reflective glasses image, real reflective pairing data can be obtained.
[0057] Step S106, randomly inputting the simulated reflection pairing data and the real reflection pairing data into the model to be trained, and performing weak supervision training using a preset loss function, adjusting the parameters of the model to be trained based on the output value of the preset loss function, and obtaining a de-reflection model.
[0058] Among them, the output value of the preset loss function can represent the difference between the predicted value output by the model to be trained and the true value. It can be understood that the smaller the difference between the predicted value output by the model to be trained and the true value, the closer the predicted output result of the model to be trained is to the ideal de-reflected image result. Furthermore, based on the output value of the preset loss function, the model parameters of the model to be trained can be updated and adjusted during the weakly supervised training to obtain a de-reflection model. After inputting the image to be processed into the de-reflection model, the de-reflection model can automatically remove the reflection of the glasses on the image to be processed and output an ideal de-reflected image.
[0059] Specifically, according to the simulated reflection pairing data and the real reflection pairing data, the model to be trained can output the corresponding prediction value; the preset loss function can calculate the output value of the preset loss function based on the above prediction value, the input simulated reflection pairing data and the real reflection pairing data; based on the output value of the preset loss function, the parameters of the model are updated and adjusted, so as to reduce the difference between the prediction result output by the model to be trained and the most ideal true value during the weak supervision training process, and thus obtain an ideal de-reflection model.
[0060] In the above-mentioned de-reflection model training method, simulated reflection pairing data can be obtained based on the non-reflective glasses image and the simulated lens reflection image, and real reflection pairing data can be obtained based on the real reflective glasses image and the real de-reflection glasses image; the simulated reflection pairing data and the real reflection pairing data are randomly input into the model to be trained for weak supervision training, and the model parameters of the model to be trained are adjusted based on the output value of the preset loss function, so as to obtain the de-reflection model.
[0061] In the conventional technology, due to the small number of real reflective glasses images and corresponding real de-reflective glasses images, it is difficult to obtain sufficient model training data, which makes it difficult to remove the reflection of glasses lenses in images based on deep learning. Compared with the conventional technology, this embodiment solves the problem of small amount of real reflective paired data in the conventional technology by fully constructing simulated reflection paired data, and provides an effective de-reflective model training framework.
[0062] In addition, compared to the traditional technology of adding polarizing filters to mobile phones or cameras to weaken or eliminate reflections, the de-reflection model obtained by weak supervision training in this embodiment can automatically remove the reflection of glasses from pictures taken by mobile phones or cameras, greatly improving the convenience of de-reflection processing. Compared with the traditional technology of manually applying de-reflection processing through image processing software such as PS, the de-reflection model trained in this embodiment can automatically remove the reflection of glasses in pictures, avoiding manual application to introduce more defects in the image and affecting the image quality.
[0063] In one embodiment, if Figure 2 As shown, based on the non-reflective glasses image and the simulated lens reflective image, the simulated reflective pairing data is obtained, including:
[0064] Step S202, taking the glasses area as the target, the non-reflective glasses image is segmented to obtain a lens mask.
[0065] The glasses area is the pixel area where the glasses are located in the non-reflective glasses image. It can be understood that due to different shooting distances and angles, the size and shape of the glasses in the captured image are uncertain. This embodiment does not specifically limit the specific shape and size of the glasses area, as long as the glasses area can correctly represent the pixel portion occupied by the glasses in the captured image.
[0066] Specifically, the glasses area may be used as a target area for image segmentation to obtain a corresponding lens mask.
[0067] Step S204, based on the simulated reflective area mask, the reflected image and the non-reflective glasses image are merged to obtain a simulated lens reflective image; the simulated reflective area mask is obtained by randomly segmenting the lens mask; the reflected image includes a plurality of pure color images of different colors.
[0068] The simulated reflective area mask can be obtained by randomly segmenting the lens mask. Exemplarily, based on a preset shape, ten or less areas are randomly segmented on the lens mask as the simulated reflective area mask, and the preset shape can be a rectangle, an ellipse, a polygon, a ring, etc. It can be understood that the above-mentioned area segmentation method is not limited to the implementation method mentioned in the above embodiment, as long as the simulated reflective area mask obtained can be close to the real reflective area on the image, the present application does not specifically limit the number of area segmentations and the preset shape.
[0069] The reflected image may include pure color images of various colors. It is understandable that in the real glasses reflection image, the reflection colors of the glasses area are mostly blue, green and other colors. For example, blue, green and purple pure color images can be used as the reflected image. It should be noted that the selection of the reflected image is not limited to the implementation methods mentioned in the above embodiments. As long as the simulated lens reflection image obtained by fusion is close to the real glasses reflection image, the present application does not make specific limitations on the selection of the reflected image. Furthermore, the edge strength of the reflected image can be enhanced based on a preset probability to simulate the effect of a flash.
[0070] The simulated lens reflection image is used to simulate the lens reflection based on the non-reflective glasses image, and the simulated lens reflection image can be obtained by fusing the reflection image with the non-reflective glasses image. For example, based on the simulated reflection area mask, the reflection image and the non-reflective glasses image can be naturally fused through the Poisson fusion algorithm, wherein the Poisson fusion algorithm can achieve seamless fusion at the joints of the images.
[0071] Specifically, the lens mask can be randomly segmented to obtain one or more simulated reflective area masks; based on the simulated reflective area masks, the reflected image and the non-reflective glasses image are naturally merged to obtain a simulated lens reflective image. By constructing a sufficient number of simulated lens reflective images, the problem of a small number of real reflective pairing data and difficulty in collecting them in traditional technologies can be effectively solved.
[0072] Step S206, obtaining simulated reflective pairing data according to the non-reflective glasses image, the simulated lens reflective image and the simulated reflective area mask.
[0073] Among them, a set of simulated reflection pairing data includes a non-reflective glasses image, a simulated lens reflection image constructed based on the non-reflective glasses image, and an artificially set simulated reflection area mask corresponding to the above-mentioned simulated lens reflection image.
[0074] Specifically, based on the non-reflective glasses image, the simulated lens reflective image and the simulated reflective area mask, a sufficient amount of simulated reflective pairing data can be constructed.
[0075] The above de-reflection model training method performs regional segmentation on the non-reflective glasses image with the glasses area as the target to obtain a lens mask; based on the simulated reflective area mask, the reflection image and the non-reflective glasses image are fused to obtain a simulated lens reflective image; based on the non-reflective glasses image, the simulated lens reflective image and the simulated reflective area mask, simulated reflective pairing data can be obtained. This embodiment can effectively solve the problem of insufficient original data of real reflective images and real de-reflected images by constructing sufficient simulated reflective pairing data.
[0076] In one embodiment, if Figure 3 As shown, the simulated reflection pairing data and the real reflection pairing data are randomly input into the model to be trained, and a preset loss function is used for weak supervision training. The parameters of the model to be trained are adjusted based on the output value of the preset loss function, including:
[0077] Step S302: Based on the input simulated reflection paired data and real reflection paired data, a predicted reflection area mask and a predicted de-reflected image output by the to-be-trained model are obtained.
[0078] It can be understood that the model to be trained can perform de-reflection processing on the input simulated lens reflection image and the real reflective glasses image, so as to output the corresponding predicted de-reflection image and the predicted reflective area mask corresponding to the predicted de-reflection image. It should be noted that in the simulated reflection pairing data, the non-reflective glasses image, the simulated reflection area mask, and the simulated lens reflection image constructed based on the non-reflective glasses image are actually known; in the input real reflection pairing data, the real reflective glasses image and the real de-reflective glasses image are also known, and the above-mentioned known data can be used as the prior information of the model to train the model.
[0079] Specifically, based on the simulated reflection pairing data and the real reflection pairing data, the predicted reflection area mask and the predicted de-reflection image corresponding to the output of the model to be trained can be obtained.
[0080] Furthermore, before the simulated reflection pairing data and the real reflection pairing data are randomly input into the model to be trained, the input simulated reflection pairing data and the real reflection pairing data may be preprocessed. Exemplarily, all the simulated reflection pairing data and the real reflection pairing data input into the model to be trained may be normalized preprocessed, including normalizing the non-reflective glasses image, the simulated lens reflection image, the real reflective glasses image, and the real de-reflective glasses image to between -1 and 1, and normalizing the simulated reflection area mask to between 0 and 1.
[0081] Step S304, using a preset loss function to calculate an output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflected image, and adjusting the parameters of the model to be trained based on the output value of the preset loss function.
[0082] It can be understood that since in the simulated reflection pairing data, the non-reflective glasses image for constructing the simulated lens reflection image and the simulated reflection area mask are both known, the difference between the output predicted de-reflective image and the corresponding non-reflective glasses image, as well as the difference between the output predicted reflection area mask and the corresponding simulated reflection area mask can be reduced by adjusting the parameters of the model to be trained.
[0083] In addition, since the real reflective pairing data input to the model to be trained also includes known real reflective glasses images and real de-reflective glasses images, the difference between the predicted de-reflective glasses image output by the model to be trained and the real de-reflective glasses image can be reduced by adjusting the parameters of the model to be trained.
[0084] Furthermore, the output value of the preset loss function can represent the difference between the predicted reflective area mask, the predicted de-reflective image and their corresponding true values (non-reflective glasses image, real de-reflective glasses and simulated reflective area mask). By adjusting the parameters of the model to be trained to reduce the output value of the preset loss function, the desired de-reflective model can be trained.
[0085] Specifically, the simulated lens reflection image in the simulated reflection pairing data and the real reflection glasses image in the real reflection pairing data are input into the model to be trained, and the predicted de-reflection image and the predicted reflection area mask corresponding to the output of the model to be trained are obtained. Based on the predicted de-reflection image, the predicted reflection area mask, the non-reflection glasses image and the real de-reflection glasses image, the output value of the preset loss function can be calculated; based on the output value of the preset loss function, the parameters of the model to be trained are adjusted to reduce the difference between the predicted result output by the model to be trained and the real value.
[0086] In one embodiment, if Figure 4As shown, the preset loss function includes a first preset loss function and a second preset loss function, and the preset loss function is used to calculate the output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflected image, including:
[0087] Step S402, in response to inputting the simulated reflection paired data into the to-be-trained model, using a first preset loss function, based on the predicted reflective area mask, the predicted de-reflected image, and the simulated reflection paired data, calculating an output value of the first preset loss function;
[0088] The non-reflective glasses image, the simulated reflective area mask, and the simulated lens reflective image contained in the simulated reflective pairing data are all known. The output value of the first preset loss function can represent the difference between the predicted de-reflective image output by the model to be tested and the known non-reflective glasses image in the simulated reflective pairing data, and the predicted reflective area mask and the known simulated reflective area mask.
[0089] Specifically, after the simulated reflection pairing data is input into the model to be trained, the model to be trained can perform de-reflection processing on the simulated lens reflection image in the input data to obtain a predicted de-reflection image and a predicted reflection area mask corresponding to the predicted de-reflection image; based on the predicted de-reflection image, the predicted reflection area mask, the non-reflective glasses image and the simulated reflection area mask in the input simulated reflection pairing data, the output value of the first preset loss function is calculated to characterize the difference between the predicted de-reflection image and the known non-reflective glasses image, as well as the difference between the predicted reflection area mask and the known simulated reflection area mask.
[0090] Step 404, in response to inputting the real reflection paired data into the model to be trained, using the second preset loss function, based on the predicted reflective area mask and the predicted de-reflected image and the real reflection paired data, calculate the output value of the second preset loss function.
[0091] The real reflective glasses image included in the real reflective pairing data inputted above and the real de-reflective glasses image after the real reflective glasses image is de-reflected are both known. The output value of the second preset loss function can then be used to represent the difference between the predicted de-reflective glasses image outputted by the to-be-trained model and the corresponding known real de-reflective glasses image.
[0092] Specifically, after the real reflection pairing data is input into the model to be trained, the model to be trained can perform de-reflection processing on the real reflection glasses image in the input data to obtain a predicted de-reflection image and a predicted reflection area mask corresponding to the predicted de-reflection image; based on the predicted de-reflection image and the real de-reflection glasses image in the real reflection pairing data, calculate the output value of the second preset loss function.
[0093] In one embodiment, the first preset loss function is as follows:
[0094]
[0095] Among them, Loss1 is the output value of the first preset loss function, is the mean square error loss, L1 is the mean absolute error loss, To predict the reflective area mask, To predict the de-reflected image; A simulated reflective area mask corresponding to a simulated lens reflective image; For non-reflective glasses images, is the weight of the mean square error loss in the first preset loss function, and y is the weight of the mean absolute error loss in the first preset loss function.
[0096] It can be understood that the mean squared error loss (MSE) can be used in this embodiment to characterize the difference between the predicted reflective area mask output by the model to be trained and the known simulated reflective area mask. The mean absolute error loss (MAE), also known as the L1 loss, can be used in this embodiment to characterize the difference between the predicted de-reflective image output by the model to be trained and the non-reflective glasses image. Exemplarily, since the reflective area of the glasses in the image is generally small, the weight of the mean square error loss in the first preset loss function can be enhanced. Preferably, the weight ratio between the weight x of the mean square error loss and the weight y of the mean absolute error loss in the first preset loss function can be x:y = 100:1.
[0097] Specifically, based on the mean square error loss between the predicted reflective area mask output by the model to be trained and the corresponding simulated reflective area mask, and the mean absolute error loss between the predicted de-reflective image output by the model to be trained and the non-reflective glasses image, the output value of the first preset loss function can be calculated.
[0098] In one embodiment, if Figure 5 As shown, using the second preset loss function, based on the predicted reflective area mask and the predicted de-reflective image and the real reflective paired data, the output value of the second preset loss function is calculated, including:
[0099] Step S502 , performing difference processing on the real reflective glasses image and the real de-reflective glasses image to obtain a difference image.
[0100] The difference image can represent the difference between the real reflective glasses image and the real de-reflective glasses image.
[0101] Specifically, the real de-reflective glasses image may be subtracted from the real reflective glasses image to obtain a difference image.
[0102] Step S504: thresholding the difference image to obtain an approximate reflective area mask.
[0103] Among them, thresholding the difference image can obtain the corresponding binary image, and based on the reflective area in the binary image, an approximate reflective area mask can be obtained. Exemplarily, the thresholding difference image includes using the Otsu threshold method OTSU to threshold the difference image to obtain a binary image; wherein the binary image contains two pixel values 0 and 1, the pixel value 1 is white to represent the reflective area, and the pixel value 0 is black to represent the non-reflective area; based on the reflective area, an approximate reflective area mask can be obtained. It can be understood that the above-mentioned thresholding processing method is not limited to the implementation method mentioned in the above-mentioned embodiment, as long as it can realize the function of thresholding the difference image to obtain an approximate reflective area mask.
[0104] Specifically, the difference image is thresholded to obtain an approximate reflective area mask. It can be understood that the real reflective pairing data includes a real reflective glasses image and a real de-reflective glasses image, and there is no artificial setting of a reflective area mask corresponding to the real reflective glasses image. In this embodiment, a binary image is obtained by thresholding the difference image between the real reflective glasses image and the real de-reflective glasses image, and an approximate reflective area mask can be obtained based on the reflective area in the binary image.
[0105] Step S506, based on the real reflection pairing data, the approximate reflection area mask, the predicted reflection area mask and the predicted de-reflection image, the output value of the preset loss function is calculated using a second preset loss function; the second preset loss function is as follows:
[0106]
[0107] Among them, Loss2 is the output value of the second preset loss function, is the mean square error loss, L1 is the mean absolute error loss, To predict the reflective area mask, To predict the de-reflected image; is the approximate reflective area mask corresponding to the real reflective glasses image; For the real de-reflective glasses image, is the weight of the mean square error loss in the second preset loss function, and y is the weight of the mean absolute error loss in the second preset loss function.
[0108] It can be understood that the mean square error loss in this embodiment can be used to characterize the difference between the predicted reflective area mask output by the model to be trained and the known approximate reflective area mask. The mean absolute error loss can be used to characterize the difference between the predicted de-reflective image output by the model to be trained and the real de-reflective glasses image. Preferably, the weight ratio between the weight x of the mean square error loss and the weight y of the mean absolute error loss in the second preset loss function can be 100:1.
[0109] Specifically, based on the mean square error loss between the predicted reflective area mask and the approximate reflective area mask output by the model to be trained, and the mean absolute error loss between the predicted de-reflective image output by the model to be trained and the corresponding known real de-reflective glasses image, the output value of the second preset loss function is obtained according to the preset weight calculation.
[0110] In order to make the purpose, technical solutions and advantages of this application clearer, Figure 6 As shown, the present application is further described in detail by taking the de-reflection model trained by screening 1000 non-reflective glasses images and 50 pairs of real reflective paired data obtained by shooting as an example. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0111] The non-reflective glasses image is segmented with the glasses area as the target to obtain the lens mask of each non-reflective glasses image. The lens mask is randomly segmented to obtain the simulated reflective area mask; based on the simulated reflective area mask, the reflection image and the non-reflective glasses image are fused by the Poisson fusion algorithm to obtain the simulated lens reflective image. Among them, a blue, green or purple pure color image can be used as the reflection image. Based on the non-reflective glasses image, the simulated lens reflective image and the simulated reflective area mask corresponding to the simulated lens reflective image, 1000 pairs of simulated reflective pairing data can be constructed.
[0112] The above 1000 pairs of simulated reflection paired data and 50 pairs of real reflection paired data are randomly input into the model to be trained. Based on the input simulated lens reflection image and real reflective glasses image, the model to be trained can output the corresponding predicted de-reflective image and the predicted reflective area mask corresponding to the predicted de-reflective image.
[0113] In response to inputting simulated reflection pairing data into the model to be trained, such as Figure 6As shown in the upper part, based on the input known simulated lens reflective image I_r and the corresponding simulated reflective area mask Mr_r, combined with the predicted de-reflective image I_predict and the predicted reflective area mask Mr_r_predict output by the model, the output value Loss1 of the first preset loss function can be obtained, wherein Loss1 includes the mean square error loss MSE_loss between the predicted reflective area mask Mr_r_predict and the simulated reflective area mask Mr_r, and the mean absolute error loss L1_loss between the predicted de-reflective image I_predict and the non-reflective glasses image I. Based on the output value Loss1 of the first preset loss function, the parameters of the model to be trained are adjusted to reduce the difference between the predicted results (I_predict, Mr_r_predict) of the model to be trained and the true values (I, Mr).
[0114] In response to inputting the real reflection pairing data into the model to be trained, such as Figure 6 As shown in the lower part. Based on the input known real reflective glasses image I_r and the real de-reflective glasses image I, the difference image between the real reflective glasses image and the real de-reflective glasses image can be obtained by difference processing, and then the difference image is thresholded by the Otsu threshold algorithm OTSU to obtain the approximate reflective area mask Mr_r_fake. Based on the input known real reflective glasses image I_r, combined with the output predicted de-reflective image I_predict and the predicted reflective area mask Mr_r_predict and the approximate reflective area mask Mr_r_fake, the output value Loss2 of the second preset loss function can be obtained, wherein Loss2 includes the mean square error loss MSE_loss between the approximate reflective area mask Mr_r_fake and the predicted reflective area mask Mr_r_predict, and the mean absolute error loss L1_loss between the predicted de-reflective image I_predict and the real de-reflective glasses image I. By adjusting the parameters of the model to be trained according to the output value Loss2 of the second preset loss function, the difference between the prediction results (I_predict, Mr_r_predict) of the model to be trained and the true values (I, Mr_r_fake) can be reduced. By adjusting the parameters of the model to be trained according to the output value of the first preset loss function and the output value of the second preset loss function, a de-reflection model that can automatically remove the reflection of the lenses of glasses pictures is obtained.
[0115] In one embodiment, Figure 7 As shown, the present application provides a method for removing reflections, the method comprising:
[0116] Step S702, obtaining an image to be processed;
[0117] Step S704: input the reflective glasses image to be processed into any one of the de-reflection models in the above embodiments to perform de-reflection processing to obtain a de-reflection image.
[0118] For example, taking the image of the glasses reflection taken by a mobile phone as an example, after the de-reflection model obtains the image of the glasses reflection taken by the mobile phone, it can perform de-reflection processing on the image of the glasses reflection to be processed, thereby obtaining the desired de-reflection image. There is no need to set a polarizing filter on the mobile phone camera, nor does the user need to manually smear the captured image through software such as PS. The de-reflection model of this embodiment can automatically remove the glasses reflection on the captured image, significantly improving the convenience and accuracy of the de-reflection processing.
[0119] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0120] Based on the same inventive concept, the embodiment of the present application also provides a de-reflection model training device for implementing the de-reflection model training method involved above. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above method, so the specific limitations in one or more de-reflection model training device embodiments provided below can refer to the limitations of the de-reflection model training method above, and will not be repeated here.
[0121] In one embodiment, Figure 8 As shown, the present application provides a reflective model training device 800, the device comprising:
[0122] A simulated reflective pairing data acquisition module 802 is used to obtain simulated reflective pairing data based on the non-reflective glasses image and the simulated lens reflective image;
[0123] A real reflection pairing data acquisition module 804 is used to obtain real reflection pairing data based on the real reflection glasses image and the real de-reflection glasses image;
[0124] The training module 806 is used to randomly input the simulated reflection pairing data and the real reflection pairing data into the model to be trained, and use a preset loss function to perform weakly supervised training, and adjust the parameters of the model to be trained based on the output value of the preset loss function to obtain a de-reflection model.
[0125] In one embodiment, the simulated reflective pairing data acquisition module 802 further includes:
[0126] A lens mask acquisition module is used to segment the non-reflective glasses image with the glasses area as the target to obtain a lens mask;
[0127] The simulated lens reflective image acquisition module is used to fuse the reflective image and the non-reflective glasses image based on the simulated reflective area mask to obtain the simulated lens reflective image; the simulated reflective area mask is obtained by randomly segmenting the lens mask; the reflective image includes a plurality of pure color images of different colors;
[0128] The simulated reflective pairing data acquisition module is used to obtain simulated reflective pairing data according to the non-reflective glasses image, the simulated lens reflective image and the simulated reflective area mask.
[0129] In one embodiment, the training module 806 further includes:
[0130] A prediction module, used for obtaining a predicted reflective area mask and a predicted de-reflected image output by a to-be-trained model based on the input simulated reflective paired data and the real reflective paired data;
[0131] The model parameter adjustment module is used to calculate the output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflective image using the preset loss function, and adjust the parameters of the model to be trained based on the output value of the preset loss function.
[0132] In one embodiment, the model parameter adjustment module further includes:
[0133] A first preset loss function output module, configured to calculate an output value of the first preset loss function based on the predicted reflective area mask, the predicted de-reflective image, and the simulated reflective paired data in response to inputting the simulated reflective paired data into the model to be trained;
[0134] The second preset loss function output module is used to calculate the output value of the second preset loss function based on the predicted reflective area mask, the predicted de-reflective image and the real reflective paired data using the second preset loss function.
[0135] In one embodiment, the first preset loss function of the first preset loss function output module is as follows:
[0136]
[0137] Among them, Loss1 is the output value of the first preset loss function, is the mean square error loss, L1 is the mean absolute error loss, To predict the reflective area mask, To predict the de-reflected image; A simulated reflective area mask corresponding to a simulated lens reflective image; For non-reflective glasses images, is the weight of the mean square error loss in the first preset loss function, and y is the weight of the mean absolute error loss in the first preset loss function.
[0138] In one embodiment, the second preset loss function output module further includes:
[0139] A difference processing module, used for performing difference processing on the real reflective glasses image and the real de-reflective glasses image to obtain a difference image;
[0140] A threshold processing module, used for threshold processing of the difference image to obtain an approximate reflective area mask;
[0141] The second preset loss function output submodule is used to calculate the output value of the preset loss function using the second preset loss function based on the real reflection pairing data, the approximate reflection area mask, the predicted reflection area mask and the predicted de-reflection image; wherein the second preset loss function is as follows:
[0142]
[0143] Among them, Loss2 is the output value of the second preset loss function, is the mean square error loss, L1 is the mean absolute error loss, To predict the reflective area mask, To predict the de-reflected image; is the approximate reflective area mask corresponding to the real reflective glasses image; For the real de-reflective glasses image, is the weight of the mean square error loss in the second preset loss function, and y is the weight of the mean absolute error loss in the second preset loss function.
[0144] Each module in the above-mentioned de-reflection model training device can be implemented in whole or in part by software, hardware, or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.
[0145] In an exemplary embodiment, an electronic device is provided. The electronic device may be a terminal, and its internal structure diagram may be as shown in FIG. Fig. 9 As shown. The electronic device includes a processor, a memory, an input / output interface, a communication interface, a display unit and an input device. The processor, the memory and the input / output interface are connected through a system bus, and the communication interface, the display unit and the input device are connected to the system bus through the input / output interface. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the electronic device is used to exchange information between the processor and an external device. The communication interface of the electronic device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner can be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a de-reflection model training method and a de-reflection method are implemented. The display unit of the electronic device is used to form a visually visible picture, which can be a display screen, a projection device or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the electronic device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the electronic device casing, or an external keyboard, touchpad or mouse.
[0146] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is merely a block diagram of a partial structure related to the scheme of the present application, and does not constitute a limitation on the electronic device to which the scheme of the present application is applied. The specific electronic device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.
[0147] In one embodiment, the present application provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0148] In one embodiment, the present application further provides a computer program product, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0149] In one embodiment, the present application further provides an electronic device, including:
[0150] A memory for storing instructions executed by one or more processors of the electronic device;
[0151] And a processor, which is one of the processors of the electronic device, is used to execute the instructions stored therein to implement the steps in the above-mentioned method embodiments.
[0152] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.
[0153] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.
[0154] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.
[0155] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A de-reflection model training method, characterized in that: The method comprises: Based on the non-reflective glasses image and the simulated lens reflective image, simulated reflective pairing data is obtained; Based on the real reflective glasses image and the real de-reflective glasses image, real reflective pairing data is obtained; The simulated reflection pairing data and the real reflection pairing data are randomly input into the model to be trained, and a preset loss function is used to perform weakly supervised training. The parameters of the model to be trained are adjusted based on the output value of the preset loss function to obtain a de-reflection model.
2. The method according to claim 1, characterized in that: The step of obtaining simulated reflective pairing data based on the non-reflective glasses image and the simulated lens reflective image includes: Taking the glasses area as the target, the non-reflective glasses image is segmented to obtain a lens mask; Based on the simulated reflective area mask, the reflected image and the non-reflective glasses image are fused to obtain the simulated lens reflective image; the simulated reflective area mask is obtained by randomly segmenting the lens mask; the reflected image includes a plurality of pure color images of different colors; The simulated reflective pairing data is obtained according to the non-reflective glasses image, the simulated lens reflective image and the simulated reflective area mask.
3. The method according to claim 2, characterized in that: The step of randomly inputting the simulated reflection pairing data and the real reflection pairing data into a model to be trained, performing weakly supervised training using a preset loss function, and adjusting parameters of the model to be trained based on an output value of the preset loss function includes: Based on the input simulated reflection paired data and the real reflection paired data, obtaining a predicted reflection area mask and a predicted de-reflected image output by the to-be-trained model; The preset loss function is used to calculate an output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflected image, and the parameters of the model to be trained are adjusted based on the output value of the preset loss function.
4. The method according to claim 3, characterized in that: The preset loss function includes a first preset loss function and a second preset loss function, and the using the preset loss function to calculate the output value of the preset loss function based on the predicted reflective area mask and the predicted de-reflected image includes: In response to inputting the simulated reflection paired data into the model to be trained, using the first preset loss function, based on the predicted reflection area mask, the predicted de-reflected image and the simulated reflection paired data, calculate an output value of the first preset loss function; In response to inputting the real reflection paired data into the model to be trained, the output value of the second preset loss function is calculated based on the predicted reflection area mask, the predicted de-reflection image and the real reflection paired data using the second preset loss function.
5. The method according to claim 4, characterized in that: The first preset loss function is as follows: Among them, Loss1 is the output value of the first preset loss function, is the mean square error loss, L1 is the mean absolute error loss, is the predicted reflective area mask, is the predicted de-reflected image; The simulated reflective area mask corresponding to the simulated lens reflective image; is the non-reflective glasses image, is the weight of the mean square error loss in the first preset loss function, and y is the weight of the mean absolute error loss in the first preset loss function.
6. The method according to claim 4, characterized in that: The using the second preset loss function to calculate the output value of the second preset loss function based on the predicted reflective area mask, the predicted de-reflective image, and the real reflective paired data includes: Performing difference processing on the real reflective glasses image and the real de-reflective glasses image to obtain a difference image; Thresholding the difference image to obtain an approximate reflective area mask; Based on the real reflection pairing data, the approximate reflection area mask, the predicted reflection area mask and the predicted de-reflection image, the output value of the preset loss function is calculated using the second preset loss function; the second preset loss function is as follows: Among them, Loss2 is the output value of the second preset loss function, is the mean square error loss, L1 is the mean absolute error loss, is the predicted reflective area mask, is the predicted de-reflected image; The approximate reflective area mask corresponding to the real reflective glasses image; is the real de-reflective glasses image, is the weight of the mean square error loss in the second preset loss function, and y is the weight of the mean absolute error loss in the second preset loss function.
7. A method for removing reflections, characterized in that: The method comprises: Get the image to be processed; The reflective glasses image to be processed is input into the de-reflection model described in claims 1 to 6 to perform de-reflection processing to obtain a de-reflection image.
8. A de-reflection model training device, characterized in that: The device comprises: A simulated reflective pairing data acquisition module, used to obtain simulated reflective pairing data based on a non-reflective glasses image and a simulated lens reflective image; A real reflective pairing data acquisition module, used to obtain real reflective pairing data based on the real reflective glasses image and the real de-reflective glasses image; A training module is used to randomly input the simulated reflection pairing data and the real reflection pairing data into a model to be trained, and use a preset loss function to perform weakly supervised training, and adjust the parameters of the model to be trained based on the output value of the preset loss function to obtain a de-reflection model.
9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 7 are implemented.
10. An electronic device, characterized in that: include: A memory for storing instructions executed by one or more processors of the electronic device; and a processor, which is one of the processors of the electronic device, configured to execute the instructions stored therein to implement the method according to any one of claims 1 to 7.
Citation Information
Cited By
Image reflection removing method based on deep learning
CN119295623A
