Method, device, equipment and medium for training network and removing image glare
By training generation and discrimination networks for infrared and grayscale images, the problem of low cost and high accuracy in infrared image glare removal was solved, and glare-free image generation was achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- HEFEI DILUSENSE TECH CORP
- Filing Date
- 2022-06-08
- Publication Date
- 2026-06-19
AI Technical Summary
Existing technologies struggle to remove glare from infrared images with low cost and high accuracy. Traditional methods are costly and ineffective, and deep learning models rely on high-quality training data that is difficult to obtain.
By acquiring infrared images and grayscale images of the same scene, generative and discriminative networks are trained, and network parameters are adjusted to remove glare, generating glare-free images.
It achieves accurate glare removal without significantly altering infrared image information, reducing resource consumption and cost in training data construction, and improving the accuracy of neural network models.
Smart Images
Figure CN117252766B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of image processing technology, and in particular to a method, apparatus, device, and medium for training a network and removing image glare. Background Technology
[0002] Currently, intelligent security systems are widely used in all aspects of people's lives, such as intelligent monitoring of pedestrians running red lights, intelligent checkpoint monitoring on highways, and smart communities. These intelligent applications all rely on cameras, primarily using camera imaging to collect environmental information and then employing relevant algorithms to monitor and judge based on the visual information collected by the camera. Therefore, the quality of camera imaging is crucial when implementing intelligent applications in the context of smart cities. It must not only be suitable for normal lighting conditions but also adapt to various challenges such as indoor low light, outdoor strong light, and nighttime. Ideally, a camera should be able to capture clear images in all scenarios. However, due to factors such as direct sunlight, wear and tear on camera lenses, and gaps between camera lenses, non-ideal scattering and reflection occur, ultimately leading to problems such as glare in the infrared images acquired by the camera based on the infrared imaging principle, affecting image quality.
[0003] To eliminate the adverse effects of flare on image quality, high-end camera lenses typically employ complex optical designs and materials to reduce flare. A common method is to coat the lens elements with an anti-reflective layer to reduce lens reflections. Algorithmic approaches to flare removal also exist, including: using deconvolution for flare removal, which is based on the assumption that the flare point spread function remains spatially constant; detecting lens flare based on its unique shape, location, or intensity, and then using restoration techniques to recover the scene behind the flare area; and employing deep learning for rain and fog removal to learn clean image processing techniques.
[0004] However, none of the above methods can universally, accurately, and cost-effectively remove flare. Applying an anti-reflective layer to each lens is expensive and can affect other coatings on the lens, such as fingerprint and scratch protection. The assumption that the point spread function of a flare does not change spatially is an ideal situation, which is usually incorrect in practical applications. Lens flare detection based on the lens is only applicable to a limited number of types of flares and is prone to misclassifying all bright areas as flares. In addition, deep learning methods heavily rely on high-quality paired clean images and rain / fog images for training. Since a person's expression and state cannot remain completely consistent at different times, it is difficult to obtain high-quality paired training data with and without flare, resulting in low accuracy of the trained deep learning model. Summary of the Invention
[0005] The purpose of this invention is to provide a method, apparatus, device, and medium for training a network and removing image glare, so that a neural network for removing glare from infrared images can be trained, which is not limited by the type of glare and can remove glare from infrared images universally, accurately, and at low cost.
[0006] To achieve the above objectives, embodiments of the present invention provide a method for training a neural network, comprising: acquiring an infrared image and a grayscale image recording the same scene as the infrared image; inputting the normalized infrared image and the grayscale image into a generator network to obtain a glare image and a glare-free image output by the generator network; inputting the glare-free image and the infrared image into a first discriminator network to obtain a first discrimination result indicating whether the glare-free image is a real infrared image; determining a reconstruction loss based on the infrared image, the glare image, and the glare-free image, and determining a first discrimination loss based on the first discrimination result; and adjusting the parameters of the generator network based on the reconstruction loss and the first discrimination loss.
[0007] To achieve the above objectives, embodiments of the present invention also provide a method for removing image glare, comprising: inputting an image to be processed into a generator network, wherein the generator network is trained according to the neural network training method described above; and obtaining a glare-free image output by the generator network as the glare-free image of the image to be processed.
[0008] To achieve the above objectives, embodiments of the present invention also provide a neural network training apparatus, comprising: an acquisition module for acquiring an infrared image and a grayscale image of the same scene recorded by the infrared image; a generation module for inputting the normalized infrared image and the grayscale image into a generation network to obtain a glare image and a glare-free image output by the generation network; a discrimination module for inputting the glare-free image and the infrared image into a first discrimination network to obtain a first discrimination result indicating whether the glare-free image is a real infrared image; a loss determination module for determining a reconstruction loss based on the infrared image, the glare image, and the glare-free image, and determining a first discrimination loss based on the first discrimination result; and an adjustment module for adjusting the parameters of the generation network based on the reconstruction loss and the first discrimination loss.
[0009] To achieve the above objectives, embodiments of the present invention also provide an apparatus for removing image glare, comprising: an input module for inputting an image to be processed into a generator network, wherein the generator network is trained according to the neural network training method described above; and a glare removal module for acquiring a glare-free image output by the generator network as the glare-free image of the image to be processed.
[0010] To achieve the above objectives, embodiments of the present invention also provide an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a neural network training method as described above, or to perform an image glare removal method as described above.
[0011] To achieve the above objectives, embodiments of the present invention also provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the neural network training method described above, or the image glare removal method described above.
[0012] The method provided in this invention uses an infrared image and a grayscale image to record the same scene. Compared to the infrared image, which may be affected by glare, the grayscale image is unaffected by glare and can therefore record complete scene information. Thus, the grayscale image can serve as supervisory information for the infrared image. After inputting the normalized infrared image and grayscale image into the generator network, a glare image and a glare-free image are obtained from the output of the generator network. The glare-free image and the infrared image are then input into a first discriminant network to obtain a first discrimination result. A reconstruction loss is determined based on the infrared image, the glare image, and the glare-free image, and a first discrimination loss is determined based on the first discrimination result. The parameters of the generator network are then adjusted based on the reconstruction loss and the first discrimination loss, enabling the generator network to learn the feature information of the glare-affected part of the infrared image without significantly changing the information in the infrared image. This allows the generator network to separate a complete glare-free image from the infrared image, i.e., the image after removing glare from the infrared image. Furthermore, since it is simple and easy to obtain grayscale images of the same scene as infrared images, such as when taking infrared images, RGB images are also taken, and grayscale is performed on the RGB images to obtain grayscale images of the same scene as infrared images, training data can be constructed conveniently, quickly and accurately through simple image processing. This reduces the resources required to construct training data, lowers the difficulty and cost of implementation, improves the accuracy of training data, and thus improves the accuracy of the trained neural network model. Attached Figure Description
[0013] One or more embodiments are illustrated by way of example with reference numerals in the accompanying drawings. These illustrations do not constitute a limitation on the embodiments. Elements with the same reference numerals in the drawings are denoted as similar elements. Unless otherwise stated, the figures in the drawings are not to be limited by scale.
[0014] Figure 1 This is a flowchart of a neural network training method provided in one embodiment of the present invention;
[0015] Figure 2 This is a flowchart of a neural network training method including the step of inputting a glare-free image into an Encoder-Decoder model, provided in another embodiment of the present invention;
[0016] Figure 3 This is a flowchart of a neural network training method including the step of inputting an infrared image into a second discriminant network, provided in another embodiment of the present invention;
[0017] Figure 4 This is a flowchart of a neural network training method provided in another embodiment of the present invention, which includes a step of adjusting the parameters of a first discriminant network according to a preset fourth expression;
[0018] Figure 5 This is a flowchart of a method for removing image glare provided in another embodiment of the present invention;
[0019] Figure 6 This is a schematic diagram of the structure of a neural network training device provided in another embodiment of the present invention;
[0020] Figure 7 This is a schematic diagram of the structure of an image glare removal device provided in another embodiment of the present invention;
[0021] Figure 8 This is a schematic diagram of the structure of an electronic device provided in another embodiment of the present invention. Detailed Implementation
[0022] As can be seen from the background technology, none of the various glare removal methods proposed so far can achieve accurate and low-cost glare removal.
[0023] Analysis revealed that the above problems occurred because: adding a coating to the camera increases costs; glare removal via deconvolution is mainly based on the fact that the point spread function of glare does not change spatially, therefore, it can only specifically remove glare where the point spread function in the infrared image does not change or remains essentially unchanged spatially; detecting lens halos based on the unique shape, position, or intensity of the lens must be implemented for different cameras, and once the camera lens changes, it needs to be redefined; and glare removal based on methods similar to rain and fog removal makes it difficult to construct accurate training data, resulting in low accuracy of the trained model.
[0024] To address the aforementioned problems, embodiments of the present invention provide a method for training a neural network, comprising: acquiring an infrared image and a grayscale image recording the same scene as the infrared image; inputting the normalized infrared image and the grayscale image into a generator network to obtain a glare image and a glare-free image output by the generator network; inputting the glare-free image and the infrared image into a first discriminator network to obtain a first discrimination result indicating whether the glare-free image is a real infrared image; determining a reconstruction loss based on the infrared image, the glare image, and the glare-free image, and determining a first discrimination loss based on the first discrimination result; and adjusting the parameters of the generator network based on the reconstruction loss and the first discrimination loss.
[0025] The method provided in this invention uses an infrared image and a grayscale image to record the same scene. Compared to the infrared image, which may be affected by glare, the grayscale image is unaffected by glare and can therefore record complete scene information. Thus, the grayscale image can serve as supervisory information for the infrared image. After inputting the normalized infrared image and grayscale image into the generator network, a glare image and a glare-free image are obtained from the output of the generator network. The glare-free image and the infrared image are then input into a first discriminant network to obtain a first discrimination result. A reconstruction loss is determined based on the infrared image, the glare image, and the glare-free image, and a first discrimination loss is determined based on the first discrimination result. The parameters of the generator network are then adjusted based on the reconstruction loss and the first discrimination loss, enabling the generator network to learn the feature information of the glare-affected part of the infrared image without significantly changing the information in the infrared image. This allows the generator network to separate a complete glare-free image from the infrared image, i.e., the image after removing glare from the infrared image. Furthermore, since it is simple and easy to obtain grayscale images of the same scene as infrared images, such as when taking infrared images, RGB images are also taken, and grayscale is performed on the RGB images to obtain grayscale images of the same scene as infrared images, training data can be constructed conveniently, quickly and accurately through simple image processing. This reduces the resources required to construct training data, lowers the difficulty and cost of implementation, improves the accuracy of training data, and thus improves the accuracy of the trained neural network model.
[0026] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the various embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, those skilled in the art will understand that many technical details have been presented in the various embodiments of the present invention to enable the reader to better understand the present invention. However, the technical solutions claimed in the present invention can be implemented even without these technical details and various changes and modifications based on the following embodiments.
[0027] The division of the following embodiments is for ease of description and should not constitute any limitation on the specific implementation of the present invention. The various embodiments can be combined with and referenced by each other without contradiction.
[0028] One embodiment of the present invention provides a method for training a neural network, applicable to electronic devices such as computers and servers. The process is as follows: Figure 1 As shown, it includes at least the following steps:
[0029] Step 101: Obtain the infrared image and a grayscale image of the same scene as the infrared image.
[0030] In this embodiment, one can first acquire an infrared image and a red-green-blue (RGB) image recording the same scene, and then convert the RGB image to grayscale to obtain a grayscale image; alternatively, one can directly acquire an infrared image and a grayscale image recording the same scene.
[0031] Based on this, in some cases, a red-green-blue-infrared (RGB-IR) camera is used to capture the scene, obtaining both RGB and infrared images. The RGB image is then converted to grayscale and normalized with the corresponding infrared image to obtain a grayscale image.
[0032] Specifically, to construct infrared images with glare as training data, the RGB-IR camera can capture scenes of direct sunlight outdoors, resulting in infrared images with glare. Of course, to improve training accuracy, images can also be captured in scenes that do not produce glare in the infrared images, obtaining glare-free infrared images, thus creating richer training data compared to infrared images with glare.
[0033] In other examples, the same scene is captured by both an RGB camera and an IR camera to obtain RGB and infrared images. The RGB and infrared images are then registered, and the registered RGB image is converted to grayscale and normalized to obtain a grayscale image.
[0034] Compared to high-quality paired training data with and without glare, which involves taking pictures of the same object in both glare-free and glare-free scenarios and using an RGB-IR camera to generate glare-free and glare-free images as training data, this method avoids the need to deliberately maintain the same object in both glare-free and glare-free scenarios, simplifies the acquisition of training data, and reduces the difficulty of implementing the training method.
[0035] In some other examples, an IR camera and a grayscale camera can be used to photograph the same scene separately to obtain an infrared image and a grayscale image.
[0036] Of course, the above is only a specific distance description. In other examples, infrared images and grayscale images can be obtained in other ways, which will not be elaborated here.
[0037] It's important to note that while infrared and grayscale images actually record the same scene, the difference lies in the presence of glare. Infrared images suffer from glare, which interferes with the image information in the affected areas, resulting in inaccurate scene information – a problem of missing information. Grayscale images, on the other hand, do not contain glare because the positions of the light rays in the spectrum differ from those in infrared imaging. Therefore, grayscale images are complete in information. Thus, the information recovered from the glare in the infrared image by the generator network can be monitored using the grayscale image as a supervisory signal to determine the accuracy of the recovered information. In other words, the generator network can learn the characteristics of glare recovery information through training, thereby avoiding or reducing the loss of infrared imaging information due to reflections from the infrared camera lens.
[0038] Step 102: Input the normalized infrared image and grayscale image into the generator network to obtain the glare image and glare-free image output by the generator network.
[0039] In this embodiment, the infrared image is considered as two parts: a glare image and a glare-free image. The glare-free image is obtained by recovering the image of any glare points that may exist in the infrared image, while the glare image is the image formed by the glare portion of the infrared image. It can be understood that the glare image and the glare-free image are superimposed in a certain way to form the infrared image. When there is no glare in the infrared image, the glare image is an empty set. Specifically, in some examples, the generator network can also be used to predict the superposition method of the glare image and the glare-free image, i.e., the fusion matrix. That is, infrared image = fusion matrix * glare image + (1 - fusion matrix) * glare-free image. Of course, in other examples, a predefined fusion matrix can also be used, which will not be elaborated here.
[0040] It should be noted that before entering the generative network, the infrared image and grayscale image are normalized to ensure that they have the same form, thus avoiding interference from differences in image format in subsequent processing.
[0041] Step 103: Input the glare-free image and the infrared image into the first discrimination network to obtain the first discrimination result used to indicate whether the glare-free image is a real infrared image.
[0042] In this embodiment, determining whether a glare-free image is a true infrared image means determining whether the glare-free image has the same mode as the infrared image, which is different from the mode of the grayscale image.
[0043] Step 104: Determine the reconstruction loss based on the infrared image, glare image, and glare-free image, and determine the first discrimination loss based on the first discrimination result.
[0044] As explained in step 102, the superposition of the glare map and the glare-free map can be considered as an infrared map. That is, the infrared map can be reconstructed based on the glare map and the glare-free map. Therefore, the information loss of the map reconstructed based on the glare map and the glare-free map relative to the infrared map is the reconstruction loss. This embodiment does not limit the discriminant loss function used to determine the first discriminant loss; it can be any function used to ensure that the glare-free map output by the generator network is judged as true by the first discriminant network, such as L... adv =-∑log(D(Pre_Ir) no_light ), where L adv For the first discriminant loss, D(Pre_Ir) no_light ) represents the first discrimination result output by the first discrimination network for the glare-free image. In particular, the summation symbol in the expression represents the summation of the result after taking the logarithm of the first discrimination results obtained from the i groups of infrared images and grayscale images.
[0045] Step 105: Adjust the parameters of the generator network based on the reconstruction loss and the first discrimination loss.
[0046] In this embodiment, the parameters of the generator network are adjusted in the direction of reducing the reconstruction loss and the first discrimination loss. The parameter adjustment can be achieved by gradient adjustment, random adjustment, etc., which will not be described in detail here.
[0047] In some embodiments, the generator network can be an encoder-decoder model, such as... Figure 2 As shown, the training method for neural networks also includes the following steps:
[0048] Step 106: Input the glare-free image into the Encoder-Decoder model to obtain the glare-free image encoding result output by the Encoder-Decoder model at the Encoder.
[0049] It should be noted that, Figure 2 Step 106 is implemented after step 102 and before step 103. In other embodiments, step 106 can also be implemented simultaneously with step 103. Step 106 can be implemented after step 102, that is, after obtaining the glare-free image.
[0050] Step 107: Determine the feature information loss based on the grayscale image encoding result, the infrared image encoding result, and the glare-free image encoding result. The grayscale image encoding result and the infrared image encoding result are the outputs of the Encoder-Decoder model at the Encoder after inputting the infrared image and the grayscale image.
[0051] It should be noted that, Figure 2Step 107 is implemented after step 104 and before step 105. In other embodiments, step 107 can also be implemented simultaneously with step 104. Step 107 can be implemented before step 105 and after step 106, that is, before parameter adjustment and after obtaining the encoding result.
[0052] Accordingly, the parameters of the generator network are adjusted based on the reconstruction loss and the first discriminant loss, including adjusting the parameters of the Encoder-Decoder model based on the feature information loss, the reconstruction loss, and the first discriminant loss.
[0053] The feature information loss is determined based on the grayscale image encoding results, infrared image encoding results, glare-free image encoding results, and a preset first expression.
[0054] In some examples, the first expression can be:
[0055] L fea =l·||G_e(Gray)-G_e(Ir)||1+||G_e(Gray)-G_e(Pre_Ir no_light )||1
[0056] Among them, L fea For feature information loss, G_e(Gray) is the grayscale image encoding result, G_e(Ir) is the infrared image encoding result, and G_e(Pre_Ir) is the feature information loss. no_light ) represents the encoding result of the glare-free image, 1 represents the label value related to the scene recorded in the infrared image. When the infrared image records a glare scene, 1 = 1, and when the infrared image records a non-glare scene, 1 = 0. ||xy||1 represents the norm distance between x and y.
[0057] Of course, the above is only a specific example of the first expression. In other embodiments, the first norm loss term in the first expression can also be replaced by the second norm loss term, the softmax loss term, etc., which will not be elaborated here.
[0058] Since the infrared image and grayscale image are not strictly paired images of the same object captured in glare-free and glare-free scenarios respectively, it is necessary to learn effective information from the grayscale image of another modality to compensate for the loss of infrared information caused by glare. Deep neural network learning is more semantically focused; therefore, it is hoped that the encoder in the Encoder-Decoder model can overcome the modal differences between the infrared image and the grayscale image, learning modality-independent deep semantic features. Through the first expression mentioned above, in glare-free dataset scenarios, the features output by the encoder and the grayscale image can be made as consistent as possible, while also ensuring that the features extracted by the encoder from the predicted glare-free image are as consistent as possible with the features extracted by the encoder from the grayscale image. In other words, the encoder learns modality-independent deep semantic features.
[0059] It is easy to see that generative networks can fully learn the heterogeneity between different modalities and use the heterogeneity between modalities to make up for the defects of their own modalities.
[0060] In some other embodiments, an additional discriminant network is introduced during training—a second discriminant network used to determine whether glare exists in the infrared image. In this case, such as... Figure 3 As shown, the training method for neural networks also includes at least the following steps:
[0061] Step 108: Input the infrared image into the second discrimination network to obtain a second discrimination result used to indicate whether there is glare in the infrared image.
[0062] It is understandable that by determining whether there is glare in the infrared image, the accuracy of the glare-free image output by the generation network can be verified. In turn, the accuracy of the glare-free image output by the generation network can be improved from another perspective by using the loss difference between the glare-free image and the infrared image.
[0063] Step 109: Based on the second discrimination result and the glare label of the infrared image, determine the second discrimination loss of the second discrimination network. The glare label is determined based on the non-glare image.
[0064] In this embodiment, the glare label is a label indicating whether the infrared image has glare. It is understood that the difference between a glare-free image and an infrared image lies primarily in glare; therefore, the difference between a glare-free image and an infrared image can be used to determine whether the infrared image has glare, i.e., the presence of a glare label.
[0065] Accordingly, the parameters of the generator network are adjusted based on the reconstruction loss and the first discriminant loss, including: adjusting the parameters of the generator network and the second discriminant network based on the second discriminant loss, the reconstruction loss, and the first discriminant loss.
[0066] In some examples, the second discrimination loss of the second discrimination network is determined based on the second discrimination result, the glare label of the infrared image, and a preset second expression. The second expression is:
[0067]
[0068] in, The second discrimination loss is defined as follows: N is the total number of infrared images, i is the infrared image input to the i-th input to the second discrimination network, and y is the loss value. iC D represents the value of the glare label of the infrared image input to the second discriminant network for the i-th input. reflection (x iC ) represents the probability value of the infrared image output by the second discriminant network with respect to the i-th input.
[0069] Of course, the above is only a specific example of the second expression. In other examples, the second expression can also be a classification loss expression in the form of a cross-entropy function, which will not be elaborated here.
[0070] Furthermore, given the second discrimination result, the reconstruction loss is determined based on the grayscale image, glare image, and glare-free image. This can be achieved as follows: the reconstruction loss is determined based on the grayscale image, glare image, glare-free image, the second discrimination result, and a preset third expression, where the third expression is:
[0071] L res =||Ir-(s·M·Pre_Ir) light +(1-s)·(1-M)·Pre_Ir no_light )||1
[0072] Among them, L res For reconstruction loss, Ir is the infrared image, s is the probability represented by the second discrimination result, M is the fusion matrix of the preset glare image and the glare-free image, and Pre_Ir light This is a glare map, Pre_Ir no_light This is a glare-free image.
[0073] Of course, the third expression above is only a specific example. In other examples, the loss generated from the infrared image to the reconstructed image from the glare-free image and the glare image, i.e. from Ir to (s·M·Pre_Ir) light +(1-s)·(1-M)·Pre_Ir no_light The losses generated can also be measured using other forms of loss, such as Ir and (s·M·Pre_Ir). light +(1-s)·(1-M)·Pre_Ir no_light The L2 distance between them, etc., will not be elaborated here.
[0074] It should be noted that, Figure 3 In other embodiments, steps 107 and 108 are implemented after step 104 and before step 105. In other embodiments, steps 107 and 108 may also be implemented before step 104, or step 107 may be implemented before step 104 and step 108 may be implemented after step 104.
[0075] It is understood that the different embodiments described above can also be combined. For example, the overall loss value of the generator network, the first discriminator network, and the second discriminator network can be determined based on the following expression:
[0076]
[0077] Among them, L gan The loss values of the generator network, the first discriminant network, and the second discriminant network as a whole are λ1, λ2, λ3, and λ4, which are adjustable parameters, such as 5, 1, 10, and 1, respectively.
[0078] In some other embodiments, the constructed generator network—the first discriminator network—actually forms an adversarial network architecture. Therefore, an adversarial network training method can also be adopted, that is, continuously adjusting the parameters of the generator network while keeping the network parameters of the first discriminator network fixed, and adjusting the parameters of the first discriminator network while keeping the generator network fixed, until both converge. Based on this, as... Figure 4 As shown, the training method for neural networks includes at least the following steps:
[0079] After adjusting the parameters of the generator network based on the reconstruction loss and the first discriminant loss, the method also includes:
[0080] Step 110: Adjust the parameters of the first discriminant network according to the preset fourth expression.
[0081] In some examples, the fourth expression is:
[0082] L dis =∑log(D(Pre_Ir) no_light ))-log(D(Ir))
[0083] Among them, L dis The loss value of the first discriminant network is D(Pre_Ir). no_light ) represents the output of the first discriminant network for the glare-free image, and D(Ir) represents the output of the first discriminant network for the infrared image.
[0084] It should be noted that this embodiment uses the example of first fixing the parameters of the first discriminator network, then adjusting the parameters of the generator network, and then fixing the parameters of the generator network again before adjusting the parameters of the first discriminator network. In other examples, after implementing step 110, steps 101-105 can be implemented next, that is, after training the generator network and adjusting its parameters until it converges, the first discriminator network is trained and its parameters are adjusted until it converges, and then the generator network is trained and its parameters are adjusted until it converges, until the preset training stopping condition is met.
[0085] One embodiment of the present invention provides a method for removing image glare, applicable to electronic devices such as computers and servers. The process is as follows: Figure 5 As shown, it includes at least the following steps:
[0086] Step 501: Input the image to be processed into the generator network.
[0087] It should be noted that the generative network used in this embodiment is obtained by training a network according to the neural network training method provided in the foregoing method embodiments.
[0088] Step 502: Obtain the glare-free image output by the generating network as the image to be processed after glare removal.
[0089] The steps of the various methods described above are only for clarity. In practice, they can be combined into one step or some steps can be split into multiple steps. As long as they include the same logical relationship, they are all within the scope of protection of this patent. Adding insignificant modifications or introducing insignificant designs to the algorithm or process, but without changing the core design of the algorithm and process, are also within the scope of protection of this patent.
[0090] Another aspect of this invention provides a neural network training device, such as... Figure 6 As shown, it includes:
[0091] The acquisition module 601 is used to acquire an infrared image and a grayscale image that records the same scene as the infrared image.
[0092] The generation module 602 is used to input the normalized infrared image and grayscale image into the generation network to obtain the glare image and glare-free image output by the generation network.
[0093] The discrimination module 603 is used to input the glare-free image and the infrared image into the first discrimination network to obtain a first discrimination result indicating whether the glare-free image is a real infrared image.
[0094] The loss determination module 604 is used to determine the reconstruction loss based on the infrared image, glare image and glare-free image and to determine the first discrimination loss based on the first discrimination result.
[0095] The adjustment module 605 is used to adjust the parameters of the generator network based on the reconstruction loss and the first discrimination loss.
[0096] Another aspect of the present invention provides an apparatus for removing image glare, such as... Figure 7 As shown, it includes:
[0097] The input module 701 is used to input the image to be processed into the generation network, which is trained according to the neural network training method provided in the above embodiments.
[0098] Glare removal module 702 is used to obtain the glare-free image output by the generating network as the image to be processed after glare removal.
[0099] It is not difficult to see that the above-described device embodiments and method embodiments correspond to each other, and the device embodiments can be implemented in conjunction with the method embodiments. The relevant technical details mentioned in the method embodiments remain valid in the device embodiments, and will not be repeated here to avoid repetition. Correspondingly, the relevant technical details mentioned in the device embodiments can also be applied to the method embodiments.
[0100] It is worth mentioning that all modules involved in this embodiment are logical modules. In practical applications, a logical unit can be a physical unit, a part of a physical unit, or a combination of multiple physical units. Furthermore, to highlight the innovative aspects of this invention, this embodiment does not introduce units that are not closely related to solving the technical problem proposed by this invention; however, this does not mean that other units are absent from this embodiment.
[0101] Another aspect of the present invention provides an electronic device, such as... Figure 8 As shown, it includes: at least one processor 801; and a memory 802 communicatively connected to at least one processor 801; wherein the memory 802 stores instructions executable by at least one processor 801, the instructions being executed by at least one processor 801 to enable at least one processor 801 to perform the method described in any of the above method embodiments.
[0102] The memory 802 and processor 801 are connected via a bus, which can include any number of interconnecting buses and bridges. The bus connects various circuits of one or more processors 801 and memory 802 together. The bus can also connect various other circuits, such as peripheral devices, voltage regulators, and power management circuits, which are well known in the art and therefore will not be described further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver can be a single element or multiple elements, such as multiple receivers and transmitters, providing a unit for communicating with various other devices over a transmission medium. Data processed by processor 801 is transmitted over a wireless medium via an antenna, which further receives data and transmits it to processor 801.
[0103] The processor 801 is responsible for managing the bus and general processing, and can also provide various functions, including timing, peripheral interfaces, voltage regulation, power management, and other control functions. The memory 802 can be used to store data used by the processor 801 during operation.
[0104] Another aspect of the present invention provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, it implements the above-described method embodiments.
[0105] That is, those skilled in the art will understand that all or part of the steps in the methods of the above embodiments can be implemented by a program instructing related hardware. This program is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, a portable hard drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.
[0106] Those skilled in the art will understand that the above embodiments are specific embodiments for implementing the present invention, and in practical applications, various changes in form and detail may be made without departing from the spirit and scope of the present invention.
Claims
1. A method for training a neural network, characterized in that, include: Acquire an infrared image and a grayscale image of the same scene as the infrared image; The normalized infrared image and the grayscale image are input into the generation network to obtain the glare image and the glare-free image output by the generation network. The glare-free image and the infrared image are input into a first discrimination network to obtain a first discrimination result indicating whether the glare-free image is a real infrared image; The reconstruction loss is determined based on the infrared image, the glare image, and the glare-free image, and the first discrimination loss is determined based on the first discrimination result. The parameters of the generator network are adjusted based on the reconstruction loss and the first discrimination loss. The reconstruction loss is the information loss of the image reconstructed based on the glare image and the glare-free image relative to the infrared image. 2.The method of Claim 1, wherein, The generator network is an encoder-decoder model. After inputting the normalized infrared image and the grayscale image into the generator network to obtain the glare image and the glare-free image output by the generator network, the method further includes: The glare-free image is input into the Encoder-Decoder model to obtain the glare-free image encoding result output by the Encoder-Decoder model at the Encoder. Before adjusting the parameters of the generator network based on the reconstruction loss and the first discriminant loss, the method further includes: The feature information loss is determined based on the grayscale image encoding result, the infrared image encoding result, and the glare-free image encoding result. The grayscale image encoding result and the infrared image encoding result are the outputs of the Encoder-Decoder model at the Encoder after inputting the infrared image and the grayscale image. The step of adjusting the parameters of the generator network based on the reconstruction loss and the first discriminant loss includes: The parameters of the Encoder-Decoder model are adjusted based on the feature information loss, the reconstruction loss, and the first discriminant loss.
3. The neural network training method according to claim 2, characterized in that, The step of determining the feature information loss based on the grayscale image encoding result, the infrared image encoding result, and the glare-free image encoding result includes: The feature information loss is determined based on the grayscale image encoding result, the infrared image encoding result, the glare-free image encoding result, and a preset first expression, wherein the first expression is: Among them, L fea The feature information loss is G_e(Gray), the grayscale image encoding result is G_e(Ir), and the infrared image encoding result is G_e(Pre_Ir). no_light ) represents the encoding result of the glare-free image, l is the tag value related to the scene recorded by the infrared image, l=1 when the infrared image records a glare scene, l=0 when the infrared image records a non-glare scene, and ||xy||1 represents the norm distance between x and y. 4.The method of Claim 1-3, wherein, After acquiring the infrared image and a grayscale image of the same scene as the infrared image, the method further includes: The infrared image is input into a second discrimination network to obtain a second discrimination result indicating whether glare exists in the infrared image. Before adjusting the parameters of the generator network based on the reconstruction loss and the first discriminant loss, the method further includes: Based on the second discrimination result and the glare label of the infrared image, the second discrimination loss of the second discrimination network is determined, wherein the glare label is information indicating whether there is glare in the infrared image, determined based on the difference between the glare-free image and the infrared image; The step of adjusting the parameters of the generator network based on the reconstruction loss and the first discriminant loss includes: The parameters of the generator network and the second discriminator network are adjusted based on the second discriminant loss, the reconstruction loss, and the first discriminant loss.
5. The method of claim 4, wherein, The step of determining the second discrimination loss of the second discrimination network based on the second discrimination result and the glare label of the infrared image includes: Based on the second discrimination result, the glare label of the infrared image, and the preset second expression, the second discrimination loss of the second discrimination network is determined, whereby the second expression is: in, The second discrimination loss is defined as N, where N is the total number of infrared images, i is the i-th infrared image input to the second discrimination network, and y is the second discrimination loss. iC This represents the value of the glare label in the infrared image of the i-th input to the second discriminant network. This represents the probability value of the infrared image output by the second discrimination network with respect to the i-th input. 6.The method of Claim 4, wherein, The determination of reconstruction loss based on the infrared image, the glare image, and the glare-free image includes: The reconstruction loss is determined based on the infrared image, the glare image, the glare-free image, the second discrimination result, and a preset third expression, wherein the third expression is: in, Let Ir be the reconstruction loss, s be the infrared image, s be the probability represented by the second discrimination result, and M be the preset fusion matrix of the glare image and the glare-free image. The glare diagram is as follows. This is the image showing the absence of glare.
7. The method for training a neural network according to any one of claims 1 to 3, characterized in that, After adjusting the parameters of the generator network based on the reconstruction loss and the first discriminant loss, the method further includes: The parameters of the first discriminant network are adjusted according to a preset fourth expression, wherein the fourth expression is: in, The loss value of the first discrimination network. The output of the first discrimination network regarding the glare-free image. This is the output result of the first discrimination network regarding the infrared image.
8. A method of removing image glare, characterized by, include: The image to be processed is input into the generator network, which is trained according to the neural network training method as described in any one of claims 1 to 7. The glare-free image output by the generating network is obtained as the glare-free image of the image to be processed.
9. A device for training a neural network, characterized by include: An acquisition module is used to acquire an infrared image and a grayscale image of the same scene as the infrared image; The generation module is used to input the normalized infrared image and the grayscale image into the generation network to obtain the glare image and the glare-free image output by the generation network. The discrimination module is used to input the glare-free image and the infrared image into the first discrimination network to obtain a first discrimination result indicating whether the glare-free image is a real infrared image; The loss determination module is used to determine the reconstruction loss based on the infrared image, the glare image, and the glare-free image, and to determine the first discrimination loss based on the first discrimination result. The adjustment module is used to adjust the parameters of the generator network based on the reconstruction loss and the first discrimination loss; The reconstruction loss is the information loss of the image reconstructed based on the glare image and the glare-free image relative to the infrared image.
10. An apparatus for removing image glare, characterized in that, include: An input module is used to input the image to be processed into a generation network, wherein the generation network is trained according to the neural network training method as described in any one of claims 1 to 7; The glare removal module is used to obtain the glare-free image output by the generating network as the glare-free image of the image to be processed.
11. An electronic device, comprising: include: At least one processor; as well as, A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform a training method for a neural network as described in any one of claims 1 to 7, or to perform a method for removing image glare as described in claim 8.
12. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the training method of the neural network as described in any one of claims 1 to 7, or the method for removing image glare as described in claim 8.