Weakly supervised image enhancement model training method and device
Through the weakly supervised image enhancement model training method, supervised training is used to obtain the noise mask and perform weak supervised training, which solves the problem of insufficient denoising performance of image enhancement models in the prior art, and significantly improves image quality and model generalization capabilities.
Patent Information
- Application Number
- CN202310649226.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-31
- Publication Date
- 2025-04-25
- Estimated Expiration
- 2043-05-31
AI Technical Summary
The existing image enhancement model training methods have limited effect on improving the noise denoising performance, resulting in low image quality after enhancement.
Weak supervised image enhancement model training method is adopted to obtain the noise mask of labeled contaminated images through supervised training, and apply it to label-free clean images for weak supervised training to improve the denoising performance of the model.
It significantly improves the denoising performance of weakly supervised image enhancement model, reduces dependence on labeled images, reduces data annotation costs, and improves the generalization ability of the model.
Smart Images
Figure CN116664439B_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image enhancement technology, and in particular to a weakly supervised image enhancement model training method and device. Background Art
[0002] Bad weather or environmental factors usually affect the imaging of visual sensors. For example, water droplets attached to the visual sensor lens on a rainy day will cause water droplet occlusions on the collected image, and mud on the visual sensor lens will cause dirt occlusions on the collected image. Images occluded by water droplets or dirt are contaminated images. When the image recognizes a contaminated image, it will not be able to normally identify the target corresponding to the image. In order to obtain a clean image, it is necessary to enhance the contaminated image through an image enhancement model to eliminate the influence of noise such as water droplets and dirt on the image. Before using the image enhancement model, the model needs to be trained to improve the image enhancement effect of the model. The current model training method has limited effect on improving the denoising performance of the model, resulting in low image quality after model enhancement. Therefore, how to better train the model to better improve the denoising performance of the model has become a problem to be solved in this field. Summary of the invention
[0003] The present invention solves the technical problem of how to improve the denoising performance of an image enhancement model by providing a weakly supervised image enhancement model training method and device.
[0004] On the one hand, the present invention provides the following technical solutions:
[0005] A weakly supervised image enhancement model training method, comprising:
[0006] Acquire a first target image and a true value image corresponding to the first target image, wherein the first target image is a contaminated image with a label;
[0007] Inputting the first target image into a weakly supervised image enhancement model for enhancement to obtain a first binary mask and a first clean image corresponding to the first target image;
[0008] Performing supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image;
[0009] Acquire a second target image, where the second target image is a clean image without labels;
[0010] Applying the first binary mask to the first target image to obtain a noise image, and fusing the noise image with the second target image to obtain a fused image;
[0011] Inputting the fused image into the weakly supervised image enhancement model for enhancement, to obtain a second binary mask and a restored image corresponding to the fused image;
[0012] The second target image and the first binary mask are used to perform weakly supervised training on the second binary mask and the restored image.
[0013] Preferably, the step of performing supervised training on the first binary mask and the first clean image using the first target image and the true value image corresponding to the first target image includes:
[0014]
[0015] is a supervised training loss function, W, H, and C are the length, width, and number of channels of the first target image, respectively, I is the first target image, gt is the true value image corresponding to the first target image, M is the first binary mask, is the first clean image.
[0016] Preferably, applying the first binary mask to the first target image to obtain a noise image, and fusing the noise image with the second target image to obtain a fused image, comprises:
[0017] U'=M.I+(1-M).U;
[0018] U′ is the fused image, M is the first binary mask, I is the first target image, and U is the second target image.
[0019] Preferably, performing weakly supervised training on the second binary mask and the restored image using the second target image and the first binary mask comprises:
[0020]
[0021] is the weakly supervised training loss function, W, H, C are the length, width and number of channels of the second target image, U is the second target image, M is the first binary mask, is the second binary mask, is the restored image.
[0022] On the other hand, the present invention also provides the following technical solutions:
[0023] A weakly supervised image enhancement model training device, comprising:
[0024] An acquisition module, used to acquire a first target image and a true value image corresponding to the first target image, wherein the first target image is a contaminated image with a label;
[0025] an enhancement module, configured to input the first target image into a weakly supervised image enhancement model for enhancement, so as to obtain a first binary mask and a first clean image corresponding to the first target image;
[0026] a training module, configured to perform supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image;
[0027] The acquisition module is further used to acquire a second target image, where the second target image is a clean image without labels;
[0028] a fusion module, configured to apply the first binary mask to the first target image to obtain a noise image, and fuse the noise image with the second target image to obtain a fused image;
[0029] The enhancement module is further used to input the fused image into the weakly supervised image enhancement model for enhancement, so as to obtain a second binary mask and a restored image corresponding to the fused image;
[0030] The training module is further used to perform weakly supervised training on the second binary mask and the restored image using the second target image and the first binary mask.
[0031] Preferably, the training module performs supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image, including:
[0032]
[0033] is a supervised training loss function, W, H, and C are the length, width, and number of channels of the first target image, respectively, I is the first target image, gt is the true value image corresponding to the first target image, M is the first binary mask, is the first clean image.
[0034] Preferably, the fusion module applies the first binary mask to the first target image to obtain a noise image, and fuses the noise image with the second target image to obtain a fused image, including:
[0035] U'=M.I+(1-M).U;
[0036] U′ is the fused image, M is the first binary mask, I is the first target image, and U is the second target image.
[0037] Preferably, the training module uses the second target image and the first binary mask to perform weakly supervised training on the second binary mask and the restored image, comprising:
[0038]
[0039] is the weakly supervised training loss function, W, H, C are the length, width and number of channels of the second target image, U is the second target image, M is the first binary mask, is the second binary mask, is the restored image.
[0040] On the other hand, the present invention also provides the following technical solutions:
[0041] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, any of the above-mentioned weakly supervised image enhancement model training methods is implemented.
[0042] On the other hand, the present invention also provides the following technical solutions:
[0043] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements any of the above-mentioned weakly supervised image enhancement model training methods.
[0044] One or more technical solutions provided by the present invention have at least the following technical effects or advantages:
[0045] The present invention inputs labeled image pairs for supervised training, so that the weakly supervised image enhancement model can autonomously extract the noise in the labeled contaminated image, and then load the noise in the labeled contaminated image into an unlabeled clean image for weakly supervised training. By training the weakly supervised image enhancement model, the denoising performance of the weakly supervised image enhancement model can be significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.
[0047] Figure 1Flow chart of a weakly supervised image enhancement model training method according to an embodiment of the present invention;
[0048] Figure 2 A schematic diagram of the principle of a weakly supervised image enhancement model training method in an embodiment of the present invention;
[0049] Figure 3 Schematic diagram of a weakly supervised image enhancement model training device in an embodiment of the present invention. DETAILED DESCRIPTION
[0050] The embodiments of the present invention solve the technical problem of how to improve the denoising performance of an image enhancement model by providing a weakly supervised image enhancement model training method and device.
[0051] In order to better understand the technical solution of the present invention, the technical solution of the present invention will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.
[0052] like Figure 1 As shown, the weakly supervised image enhancement model training method of this embodiment includes:
[0053] Step S1, obtaining a first target image and a true value image corresponding to the first target image, wherein the first target image is a contaminated image with a label;
[0054] Step S2, inputting the first target image into a weakly supervised image enhancement model for enhancement, to obtain a first binary mask and a first clean image corresponding to the first target image;
[0055] Step S3, performing supervised training on the first binary mask and the first clean image using the first target image and the true value image corresponding to the first target image;
[0056] Step S4, obtaining a second target image, where the second target image is a clean image without labels;
[0057] Step S5, applying the first binary mask to the first target image to obtain a noise image, and fusing the noise image with the second target image to obtain a fused image;
[0058] Step S6, inputting the fused image into a weakly supervised image enhancement model for enhancement, and obtaining a second binary mask and a restored image corresponding to the fused image;
[0059] Step S7: Perform weak supervision training on the second binary mask and the restored image using the second target image and the first binary mask.
[0060] In this embodiment, the first target image and the true value image corresponding to the first target image and the true value image corresponding to the first target image are obtained in the following manner: in a static scene, a clean image is first collected by using a visual sensor, which is the true value image corresponding to the first target image; then water or dark dye is sprayed on the visual sensor lens to simulate noise such as raindrops, dirt, etc., and then an image is collected, which is the first target image. The first target image and the true value image corresponding to the first target image constitute a labeled image pair, such as Figure 2 shown.
[0061] A traditional image enhancement model generally consists of an encoder and an image enhancement decoder. The weakly supervised image enhancement model of this embodiment consists of an encoder and two decoders, and has an additional region recognition decoder compared to the traditional image enhancement model.
[0062] Figure 2 In it, I is the first target image, gt is the true value image corresponding to the first target image, M is the first binary mask, is the first clean image. The first binary mask M is obtained by marking the polluted area in the first target image I as 1 and marking the unpolluted area as 0. Step S2 actually extracts the polluted area (corresponding to M) of the polluted image and restores the polluted image to a clean image (corresponding to ).
[0063] Step S3 may specifically include:
[0064]
[0065] is the supervised training loss function, W, H, and C are the length, width, and number of channels of the first target image, respectively. Through supervised training, the model can autonomously learn to identify and extract all contaminated areas and generate corresponding binary masks without labeling the contaminated areas in the first target image I.
[0066] In step S4, the second target image is a single-frame image captured by the visual sensor in any scene. Figure 2 U in is the second target image, and U′ is the fused image (not shown).
[0067] Step S5 may specifically include: U'=M·I+(1-M)·U. M·I is a noise image, which is equivalent to the noise in the first target image I. 1-M is the inversion of M, and (1-M)·U is equivalent to setting the area in the second target image U that is the same as the position (contaminated position) of the noise in the first target image I in the first target image I to 0, and setting the other areas in the second target image U to 1, and the area set to 1 is the background image after removing the contaminated position. M·I+(1-M)·U is equivalent to loading the noise in the first target image I into the second target image U.
[0068] Figure 2 In is the second binary mask, To restore the image, is basically the same as M but not completely the same. Step S6 is actually to extract the contaminated area of the fused image U′ (corresponding to ) and restore the fused image U′ to a clean image (corresponding to ). Figure 2 The dotted box on the left is the supervised training part, and the dotted box on the right is the weakly supervised training part.
[0069] Step S7 may specifically include: is the weakly supervised training loss function, W, H, and C are the length, width, and number of channels of the second target image, respectively.
[0070] From the above, it can be seen that in this embodiment, by inputting labeled image pairs for supervised training, the weakly supervised image enhancement model can autonomously extract the noise in the labeled contaminated image, and then load the noise in the labeled contaminated image into the unlabeled clean image for weakly supervised training. By training the weakly supervised image enhancement model, the denoising performance of the weakly supervised image enhancement model can be significantly improved.
[0071] In addition, the traditional model training method is to input only labeled image pairs for supervised training, that is, input labeled contaminated images and true value images corresponding to the contaminated images, and output clean images, which requires a large number of labeled image pairs and high data annotation costs. This embodiment inputs labeled image pairs and unlabeled images, does not require a large number of labeled image pairs, reduces the model's dependence on labeled images, greatly reduces the data annotation cost, and improves the generalization ability of the model.
[0072] like Figure 3 As shown, this embodiment also provides a weakly supervised image enhancement model training device, including:
[0073] An acquisition module, used to acquire a first target image and a true value image corresponding to the first target image, wherein the first target image is a contaminated image with a label;
[0074] An enhancement module, used for inputting the first target image into a weakly supervised image enhancement model for enhancement, to obtain a first binary mask and a first clean image corresponding to the first target image;
[0075] A training module, configured to perform supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image;
[0076] The acquisition module is further used to acquire a second target image, where the second target image is a clean image without labels;
[0077] a fusion module, configured to apply the first binary mask to the first target image to obtain a noise image, and fuse the noise image with the second target image to obtain a fused image;
[0078] The enhancement module is further used to input the fused image into the weakly supervised image enhancement model for enhancement, so as to obtain a second binary mask and a restored image corresponding to the fused image;
[0079] The training module is also used to perform weakly supervised training on the second binary mask and the restored image using the second target image and the first binary mask.
[0080] Furthermore, the training module performs supervised training on the first binary mask and the first clean image using the first target image and the true value image corresponding to the first target image, which may include:
[0081]
[0082] is the supervised training loss function, W, H, C are the length, width and number of channels of the first target image respectively, I is the first target image, gt is the true value image corresponding to the first target image, M is the first binary mask, is the first clean image.
[0083] Furthermore, the fusion module applies the first binary mask to the first target image to obtain a noise image, and fuses the noise image with the second target image to obtain a fused image, which may include:
[0084] U'=M.I+(1-M).U;
[0085] U′ is the fused image, M is the first binary mask, I is the first target image, and U is the second target image.
[0086] Furthermore, the training module uses the second target image and the first binary mask to perform weakly supervised training on the second binary mask and the restored image, which may include:
[0087]
[0088] is the weakly supervised training loss function, W, H, C are the length, width and number of channels of the second target image respectively, U is the second target image, M is the first binary mask, is the second binary mask, To restore the image.
[0089] Based on the same inventive concept as the weakly supervised image enhancement model training method described above, this embodiment also provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the weakly supervised image enhancement model training method described above when executing the program. In which, the bus architecture (represented by a bus) may include any number of interconnected buses and bridges, and the bus links together various circuits including one or more processors represented by a processor and a memory represented by a memory. The bus may also link together various other circuits such as peripherals, voltage regulators, and power management circuits, which are well known in the art and are therefore not further described herein. The bus interface provides an interface between the bus and the receiver and transmitter. The receiver and transmitter may be the same element, i.e., a transceiver, providing a unit for communicating with various other devices on a transmission medium. The processor is responsible for managing the bus and general processing, and the memory may be used to store data used by the processor when performing operations.
[0090] Since the electronic device introduced in this embodiment is an electronic device used to implement the weakly supervised image enhancement model training method in the embodiment of the present invention, based on the weakly supervised image enhancement model training method introduced in the embodiment of the present invention, a person skilled in the art can understand the specific implementation of the electronic device of the present embodiment and its various variations, so how the electronic device implements the method in the embodiment of the present invention is not described in detail here. As long as a person skilled in the art implements the electronic device used in the weakly supervised image enhancement model training method in the embodiment of the present invention, it belongs to the scope of protection of the present invention.
[0091] Based on the same inventive concept as the above-mentioned weakly supervised image enhancement model training method, the present invention also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements any of the above-mentioned weakly supervised image enhancement model training methods.
[0092] Those skilled in the art will appreciate that embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0093] The present invention is described with reference to flowcharts and / or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.
[0094] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.
[0095] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.
[0096] Although the preferred embodiments of the present invention have been described, those skilled in the art may make other changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention.
[0097] Obviously, those skilled in the art can make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.
Claims
1. A weakly supervised image enhancement model training method, characterized in that: include: Acquire a first target image and a true value image corresponding to the first target image, wherein the first target image is a contaminated image with a label; Inputting the first target image into a weakly supervised image enhancement model for enhancement to obtain a first binary mask and a first clean image corresponding to the first target image; Performing supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image; Acquire a second target image, where the second target image is a clean image without labels; Applying the first binary mask to the first target image to obtain a noise image, and fusing the noise image with the second target image to obtain a fused image; Inputting the fused image into the weakly supervised image enhancement model for enhancement, to obtain a second binary mask and a restored image corresponding to the fused image; Performing weakly supervised training on the second binary mask and the restored image using the second target image and the first binary mask; The using the first target image and the true value image corresponding to the first target image to perform supervised training on the first binary mask and the first clean image includes: is a supervised training loss function, W, H, and C are the length, width, and number of channels of the first target image, respectively, I is the first target image, gt is the true value image corresponding to the first target image, M is the first binary mask, is the first clean image.
2. The weakly supervised image enhancement model training method according to claim 1, characterized in that: The step of applying the first binary mask to the first target image to obtain a noise image, and fusing the noise image with the second target image to obtain a fused image includes: U′=M·I+(1-M)·U; U′ is the fused image, M is the first binary mask, I is the first target image, and U is the second target image.
3. The weakly supervised image enhancement model training method according to claim 1, characterized in that: The performing weak supervision training on the second binary mask and the restored image by using the second target image and the first binary mask comprises: is the weakly supervised training loss function, W, H, C are the length, width and number of channels of the second target image, U is the second target image, M is the first binary mask, is the second binary mask, is the restored image.
4. A weakly supervised image enhancement model training device, characterized in that: include: An acquisition module, used to acquire a first target image and a true value image corresponding to the first target image, wherein the first target image is a contaminated image with a label; an enhancement module, configured to input the first target image into a weakly supervised image enhancement model for enhancement, so as to obtain a first binary mask and a first clean image corresponding to the first target image; a training module, configured to perform supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image; The acquisition module is further used to acquire a second target image, where the second target image is a clean image without labels; a fusion module, configured to apply the first binary mask to the first target image to obtain a noise image, and fuse the noise image with the second target image to obtain a fused image; The enhancement module is further used to input the fused image into the weakly supervised image enhancement model for enhancement, so as to obtain a second binary mask and a restored image corresponding to the fused image; The training module is further used to perform weak supervision training on the second binary mask and the restored image using the second target image and the first binary mask; The training module performs supervised training on the first binary mask and the first clean image using the first target image and a true value image corresponding to the first target image, including: is a supervised training loss function, W, H, and C are the length, width, and number of channels of the first target image, respectively, I is the first target image, gt is the true value image corresponding to the first target image, M is the first binary mask, is the first clean image.
5. The weakly supervised image enhancement model training device according to claim 4, characterized in that: The fusion module applies the first binary mask to the first target image to obtain a noise image, and fuses the noise image with the second target image to obtain a fused image, including: U′=M·I+(1-M)·U; U′ is the fused image, M is the first binary mask, I is the first target image, and U is the second target image.
6. The weakly supervised image enhancement model training device according to claim 4, characterized in that: The training module performs weak supervision training on the second binary mask and the restored image using the second target image and the first binary mask, including: is the weakly supervised training loss function, W, H, C are the length, width and number of channels of the second target image, U is the second target image, M is the first binary mask, is the second binary mask, is the restored image.
7. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the weakly supervised image enhancement model training method as described in any one of claims 1 to 3 is implemented.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the computer program implements the weakly supervised image enhancement model training method described in any one of claims 1 to 3.