Unsupervised image restoration method and device for improving high-level visual task performance

By building a joint distribution and lightweight network of images and advanced visual tasks, the connection problems of image restoration and advanced visual tasks are solved, and performance and quality are improved, suitable for degraded environments.

CN119048375AActive Publication Date: 2024-11-29SUN YAT SEN UNIV
View PDF 2 Cites 0 Cited by

Patent Information

Application Number
CN202411101434.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-08-12
Publication Date
2024-11-29
Estimated Expiration
2044-08-12

AI Technical Summary

Technical Problem

The prior art is unable to effectively connect image recovery with advanced visual tasks, resulting in poor image recovery quality and low performance of advanced visual tasks, especially in degraded environments.

Method used

By constructing the joint distribution of the initial recovery image and the expected input image of the high-level visual task, the optimization target is obtained based on variational inference, a lightweight network is built using the gated fusion module and the transformation module, an unsupervised loss function is constructed, and unsupervised images are performed. Training to recover the target input image.

Benefits of technology

It significantly improves the performance and image restoration quality of advanced visual tasks, reduces the training cost and difficulties of pre-trained networks, and can be applied in low light, fog, rain and other scenarios.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119048375A_ABST
    Figure CN119048375A_ABST
Patent Text Reader

Abstract

The invention discloses an unsupervised image restoration method and device for improving high-level visual task performance, and the method comprises the steps: constructing the joint distribution of an initial restoration image and a high-level visual task expectation input image, and obtaining an optimization target based on variational inference; constructing a lightweight network based on the gating fusion module and the transformation module; constructing an unsupervised loss function according to the optimization target and the lightweight network; and according to the lightweight network and the unsupervised loss function, unsupervised training is performed through unpaired images, and then a target input image is restored to obtain a target restored image. The method can improve the performance of a high-level visual task, improves the performance of image restoration, and can be widely applied to the technical field of computer vision.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and in particular to an unsupervised image restoration method and device for improving the performance of high-level visual tasks. Background Art

[0002] In degraded environments such as fog and night scenes, the performance of high-level visual tasks (target detection, recognition, and segmentation) will be seriously affected. The currently proposed AODNet method of using it as a pre-step to enhance the performance of target detection tasks under synthetic fog conditions is not suitable for improving the performance of high-level visual tasks in real-world scenarios, and may even exacerbate the problem. Therefore, researchers use restoration as a module integration to improve the performance of high-level vision in degraded environments, but due to the lack of supervision of the restoration results, the restoration results produced by these joint methods perform poorly in human perception. In addition, some researchers have proposed using adversarial attacks to embed invisible information preferred by high-level visual tasks as restoration labels, attempting to simultaneously maintain image restoration performance and high-level visual performance, but experiments have shown that it is difficult for the restoration network to adapt to these transformed labels, and the performance of high-level visual tasks is still low. It can be seen that current research cannot effectively connect image restoration with high-level visual tasks, resulting in poor image restoration quality and poor performance of high-level visual tasks. Summary of the invention

[0003] In view of this, the main purpose of the embodiments of the present invention is to provide an unsupervised image restoration method and device for improving the performance of high-level visual tasks, in order to solve at least one problem of the prior art. The present invention can improve the performance of high-level visual tasks and improve the performance of image restoration.

[0004] To achieve the above object, an embodiment of the present invention provides an unsupervised image restoration method for improving the performance of high-level visual tasks, the method comprising the following steps:

[0005] Construct the joint distribution of the initial restored image and the expected input image of the high-level visual task, and obtain the optimization target based on variational inference;

[0006] Build a lightweight network based on the gated fusion module and the transformation module;

[0007] Constructing an unsupervised loss function according to the optimization objective and the lightweight network;

[0008] According to the lightweight network and the unsupervised loss function, unsupervised training is performed through unpaired images, and then the target input image is restored to obtain a target restored image.

[0009] In some embodiments, constructing a joint distribution of an initial restored image and an expected input image for a high-level visual task, and obtaining an optimization target based on variational inference, comprises the following steps:

[0010] Inputting the original degraded image into the restoration model to obtain the initial restored image;

[0011] Approximating the joint distribution of the initial restored image and the expected input image of the high-level visual task by variational inference to obtain a parameterized distribution;

[0012] The KL divergence between the joint distribution and the parameterized distribution is minimized to obtain an optimization target.

[0013] In some embodiments, the optimization target includes a first optimization item and a second optimization item; the expression of the optimization target is:

[0014]

[0015] The content retention item is the first optimization item, and the maximum boundary likelihood item is the second optimization item;

[0016] In the formula, Represents the optimization goal; I R represents the initial restored image; I HQ Represents a clean image, i.e., the input image expected by high-level visual tasks; represents a random variable I that follows a distribution Q defined by parameters θ R Expected value; represents the expected value of the parameterized distribution that the clean image obeys under the condition of a given initial restored image; P(I R |I HQ ) represents the joint distribution of the initial restored image given the clean image; represents the joint distribution of clean images conditioned on the high-level visual labels associated with i; represents the high-level visual label associated with i; m represents the object of interest for the high-level visual task in all the data.

[0017] In some embodiments, an unsupervised image restoration method for improving high-level visual task performance further includes the following steps:

[0018] Inputting the original degraded image and the initial restored image into the gated fusion module to extract weights to obtain initial weights;

[0019] The initial weights are processed by an activation function to obtain gated fusion weights;

[0020] A first fused image is generated according to the original degraded image, the initial restored image and the gated fusion weights.

[0021] In some embodiments, an unsupervised image restoration method for improving high-level visual task performance further includes the following steps:

[0022] Inputting the first fused image into the transformation module;

[0023] The first fused image is transformed to obtain a transformed image.

[0024] In some embodiments, constructing an unsupervised loss function according to the optimization target and the lightweight network includes the following steps:

[0025] Acquire a mapping process in the gated fusion module of the lightweight network;

[0026] Based on the cycle consistency loss, a first loss function is constructed according to the optimization objective and the mapping process.

[0027] In some embodiments, constructing an unsupervised loss function according to the optimization target and the lightweight network further includes the following steps:

[0028] Predicting an expected input image for the high-level visual task to obtain a first pseudo label;

[0029] Predicting and filtering the original degraded image and the initial restored image to obtain a second pseudo label;

[0030] Obtaining a fusion label according to the first pseudo label and the second pseudo label;

[0031] Constructing a second fused image through uncertainty estimation according to the original degraded image, the initial restored image and the expected input image of the high-level visual task;

[0032] Obtaining a first loss term according to the expected input image of the high-level visual task and the first pseudo label;

[0033] Obtaining a second loss term according to the second fused image and the fused label;

[0034] A second loss function is obtained according to the first loss term and the second loss term.

[0035] To achieve the above object, another aspect of an embodiment of the present invention provides an unsupervised image restoration device for improving the performance of high-level visual tasks, the device comprising:

[0036] The first module is used to construct the joint distribution of the initial restored image and the expected input image of the high-level visual task, and obtain the optimization target based on variational inference;

[0037] The second module is used to build a lightweight network based on the gated fusion module and the transformation module;

[0038] The third module is used to construct an unsupervised loss function according to the optimization target and the lightweight network;

[0039] The fourth module is used to perform unsupervised training through unpaired images according to the lightweight network and the unsupervised loss function, and then restore the target input image to obtain a target restored image.

[0040] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, it implements the unsupervised image restoration method for improving the performance of high-level visual tasks as described above.

[0041] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, it implements the above-mentioned unsupervised image restoration method for improving the performance of high-level visual tasks.

[0042] To achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the aforementioned unsupervised image restoration method for improving the performance of high-level visual tasks.

[0043] The embodiments of the present invention include at least the following beneficial effects: the present invention provides an unsupervised image restoration method and device for improving the performance of high-level visual tasks. The scheme constructs a joint distribution of an initial restored image and an expected input image of a high-level visual task, so that restoration promotes the high-level visual task, obtains a simplified optimization target to focus on the object of interest of the high-level visual task, so that the converted image can not only maintain the similarity of the original content, but also significantly improve the performance of the high-level visual task; based on a gated fusion module and a transformation module, a lightweight network is constructed to connect an image restoration network and a high-level visual task network, thereby avoiding the training of these pre-trained networks; according to the optimization target and the lightweight network, an unsupervised loss function is constructed; according to the lightweight network and the unsupervised loss function, unsupervised training is performed through unpaired images, and then the target input image is restored to obtain a target restored image, thereby improving the performance of image restoration. BRIEF DESCRIPTION OF THE DRAWINGS

[0044] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0045] Figure 1 is a flow chart of an unsupervised image restoration method for improving the performance of high-level visual tasks provided by an embodiment of the present invention;

[0046] Figure 2 is a schematic diagram of the overall structure of a lightweight network VaT provided by an embodiment of the present invention;

[0047] Figure 3a-Figure 3b is a schematic diagram of cycle consistency provided by an embodiment of the present invention;

[0048] Figure 4 It is a schematic diagram of the hardware structure of an electronic device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the embodiments of the present invention, and they are only examples of devices and methods consistent with some aspects of the embodiments of the present invention as detailed in the attached claims.

[0050] It should be noted that, although the functional modules are divided in the system schematic diagram and the logical order is shown in the flow chart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the flow chart. The terms "first / S100" and "second / S200" in the specification and claims and the above-mentioned drawings may be used to describe various concepts in this article, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, without departing from the scope of the embodiment of the present invention, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the words "if" and "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determination".

[0051] The terms "at least one", "multiple", "each", "any", etc. used in the present invention, at least one includes one, two or more, multiple includes two or more, each refers to each of the corresponding multiple, and any refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which the present invention belongs. The terms used herein are only for the purpose of describing the embodiments of the present invention and are not intended to limit the present invention.

[0053] Before describing the embodiments of the present invention in detail, some nouns and terms involved in the embodiments of the present invention are first described. The nouns and terms involved in the embodiments of the present invention are subject to the following explanations.

[0054] Degraded image. In the field of image degradation and restoration, image degradation refers to the degradation of image quality during the imaging, transmission and recording process due to factors such as imperfections in the imaging system, transmission media, equipment or specific environmental conditions (such as in rain, fog or night scenes), which manifests as blur, distortion or noise.

[0055] Unpaired degraded images usually refer to the high-quality (HQ) and low-quality (LQ) image pairs used for training and testing in image processing and computer vision that are not collected in pairs, that is, there is no direct correspondence between them.

[0056] At present, the performance of the target detection task can be successfully enhanced under synthetic fog conditions by using AODNet as a pre-step, but this dehazing method as a pre-processing step is not suitable for improving the performance of high-level visual tasks in real-world scenes, and may even aggravate the problem. In addition, due to the lack of supervision of the restoration results, the restoration results produced by the method of improving the high-level visual performance in degraded environments by integrating restoration as a module do not perform well in human perception. The method of using adversarial attacks to embed invisible information preferred by high-level visual tasks as restoration labels, attempting to simultaneously maintain image restoration performance and high-level task visual performance, its restoration network is difficult to adapt to these converted labels, resulting in only slight improvements in improving performance. Therefore, existing research cannot effectively connect image restoration with high-level visual tasks. The above methods all assume that the restoration or high-level visual model needs to be trained and requires paired training data. However, this is often difficult to achieve in many real-world scenarios. Today's restoration models and high-level visual models are both of considerable size, and training is bound to incur huge overhead. At the same time, it is usually challenging to obtain paired data for degraded scenes in the real world.

[0057] In view of this, if Figure 1 As shown, the embodiment of the present invention provides an unsupervised image restoration method for improving the performance of high-level visual tasks. The method may include but is not limited to steps S100 to S400:

[0058] Step S100, constructing a joint distribution of an initial restored image and an expected input image for a high-level visual task, and obtaining an optimization target based on variational inference;

[0059] Step S200, constructing a lightweight network based on the gated fusion module and the transformation module;

[0060] Step S300, constructing an unsupervised loss function according to the optimization target and the lightweight network;

[0061] Step S400, performing unsupervised training through unpaired images according to the lightweight network and the unsupervised loss function, and then restoring the target input image to obtain a target restored image.

[0062] In some embodiments, step S100 may include but is not limited to steps S110 to S130:

[0063] Step S110, inputting the original degraded image into the restoration model to obtain the initial restored image;

[0064] Step S120, approximating the joint distribution of the initial restored image and the expected input image of the high-level visual task by variational inference to obtain a parameterized distribution;

[0065] Step S130, minimizing the KL divergence between the joint distribution and the parameterized distribution to obtain an optimization target.

[0066] In step S110 of some embodiments, Figure 2 As shown, the original degraded image I LQ Input the restoration model to get the initial restoration image I R , then I R =R(I LQ ) indicates restore output.

[0067] In step S120 of some embodiments, by constructing a joint distribution P(I R ,I HQ ), so that the restoration promotes high-level visual tasks, but in practice it is difficult to obtain an analytical expression for the joint distribution, so the parameterized distribution Q is used through variational inference θ (I R ,I HQ ) to approximate the true joint distribution.

[0068] In step S130 of some embodiments, by minimizing the Kullback-Leibler divergence (KL divergence for short) between the joint distribution and the parameterized distribution, the parameterized distribution is adjusted to be as close as possible to the true joint distribution, and the following expression is obtained:

[0069]

[0070] After derivation, the initial optimization goal can be obtained as:

[0071]

[0072] in, Represents the optimization goal; I R represents the initial restored image; I HQ represents the clean image, that is, the input image expected by the high-level visual task; Q θ (I R ,I HQ ) represents the joint distribution of the initial restored image and the clean image using parameter fitting, that is, the parameterized distribution; P(I R ,I HQ ) represents the true joint distribution; represents a random variable I that follows a distribution Q defined by parameters θ R Expected value; represents the expected value of the parameterized distribution that the clean image obeys under the condition of a given initial restored image; P(I R |IHQ ) represents the true distribution that the initial restored image obeys given the clean image; P(I HQ ) represents the true distribution that the clean image obeys.

[0073] Since the number of target objects is much smaller than the number of all objects in high-level vision tasks in complex real-world scenes, it is difficult for unpaired learning methods to complete accurate mapping between many distracting objects. The original optimization objective of existing unpaired learning methods only considers the global mapping, which leads to a tendency to learn the average mapping of the two domains.

[0074] In order to better focus on the target object of the high-level visual task, in some embodiments, the optimization objective is updated to the following expression by ignoring interference terms to maximize the boundary likelihood:

[0075]

[0076] The optimization target includes a first optimization item and a second optimization item; the content retention item is the first optimization item, and the maximum boundary likelihood item is the second optimization item;

[0077] In the formula, represents the joint distribution of clean images conditioned on the high-level visual labels associated with i; represents the high-level visual label associated with i; m represents the object of interest for the high-level visual task in all the data.

[0078] Due to the performance limitations of the image restoration network, using the restored output of the image restoration network as the input of the connection network may not even be as good as using the original degraded image as the input. In step S200 of some embodiments, a lightweight network VaT is constructed to connect the image restoration network and the high-level visual task network, thereby avoiding the training of these pre-trained networks and reducing the cost and difficulty of retraining these networks. Figure 2 As shown in Figure 3, the lightweight network VaT consists of two parts: a gated fusion module (GFM) and a transformation module (TM).

[0079] In some embodiments, the gated fusion module (GFM) adaptively combines the input original degraded image I LQ And the restored image I output by the restoration network model R , to obtain the best network input for high-level visual tasks. For example, Figure 2As shown in FIG. 1 , the original degraded image and the initial restored image are input into a Transformer block and a point convolution layer with only one output channel through the gated fusion module to extract the initial weight w. Then, the initial weight is processed by the activation function σ(·) to obtain the gated fusion weight σ(w). Then, according to the original degraded image, the initial restored image and the gated fusion weight, a first fused image I can be generated. F , the expression of the first fused image is:

[0080] I F =σ(w)⊙I LQ +(1-σ(w))⊙I R

[0081] In the formula, I F represents the first fused image; w represents the initial weight; σ(w) represents the gated fusion weight; I LQ represents the original degraded image; ⊙ represents the element-wise product.

[0082] In some embodiments, by introducing a transformation module (TM), a Transformer block is used to convert the fused image into an image that is friendly to the advanced visual model. For example, Figure 2 As shown, the first fused image is input into the transformation module, the image details are retained through residual connection, and the image semantics are changed through the encoder-decoder framework, and finally the transformed image is converted. The Transformer structure has a good global receptive field to enhance learning ability, enabling VaT to understand contextual information.

[0083] In step S300 of some embodiments, an unsupervised loss function is designed based on the two optimization items of the optimization target and the established VaT network structure to achieve unsupervised learning.

[0084] In some embodiments, step S300 may include but is not limited to steps S310 to S320:

[0085] Step S310, obtaining a mapping process in the gated fusion module of the lightweight network;

[0086] Step S320: construct a first loss function based on the cycle consistency loss, according to the optimization target and the mapping process.

[0087] In some embodiments, in step S310 to step S320, in order to meet the content preservation item, a cycle consistency loss is introduced. Since the VaT network includes a gated fusion module, the original cycle consistency loss cannot be directly applied. Therefore, by modifying the cycle consistency training process, it will tend to maintain the cycle consistency between the first fused image and the VaT output, such as Figure 3a-Figure 3b As shown, this actually includes 3 mapping processes:

[0088]

[0089] in, Represents the original degraded image I LQ With the initial restored image I R The first fused image I is generated by the gated fusion module F process; Represents the first fused image I F To the clean image I HQ process; Represents the clean image I HQ To the first fused image I F According to the cycle consistency loss, such as Figure 3a The cycle consistency of the first fused image is shown as follows. F Through the process and The result should be the same as the original F Stay consistent; Figure 3b The cycle consistency of the clean image is shown. HQ Through the process and The result should be the same as the original HQ Keep consistent. The corresponding cycle consistency loss is:

[0090]

[0091] In the formula, Represents the first loss function.

[0092] In some embodiments, step S300 may also include but is not limited to steps S330 to S390:

[0093] Step S330, predicting the expected input image of the high-level visual task to obtain a first pseudo label;

[0094] Step S340, predicting and filtering the original degraded image and the initial restored image to obtain a second pseudo label;

[0095] Step S350, obtaining a fusion label according to the first pseudo label and the second pseudo label;

[0096] Step S360, constructing a second fused image through uncertainty estimation according to the original degraded image, the initial restored image and the expected input image of the high-level visual task;

[0097] Step S370, obtaining a first loss term according to the expected input image of the high-level visual task and the first pseudo label;

[0098] Step S380, obtaining a second loss term according to the second fused image and the fused label;

[0099] Step S390: Obtain a second loss function according to the first loss term and the second loss term.

[0100] In some embodiments, in steps S330 to S360, an effective unsupervised optimization method is proposed based on the inspiration of the self-training paradigm to improve the performance of high-level visual tasks. Figure 2 As shown, in the pseudo-label generation module, for the clean image I HQ ~P(I HQ ), which belongs to the same domain as the high-order vision model training data, the prediction of the original high-order vision model is used as the high-order vision task label (i.e., the first pseudo label Y HQ ); For the original degraded image and its corresponding initial restored image, the predictions of the high-order visual model for both the original degraded image and the initial restored image may contain potential true labels. In order to make full use of these potential true labels, non-parametric uncertainty estimation is used to filter these predictions, and the predictions with high certainty are regarded as the pseudo labels of the original degraded image (i.e., the second pseudo label Y LQ ). Based on the inspiration that weak enhancement can supervise strong enhancement, the second fused image I is constructed through uncertainty-guided fusion based on the original degraded image, the initial restored image and the clean image. mix , combined with the first pseudo label and the second pseudo label, the fusion label Y corresponding to the second fused image is obtained mix . Then the expression of the second fused image is:

[0101]

[0102] The expression of the fusion label is:

[0103] Y mix =U LQ ·λ·Y LQ +U HQ ·(1-λ)·Y HQ

[0104] In the formula, I mix represents the second fused image; represents the gated fusion module; λ follows the Beta distribution with parameters (α,α); Y mix U stands for fusion label; LQ represents the high-level visual prediction uncertainty obtained when predicting the original degraded image; U HQ represents the high-level visual prediction uncertainty obtained when predicting a clean image; Y HQ represents the pseudo label assigned to the clean image, i.e., the first pseudo label; Y LQ represents the pseudo label assigned to the original degraded image, i.e., the second pseudo label.

[0105] In steps S370 to S390 of some embodiments, the original high-level visual loss To constrain the prediction results of the high-level visual model, wherein, according to the clean image and the first pseudo label, a first loss term can be obtained; according to the second fused image and the fused label, a second loss term can be obtained; according to the first loss term and the second loss term, the expression of the second loss function is obtained as follows:

[0106]

[0107] In the formula, represents the second loss function; represents the first loss term; Represents the second loss term. For high-level vision tasks involving dense predictions (such as object detection), pseudo labels may miss potential true labels. Therefore, in dense tasks, the second loss term can be calculated only for labels containing objects.

[0108] In some embodiments, an unpaired degraded image is input into the lightweight network VaT of the embodiment of the present invention, and the lightweight network is unsupervisedly trained in combination with an unsupervised loss function. The target input image to be restored is sent to the trained lightweight network for restoration, and the target restored image can be obtained. After experimental comparison, in the dehazing detection task, the lightweight network VaT of the embodiment of the present invention is 15% higher than the mean average precision (mAP) of the existing unsupervised method. In the real degraded scene, the lightweight network VaT of the embodiment of the present invention is even about 4% higher than the mAP of the existing supervised method. In addition, the performance of the embodiment of the present invention in image quality enhancement in real scenes is also ahead of other methods, and it can improve the performance of high-level visual tasks and improve the performance of image restoration with the advantage of low training cost. The lightweight network VaT of the embodiment of the present invention can be applied to target detection, recognition, and segmentation in low-light scenes, foggy scenes, or rainy scenes, but is not limited to this.

[0109] The embodiment of the present invention further provides an unsupervised image restoration device for improving the performance of high-level visual tasks, which can implement the above-mentioned unsupervised image restoration method for improving the performance of high-level visual tasks. The device includes:

[0110] The first module is used to construct the joint distribution of the initial restored image and the expected input image of the high-level visual task, and obtain the optimization target based on variational inference;

[0111] The second module is used to build a lightweight network based on the gated fusion module and the transformation module;

[0112] The third module is used to construct an unsupervised loss function according to the optimization target and the lightweight network;

[0113] The fourth module is used to perform unsupervised training through unpaired images according to the lightweight network and the unsupervised loss function, and then restore the target input image to obtain a target restored image.

[0114] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] The embodiment of the present invention further provides an electronic device, which includes a processor and a memory, wherein the memory stores a computer program, and when the processor executes the computer program, the above-mentioned unsupervised image restoration method for improving the performance of high-level visual tasks is implemented. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0116] It can be understood that the contents of the above method embodiments are all applicable to the present device embodiments, the functions specifically implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] refer to Figure 4 , Figure 4 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0118] The processor 501 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention;

[0119] The memory 502 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 502 can store an operating system and other application programs. When the technical solution provided in the embodiments of this specification is implemented by software or firmware, the relevant program code is stored in the memory 502, and the processor 501 calls and executes an unsupervised image restoration method for improving the performance of high-level visual tasks in an embodiment of the present invention;

[0120] Input / output interface 503, used to implement information input and output;

[0121] Communication interface 504, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0122] A bus 505 that transmits information between the various components of the device (e.g., the processor 501, the memory 502, the input / output interface 503, and the communication interface 504);

[0123] The processor 501 , the memory 502 , the input / output interface 503 and the communication interface 504 are connected to each other in communication within the device via the bus 505 .

[0124] An embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned unsupervised image restoration method for improving the performance of high-level visual tasks is implemented.

[0125] It can be understood that the contents of the above method embodiments are all applicable to the present storage medium embodiments, the functions specifically implemented by the present storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0126] The embodiment of the present invention also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the aforementioned unsupervised image restoration method for improving the performance of high-level visual tasks.

[0127] In summary, the unsupervised image restoration method and device for improving the performance of high-level visual tasks according to the embodiments of the present invention have the following advantages:

[0128] 1. The embodiment of the present invention establishes a joint distribution model between the initial restored image output by image restoration and the clean image of high-level visual input through variational reasoning, optimizes the conversion process of the output image of the restoration model to the input image that is beneficial to machine vision, simplifies the optimization target to focus on the objects of interest in high-level visual tasks, and achieves the situation that the converted image can maintain the similarity of the original content and significantly improve the performance of image restoration and high-level visual tasks without relying on paired images.

[0129] 2. The embodiment of the present invention uses variational inference to approximate the joint distribution of the recovery network output and the high-level visual network input, and decomposes the optimization objective into a content preservation term and a maximum boundary likelihood term related to high-level visual tasks. Uncertainty filtering is used to obtain reliable pseudo-labels for supervising the image restoration process guided by uncertainty, thereby realizing unsupervised learning that does not require paired data and improving the performance of image restoration and high-level visual tasks.

[0130] 3. The embodiment of the present invention constructs a lightweight network VaT to connect the image restoration network and the high-level visual task network, thereby avoiding the training of these pre-trained networks and reducing the cost and difficulty of retraining these networks. The lightweight network VaT also converts the fused image into an image that is friendly to the high-level visual model through a Transformer structure with a good global receptive field, thereby enhancing the learning ability and enabling VaT to understand contextual information.

[0131] 4. The embodiment of the present invention designs a loss function based on the two optimization items of the optimization target and the established VaT network structure to achieve unsupervised learning. By introducing the cycle consistency loss to meet the content preservation item, inspired by the self-training paradigm, an effective unsupervised optimization method is proposed to improve the performance of high-level visual tasks, and the uncertainty-guided self-training strategy is used to optimize the network.

[0132] 5. The embodiment of the present invention effectively connects the image restoration task and the high-level visual task through the established VaT network structure and the designed loss function without retraining the existing image restoration network and high-level visual tasks, thereby enhancing the performance of high-level visual tasks in degraded scenarios.

[0133] In some selectable embodiments, the function / operation mentioned in the block diagram may not occur in the order mentioned in the operation diagram. For example, depending on the function / operation involved, the two boxes shown in succession can actually be executed substantially simultaneously or the boxes can sometimes be executed in reverse order. In addition, the embodiment presented and described in the flow chart of the present invention is provided by way of example, for the purpose of providing a more comprehensive understanding of technology. The disclosed method is not limited to the operation and logic flow presented herein. Selectable embodiments are expected, wherein the order of various operations is changed and the sub-operation of a part for which is described as a larger operation is performed independently.

[0134] In addition, although the present invention is described in the context of functional modules, it should be understood that, unless otherwise specified, one or more of the functions and / or features described may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in separate physical devices or software modules. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the present invention. More specifically, in view of the properties, functions, and internal relationships of the various functional modules in the device disclosed herein, the actual implementation of the module will be understood within the conventional skills of the engineer. Therefore, those skilled in the art can implement the present invention set forth in the claims without excessive experimentation using ordinary techniques. It is also understood that the specific concepts disclosed are merely illustrative and are not intended to limit the scope of the present invention, which is determined by the full scope of the appended claims and their equivalents.

[0135] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc., which can store program codes.

[0136] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or in conjunction with such instruction execution systems, devices or apparatuses. For the purposes of this specification, "computer-readable medium" can be any device that can contain, store, communicate, propagate or transmit a program for use by an instruction execution system, device or apparatus, or in conjunction with such instruction execution systems, devices or apparatuses.

[0137] More specific examples of computer-readable media (a non-exhaustive list) include the following: an electrical connection with one or more wires (electronic device), a portable computer disk case (magnetic device), a random access memory (RAM), a read-only memory (ROM), an erasable and programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disk read-only memory (CDROM). In addition, the computer-readable medium may even be a paper or other suitable medium on which the program is printed, since the program may be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, deciphering or, if necessary, processing in another suitable manner, and then stored in a computer memory.

[0138] It should be understood that the various parts of the present invention can be implemented by hardware, software, firmware or a combination thereof. In the above-mentioned embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented by hardware, as in another embodiment, it can be implemented by any one of the following technologies known in the art or their combination: a discrete logic circuit having a logic gate circuit for implementing a logic function for a data signal, a dedicated integrated circuit having a suitable combination of logic gate circuits, a programmable gate array (PGA), a field programmable gate array (FPGA), etc.

[0139] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "examples", "specific examples", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present invention. In this specification, the schematic representation of the above terms does not necessarily refer to the same embodiment or example. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner.

[0140] Although the embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the claims and their equivalents.

[0141] The above is a specific description of the preferred implementation of the present invention, but the present invention is not limited to the described embodiments. Those skilled in the art may make various equivalent modifications or substitutions without violating the spirit of the present invention. These equivalent modifications or substitutions are all included in the scope defined by the claims of the present invention.

Claims

1. An unsupervised image restoration method for improving the performance of high-level visual tasks, characterized in that: The following steps are involved: Construct the joint distribution of the initial restored image and the expected input image of the high-level visual task, and obtain the optimization target based on variational inference; Build a lightweight network based on the gated fusion module and the transformation module; Constructing an unsupervised loss function according to the optimization objective and the lightweight network; According to the lightweight network and the unsupervised loss function, unsupervised training is performed through unpaired images, and then the target input image is restored to obtain a target restored image.

2. The unsupervised image restoration method for improving high-level visual task performance according to claim 1, characterized in that: The method of constructing a joint distribution of an initial restored image and an expected input image of a high-level visual task, and obtaining an optimization target based on variational inference, includes the following steps: Inputting the original degraded image into the restoration model to obtain the initial restored image; Approximating the joint distribution of the initial restored image and the expected input image of the high-level visual task by variational inference to obtain a parameterized distribution; The KL divergence between the joint distribution and the parameterized distribution is minimized to obtain an optimization target.

3. The unsupervised image restoration method for improving high-level visual task performance according to claim 1, characterized in that: The optimization target includes a first optimization item and a second optimization item; The expression of the optimization objective is: The content retention item is the first optimization item, and the maximum boundary likelihood item is the second optimization item; In the formula, Represents the optimization goal; I R represents the initial restored image; I HQ Represents a clean image, i.e., the input image expected by high-level visual tasks; represents a random variable I that follows a distribution Q defined by parameters θ R Expected value; represents the expected value of the parameterized distribution that the clean image obeys under the condition of a given initial restored image; P(I R |I HQ ) represents the joint distribution of the initial restored image given the clean image; represents the joint distribution of clean images conditioned on the high-level visual labels associated with i; represents the high-level visual label associated with i; m represents the object of interest for the high-level visual task in all the data.

4. The unsupervised image restoration method for improving high-level visual task performance according to claim 1, characterized in that: The following steps are also included: Inputting the original degraded image and the initial restored image into the gated fusion module to extract weights to obtain initial weights; The initial weights are processed by an activation function to obtain gated fusion weights; A first fused image is generated according to the original degraded image, the initial restored image and the gated fusion weights.

5. The unsupervised image restoration method for improving high-level visual task performance according to claim 4, characterized in that: The following steps are also included: Inputting the first fused image into the transformation module; The first fused image is transformed to obtain a transformed image.

6. The unsupervised image restoration method for improving high-level visual task performance according to claim 1, characterized in that: According to the optimization target and the lightweight network, an unsupervised loss function is constructed, comprising the following steps: Acquire a mapping process in the gated fusion module of the lightweight network; Based on the cycle consistency loss, a first loss function is constructed according to the optimization objective and the mapping process.

7. The unsupervised image restoration method for improving high-level visual task performance according to claim 1, characterized in that: According to the optimization target and the lightweight network, an unsupervised loss function is constructed, and the following steps are also included: Predicting an expected input image for the high-level visual task to obtain a first pseudo label; Predicting and filtering the original degraded image and the initial restored image to obtain a second pseudo label; Obtaining a fusion label according to the first pseudo label and the second pseudo label; Constructing a second fused image through uncertainty estimation according to the original degraded image, the initial restored image and the expected input image of the high-level visual task; Obtaining a first loss term according to the expected input image of the high-level visual task and the first pseudo label; Obtaining a second loss term according to the second fused image and the fused label; A second loss function is obtained according to the first loss term and the second loss term.

8. An unsupervised image restoration device for improving the performance of high-level visual tasks, characterized in that: include: The first module is used to construct the joint distribution of the initial restored image and the expected input image of the high-level visual task, and obtain the optimization target based on variational inference; The second module is used to build a lightweight network based on the gated fusion module and the transformation module; The third module is used to construct an unsupervised loss function according to the optimization target and the lightweight network; The fourth module is used to perform unsupervised training through unpaired images according to the lightweight network and the unsupervised loss function, and then restore the target input image to obtain a target restored image.

9. An electronic device, characterized in that: including a processor and a memory; The memory is used to store programs; The processor executes the program to implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that: The storage medium stores a program, and the program is executed by a processor to implement the method according to any one of claims 1 to 7.

Citation Information

Patent Citations

  • Blurred image restoration method based on unsupervised generative adversarial network

    CN112884671A

  • Image defogging method based on semi-supervision

    CN114155165A