An unsupervised image restoration method and device for improving high-level visual task performance

Abstract

The application discloses an unsupervised image restoration method and device for improving high-level visual task performance, and the method comprises the following steps: constructing a joint distribution of an initial recovery image and a high-level visual task expected input image, and obtaining an optimization target based on variational inference; constructing a light-weight network based on a gating fusion module and a transformation module; constructing an unsupervised loss function according to the optimization target and the light-weight network; and performing unsupervised training on unpaired images according to the light-weight network and the unsupervised loss function, and then performing recovery on a target input image to obtain a target recovery image. The application can improve the performance of high-level visual tasks and improve the performance of image restoration, and can be widely applied to the technical field of computer vision.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to an unsupervised image restoration method and apparatus for improving the performance of high-level vision tasks. Background Technology

[0002] In degraded environments such as fog and nighttime scenes, the performance of advanced vision tasks (object detection, recognition, and segmentation) is severely impacted. Current methods, such as AODNet as a pre-step to enhance object detection performance under synthetic fog conditions, are not applicable to improving advanced vision performance in real-world scenes and may even exacerbate the problem. Therefore, researchers have integrated image restoration as a module to improve advanced vision performance in degraded environments. However, due to a lack of supervision over the restoration results, these joint methods produce restorations that perform poorly in human perception. Furthermore, some researchers have proposed using adversarial attacks to embed invisible information preferred by advanced vision tasks as restoration labels, attempting to maintain both image restoration performance and advanced vision performance. However, experiments show that restoration networks struggle to adapt to these transformed labels, and performance on advanced vision tasks remains low. Thus, current research fails to effectively connect image restoration with advanced vision tasks, resulting in poor image restoration quality and poor performance on advanced vision tasks. Summary of the Invention

[0003] In view of this, the main objective of the embodiments of the present invention is to provide an unsupervised image restoration method and apparatus for improving the performance of high-level vision tasks, in order to solve at least one problem of the prior art. The present invention can improve the performance of high-level vision tasks and enhance the performance of image restoration.

[0004] To achieve the above objectives, one aspect of the present invention provides an unsupervised image restoration method to improve the performance of high-level vision tasks, the method comprising the following steps:

[0005] Construct a joint distribution of the initial restored image and the expected input image for the high-level vision task, and obtain the optimization objective based on variational inference;

[0006] A lightweight network is constructed based on the gating fusion module and the transformation module;

[0007] Based on the optimization objective and the lightweight network, an unsupervised loss function is constructed.

[0008] Based on the lightweight network and the unsupervised loss function, unsupervised training is performed using unpaired images to recover the target input image, thereby obtaining the target recovered image.

[0009] In some embodiments, constructing the joint distribution of the initial restored image and the expected input image for the high-level vision task, and obtaining the optimization objective based on variational inference, includes the following steps:

[0010] The original degraded image is input into the restoration model to obtain the initial restored image;

[0011] A parameterized distribution is obtained by approximating the joint distribution of the initial restored image and the expected input image of the high-level vision task through variational inference;

[0012] The optimization objective is obtained by minimizing the KL divergence between the joint distribution and the parameterized distribution.

[0013] In some embodiments, the optimization objective includes a first optimization term and a second optimization term; the expression for the optimization objective is:

[0014]

[0015] Among them, the content retention term is the first optimization term, and the maximum boundary likelihood term is the second optimization term;

[0016] In the formula, Represents the optimization objective; I R Represents the initial restored image; I HQ This represents a clean image, i.e., the input image expected by high-level vision tasks; I represents a random variable that follows a distribution Q defined by parameter θ. R Expected value; P(I) represents the expected value of the parameterized distribution that the clean image follows, given an initial restored image; R |I HQ ) represents the joint distribution that the initially recovered image follows, given a clean image; This represents the joint distribution that a clean image follows under conditions of high-level visual labels associated with i; represents the high-level visual label associated with i; m represents the object of interest for high-level visual tasks across all data.

[0017] In some embodiments, an unsupervised image restoration method for improving the performance of high-level vision tasks further includes the following steps:

[0018] The original degraded image and the initial restored image are input into the gated fusion module to extract weights and obtain initial weights;

[0019] The initial weights are processed by an activation function to obtain gated fusion weights;

[0020] A first fused image is generated based on the original degraded image, the initial restored image, and the gating fusion weights.

[0021] In some embodiments, an unsupervised image restoration method for improving the performance of high-level vision tasks further includes the following steps:

[0022] The first fused image is input into the transformation module;

[0023] The first fused image is transformed to obtain a transformed image.

[0024] In some embodiments, an unsupervised loss function is constructed based on the optimization objective and the lightweight network, including the following steps:

[0025] Obtain the mapping process in the gated fusion module of the lightweight network;

[0026] Based on the cycle consistency loss, a first loss function is constructed according to the optimization objective and the mapping process.

[0027] In some embodiments, constructing an unsupervised loss function based on the optimization objective and the lightweight network further includes the following steps:

[0028] The first pseudo-label is obtained by predicting the expected input image for the high-level vision task.

[0029] The original degraded image and the initial restored image are predicted and filtered to obtain the second pseudo-label;

[0030] Based on the first pseudo-tag and the second pseudo-tag, a fused tag is obtained;

[0031] Based on the original degraded image, the initial restored image, and the expected input image for the high-level vision task, a second fused image is constructed through uncertainty estimation;

[0032] Based on the expected input image of the high-level vision task and the first pseudo-label, a first loss term is obtained;

[0033] Based on the second fused image and the fused label, a second loss term is obtained;

[0034] The second loss function is obtained based on the first loss term and the second loss term.

[0035] To achieve the above objectives, another aspect of the present invention provides an unsupervised image restoration apparatus for improving the performance of high-level vision tasks, the apparatus comprising:

[0036] The first module is used to construct the joint distribution of the initial restored image and the expected input image for the high-level vision task, and to obtain the optimization objective based on variational inference;

[0037] The second module is used to build a lightweight network based on the gating fusion module and the transformation module;

[0038] The third module is used to construct an unsupervised loss function based on the optimization objective and the lightweight network.

[0039] The fourth module is used to perform unsupervised training on unpaired images based on the lightweight network and the unsupervised loss function, and then recover the target input image to obtain the target recovered image.

[0040] To achieve the above objectives, another aspect of the present invention provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the aforementioned unsupervised image restoration method for improving the performance of high-level vision tasks.

[0041] To achieve the above objectives, another aspect of the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned unsupervised image restoration method for improving the performance of high-level vision tasks.

[0042] To achieve the above objectives, another aspect of the present invention provides a computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned unsupervised image restoration method for improving the performance of high-level vision tasks.

[0043] The embodiments of the present invention include at least the following beneficial effects: The present invention provides an unsupervised image restoration method and apparatus for improving the performance of high-level vision tasks. This scheme constructs a joint distribution of an initial restored image and the expected input image for the high-level vision task, enabling restoration to facilitate the high-level vision task, obtaining a simplified optimization objective to focus on the object of interest for the high-level vision task, so that the transformed image can maintain the similarity of the original content and significantly improve the performance of the high-level vision task; a lightweight network is constructed based on a gated fusion module and a transformation module to connect the image restoration network and the high-level vision task network, thereby avoiding the training of these pre-trained networks; an unsupervised loss function is constructed according to the optimization objective and the lightweight network; unsupervised training is performed on unpaired images according to the lightweight network and the unsupervised loss function, and then the target input image is restored to obtain the target restored image, thereby improving the performance of image restoration. Attached Figure Description

[0044] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0045] Figure 1 This is a flowchart of an unsupervised image restoration method for improving the performance of high-level vision tasks, provided by an embodiment of the present invention;

[0046] Figure 2 This is a schematic diagram of the overall structure of the lightweight VaT network provided in an embodiment of the present invention;

[0047] Figures 3a-3b This is a schematic diagram of cycle consistency provided in an embodiment of the present invention;

[0048] Figure 4 This is a schematic diagram of the hardware structure of the electronic device provided in an embodiment of the present invention. Detailed Implementation

[0049] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with those of this invention; they are merely examples of apparatuses and methods consistent with some aspects of the embodiments of this invention as detailed in the appended claims.

[0050] It should be noted that although functional modules are divided in the system diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the system or the order in the flowchart. The terms "first / S100" and "second / S200" in the specification, claims, and the foregoing drawings may be used herein to describe various concepts, but unless specifically stated otherwise, these concepts are not limited by these terms. These terms are used only to distinguish one concept from another. For example, first information may also be referred to as second information without departing from the scope of the embodiments of the invention, and similarly, second information may also be referred to as first information. Depending on the context, the words "if" or "when" as used herein may be interpreted as "when," "in response to a determination," or "in the event of a determination."

[0051] The terms “at least one,” “multiple,” “each,” “any,” etc., used in this invention, “at least one” includes one, two, or more than two; “multiple” includes two or more than two; “each” refers to each of the corresponding multiple; and “any” refers to any one of the multiple.

[0052] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention pertains. The terminology used herein is for the purpose of describing embodiments of the invention only and is not intended to limit the invention.

[0053] Before providing a detailed description of the embodiments of the present invention, some of the nouns and terms involved in the embodiments of the present invention will be explained first. The nouns and terms involved in the embodiments of the present invention are subject to the following interpretations.

[0054] In the field of image degradation and restoration, image degradation refers to the decline in image quality during the imaging, transmission, and recording process due to imperfections in the imaging system, transmission medium, equipment, or specific environmental conditions (such as in rain, fog, or nighttime scenes), manifesting as blurriness, distortion, or noise.

[0055] Unpaired degraded images generally refer to high-quality (HQ) and low-quality (LQ) image pairs used for training and testing in image processing and computer vision that are not acquired in pairs, meaning there is no direct correspondence between them.

[0056] Currently, using AODNet as a preprocessing step has successfully enhanced the performance of object detection tasks under synthetic fog conditions. However, this dehazing method as a preprocessing step is not suitable for improving the performance of high-level vision tasks in real-world scenes, and may even exacerbate the problem. Furthermore, due to the lack of supervision over the restoration results, methods that integrate restoration as a module to improve high-level vision performance in degraded environments produce restoration results that perform poorly in human perception. Methods that use adversarial attacks to embed invisible information preferred by high-level vision tasks as restoration labels, attempting to maintain both image restoration performance and high-level vision performance, struggle to adapt their restoration networks to these transformed labels, resulting in only slight performance improvements. Therefore, existing research cannot effectively connect image restoration with high-level vision tasks. All the above methods assume that the restoration or high-level vision model needs to be trained and requires paired training data. However, this is often difficult to achieve in many real-world scenarios. Current restoration and high-level vision models are quite large, and training them would inevitably incur huge overhead. At the same time, obtaining paired data for degraded scenes in the real world is usually challenging.

[0057] In view of this, such as Figure 1 As shown, this embodiment of the invention provides an unsupervised image restoration method to improve the performance of high-level vision tasks. This method may include, but is not limited to, steps S100 to S400:

[0058] Step S100: Construct the joint distribution of the initial restored image and the expected input image for the high-level vision task, and obtain the optimization objective based on variational inference;

[0059] Step S200: Construct a lightweight network based on the gated fusion module and the transformation module;

[0060] Step S300: Construct an unsupervised loss function based on the optimization objective and the lightweight network;

[0061] Step S400: Based on the lightweight network and the unsupervised loss function, unsupervised training is performed using unpaired images to recover the target input image, thereby obtaining the recovered target image.

[0062] In some embodiments, step S100 may include, but is not limited to, steps S110 to S130:

[0063] Step S110: Input the original degraded image into the restoration model to obtain the initial restored image;

[0064] Step S120: Approximate the joint distribution of the initial restored image and the expected input image of the high-level vision task through variational inference to obtain the parameterized distribution;

[0065] Step S130: Minimize the KL divergence between the joint distribution and the parameterized distribution to obtain the optimization objective.

[0066] In step S110 of some embodiments, such as Figure 2 As shown, the original degraded image I LQ By inputting the restoration model, we can obtain the initial restored image I. R Then I R =R(I LQ ) indicates the restored output.

[0067] In step S120 of some embodiments, a joint distribution P(I0) of the original degraded image and the high-level vision task expected input image (i.e., the clean image) is constructed. R ,I HQ This allows for reconstruction to facilitate advanced visual tasks, but in practice, it is difficult to obtain an analytical expression for the joint distribution. Therefore, variational inference is used to employ the parameterized distribution Q. θ (I R ,I HQ To approximate the true joint distribution.

[0068] In step S130 of some embodiments, the parameterized distribution is adjusted to be as close as possible to the true joint distribution by minimizing the Kullback-Leibler divergence (KL divergence) between the joint distribution and the parameterized distribution, thus yielding the following expression:

[0069]

[0070] Through derivation, the initial optimization objective can be obtained as follows:

[0071]

[0072] in, Represents the optimization objective; I R Represents the initial restored image; I HQ Represents a clean image, i.e., the input image expected for high-level vision tasks; Q θ (I R ,I HQ P(I) represents the joint distribution of the initial restored image and the clean image fitted using parameters, i.e., the parameterized distribution; R ,I HQ ) represents the true joint distribution; I represents a random variable that follows a distribution Q defined by parameter θ. R Expected value; P(I) represents the expected value of the parameterized distribution that the clean image follows, given an initial restored image; R |IHQ P(I) represents the true distribution that the initially recovered image follows, given a clean image; HQ ) represents the true distribution that a clean image follows.

[0073] In advanced vision tasks set in complex real-world scenarios, the number of target objects is far less than the total number of objects, making it difficult for unpaired learning methods to achieve accurate mappings among many interfering objects. Existing unpaired learning methods' original optimization objective only considers the global mapping, leading to a tendency to learn the average mapping between two domains.

[0074] To better focus on the target object in advanced vision tasks, in some embodiments, the optimization objective is updated to the following expression by ignoring distractors to maximize the boundary likelihood:

[0075]

[0076] The optimization objective includes a first optimization term and a second optimization term; the content retention term is the first optimization term, and the maximum boundary likelihood term is the second optimization term.

[0077] In the formula, This represents the joint distribution that a clean image follows under conditions of high-level visual labels associated with i; represents the high-level visual label associated with i; m represents the object of interest for high-level visual tasks across all data.

[0078] Due to performance limitations of image restoration networks, using the restored output of an image restoration network as input to a connecting network may not even be as effective as using the original degraded image as input. In some embodiments, step S200 connects the image restoration network and the high-level vision task network by constructing a lightweight network VaT, thereby avoiding the need to train these pre-trained networks and reducing the cost and difficulty of retraining them. Figure 2 As shown, the lightweight network VaT consists of two parts: a gated fusion module (GFM) and a transformation module (TM).

[0079] In some embodiments, the gated fusion module (GFM) adaptively combines the input original degraded image I LQ The restored image I output by the restoration network model R To obtain optimal network input for high-level vision tasks. For example, such as... Figure 2As shown, the gated fusion module inputs the original degraded image and the initial restored image into a Transformer block and a point convolutional layer with only one output channel to extract initial weights w. Then, the initial weights are processed using the activation function σ(·) to obtain the gated fusion weights σ(w). Based on the original degraded image, the initial restored image, and the gated fusion weights, a first fused image I can be generated. F The expression for the first fused image is:

[0080] I F =σ(w)⊙I LQ +(1-σ(w))⊙I R

[0081] In the formula, I F Represents the first fused image; w represents the initial weights; σ(w) represents the gated fusion weights; I LQ ⊙ represents the original degraded image; ⊙ represents element-wise product.

[0082] In some embodiments, a Transformer block is used to transform the fused image into an image that is compatible with high-level vision models by introducing a Transformer module (TM). For example, such as... Figure 2 As shown, the first fused image is input into the transformation module. Residual connections preserve image details, while an encoder-decoder framework alters the image semantics, ultimately yielding a transformed image. The Transformer structure possesses a good global receptive field to enhance learning capabilities, enabling VaT to understand contextual information.

[0083] In step S300 of some embodiments, an unsupervised loss function is designed based on the two optimization terms of the optimization objective and the established VaT network structure to achieve unsupervised learning.

[0084] In some embodiments, step S300 may include, but is not limited to, steps S310 to S320:

[0085] Step S310: Obtain the mapping process in the gated fusion module of the lightweight network;

[0086] Step S320: Based on the cycle consistency loss, construct a first loss function according to the optimization objective and the mapping process.

[0087] In steps S310 to S320 of some embodiments, a cycle consistency loss is introduced to satisfy the content retention requirement. Since the VaT network includes a gated fusion module, the original cycle consistency loss cannot be directly applied. Therefore, by modifying the cycle consistency training process, it is preferable to maintain cycle consistency between the first fused image and the VaT output, such as... Figures 3a-3b As shown, this essentially involves three mapping processes:

[0088]

[0089] in, Represents the original degraded image I LQ Compared with the initial restored image I R The first fused image I is generated through the gated fusion module. F The process; Represents the first fused image I F To clean image I HQ The process; Represents a clean image I HQ To the first fused image I F The process. Based on the cycle consistency loss, such as... Figure 3a The cycle consistency shown is for the first fused image I. F Process and The result should be consistent with the original I. F Maintain consistency; such as Figure 3b The cycle consistency shown is for a clean image, clean image I. HQ Process and The result should be consistent with the original I. HQ Maintain consistency. Therefore, the corresponding cycle consistency loss is:

[0090]

[0091] In the formula, This represents the first loss function.

[0092] In some embodiments, step S300 may also include, but is not limited to, steps S330 to S390:

[0093] Step S330: Predict the expected input image for the high-level vision task to obtain the first pseudo-label;

[0094] Step S340: Predict and filter the original degraded image and the initial restored image to obtain the second pseudo-label;

[0095] Step S350: Obtain the fused tag based on the first pseudo tag and the second pseudo tag;

[0096] Step S360: Based on the original degraded image, the initial restored image, and the expected input image for the high-level vision task, a second fused image is constructed through uncertainty estimation;

[0097] Step S370: Based on the input image expected by the high-level vision task and the first pseudo-label, obtain the first loss term;

[0098] Step S380: Obtain the second loss term based on the second fused image and the fused label;

[0099] Step S390: Obtain the second loss function based on the first loss term and the second loss term.

[0100] In steps S330 to S360 of some embodiments, inspired by the self-training paradigm, an effective unsupervised optimization method is proposed to improve the performance of high-level vision tasks. For example... Figure 2 As shown, in the pseudo-label generation module, for a clean image I... HQ ~P(I HQ If the data belongs to the same domain as the training data of the high-order visual model, then the prediction of the original high-order visual model is used as the label for the high-order visual task (i.e., the first pseudo-label Y). HQ For the original degraded image and its corresponding initial restored image, the predictions made by the high-order visual model for both the original degraded image and the initial restored image may contain potential true labels. To fully utilize these potential true labels, nonparametric uncertainty estimation is used to filter these predictions, and the predictions with high determinism are regarded as pseudo-labels of the original degraded image (i.e., the second pseudo-label Y). LQ Inspired by the idea that weak enhancement can supervise strong enhancement, a second fused image I is constructed from the original degraded image, the initial restored image, and the clean image through uncertainty-guided fusion. mix By combining the first pseudo-label and the second pseudo-label, the fusion label Y corresponding to the second fused image is obtained. mix The expression for the second fused image is then:

[0101]

[0102] The expression for the fusion tag is:

[0103] Y mix =U LQ ·λ·Y LQ +U HQ ·(1-λ)·Y HQ

[0104] In the formula, I mix Represents the second fused image; Represents the gated fusion module; λ follows a Beta distribution with parameters (α, α); Y mix Represents a fusion tag; U LQ U represents the uncertainty in high-level visual predictions obtained when predicting the original degraded image; HQ Y represents the high-level visual prediction uncertainty obtained when predicting clean images; HQ Y represents the pseudo-label assigned to the clean image, i.e., the first pseudo-label; LQ This represents the pseudo-label assigned to the original degraded image, i.e., the second pseudo-label.

[0105] In steps S370 to S390 of some embodiments, the original high-level visual loss is used. To constrain the prediction results of the high-level vision model, a first loss term is obtained based on the clean image and the first pseudo-label; a second loss term is obtained based on the second fused image and the fused label; and the expression for the second loss function is obtained based on the first loss term and the second loss term:

[0106]

[0107] In the formula, This represents the second loss function; This represents the first loss item; This represents the second loss term. For advanced vision tasks involving dense prediction (such as object detection), false labels may miss potential true labels. Therefore, in dense tasks, the second loss term can be computed only for labels containing the target.

[0108] In some embodiments, unpaired degraded images are input into the lightweight VaT network of this invention, and unsupervised training is performed on the lightweight network using an unsupervised loss function. The target input image to be restored is then fed into the trained lightweight network for restoration, resulting in a restored target image. Experimental comparisons show that in dehazing detection tasks, the lightweight VaT network of this invention achieves a 15% higher mean accuracy (mAP) than existing unsupervised methods. In real-world degraded scenarios, the lightweight VaT network of this invention even outperforms existing supervised methods by approximately 4% in mAP. Furthermore, this invention also leads other methods in image quality enhancement in real-world scenarios, improving the performance of high-level vision tasks and enhancing image restoration performance with low training costs. The lightweight VaT network of this invention can be applied to target detection, recognition, and segmentation in low-light, foggy, or rainy scenarios, but is not limited to these applications.

[0109] This invention also provides an unsupervised image restoration apparatus for improving the performance of high-level vision tasks, which can implement the above-mentioned unsupervised image restoration method for improving the performance of high-level vision tasks. The apparatus includes:

[0110] The first module is used to construct the joint distribution of the initial restored image and the expected input image for the high-level vision task, and to obtain the optimization objective based on variational inference;

[0111] The second module is used to build a lightweight network based on the gating fusion module and the transformation module;

[0112] The third module is used to construct an unsupervised loss function based on the optimization objective and the lightweight network.

[0113] The fourth module is used to perform unsupervised training on unpaired images based on the lightweight network and the unsupervised loss function, and then recover the target input image to obtain the target recovered image.

[0114] It is understood that the content of the above method embodiments is applicable to the present device embodiments. The specific functions implemented by the present device embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0115] This invention also provides an electronic device, which includes a processor and a memory. The memory stores a computer program, and when the processor executes the computer program, it implements the aforementioned unsupervised image restoration method for improving the performance of high-level vision tasks. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0116] It is understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0117] refer to Figure 4 , Figure 4 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0118] The processor 501 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present invention.

[0119] The memory 502 can be implemented as a read-only memory (ROM), static storage device, dynamic storage device, or random access memory (RAM). The memory 502 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 502 and is called and executed by the processor 501 to implement an unsupervised image restoration method for improving the performance of high-level vision tasks according to an embodiment of the present invention.

[0120] The input / output interface 503 is used to implement information input and output;

[0121] The communication interface 504 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, network cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0122] Bus 505 transmits information between various components of the device (e.g., processor 501, memory 502, input / output interface 503, and communication interface 504);

[0123] The processor 501, memory 502, input / output interface 503, and communication interface 504 are connected to each other within the device via bus 505.

[0124] This invention also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the aforementioned unsupervised image restoration method for improving the performance of high-level vision tasks.

[0125] It is understood that the content of the above method embodiments is applicable to this storage medium embodiment. The specific functions implemented in this storage medium embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0126] This invention also provides a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device can read the computer instructions from the computer-readable storage medium and execute the computer instructions, causing the computer device to perform the aforementioned unsupervised image restoration method for improving the performance of high-level vision tasks.

[0127] In summary, the unsupervised image restoration method and apparatus for improving the performance of high-level vision tasks according to embodiments of the present invention have the following advantages:

[0128] 1. The embodiments of the present invention establish a joint distribution model between the initial restored image output by the image restoration output and the clean image of the high-level vision input through variational inference, so as to optimize the conversion process of the output image of the restoration model to the input image that is beneficial to machine vision. The optimization objective is simplified to focus on the object of interest for high-level vision tasks. It achieves that the converted image can maintain the similarity of the original content and significantly improve the performance of image restoration and high-level vision tasks without relying on paired images.

[0129] 2. The embodiments of the present invention utilize variational inference to approximate the joint distribution of the network output and the input of the high-level vision network, and decompose the optimization objective into a content preservation term and a maximum boundary likelihood term related to the high-level vision task. Reliable pseudo-labels are obtained by using uncertainty filtering to supervise the image restoration process guided by uncertainty. This achieves unsupervised learning without pairwise data and improves the performance of image restoration and high-level vision tasks.

[0130] 3. In this embodiment of the invention, a lightweight network VaT is constructed to connect the image restoration network and the high-level vision task network, thereby avoiding the training of these pre-trained networks, reducing the cost and difficulty of retraining these networks. Furthermore, the lightweight network VaT also uses a Transformer structure with a good global receptive field to convert the fused image into an image that is friendly to the high-level vision model, thereby enhancing the learning ability and enabling VaT to understand contextual information.

[0131] 4. Based on the two optimization terms of the optimization objective and the established VaT network structure, this invention designs a loss function to achieve unsupervised learning. By introducing a cycle consistency loss to satisfy the content retention term, inspired by the self-training paradigm, an effective unsupervised optimization method is proposed to improve the performance of high-level vision tasks, utilizing an uncertainty-guided self-training strategy to optimize the network.

[0132] 5. The embodiments of the present invention, through the established VaT network structure and the designed loss function, effectively connect the image restoration task and the advanced vision task without retraining the existing image restoration network and the advanced vision task, thereby enhancing the performance of the advanced vision task in the degraded scene.

[0133] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this invention are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is altered and sub-operations described as part of a larger operation are executed independently.

[0134] Furthermore, although the invention has been described in the context of functional modules, it should be understood that, unless otherwise stated, one or more of the described functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding the invention. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional skill of an engineer. Therefore, those skilled in the art can implement the invention as set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of the invention, which is determined by the full scope of the appended claims and their equivalents.

[0135] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, essentially, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0136] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0137] More specific examples of computer-readable media (a non-exhaustive list) include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which the program can be printed, because the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0138] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0139] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0140] Although embodiments of the invention have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the claims and their equivalents.

[0141] The above is a detailed description of the preferred embodiments of the present invention. However, the present invention is not limited to the embodiments described. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of the present invention. All such equivalent modifications or substitutions are included within the scope defined by the claims of the present invention.

Claims

1. An unsupervised image restoration method for improving performance in high-level vision tasks, characterized in that, Includes the following steps: Constructing a joint distribution of the initial restored image and the expected input image for the high-level vision task, and obtaining an optimization objective based on variational inference, includes: inputting the original degraded image into the restoration model to obtain the initial restored image; approximating the joint distribution of the initial restored image and the expected input image for the high-level vision task through variational inference to obtain a parameterized distribution; minimizing the KL divergence between the joint distribution and the parameterized distribution to obtain the optimization objective; the optimization objective includes a first optimization term and a second optimization term; the expression of the optimization objective is: ； Among them, the content retention term is the first optimization term, and the maximum boundary likelihood term is the second optimization term; In the formula, Represents the optimization goal; Represents the initial restored image; This represents a clean image, i.e., the input image expected by high-level vision tasks; The representative obeys the parameters Defined distribution random variables Expected value; This represents the expected value of the parameterized distribution that the clean image follows, given an initial restored image. This represents the joint distribution that the initially recovered image follows, given a clean image. Representative in The joint distribution that clean images follow under the condition of relevant high-level visual labels; Representative and Related advanced visual tags; Represents objects of interest for advanced vision tasks across all data; A lightweight network is constructed based on the gating fusion module and the transformation module; Based on the optimization objective and the lightweight network, an unsupervised loss function is constructed. Based on the lightweight network and the unsupervised loss function, unsupervised training is performed using unpaired images to recover the target input image, thereby obtaining the target recovered image; The original degraded image and the initial restored image are input into the gated fusion module to extract weights and obtain initial weights; The initial weights are processed by an activation function to obtain gated fusion weights; A first fused image is generated based on the original degraded image, the initial restored image, and the gated fusion weights; The first fused image is input into the transformation module; The first fused image is transformed to obtain a transformed image.

2. The unsupervised image restoration method for improving the performance of high-level vision tasks according to claim 1, characterized in that, Based on the optimization objective and the lightweight network, an unsupervised loss function is constructed, including the following steps: Obtain the mapping process in the gated fusion module of the lightweight network; Based on the cycle consistency loss, a first loss function is constructed according to the optimization objective and the mapping process.

3. The unsupervised image restoration method for improving the performance of high-level vision tasks according to claim 1, characterized in that, Based on the optimization objective and the lightweight network, an unsupervised loss function is constructed, which further includes the following steps: The first pseudo-label is obtained by predicting the expected input image for the high-level vision task. The original degraded image and the initial restored image are predicted and filtered to obtain the second pseudo-label; Based on the first pseudo-tag and the second pseudo-tag, a fused tag is obtained; Based on the original degraded image, the initial restored image, and the expected input image for the high-level vision task, a second fused image is constructed through uncertainty estimation; Based on the expected input image of the high-level vision task and the first pseudo-label, a first loss term is obtained; Based on the second fused image and the fused label, a second loss term is obtained; The second loss function is obtained based on the first loss term and the second loss term.

4. An unsupervised image restoration device for improving the performance of high-level vision tasks, characterized in that, include: The first module is used to construct a joint distribution of the initial restored image and the expected input image for the high-level vision task, and to obtain an optimization objective based on variational inference. Specifically, the first module is used to: input the original degraded image into the restoration model to obtain the initial restored image; approximate the joint distribution of the initial restored image and the expected input image for the high-level vision task through variational inference to obtain a parameterized distribution; minimize the KL divergence between the joint distribution and the parameterized distribution to obtain the optimization objective; the optimization objective includes a first optimization term and a second optimization term; the expression of the optimization objective is: ； Among them, the content retention term is the first optimization term, and the maximum boundary likelihood term is the second optimization term; In the formula, Represents the optimization goal; Represents the initial restored image; This represents a clean image, i.e., the input image expected by high-level vision tasks; The representative obeys the parameters Defined distribution random variables Expected value; This represents the expected value of the parameterized distribution that the clean image follows, given an initial restored image. This represents the joint distribution that the initially recovered image follows, given a clean image. Representative in The joint distribution that clean images follow under the condition of relevant high-level visual labels; Representative and Related advanced visual tags; Represents objects of interest for advanced vision tasks across all data; The second module is used to build a lightweight network based on the gating fusion module and the transformation module; The third module is used to construct an unsupervised loss function based on the optimization objective and the lightweight network. The fourth module is used to perform unsupervised training on unpaired images based on the lightweight network and the unsupervised loss function, and then recover the target input image to obtain the target recovered image; The original degraded image and the initial restored image are input into the gated fusion module to extract weights and obtain initial weights; The initial weights are processed by an activation function to obtain gated fusion weights; A first fused image is generated based on the original degraded image, the initial restored image, and the gated fusion weights; The first fused image is input into the transformation module; The first fused image is transformed to obtain a transformed image.

5. An electronic device, characterized in that, Including the processor and memory; The memory is used to store programs; The processor executes the program to implement the method as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, The storage medium stores a program that is executed by a processor to implement the method as described in any one of claims 1 to 3.

Images