System and method for self-supervised training of image restoration networks

Abstract

A system and method for self-supervised training of an image restoration network. An image capture system for controlling exposure of each captured image by adjusting at least one exposure parameter. The system is configured to control the exposure parameters of an image captured by the image capture system. The system is configured to generate a training image pair from the captured image. Each training image pair includes a first training image and a second training image of a scene, the first training image and the second training image having different values on at least one exposure parameter. The system is used for collecting training image block pairs from the training image pairs. Each training image block pair comprises a source image block and a target image block. The system is used for training the image restoration network based on the training image block pairs.

Description

TECHNICAL FIELD

[0001] The present disclosure generally relates to self-supervised training of image restoration networks. BACKGROUND

[0002] In recent years, the application of image capture has been greatly expanded. In this document, the term “image capture” can refer to still images and / or videos. Image capture is very important in many consumer electronics, surveillance, automotive, and industrial applications. Image degradation (such as noise, defocus blur, motion blur, overexposure, and underexposure) is extremely detrimental to both photography (leading to loss of details and artistic quality) and machine vision (leading to reduced accuracy and reliability) application scenarios. Image noise often arises throughout the image capture and processing pipeline, with diverse sources and characteristics. Motion blur is caused by camera shake and / or object movement within the exposure interval. Defocus blur is caused by the limited depth of field, leading to objects at different depths being focused on different focal planes. Overexposure and underexposure are caused by improper exposure parameter settings and / or the inability of the image capture device to cope with high dynamic range scenes.

[0003] For most mass-produced camera-equipped products, such as smartphones, tablets, drones, robots, and augmented reality / virtual reality (AR / VR) devices, image denoising, deblurring, and / or exposure correction need to be performed to achieve acceptable image quality. Image denoising refers to the process of removing noise from a noisy image. Image deblurring refers to the process of removing blur (such as motion blur or defocus blur) from a blurred image. Image exposure correction refers to the process of correcting overexposed or underexposed images.

[0004] In recent years, it has been proposed to apply machine learning (such as deep learning) to image restoration. SUMMARY

[0005] One aspect includes a system including an image capture system to capture images. The image capture system is to control an exposure of each captured image by adjusting at least one of a plurality of exposure parameters. The system includes one or more control circuits coupled to and in communication with the image capture system. The one or more control circuits are to control the plurality of exposure parameters of the images captured by the image capture system. The one or more control circuits are to generate pairs of training images from the captured images. Each pair of training images includes a first training image of a scene and a second training image of the scene having different values for at least one of the plurality of exposure parameters. The one or more control circuits are to collect pairs of training image patches from the pairs of training images. Each pair of training image patches includes a source image patch and a target image patch. The one or more control circuits are to train an image restoration network based on the pairs of training image patches.

[0006] Implementations can include the pairs of training image patches including pairs of denoising training image patches, each pair of denoising training image patches including a noisy source image patch and a ground truth clean target image patch. The image restoration network can include a blind denoising network. The one or more control circuits can be to train the blind denoising network based on the pairs of denoising training image patches.

[0007] Implementations can include the pairs of training image patches including pairs of deblurring training image patches, each pair of deblurring training image patches including a blurred source image patch and a ground truth sharp target image patch. The image restoration network can include a blind deblurring network. The one or more control circuits are to train the blind deblurring network based on the pairs of deblurring training image patches.

[0008] Implementations can include the pairs of training image patches including pairs of exposure training image patches, each pair of exposure training image patches including an abnormal exposure source image patch and a ground truth normal exposure target image patch. The image restoration network can include an exposure correction network. The one or more control circuits can be to train the exposure correction network based on the pairs of exposure training image patches.

[0009] Implementations can include the one or more control circuits to collect a first set of pairs of exposure training image patches, where each pair of exposure training image patches includes an underexposed source image patch and a ground truth normal exposure target image patch; collect a second set of pairs of exposure training image patches, where each pair of exposure training image patches includes an overexposed source image patch and a ground truth normal exposure target image patch; train the exposure correction network based on the first set of pairs of exposure training image patches and the second set of pairs of exposure training image patches.

[0010] Implementations can include that the one or more control circuits are to: set a first aperture setting for the first training image and a second aperture setting for the second training image, wherein a particular pair of training images has substantially the same exposure value (EV); analyze a pixel difference between a first pixel in a first image block in the first training image and a second pixel in a second image block in the second training image within the particular pair of training images; based on the pixel difference, the first aperture setting, and the second aperture setting, classify one of the first image block or the second image block as a source image block and the other of the first image block or the second image block as a target image block.

[0011] Implementations can include that the one or more control circuits are to: set a substantially same shutter speed for the first training image and the second training image within the particular pair of training images, wherein the first training image and the second training image within the particular pair of training images have different ISO levels.

[0012] Implementations can include that the first aperture setting is lower than the second aperture setting. The one or more control circuits can be to: classify the first image block as the source image block and the second image block as the target image block when the difference between the first pixel in the first image block and the second pixel in the second image block is less than a low detection threshold; and classify the first image block as the target image block and the second image block as the source image block when the difference between the first pixel in the first image block and the second pixel in the second image block is greater than a high detection threshold.

[0013] Implementations can include that the one or more control circuits are to: train the image restoration network to implement blind de-noising based on the source image block and the target image block when the difference between the first pixel in the first image block and the second pixel in the second image block is less than the low detection threshold; and train the image restoration network to implement blind de-blurring based on the source image block and the target image block when the difference between the first pixel in the first image block and the second pixel in the second image block is greater than the high detection threshold.

[0014] Implementations can include that the one or more control circuits are to: set a first shutter speed for the first training image and a second shutter speed for the second training image, wherein a particular pair of training images has substantially the same exposure value (EV); analyze a pixel difference between a first pixel in a first image block in the first training image and a second pixel in a second image block in the second training image within the particular pair of training images; based on the pixel difference, the first shutter speed, and the second shutter speed, classify one of the first image block or the second image block as a source image block and the other of the first image block or the second image block as a target image block.

[0015] Implementations can include that the one or more control circuits are to: set substantially the same aperture setting for the first training image and the second training image within the particular pair of training images, wherein the first training image and the second training image within the particular pair of training images have different ISO ratings.

[0016] Implementations can include that the first shutter speed is higher than the second shutter speed. The one or more control circuits can be to: classify the first image block as the source image block and the second image block as the target image block when the difference between the first pixel in the first image block and the second pixel in the second image block is less than a low detection threshold; and classify the first image block as the target image block and the second image block as the source image block when the difference between the first pixel in the first image block and the second pixel in the second image block is greater than a high detection threshold.

[0017] Implementations can include that the one or more control circuits are to: train the image restoration network to implement blind de-noising based on the source image block and the target image block when the difference between the first pixel in the first image block and the second pixel in the second image block is less than the low detection threshold; and train the image restoration network to implement blind de-blurring based on the source image block and the target image block when the difference between the first pixel in the first image block and the second pixel in the second image block is greater than the high detection threshold.

[0018] Implementations can include the one or more control circuits to: for a particular training image pair, set a first exposure value for the first training image and a second exposure value for the second training image; for the particular training image pair, analyze a first luminance distribution generated based on first pixels within a first image block in the first training image and a second luminance distribution generated based on second pixels within a second image block in the second training image; based on the first luminance distribution, the second luminance distribution, the first exposure value, and the second exposure value, determine whether to classify one of the first image block or the second image block as a source image block and whether to classify the other of the first image block or the second image block as a target image block.

[0019] Implementations can include the one or more control circuits to: for the particular training image pair, set a first ISO value for the first training image and a second ISO value for the second training image, where an aperture setting is substantially the same and a shutter speed is substantially the same.

[0020] Implementations can include the first ISO value to be less than the second ISO value. The one or more control circuits can be to: when the first luminance distribution satisfies an underexposure condition and the second luminance distribution satisfies a normal exposure condition, classify the first image block as an underexposed source image block and the second image block as a ground truth normal exposure target image block; when the first luminance distribution satisfies a normal exposure condition and the second luminance distribution satisfies an overexposure condition, classify the first image block as a ground truth normal exposure target image block and the second image block as an overexposed source image block.

[0021] Implementations can include the one or more control circuits to: when the first luminance distribution satisfies the underexposure condition and the second luminance distribution satisfies the normal exposure condition, train the image restoration network to implement underexposure correction based on the underexposed source image block and the ground truth normal exposure target image block; when the first luminance distribution satisfies the normal exposure condition and the second luminance distribution satisfies the overexposure condition, train the image restoration network to implement overexposure correction based on the overexposed source image block and the ground truth normal exposure target image block.

[0022] Implementations can include the one or more control circuits to: operate the system in a training mode, where the training image block pairs are used for self-supervised training of the image restoration network in the training mode; operate the system in an inference mode, where the first training image and the second training image within each training image pair are merged into a merged image and provided to the image restoration network in the inference mode.

[0023] The implementation may include: the image capture system includes a binocular camera, which includes a first image sensor and a second image sensor. The first image sensor and the second image sensor can be used to capture frames, wherein each frame has one exposure. The one or more control circuits can be used to generate training image pairs from the first frame of the first image sensor and the second frame of the second image sensor, wherein the first frame and the second frame are exposed within the same time interval.

[0024] The implementation may include: the image capture system includes a monocular camera for capturing frames, wherein each frame has a first exposure and a second exposure. The one or more control circuits may be used to generate training image pairs from the first exposure and the second exposure within each frame.

[0025] The implementation may include: the one or more control circuits being used to: determine local updates of the image restoration network based on the training image patch pairs; send the local updates to a central server, wherein the central server receives local updates from multiple edge devices, and the multiple edge devices determine local updates of the image restoration network based on the training image patch pairs respectively; aggregate the local updates at the central server, and update the global image restoration network based on the aggregated updates; and forward the updated global image restoration network to the edge devices.

[0026] One general aspect includes a method for self-supervised training of an image restoration network. The method includes: controlling multiple exposure parameters of images captured by an image capture system; generating training image pairs from the captured images, wherein each training image pair includes a first training image of a scene and a second training image of the scene, the first training image and the second training image having different values ​​on at least one of the multiple exposure parameters; generating training image patch pairs from the training image pairs, wherein each training image patch pair includes a source image patch and a target image patch; providing the training image patch pairs to the image restoration network; and training the image restoration network based on the training image patch pairs.

[0027] The implementation may include: using the trained image restoration network to perform image restoration on the images captured by the image capture system.

[0028] The implementation may include: receiving the global model of the image restoration network from a central server at an edge device; calculating a local update of the global model of the image restoration network at the edge device based on the training of the image restoration network using the training image patch pairs; and sending the local update to the central server.

[0029] The implementation may include: receiving local updates from multiple edge devices at the central server; aggregating the local updates received from the multiple edge devices; updating the global model of the image restoration network based on the aggregated local updates from the multiple edge devices; and sending the updated global model of the image restoration network to at least one of the multiple edge devices.

[0030] A general aspect includes a user equipment. The user equipment includes an image capture system for capturing images. The user equipment includes a training image generator coupled to the image capture system for generating training image pairs, wherein each training image pair includes a first training image of a scene and a second training image of the scene, and the image capture system controls the exposure of each training image by adjusting at least one of a plurality of exposure parameters. The user equipment includes an exposure controller coupled to the image capture system, wherein the exposure controller controls the exposure of the training images such that the first training image and the second training image within each training image pair have different values ​​on at least one of the plurality of exposure parameters. The user equipment includes an image restoration network coupled to the image capture system, wherein the image restoration network restores the images captured by the image capture system. The user equipment includes a training controller coupled to the training image generator, the exposure controller, and the image restoration network. The training controller is configured to collect training image patch pairs from the training image pairs, each training image patch pair including a source image patch and a target image patch. The training controller is also configured to provide the training image patch pairs to the image restoration network and control the supervised training of the image restoration network based on the training image patch pairs.

[0031] This invention provides a brief overview of some concepts, which will be further described in the specific embodiments. This invention is not intended to identify key or essential features of the claimed subject matter, nor is it intended to help define the scope of the claimed subject matter. The claimed subject matter is not limited to implementations that address any or all of the shortcomings mentioned in the background art. Attached Figure Description

[0032] Various aspects of the invention are illustrated by way of example and are not limited to the accompanying drawings, in which similar reference numerals indicate the same or similar elements.

[0033] Figure 1 An exemplary iso-exposure surface is shown in a three-dimensional (3D) exposure parameter space.

[0034] Figure 2 An example iso-exposure surface is shown.

[0035] Figure 3 An exemplary isobars are shown in a two-dimensional (2D) exposure parameter space.

[0036] Figure 4A , Figure 4B , Figure 5A , Figure 5B , Figure 6A and Figure 6B An example of generating training image pairs for the same scene is shown.

[0037] Figure 7 An embodiment of a system for self-supervised training of image restoration networks is described.

[0038] Figure 8 A flowchart depicts an embodiment of the self-supervised training process for an image restoration network.

[0039] Figure 9 A flowchart depicts an embodiment of the process of restoring an image using a trained image restoration network.

[0040] Figure 10A An example of the front end of a system for self-supervised training of image restoration networks is described.

[0041] Figure 10B An example of the front end of a system for self-supervised training of image restoration networks is described.

[0042] Figure 11 This is a diagram of an embodiment of the backend of a system for self-supervised training of image restoration networks.

[0043] Figure 12A An embodiment of a binocular camera system is described, which can be used in a self-supervised training system for image restoration networks.

[0044] Figure 12B An embodiment of a binocular camera system equipped with a beam splitter is described, which can be used in a system for self-supervised training of image restoration networks.

[0045] Figure 13 An embodiment of a monocular camera system is described, which can be used in a system for self-supervised training of image restoration networks.

[0046] Figure 14A , Figure 14B and Figure 14C Several examples of real-time controllable aperture structures for controlling aperture settings are described.

[0047] Figure 15 An embodiment of a system for generating training image patch pairs is described, which are used for self-supervised training of an image restoration network.

[0048] Figure 16 An example of a system for self-supervised training of an image restoration network in an uncontrolled environment (such as a field environment or a natural environment) is described.

[0049] Figure 17 This is a flowchart of an embodiment of the process of controlling the test environment during the self-supervised training of an image restoration network.

[0050] Figure 18 This is a flowchart of an embodiment of the self-supervised training process of an image restoration network based on a pair of training images.

[0051] Figure 19 This is a flowchart of one embodiment of the process of generating training image pairs using a binocular camera system or a monocular camera system.

[0052] Figure 20 This is a flowchart of an embodiment of the process of generating training image patch pairs from training image pairs.

[0053] Figure 21A This is a flowchart of an embodiment of the process of generating training image patch pairs from training image pairs with exposure diversity.

[0054] Figure 21B This is a flowchart of an embodiment of the process of generating training image patch pairs from training image pairs with shutter diversity.

[0055] Figure 21C This is a flowchart of an embodiment of the process of generating training image patch pairs from training image pairs with aperture diversity.

[0056] Figure 22 This is a diagram of an embodiment of a federated learning system implemented through self-supervised training of an image restoration network.

[0057] Figure 23 This is a flowchart of an example of federated learning achieved through self-supervised training of an image restoration network.

[0058] Figure 24 It is a block diagram of the processing equipment.

[0059] Figure 25 This is a block diagram of a service host that can be used to implement various embodiments of a central server. Detailed Implementation

[0060] This invention and its embodiments provide a system and method for self-supervised training of image restoration networks. Machine learning algorithms construct and / or update models based on training data in training mode. This model can then be applied in inference mode. For example, a denoising network can be trained using training data to construct / update a denoising model, which can be used for image denoising in inference mode. Similarly, a deblurring network can be trained using training data to construct / update a deblurring model, which can be used for image deblurring in inference mode. Likewise, an exposure correction network can be trained using training data to construct / update an exposure correction model, which can be used for image exposure correction in inference mode. As used herein, the term "image restoration network" refers to a system that uses machine learning algorithms to construct and / or update models based on training data in training mode and uses these models in inference mode to restore the image quality of an input image. Image restoration networks may include, but are not limited to, denoising networks, deblurring networks, and / or exposure correction networks.

[0061] Generally, machine learning can be either supervised or unsupervised. In supervised training (also known as supervised learning), the training data needs to include ground truth. In unsupervised training, the training data does not need to include ground truth. Some machine learning techniques use two image sets for training. The source image set has one or more image quality problems (such as noise, blur, and / or abnormal exposure). The corresponding target image set represents the target of image restoration (such as ground truth). For example, an image denoising network can be trained using a set including a clean (target) image as ground truth and a corresponding noisy (source) image. Similarly, an image deblurring network can be trained using a set including a sharp (target) image as ground truth and a corresponding blurred (source) image. Attributing image quality problems to the source image and indicating that the target image is the target of image restoration (such as ground truth) helps to constrain the training data.

[0062] The architecture of such image restoration networks can be divided into non-blind networks and blind networks. In non-blind image restoration networks, prior information about the image is known. For example, non-blind denoising networks can reduce noise in noisy input images based on known or estimated noise levels or parameters. In blind image restoration networks, prior information about the image is unknown. For example, blind denoising networks can use a model-free, direct end-to-end approach to handle various unknown noisy image sources.

[0063] Image restoration networks based on deep learning and other technologies can be effective, but providing training datasets presents a challenge. This is especially true for supervised training that uses ground truth data in the training data. Therefore, training image restoration networks is also challenging. The performance of image restoration networks largely depends on the quality and suitability of the training dataset, which is often difficult to obtain.

[0064] Acquiring, defining, and storing massive amounts of real-world training data is extremely difficult. For example, training an image denoising network with a set of clean (target) images serving as ground truth and their corresponding noisy (source) images is highly challenging. Similarly, training an image deblurring network with a set of sharp (target) images serving as ground truth and their corresponding blurred (source) images is equally challenging. The acquired images can be enhanced through post-processing and synthetic training images generated by artificial intelligence (AI). However, such enhanced training images may lack realism.

[0065] Furthermore, image capture systems vary significantly across different devices, even those of the same model. For example, lens systems and image sensors may have inherent defects. Lens system issues might include achromatic / chromatic aberration, lens shading, and depth-of-field problems. Image sensor issues might include spectral response, channel crosstalk, color filter array (CFA) patterns, and pixel orientation. These significant problems can cause a globally trained image restoration network to perform poorly on a specific electronic device equipped with an image capture system.

[0066] One embodiment of the above system includes an image capture system capable of capturing images. This system may have an image restoration network capable of deblurring, denoising, and / or correcting exposure on the captured images. The system can train the image restoration network based on images captured by the image capture system. Therefore, training the image restoration network does not require pre-generated training data. Furthermore, the system can define the training data (e.g., generating or identifying ground truth). Moreover, a specific user device can generate training data to be used by that specific user device. Therefore, differences specific to each user device can be taken into account. For example, there may be subtle differences in the lenses within each specific user device. Such differences are considered in embodiments where the user device generates its own training data. Therefore, domain offset errors are avoided. Domain offset errors refer to the errors that may occur when an image restoration model is trained in one domain (such as a specific device or application scenario) and then transferred to another domain (such as a different device or different application scenario). The disclosed system and method enable image restoration networks to adapt to new domains through domain adaptation by retraining or fine-tuning the network using training data captured directly in the new domain. This eliminates the need to pre-collect a training dataset including ground truth values ​​for the new domain, which can be very costly and time-consuming.

[0067] The system described above is capable of generating pairs of training images (referred to herein as "training image pairs") from captured images. Each training image pair includes a first training image and a second training image of the same scene. However, as detailed below, the first training image and the second training image differ in at least one exposure parameter. The image capture system can control the exposure of each training image by adjusting at least one exposure parameter. Exposure parameters are defined herein as parameters that control image exposure. These exposure parameters may include, but are not limited to, ISO value (the sensitivity of the image capture system to light), aperture setting, and / or shutter speed. Shutter speed may also be expressed as shutter interval. These exposure parameters collectively determine the exposure.

[0068] Exposure refers to how a camera sensor records an image and the amount of light it captures. A camera's sensitivity to light can be expressed as an ISO value. The lower the ISO value, the lower the sensitivity to light. Furthermore, for a given scene, a lower ISO value may result in a lower noise level in the captured image. The camera's aperture is the opening on the camera lens used to control the amount of light reaching the camera's sensor. Therefore, the aperture setting controls the amount of light reaching the camera's sensor. The aperture setting can be specified as the f-stop value, which is defined as the ratio of the focal length to the aperture diameter. The smaller the f-stop value, the larger the corresponding aperture diameter relative to the focal length. Therefore, a smaller f-stop value allows more light to reach the camera's sensor. However, a smaller f-stop value may reduce the camera's depth of field. Depth of field refers to the distance between the nearest and farthest objects in a photograph that present acceptable focus and sharpness. Shutter speed controls the exposure interval. The slower the shutter speed (i.e., the longer the exposure interval), the more likely motion blur of moving objects will occur.

[0069] In one embodiment, the exposure value (EV) of a training image is determined by the ISO value, aperture setting, and shutter speed of the image capture system, and can be derived using the following formula: EV = ISO value × aperture area × shutter time interval. The ISO value, aperture setting, and shutter speed are referred to herein as "exposure parameters." In this document, the terms "shutter diversity," "aperture diversity," and "exposure diversity" can be used to refer to differences in exposure parameters between two images within a training image pair. Therefore, "diversity" means that two different training images of the same scene have different values ​​for at least one of the exposure parameters.

[0070] The "shutter diversity" between two images within a training image pair is defined as: using different shutter speeds, but with essentially the same aperture setting and exposure (achieving essentially the same exposure using different ISO values). The "aperture diversity" between two images within a training image pair is defined as: using different aperture settings, but with essentially the same shutter speed and exposure (achieving essentially the same exposure using different ISO values). The "exposure diversity" between two training images within a training image pair is defined as: using different exposure values ​​(EV), but with essentially the same aperture setting and shutter speed.

[0071] The system described above can be equipped with an exposure controller that controls the exposure of training images, such that the first and second training images within each training image pair are captured using different values ​​for at least one exposure parameter. For example, the first and second training images can be captured using different aperture settings, but with essentially the same EV (similar shutter speeds but different ISO values, resulting in essentially the same EV). Alternatively, the first and second training images can be captured using different shutter speeds, but with essentially the same EV (similar aperture settings but different ISO values, resulting in essentially the same EV). Furthermore, the first and second training images can be captured using different EVs (e.g., different ISO values, but essentially the same shutter speed and aperture settings). The exposure controller provides a diversity indicator to the training controller. This diversity indicator indicates which exposure parameters differ between specific training image pairs (e.g., aperture diversity, shutter speed diversity, exposure diversity).

[0072] The system described above is capable of collecting training image patch pairs from training image pairs. In this paper, a training image patch is defined as a portion of the training images used to train the image restoration network. Each training image patch pair includes a source image patch and a target image patch. These image patches are taken from corresponding regions of the training image pair. Therefore, the two image patches within a training image patch pair comprise the same portion of the scene captured by the training image pair. Generally, the source image patch has some image quality issues or corruption (blurred, noisy, underexposed, overexposed), while the target image patch is a "high-quality" sample (sharp, clean, properly exposed). The target image patch can be defined as the target or ground truth for image restoration. The system described above uses training image patch pairs to supervise the training of the image restoration network. The generation of target image patches eliminates the need for pre-collected training data. Therefore, there is no need to acquire, qualify, and store pre-collected training data. Furthermore, designating the target image patch as either a source image patch or a target image patch can help qualify the training image patches used for machine learning.

[0073] In one embodiment, the system has a training mode and an inference mode. In training mode, the system collects source / target image patch pairs. These source / target image patch pairs can be input into an image restoration network and used to train the network. In inference mode, captured images are input into the image restoration network. The trained image restoration network can be used to denoise, deblur, and / or correct exposure on the captured images. In one embodiment, an image merger merges each pair of training images into a "merged image," which is the input to the image restoration network. When training mode is selected, source image patches can be provided to the image restoration network (along with target image patches used for training as ground truth). When inference mode is selected, the merged image can be provided to the image restoration network. Therefore, the trained image restoration network can perform deblurring, denoising, and / or exposure correction on the merged image.

[0074] Figure 1 An exemplary isostatic plane is shown within the 3D exposure parameter space. The three axes correspond to the ISO value, shutter speed, and aperture setting (e.g., specified via f-stop), representing the aperture area. The values ​​of these three exposure parameters increase along the arrows on their respective axes. Shutter speed can also be expressed as exposure time (i.e., shutter interval); a larger exposure time value results in greater exposure. A longer exposure time corresponds to a slower shutter speed. Therefore, shutter speed decreases (exposure time increases) along the arrows on the shutter speed axis. Aperture area increases along the aperture axis (f-stop value decreases). ISO value increases along the ISO axis.

[0075] Each triangle 102-1, 102-2, 102-3, and 102-4 represents a surface where all points have the same EV. Arrow 104 indicates the direction of EV increase for the isotopic surfaces. In one embodiment, each image in the training image pair has the same EV but a different shutter speed. In another embodiment, each image in the training image pair has the same EV but a different aperture setting. In yet another embodiment, each image in the training image pair has a different ISO value but the same shutter speed and aperture setting.

[0076] Figure 2An exemplary isostatic plane 102 is shown. Each point on isostatic plane 102 is represented by a triplet of ISO value, aperture setting, and shutter speed. This example illustrates a camera system capable of achieving full exposure control within a 3D exposure parameter space. Line 202, located on the exemplary isostatic plane 102, represents an example of shutter speed diversity. Each point on line 202 has a different shutter speed value. However, all points on line 202 have the same EV. The reason for achieving the same EV in this example is that all points on line 202 have the same aperture setting but different ISO values, thus resulting in the same EV. Line 204, located on the exemplary isostatic plane 102, represents an example of aperture diversity. Each point on line 204 has a different aperture setting value. However, all points on line 204 have the same EV. The reason for achieving the same EV in this example is that all points on line 204 have the same shutter speed but different ISO values, thus resulting in the same EV. Line 206, not located on the exemplary isostatic plane 102, represents an example of exposure diversity. Point 208 is located on the exemplary iso-exposure surface 102, but other points on line 206 are located on other iso-exposure surfaces. Figure 2 (Not depicted in the text).

[0077] Figure 2 Point 210 on the exemplary iso-exposure plane 102 corresponds to the slowest shutter speed among all the exposure parameter triplets on the triangle. Figure 2 Point 212 on the exemplary iso-exposure plane 102 corresponds to the largest aperture area among all exposure parameter triplets on the triangle. Figure 2 Point 214 on the exemplary iso-exposure plane 102 corresponds to the largest ISO value among all exposure parameter triplets on the triangle. An image captured using the exposure parameters corresponding to point 210 may have very severe motion blur, but very little noise (relative to other points on the triangle). An image captured using the exposure parameters corresponding to point 212 may have very severe defocus blur, but very little noise (relative to other points on the triangle). An image captured using the exposure parameters corresponding to point 214 may have a lot of noise, but very low levels of motion blur and defocus blur (relative to other points on the triangle).

[0078] Figure 3 An exemplary isobars are shown in the 2D exposure parameter space. Figure 3 The system contains ISO and shutter speed axes. Within this exemplary 2D exposure parameter space, the aperture setting remains constant. In this example, each point within the 2D exposure parameter space is represented by a unique binary tuple including the ISO value and shutter speed. This example illustrates a camera system that can achieve partial exposure control within a 2D exposure parameter space. Figure 3Several exemplary isobars 302-1, 302-2, and 302-3 are shown. All points on a specific isobar 302 have the same EV. Line 304 represents exposure diversity. Points on line 304 have different exposure values, but the same shutter speed and aperture setting.

[0079] As described above, the exemplary isobars 302-1, 302-2, and 302-3 each have the same EV. However, each point on a specific isobar 302 has a different shutter speed (and a different ISO value). Therefore, two different points on a specific isobar 302 represent an example of shutter speed diversity. Point 306 represents the slowest shutter speed among all exposure parameter pairs on isobar 302-2. Point 308 represents the largest ISO value among all exposure parameter pairs on isobar 302-2. The image captured using the exposure parameters corresponding to point 306 has the most motion blur but the least noise compared to other points on line 302-2. The image captured using the exposure parameters corresponding to point 308 has the least motion blur but the most noise compared to other points on line 302-2.

[0080] Figures 4A to 6B An example for generating training image pairs of the same scene is shown. Each figure illustrates how the shutter interval, ISO value, and aperture area are controlled over time. Each figure covers a time period of two frames (frame n, frame n+1). A training image pair is generated within each frame. For ease of discussion, these two training images are referred to as Training Image 1 (or the first training image) and Training Image 2 (or the second training image), respectively. The terms "first" and "second" are arbitrary, and there is no order relationship between the images; either image can be referred to as the first image. The term "member" can be used to refer to any image within a training image pair. The two members within a particular training image pair differ in at least one exposure parameter.

[0081] Figure 4A An example is depicted where frame n is used to generate a first training image with a first aperture area and a second training image with a second aperture area (as in aperture diversity setting A). The pair of training images using two different aperture areas is an example of aperture diversity. Figure 4A Frame n+1 in the dataset is used to generate another pair of training images with different aperture diversity (such as aperture diversity setting B).

[0082] Figure 4AThis corresponds to one embodiment of an image capture system equipped with a binocular camera. The binocular camera includes two cameras, referred to as binocular camera 1 and binocular camera 2. Both cameras capture a single exposure per frame, thus enabling the binocular camera to achieve two exposures per frame. In one embodiment, the binocular camera is equipped with two image sensors. These image sensors may include, but are not limited to, charge-coupled devices (CCDs) and active pixel sensors (such as CMOS).

[0083] exist Figure 4A Within frame n, the aperture area of ​​binocular camera 1 is a1, while the aperture area of ​​binocular camera 2 is a2, thus achieving aperture diversity (aperture diversity A). Figure 4A Within frame n, the shutter interval (s1) of the stereo cameras is the same. Generally, the shutter interval is between t0 and t1 (see scan lines 402(1a) and 402(2a)). Each individual scan line may be active within a subset of this interval. To use the same EV within frame n, the ISO value of stereo camera 1 is I1, and the ISO value of stereo camera 2 is I2. Within frame n, the left side of Equation 1 shows how the EV of stereo camera 1 is calculated, and the right side of Equation 1 shows how the EV of stereo camera 2 is calculated.

[0084] Formula 1 Equation 1 shows that within frame n, the EV of stereo camera 1 is equal to the EV of stereo camera 2. It should be noted that to achieve the same EV, the two training images may have different ISO values. Figure 4A Scan lines 402 are also shown to indicate the operation of the image sensor. Scan line 402(1a) corresponds to the stereo camera 1 in frame n, scan line 402(2a) corresponds to the stereo camera 2 in frame n, scan line 402(1b) corresponds to the stereo camera 1 in frame n+1, and scan line 402(2b) corresponds to the stereo camera 2 in frame n+1. Scan lines 402 correspond to a rolling scan line sensor, where the shutter start time (and shutter end time) of each line are staggered. Each line in scan line 402 represents the start and end points of the exposure corresponding to that line.

[0085] The following is for reference. Figure 4A In frame n+1, the aperture area of ​​binocular camera 1 is a3, while the aperture area of ​​binocular camera 2 is a4, thus achieving aperture diversity (aperture diversity B). Figure 4AWithin frame n+1, the shutter interval (s2) of the stereo cameras is the same. Generally, the shutter interval is between t2 and t3 (see scan lines 402(1b) and 402(2b)). Each individual scan line may be active within a subset of this time interval. To use the same EV within frame n+1, the ISO value of stereo camera 1 is I3, while the ISO value of stereo camera 2 is I4. Within frame n+1, the left side of Equation 2 shows how the EV of stereo camera 1 is calculated, and the right side of Equation 2 shows how the EV of stereo camera 2 is calculated.

[0086] Formula 2 Figure 4B The operation of one embodiment of a monocular camera used to capture training image pairs with aperture diversity is described. In this monocular embodiment, multiple exposures occur within each frame. Figure 4B This illustrates an example of how a monocular camera can capture multiple exposures within frame n to achieve aperture diversity A, and how a monocular camera can capture multiple exposures within frame n+1 to achieve aperture diversity B. It is important to note that... Figure 4B An example is depicted where there are two exposures within each frame; however, there may be more than two exposures within each frame. Within frame n, the exposure parameters of the monocular camera between t0 and t1 are similar to those of the stereo camera 1 between t0 and t1 (s1, I1, a1). Within frame n, the exposure parameters of the monocular camera between t1 and t1' are similar to those of the stereo camera 2 between t0 and t1 (s1, I2, a2). Figure 4B Scan line 402 (1a) in the diagram represents the exposure used to capture the first training image, and... Figure 4A The scan line 402 (1a) is similar. Figure 4B Scan line 402 (2a) in the diagram represents the exposure used to capture the second training image, and... Figure 4A The scan line 402 (2a) is similar.

[0087] exist Figure 4B Within frame n+1, the exposure parameters of the monocular camera between t2 and t3 are similar to those of the stereo camera 1 between t2 and t3 (s2, I3, a3). Within frame n+1, the exposure parameters of the monocular camera between t3 and t3' are similar to those of the stereo camera 2 between t2 and t3 (s2, I4, a4). Figure 4B Scan line 402 (1b) in the diagram represents the exposure used to capture the first training image, and... Figure 4A The scan line 402 (1b) is similar. Figure 4B Scan line 402 (2b) in the diagram represents the exposure used to capture the second training image, and... Figure 4AThe scan line 402 (2b) is similar. Therefore, the training image pairs that a monocular camera system can capture are similar to those that a binocular camera system can capture.

[0088] It is clear that the two exposures within a single frame of a monocular camera are not captured exactly simultaneously (i.e., at the same time), but rather nearly simultaneously (i.e., close to the same time). If a fast-moving object is present in a region of the image, this slight temporal difference can cause problems. Therefore, in one embodiment, the presence of a fast-moving object in a region of the image is analyzed, and that region is not used for training purposes. For example, in one embodiment equipped with a monocular camera, training image patches are not generated from regions containing fast-moving objects.

[0089] Figure 5A An example is depicted where frame n is used to generate a first training image with a first shutter interval and a second training image with a second shutter interval (as in shutter diversity setting A). The pair of training images using two different shutter intervals is an example of shutter diversity. Figure 5A Frame n+1 in the dataset is used to generate another pair of training images with different shutter diversity (such as shutter diversity setting B).

[0090] Figure 5A This corresponds to an embodiment of an image capture system equipped with a binocular camera. Figure 5A Within frame n, the shutter interval of binocular camera 1 is s3, while the shutter interval of binocular camera 2 is s4, thus achieving shutter diversity (shutter diversity A). The shutter interval of binocular camera 1 is between t0 and t3 (see scan line 502 (1a)). The shutter interval of binocular camera 2 is between t1 and t2 (see scan line 502 (2a)). Figure 5A Within frame n, the aperture area (a5) of the two cameras is the same. To use the same EV within frame n, the ISO value of binocular camera 1 is I5, while the ISO value of binocular camera 2 is I6. Within frame n, the left side of Equation 3 shows how the EV of binocular camera 1 is calculated, and the right side of Equation 3 shows how the EV of binocular camera 2 is calculated.

[0091] Formula 3 Equation 3 shows that within frame n, the EV of stereo camera 1 is equal to the EV of stereo camera 2. It should be noted that to achieve the same EV, the two training images may have different ISO values.

[0092] The following is for reference. Figure 5AIn frame n+1, the shutter interval of binocular camera 1 is s5, while the shutter interval of binocular camera 2 is s6, thus achieving shutter diversity (shutter diversity B). The shutter interval of binocular camera 1 is between t3' and t5' (see scan line 502 (1b)). The shutter interval of binocular camera 2 is between t4 and t5 (see scan line 502 (2b)). Figure 5A Within frame n+1, the aperture area of ​​the stereo camera is a6. To use the same EV within frame n+1, the ISO value of stereo camera 1 is I7, while the ISO value of stereo camera 2 is I8. Within frame n+1, the left side of Equation 4 shows the EV calculation method for stereo camera 1, and the right side of Equation 4 shows the EV calculation method for stereo camera 2.

[0093] Formula 4 Figure 5B The operation of one embodiment of a monocular camera used to capture training image pairs with shutter diversity is described. Figure 5B This illustrates an example of how a monocular camera can capture multiple exposures within frame n to achieve shutter diversity A, and how a monocular camera can capture multiple exposures within frame n+1 to achieve shutter diversity B. It is important to note that... Figure 5B The time axis in the middle corresponds to Figure 5A The timeline within. However... Figure 5A and Figure 5B The timeline in the text does not necessarily correspond to Figure 4A and Figure 4B The timeline in the middle.

[0094] exist Figure 5B Within frame n, the exposure parameters of the monocular camera between t1 and t2 are similar to those of the stereo camera 2 between t1 and t2 (s4, I6, a5). The image sensor in the monocular camera has... Figure 5B The image captured at scan line 502 (2a) has the same characteristics as the image sensor in the binocular camera 2. Figure 5A The images captured at scan line 502 (2a) have similar exposure parameters (s4, I6, a5).

[0095] exist Figure 5B Within frame n, the ISO and aperture values ​​of the monocular camera between t0 and t1 are similar to those of the stereo camera 1 between t0 and t1 (I5, a5). Furthermore, in Figure 5BWithin frame n, the ISO and aperture values ​​of the monocular camera between t2 and t3 are similar to those of the stereo camera 1 between t2 and t3 (I5, a5). The shutter intervals between t0 and t1 and between t2 and t3 are shown as s3', indicating that within frame n, the combined three shutter intervals of the monocular camera are similar to those of the stereo camera 1, for example, s3'+s4+s3'≈s3. Within frame n, scan line 502 (1c) represents the first exposure, scan line 502 (2a) represents the second exposure, and scan line 502 (1d) represents the third exposure. In one embodiment, image processing is used to combine the three exposures of the monocular camera within frame n to generate an image similar to that of the stereo camera 1. Figure 5A The training images generated within frame n are similar to the training images generated by the monocular camera. An exemplary method for merging the three exposures of the monocular camera within frame n is to calculate the normalized sum of the shutter intervals for equalizing the three ISO values, for example, training image = [exposure 502 (1c) + I5 / I6 × exposure 502 (2a) + exposure 502 (1d)] × s3 / (s3' + s4 + s3'). Merging the three exposures of the monocular camera within frame n can achieve shutter intervals (t0 to t3) similar to those of the stereo camera 1. The training images generated by the monocular camera do not need to have the same exposure parameters as those generated by the stereo camera. For example, to compensate for the shorter shutter intervals, exposures 502 (1c) and 502 (1d) in the monocular camera can use a larger ISO value I5 than exposure 502 (1a) in the stereo camera 1. However, there should be shutter diversity between the first and second training images generated by the monocular camera.

[0096] exist Figure 5B Within frame n+1, the exposure parameters of the monocular camera between t4 and t5 are similar to those of the stereo camera 2 between t4 and t5 (s6, I8, a6). The image sensor in the monocular camera has... Figure 5B The image captured at scan line 502 (2b) has the same characteristics as the image sensor in the binocular camera 2. Figure 5A The images captured at scan line 502 (2b) have similar exposure parameters (s6, I8, a6).

[0097] exist Figure 5B Within frame n+1, the ISO and aperture values ​​of the monocular camera between t3' and t4 are similar to those of the stereo camera 1 between t3' and t4 (I7, a6). Furthermore, in Figure 5BWithin frame n+1, the ISO and aperture values ​​of the monocular camera between t5 and t5' are similar to those of the stereo camera 1 between t5 and t5' (I7, a6). The shutter intervals between t3' and t4 and between t5 and t5' are shown as s5', indicating that within frame n+1, the three shutter intervals of the monocular camera combination are similar to those of the stereo camera 1, for example, s5'+s6+s5'≈s5. Within frame n+1, scan line 502 (1e) represents the first exposure, scan line 502 (2b) represents the second exposure, and scan line 502 (1f) represents the third exposure. In one embodiment, image processing is used to combine the three exposures of the monocular camera within frame n+1 to generate an image similar to that of the stereo camera 1. Figure 5A The training images generated within frame n+1 are similar to the training images generated by the monocular camera. An exemplary method for merging the three exposures of the monocular camera within frame n+1 is to calculate the normalized sum of the shutter intervals for equalizing the three ISO values, for example, training image = [exposure 502 (1e) + I7 / I8 × exposure 502 (2b) + exposure 502 (1f)] × s5 / (s5' + s6 + s5'). Merging the three exposures of the monocular camera within frame n+1 can achieve shutter intervals (t3' to t5') similar to those of the stereo camera 1. The training images generated by the monocular camera do not need to have the same exposure parameters as those generated by the stereo camera. For example, to compensate for the shorter shutter intervals, exposures 502 (1e) and 502 (1f) in the monocular camera can use a larger ISO value I7 than exposure 502 (1b) in the stereo camera 1. However, there should be shutter diversity between the first and second training images generated by the monocular camera.

[0098] Figure 6A An example is described where frame n is used to generate a first training image with a first EV (such as a first ISO value) and a second training image with a second EV (such as a second ISO value), which may be referred to herein as exposure diversity (such as exposure diversity setting A). Figure 6A Frame n+1 in the dataset is used to generate another pair of training images with different exposure diversity (such as exposure diversity setting B).

[0099] Figure 6A This corresponds to an embodiment of an image capture system equipped with a binocular camera. Figure 6AWithin frame n, the ISO value of stereo camera 1 is I9, while the ISO value of stereo camera 2 is I10, thus achieving exposure diversity (exposure diversity A). Furthermore, the shutter interval for both stereo cameras 1 and 2 is s7. Also, the aperture area for both stereo cameras 1 and 2 is a7. Within frame n, the left side of Equation 5 represents the EV calculation method for stereo camera 1, and the right side of Equation 5 represents the EV calculation method for stereo camera 2.

[0100] Formula 5 Equation 5 shows that within frame n, the EV of the image captured by binocular camera 1 is not equal to the EV of the image captured by binocular camera 2. In this paper, this can be referred to as exposure diversity. Scan line 602 (1a) shows that, between t0 and t1, the image sensor in binocular camera 1 is exposed using ISO value I9. Scan line 602 (2a) shows that, between t0 and t1, the image sensor in binocular camera 2 is exposed using ISO value I10.

[0101] The following is for reference. Figure 6A In frame n+1, the ISO value of stereo camera 1 is I11, while the ISO value of stereo camera 2 is I12, thus achieving exposure diversity (exposure diversity B). Furthermore, the shutter interval for both stereo cameras 1 and 2 is s8. Also, the aperture area for both stereo cameras 1 and 2 is a8. Within frame n+1, the left side of Equation 6 shows the EV calculation method for stereo camera 1, and the right side of Equation 6 shows the EV calculation method for stereo camera 2.

[0102] Formula 6 Equation 6 shows that within frame n+1, the EV of the image captured by binocular camera 1 is not equal to the EV of the image captured by binocular camera 2. Scan line 602 (1b) shows that between t2 and t3, the image sensor in binocular camera 1 is exposed using the ISO value I11. Scan line 602 (2b) shows that between t2 and t3, the image sensor in binocular camera 2 is exposed using the ISO value I12.

[0103] Figure 6B The operation of one embodiment of a monocular camera used to capture training image pairs with exposure diversity is described. Figure 6B This illustrates an example of how a monocular camera can capture multiple exposures within frame n to achieve exposure diversity A, and how a monocular camera can capture multiple exposures within frame n+1 to achieve exposure diversity B. It is important to note that... Figure 6B The time axis in the middle corresponds to Figure 6A The timeline in the middle.

[0104] There is a first shutter interval between t0 and t1 within frame n, corresponding to Figure 6B Scan line 602 (1a) in the image. Within the first shutter interval of frame n, the shutter interval is s7, the ISO value is I9, and the aperture is a7. Therefore, the exposure parameters of the monocular camera between t0 and t1 of the first shutter interval of frame n are similar to those of the binocular camera 1 between t0 and t1 of frame n (s7, I9, a7). The monocular camera has a second shutter interval between t1 and t1' of frame n, corresponding to... Figure 6B Scan line 602 (2a) in frame n. During the second shutter interval of frame n, the shutter interval is s7, the ISO value is I10, and the aperture is a7. Therefore, the exposure parameters of the monocular camera during the second shutter interval of frame n are similar to the exposure parameters (s7, I10, a7) of the stereo camera 2 between t0 and t1 of frame n.

[0105] There is a first shutter interval between t2 and t3 within frame n+1, corresponding to Figure 6B Scan line 602 (1b) in the image. Within the first shutter interval of frame n+1, the shutter interval is s8, the ISO value is I11, and the aperture is a8. Therefore, the exposure parameters of the monocular camera between t2 and t3 of the first shutter interval of frame n+1 are similar to those of the binocular camera 1 between t2 and t3 of frame n+1 (s8, I11, a8). The monocular camera has a second shutter interval between t3 and t3' of frame n+1, corresponding to... Figure 6B Scan line 602 (2b) is used. During the second shutter interval of frame n+1, the shutter interval is s8, the ISO value is I12, and the aperture is a8. Therefore, the exposure parameters of the monocular camera during the second shutter interval of frame n+1 are similar to the exposure parameters (s8, I12, a8) of the stereo camera 2 between t2 and t3 of frame n+1.

[0106] Figure 7 An embodiment of a system 700 for self-supervised training of an image restoration network is depicted. System 700 may include, but is not limited to, cameras (such as surveillance cameras, vehicle cameras, consumer cameras), smartphones, tablets, laptops, desktop computers, AR / VR devices, drones (equipped with cameras), robots, etc. An image capture system 702 is used to capture images. In one embodiment, image capture system 702 is a monocular system equipped with a lens system and image sensors. In another embodiment, image capture system 702 is a binocular system equipped with a pair of lens systems and a pair of image sensors. These image sensors may include, but are not limited to, charge-coupled devices (CCDs) and active pixel sensors (such as CMOS).

[0107] The training image generator 703 is used to generate pairs of training images from the captured images. Each pair of training images includes a first training image of the scene and a second training image of the same scene. For example, the image capture system 702 captures one or more images of scene 704, and the training image generator 703 generates training image 1 and training image 2 of the same scene. The image capture system 702 can control the exposure of each training image by adjusting at least one exposure parameter according to the exposure settings. These exposure parameters include the ISO value (the sensitivity of the image capture system to light), aperture setting, and shutter speed. These exposure parameters collectively determine the exposure of the images captured by the image capture system 702.

[0108] Exposure controller 706 sends exposure settings to image capture system 702 to establish the exposure of training images. The exposure settings can specify that two images within a training image pair have different values ​​for at least one exposure parameter. Therefore, the first and second training images within each training image pair have different values ​​for at least one exposure parameter. For example, the first and second training images can be captured using different aperture settings to achieve aperture diversity, but with essentially the same EV (same shutter speed, but different ISO values, thus making the EV the same). Similarly, the first and second training images can be captured using different shutter speeds to achieve shutter diversity, but with essentially the same EV (same aperture setting, but different ISO values, thus making the EV the same). Or, the first and second training images can be captured using different EVs (e.g., different ISO values, but the same shutter speed and the same aperture setting) to achieve exposure diversity. Exposure controller 706 provides a diversity indicator to training controller 710. This diversity indicator indicates which exposure parameter differs between a particular training image pair (e.g., aperture diversity, shutter diversity, exposure diversity).

[0109] Image restoration network 708 is used to deblur, denoise, and / or correct exposure of images captured by image capture system 702. Training controller 710 is used to collect training image patch pairs from training image pairs. Each training image patch pair includes a source image patch and a target image patch. These image patches are taken from the same region of the training image pair. Generally, the source image patch may have some image quality problems or damage (blurred, noisy, underexposed, overexposed), while the target image patch may be a "high-quality" sample (sharp, clean, properly exposed). Training controller 710 also controls the training of image restoration network 708 using the collected training image patch pairs by sending a training enable indicator (EN). The generation of target image patches eliminates the need for pre-collected training data. Therefore, there is no need to acquire, qualify, and store pre-collected training data.

[0110] System 700 has a training mode and an inference mode. A training / inference scheduler 712 selects either the training mode or the inference mode. The training / inference scheduler 712 can send a mode selection signal to the training controller 710 to select either the training mode or the inference mode. In training mode, the training controller 710 collects source / target image patch pairs. These source / target image patch pairs can be input to the image restoration network 708 and used to train the image restoration network 708. The training / inference scheduler 712 can send the mode selection signal as an input selection control (SEL) to the MUX 716 to select source image patches in training mode. In inference mode, the image merger 714 can merge each training image pair into a merged image. In inference mode, the training / inference scheduler 712 can send the mode selection signal as an input selection control (SEL) to the MUX 716 to select the merged image. Therefore, the merged image can replace the source image patch and be input to the image restoration network 708 in inference mode. In inference mode, the trained image restoration network 708 can be used to denoise, deblur, and / or correct exposure on the merged image. In inference mode, the image restoration network 708 outputs a restored image as the output of system 700. The image restoration network 708 performs supervised training on its independent models, and in training mode, it does not need to output a "restored" image as the output of system 700. The training controller 710 also sends an enable indicator (EN) to control the training of the independent models of the image restoration network 708.

[0111] In one embodiment, system 700 can quickly switch between training mode and inference mode. For example, system 700 can operate in training mode within the blanking interval of a frame and in inference mode outside the blanking interval of a frame.

[0112] The training controller 710, exposure controller 706, training image generator 703, training / inference scheduler 712, image merger 714, MUX 716 and / or image restoration network 708 can be implemented in software, hardware or a combination of hardware and software. Figure 24 An embodiment of a processing device 2400 is depicted, which can be used to implement one or more of these components in system 700. In one embodiment, system 700 includes one or more processors that execute code to implement one or more of these components. Various types of processors can be used. In one embodiment, the system described above includes a graphics processing unit (GPU) that executes deep learning algorithms to implement image restoration network 708. However, the system described above does not need to include a GPU.

[0113] Generally, system 700 may include a camera system and one or more control circuits. These control circuits may be used to implement any combination of a training controller 710, an exposure controller 706, a training image generator 703, a training / inference scheduler 712, an image merger 714, a MUX 716, and / or an image restoration network 708. The control circuits may be implemented in hardware, software, or a combination of both.

[0114] In one embodiment, software (stored in a storage device) implementing one or more embodiments is used to program one or more processors to implement some or all of the components in system 700. The one or more processors may communicate with one or more computer-readable media / storage devices, peripheral devices, and / or communication interfaces. In an alternative embodiment, some or all of the components in system 700 may be implemented by dedicated hardware logic components. Illustrative types of hardware logic components that may be used include, for example, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), dedicated computers, etc.

[0115] Figure 8 A flowchart depicts one embodiment of a self-supervised training process 800 for an image restoration network. Process 800 can be executed by system 700, but is not limited thereto. References can be made to process 800. Figure 7 The elements in; however, process 800 is not limited to combining. Figure 7 Elements to be depicted and described. Step 802 includes: determining a diverse set of exposure settings. In other words, determining the values ​​of exposure parameters. These exposure parameters may include, but are not limited to, ISO values, shutter speeds, and / or aperture settings. Each frame may include two sets of exposure parameters. Each set may include a triplet (such as ISO value, shutter speed, aperture setting), where the first and second sets differ in at least one exposure parameter within a given frame.

[0116] Step 804 includes controlling the exposure parameters of the image captured by the image capture system 702. Step 804 may include the exposure controller 706 providing exposure settings (such as the values ​​of exposure parameters) to the image capture system 702.

[0117] Step 806 includes: generating a training image pair comprising a first training image and a second training image of the same scene 704, wherein the first training image and the second training image have different values ​​for at least one exposure parameter. In one embodiment, the training image pair has aperture diversity. In one embodiment, the training image pair has shutter diversity. In one embodiment, the training image pair has exposure diversity.

[0118] Step 808 includes collecting training image patch pairs from the training image pairs. In one embodiment, each training image patch pair includes a source image patch and a target image patch. It should be noted that the training image pairs may generate many training image patch pairs. In one embodiment, the training image patch pairs are customized for a specific aspect of the image restoration network 708. The training image patch pairs may include, but are not limited to, training image patch pairs customized for a denoising network of the image restoration network 708, training image patch pairs customized for a deblurring network of the image restoration network 708, and / or training image patch pairs customized for an exposure correction network of the image restoration network 708.

[0119] Step 810 includes providing training image patch pairs to the image restoration network 708. Step 812 includes supervising the training of the image restoration network 708 based on the training image patch pairs. Step 814 includes determining whether to collect training image patches for another set of exposure settings. If yes, the above process returns to step 802. If no, training ends. Therefore, process 800 enables self-supervision of the image restoration network 708 without the need for pre-collection of training data. It should be noted that one option is to store the training image patch pairs collected in step 808. The stored training image patch pairs can be used later to train the image restoration network.

[0120] Figure 9 A flowchart depicts an embodiment of a process 900 for restoring an image using a trained image restoration network. Process 900 can be performed by... Figure 7 The system 700 executes the process, but is not limited to it. In one embodiment, process 900 executes in inference mode. In one embodiment, process 900 executes after training the image restoration network 708 using process 800. However, different processes may also be used to train the image restoration network 708. In one embodiment, process 900 executes concurrently with the training of the image restoration network 708. Process 900 describes the processing within a frame and can be repeated for other frames. In one embodiment, both process 800 and process 900 execute within a frame. For example, process 800 may execute within the blanking interval of a frame, while process 900 may execute outside the blanking interval of a frame.

[0121] Step 902 includes generating a training image pair comprising a first training image and a second training image of the same scene 704. Step 904 includes merging the training image pairs into a merged image. Step 906 includes providing the merged image to an image restoration network 708. Step 908 includes performing image restoration on the merged image using the trained image restoration network 708. Step 908 may include, but is not limited to, denoising, deblurring, and / or exposure correction of the merged image. Step 910 includes outputting the restored image.

[0122] System 700 can be broadly divided into two parts: a front-end that captures images and generates training images, and a back-end that collects training image patch pairs and trains an image restoration network. The front-end may include an image capture system 702, a training image generator 703, and an exposure controller 706. The front-end is responsible for generating training image pairs. Figure 10A An embodiment of the front end 1000 of a system for self-supervised training of an image restoration network is described. Figure 10A The front end 1000 is equipped with a binocular camera system. This binocular camera system is equipped with a first lens system 1002(1) connected to a first image sensor 1004(1), which outputs a first raw image (raw image 1). The binocular camera system is also equipped with a second lens system 1002(2) connected to a second image sensor 1004(2), which outputs a second raw image (raw image 2). These two lens systems and two image sensors constitute one embodiment of the image capture system 702. The binocular camera system can simultaneously capture the first and second raw images within the same frame. A frame synchronizer 1006 synchronizes the first and second raw images, keeping them synchronized within each frame interval and pairing them after further processing to form training image pairs. As is known to those skilled in the art, an image signal processor 1008 performs image signal processing on the raw images. An optional image distortion remover 1010 removes lens geometry distortion and shading from the raw images. An optional parallax corrector 1012 compensates for parallax problems between the two raw images, which arise from viewing angle differences between the two lens systems and image sensors. The parallax corrector 1012 outputs training image pairs in each frame. The frame synchronizer 1006, the image signal processor 1008, the image distortion remover 1010, and the parallax corrector 1012 constitute one embodiment of the training image generator 703.

[0123] As is known to those skilled in the art, the automatic exposure module 1014 performs an automatic exposure function. The automatic exposure module 1014 can provide the current values ​​of the exposure parameters to the exposure controller 706. The exposure controller 706 determines the values ​​of the exposure parameters (ISO value, shutter speed, aperture setting) to be used within one or more frames. In one embodiment, the exposure parameters are selected to achieve diversity in shutter speed, aperture setting, or exposure. Therefore, the exposure controller 706 determines two sets of exposure parameters (ISO1, shutter speed 1, aperture 1) and (ISO2, shutter speed 2, aperture 2). The exposure controller 706 provides ISO1 and shutter speed 1 to the first image sensor 1004 (1) and aperture 1 to the first lens system 1002 (1). The exposure controller 706 provides ISO2 and shutter speed 2 to the second image sensor 1004 (2) and aperture 2 to the second lens system 1002 (2). The exposure controller 706 outputs a diversity indicator based on the exposure parameters within one or more frames. In one embodiment, the diversity indicator is used to specify the type of diversity (such as shutter diversity, aperture diversity, exposure diversity) created in training image pairs associated with one or more frames.

[0124] Figure 10B An embodiment of a front-end 1050 for a system used for self-supervised training of an image restoration network is depicted. The front-end 1050 is equipped with a monocular camera system. This monocular camera system is equipped with a lens system 1052 connected to an image sensor 1054, which outputs a raw image captured using one or more exposures within a frame. This lens system and sensor constitute an embodiment of an image capture system 702. A frame decomposer 1056 decomposes the raw image captured using one or more exposures within a frame to generate a first raw image (raw image 1) and a second raw image (raw image 2) within that frame. Therefore, the monocular camera system can simultaneously capture the first and second raw images within the same frame. As is known to those skilled in the art, an image signal processor 1058 performs image signal processing on the raw images. The frame decomposer 1056 and the image signal processor 1058 constitute an embodiment of a training image generator 703.

[0125] As known to those skilled in the art, the automatic exposure module 1060 performs an automatic exposure function. The automatic exposure module 1060 can provide the current values ​​of the exposure parameters to the exposure controller 706. The exposure controller 706 determines the values ​​of the exposure parameters (ISO value, shutter speed, aperture setting) to be used within one or more frames. In one embodiment, the exposure parameters are selected to achieve diversity in shutter speed, aperture, or exposure. Therefore, the exposure controller 706 determines two sets of exposure parameters (ISO1, shutter speed 1, aperture 1) and (ISO2, shutter speed 2, aperture 2). The exposure controller 706 provides ISO1, shutter speed 1, ISO2, and shutter speed 2 to the image sensor 1054. The exposure controller 706 provides aperture 1 and aperture 2 to the lens system 1052. The exposure controller 706 outputs a diversity indicator based on the exposure parameters within one or more frames.

[0126] Figure 11 This is a diagram of an embodiment of the backend 1100 of a system for self-supervised training of an image restoration network. Figure 11An embodiment of the training controller 710 and the image restoration network 708 is described in further detail. The training controller 710 includes an image difference generator 1102, a noise threshold generator 1104, a multi-scale difference detector 1106, and a training image patch collector 1108. The image difference generator 1102 takes training image 1 and training image 2 as input and outputs a difference image (Diff Image) between them. As those skilled in the art will appreciate, the image difference generator 1102 can perform pixel subtraction. In one embodiment, when the exposure values ​​(EV) of training image 1 and training image 2 are not substantially the same, the training image pair is normalized to the same exposure value (EV) by their inverse exposure ratio before pixel subtraction is performed. For example, when the exposure value (EV) of training image 1 is 5% greater than that of training image 2, training image 1 is multiplied by 100 / 105 before pixel subtraction is performed. The difference image can also be referred to as a residual image. Noise threshold generator 1104 takes training image 1 and training image 2 as input and generates a noise threshold for each image. Those skilled in the art will understand that these noise thresholds can be used for adaptive difference detection based on the noise distribution of the image sensor. In the presence of noise, these detection thresholds are crucial for robust difference detection of difference images. Since typical image sensors have signal-dependent noise characteristics, the detection threshold for each pixel in the difference image should adaptively depend on the noise threshold of each corresponding pixel in each image within the training image pair. These noise thresholds can be determined by the pixel intensity of the corresponding pixel in each image within the training image pair and the ISO value of each image within the training image pair. Therefore, the noise threshold for each pixel is adaptively calculated using a predefined sensor noise model and parameters obtained through the image sensor calibration process. The predefined sensor noise model and the image sensor calibration process are known to those skilled in the art. In one embodiment, when the exposure values ​​(EV) of training image 1 and training image 2 are not substantially the same, the corresponding noise thresholds are normalized to the same exposure value (EV) by using the inverse exposure ratio of the two. The image difference generator 1102 and the noise threshold generator 1104 can operate on a per-image-block basis. Therefore, difference images can be generated for image blocks. An image may consist of many (e.g., hundreds or thousands) image blocks.

[0127] A multi-scale difference detector 1106 determines the difference between two training images based on the difference image and a noise threshold. The multi-scale difference detector 1106 can also operate on a per-image-patch basis. In one embodiment, the multi-scale difference detector 1106 analyzes the differences between pixels in corresponding image patches within a training image pair. In one embodiment, the multi-scale difference detector 1106 uses a coarse-to-fine decision rule to detect image patch differences based on noise thresholds generated by a noise threshold generator 1104. In one embodiment, the multi-scale difference detector 1106 detects a first scale in order from the coarsest to the finest scale, wherein the difference image downscaled to this scale has the sum of absolute pixel values ​​of the image patches (i.e., the local sum within the image patch), which exceeds the adaptive detection threshold corresponding to that scale. The adaptive detection threshold of the multi-scale difference detector 1106 can be assigned by multiplying a predetermined constant by the local root-mean-square (RMS) values ​​of the noise thresholds for corresponding pixels in image patches within the two images of the training image pair, wherein these noise thresholds are generated by the noise threshold generator 1104. For multi-scale difference detection, as is known to those skilled in the art, the adaptive detection thresholds corresponding to different scales can be determined by a downscaled version of the adaptive detection threshold corresponding to the original finest scale. The multi-scale difference detector 1106 outputs image patch type, image patch scale, and confidence score to a training image patch collector 1108 and an image merger 714. In one embodiment, the image patch type indicates which image patch is the source image patch, which image patch is the target image patch, and which network (e.g., a denoising network, a deblurring network, an exposure correction network) to be trained. In one embodiment, the image patch scale indicates whether the image patch should be scaled down (e.g., reduced in size) for deblurr training, and if so, what downscaling ratio is used to provide a sharp target image patch. The confidence score indicates the level of confidence in using the image patch for training purposes at the indicated downscaling ratio. In one embodiment, the confidence level can be a value positively correlated with the total amount of spatial features (e.g., details, textures, edges) included in the image patch. The total amount of such spatial features can be reasonably estimated using methods known to those skilled in the art. In one embodiment, as known to those skilled in the art, the image merger 714 performs multi-scale image fusion using training image pairs for each image patch at each scale. During image fusion, target image patches with higher confidence scores are assigned higher weights than their corresponding source image patches. Target image patches with lower confidence scores are assigned similar weights to their corresponding source image patches. When target image patches and corresponding source image patches are not determined, similar weights are assigned to two corresponding image patches within the training image pair during image fusion.

[0128] The training image patch collector 1108 takes training image pairs as input and collects training image patch pairs (source image patches and target image patches) based on the signal from the multi-scale difference detector 1106. The training image patch collector 1108 provides the target image patches to the image restoration network 708 and the source image patches to the MUX 716. The target image patches are provided to various ground truth loss function modules in the image restoration network 708. The training image patch collector 1108 also sends a training enable indicator (EN) to control the training of independent models within the image restoration network 708.

[0129] The image restoration network 708 comprises three distinct networks. The blind denoising network includes a blind image denoiser 1110 and a denoiser ground truth loss function module 1112. The blind image denoiser 1110 may include a denoiser model. Supervised training updates the denoiser model. During the training of the blind image denoiser 1110, source image patches are provided as input image patches, and target image patches and denoised output image patches from the blind image denoiser 1110 are provided to the denoiser ground truth loss function module 1112. These source and target image patches are referred to herein as denoised training image patch pairs, each pair comprising a noisy source image patch and a ground truth clean target image patch. The source image patch can be used as a noisy input image, while the corresponding target image patch can be used as a ground truth clean image. The target image patch does not need to be completely clean, as long as it is cleaner (i.e., less noisy) than the corresponding source image patch. Practice has shown that, under the constraint of using data augmentation methods known to those skilled in the art, a denoiser model trained using noisy (source) training image patches and noisy (target) training image patches can achieve denoising performance comparable to a denoiser model trained using noisy (source) training image patches and clean (target) training image patches. The denoiser ground truth loss function module 1112 generates loss values ​​for supervised training of the blind image denoiser 1110. These loss values ​​are provided to the blind image denoiser 1110 to update the denoiser model parameters using a training algorithm or optimization algorithm, thereby improving its denoising performance. Those skilled in the art should understand supervised training of the blind denoiser model.

[0130] The blind deblurring network comprises a blind image deblurr 1114 and a deblurr ground truth loss function module 1116. The blind image deblurr 1114 may include a deblurr model. Supervised training can update the deblurr model. During the training of the blind image deblurr 1114, source image patches are provided as input image patches to the blind image deblurr 1114, and target image patches and the deblurred image patches from the output of the blind image deblurr 1114 are provided to the deblurr ground truth loss function module 1116. These source and target image patches are referred to herein as deblurring training image patch pairs, each pair comprising a blurred source image patch and a ground truth sharp target image patch. The source image patch can be used as the blurred input image, while the corresponding target image patch can be used as the ground truth sharp image. The target image patch does not need to be a completely sharp image at the original scale, as long as it is sharper (i.e., less blurred) than the corresponding source image patch at the original scale. The multi-scale difference detector 1106 can specify the image patch scale to indicate whether the training image patch pairs should be scaled down, and if so, by what downscaling ratio; then, the training image patch pairs can be scaled down by the specified downscaling ratio to provide sharp target image patches for deblurring training at the specified spatial scale. The deblurring ground truth loss function module 1116 generates loss values ​​for supervised training of the blind image deblurring device 1114. These loss values ​​are provided to the blind image deblurring device 1114 to update the deblurring device model parameters using a training algorithm or an optimization algorithm, thereby improving its deblurring performance. Supervised training of the blind deblurring device model will be understood by those skilled in the art.

[0131] The image exposure correction network includes an image exposure corrector 1118 and an exposure corrector ground truth loss function module 1120. The image exposure corrector 1118 may include an exposure corrector model. Supervised training can update the exposure corrector model. During the training of the image exposure corrector 1118, source image patches are provided as input image patches, and target image patches and the output exposure-corrected image patches from the image exposure corrector 1118 are provided to the exposure corrector ground truth loss function module 1120. These source and target image patches are referred to herein as exposure training image patch pairs, each pair including an aberrantly exposed source image patch and a ground truth normally exposed target image patch. The source image patch can be used as an aberrantly exposed (e.g., overexposed or underexposed) input image, while the corresponding target image patch can be used as a ground truth normally exposed image. The target image patch does not need to be a perfectly exposed image, as long as it is more appropriately exposed than the corresponding source image patch (i.e., fewer overexposed and underexposed pixels). Practice shows that, based on underlying statistical characteristics, the exposure correction performance of an image exposure corrector model can be improved by training it with a less-than-perfectly exposed (source) training image patch and a more-perfectly exposed (target) training image patch. The exposure corrector ground truth loss function module 1120 generates loss values ​​for supervised training of the image exposure corrector 1118. These loss values ​​are provided to the image exposure corrector 1118 to update the exposure corrector model parameters using a training algorithm or an optimization algorithm, thereby improving its exposure correction performance. Those skilled in the art will understand supervised training of the image exposure corrector model.

[0132] Figure 12AAn embodiment of a binocular camera system is described, which can be used in a system for self-supervised training of image restoration networks. The binocular camera system is equipped with two cameras 1202(1) and 1202(2). The ISO value, aperture setting, and shutter speed of each camera can be controlled independently. Each camera has a controllable aperture. The first camera 1202(1) has a first controllable aperture 1204(1). The second camera 1202(2) has a second controllable aperture 1204(2). The aperture diameter can be dynamically controlled using an electronic, electrochromic, or electromechanical aperture structure. In one embodiment, each camera 1202(1) and 1202(2) is equipped with an image sensor that supports control of the ISO value and shutter speed. In one embodiment, the ISO value and shutter speed are controlled by software. For example, for a CMOS image sensor, the shutter speed can be controlled frame-by-frame or exposure-by-exposure. In one aspect, the ISO value can be dynamically controlled frame-by-frame or exposure-by-exposure using the analog or digital gain of the CMOS image sensor. Each camera 1202(1), 1202(2) captures images of the same scene. However, a baseline 1206 exists between the incident light rays captured by the two cameras 1202(1) and 1202(2), which may cause image alignment and parallax problems. In one embodiment, image registration is used to register (align) the images of each camera 1202(1), 1202(2) with each other. In one embodiment, parallax correction is used to correct for parallax problems.

[0133] Figure 12B An embodiment of a binocular camera system equipped with a beam splitter is described, which can be used in a system for self-supervised training of image restoration networks. The binocular camera system is equipped with two cameras 1222(1) and 1222(2). The ISO value, aperture setting, and shutter speed of each camera can be controlled independently. The system is equipped with three apertures. The zero aperture 1214(0) is the first aperture to receive incident light. The first camera 1222(1) has a first controllable aperture 1214(1). The second camera 1222(2) has a second controllable aperture 1214(2). The system is equipped with a 50% beam splitter 1216, which sends 50% of the light from the zero aperture 1214(0) to the second aperture 1214(2) of the second camera 1222(2). The remaining 50% of the light is sent to a reflector 1218, which reflects this portion of the light back to the first aperture 1214(1) of the first camera 1222(1). The second camera 1222(2) can be offset relative to the first camera 1222(1) so that the light travels the same distance from the zero aperture 1214(0) to each camera 1222(1) and 1222(2).

[0134] In one embodiment, each camera 1222(1), 1222(2) is equipped with an image sensor that supports control of ISO value and shutter speed. In one embodiment, ISO value and shutter speed are controlled by software. Each camera 1222(1), 1222(2) captures an image of the same scene. Since both cameras 1222(1) and 1222(2) capture the same incident light, therefore Figure 12B The configuration avoids Figure 12A Parallax issues may occur in the configuration. However, in one embodiment, image registration is used to register (align) the images of each camera 1222(1), 1222(2) with each other.

[0135] Figure 13 An embodiment of a monocular camera system is depicted, which can be used in a system for self-supervised training of image restoration networks. The monocular camera 1302 has a controllable aperture 1314. In one embodiment, the monocular camera 1302 is equipped with an image sensor that supports control of ISO value and shutter speed. In another embodiment, the ISO value and shutter speed are controlled by software. In one embodiment, two different sets of exposure parameters (ISO value, aperture setting, and shutter speed) are used per frame through time multiplexing. In some embodiments, the controllable aperture 1314 has a real-time controllable aperture structure that allows the aperture 1314 setting to be changed rapidly frame-by-frame or exposure-by-exposure.

[0136] Figure 14A , Figure 14B and Figure 14C Several examples of real-time controllable aperture structures for controlling aperture settings are depicted. These real-time controllable aperture structures can all be used for aperture 1314 in a monocular camera system; however, aperture 1314 in a monocular camera system is not limited to these examples. Apertures in binocular camera systems can also employ real-time controllable aperture structures.

[0137] Figure 14A A 2D aperture array 1410 is shown. The 2D aperture array 1410 includes units that can transmit light, block light, or partially block light to control the aperture. For example, unit 1412 is in a transmitting state, unit 1414 is in a blocking state, and unit 1416 is in a partially blocking state. Figure 14B The concentric annular aperture 1420 is shown. The aperture can be controlled by operating the concentric rings 1422 and 1424. Figure 14C The radial blade aperture 1430 is shown. The aperture 1434 can be controlled by operating the radial blade 1432.

[0138] Figure 15An embodiment of a system 1500 for generating training image patch pairs is described, which are used for self-supervised training of an image restoration network. System 1500 can be located in a controlled environment such as a laboratory, where camera and / or object motion, object brightness, lighting conditions, depth, and / or surface texture corresponding to a test scene can be controlled. System 1500 can generate training data for self-supervised training of the image restoration network. System 1500 is capable of collecting high-quality training data.

[0139] System 1500 is equipped with a camera 1502 mounted on a movable camera platform 1504. Camera 1502 captures images of a calibration pattern 1506 illuminated by a controlled light source 1508. Distance controller 1510 controls the distance between the calibration pattern 1506 and the camera 1502. Self-supervised training manager 1512 controls system 1500.

[0140] The self-supervised training manager 1512 determines the ISO value, aperture setting, and shutter speed of the camera 1502. In one embodiment, the self-supervised training manager 1512 includes a training controller 710 and an exposure controller 706. The self-supervised training manager 1512 sends a brightness control signal to a controlled light source 1508 to control brightness. The self-supervised training manager 1512 sends a depth control signal to a distance controller 1510 to control the distance between the calibration pattern 1506 and the camera 1502. The self-supervised training manager 1512 sends a motion control signal to a movable camera platform 1504 to control the movement of the camera 1502. The camera control signals sent by the self-supervised training manager 1512 can also be extended to lens focusing, neutral density (ND) filters, spectral filters, and / or polarizers, thereby enabling additional diversity in generating training image patch pairs for self-supervised training of the image restoration network.

[0141] The self-supervised training manager 1512 can traverse numerous test case triples (such as motion, depth, and brightness) during the overall training process. Overall training may include controlling the exposure parameters described herein (ISO value, aperture setting, shutter speed). For example, process 800 may be executed. The self-supervised training manager 1512 outputs training image patch pairs. In one embodiment, these are densely packed training image patch pairs. In one embodiment, the training image patch pairs are used to train an image restoration network for a user device such as a cellular phone. In one embodiment, system 1500 is used for initial training of the image restoration network. This initial training may include learning the parameters of the image restoration network. In other words, the initial training can learn a model of the image restoration network. The same model can be used in many different electronic devices. In one embodiment, the model is updated in a specific electronic device within a field environment.

[0142] Figure 16 An embodiment of a system for self-supervised training of an image restoration network in an uncontrolled environment (such as a field environment or a natural environment) is described. The system includes a user device 1600, which may include, but is not limited to, cameras (such as surveillance cameras, vehicle cameras, consumer cameras), smartphones, tablets, laptops, desktop computers, AR / VR devices, drones (equipped with cameras), robots, etc. The user device 1600 is equipped with a camera 1602, a self-supervised training manager 1612, and an image restoration network 708. In one embodiment, the image restoration network 708 is trained before the user owns the user device 1600. For example, Figure 15 System 1500 can be used for the initial training of image restoration network 708. In one embodiment, self-supervised training manager 1612 updates the training of image restoration network 708. These updates can be used for retraining, fine-tuning, and / or model adaptation. Model adaptation can be used to adapt the model to a specific user device 1600.

[0143] Camera 1602 captures images of an uncontrolled scene 1606 illuminated by an uncontrolled light source 1608. User equipment 1600 may be handheld or supported by some mechanism. Therefore, camera 1602 may experience uncontrolled movement. Self-supervised training manager 1612 determines the ISO value, aperture setting, and shutter speed of camera 1602. In one embodiment, self-supervised training manager 1612 includes training controller 710 and exposure controller 706. Self-supervised training manager 1612 outputs training image patch pairs. In one embodiment, these are sparse training image patch pairs. In one embodiment, the training image patch pairs are used to update the model of an image restoration network for a specific user device, such as a cellular phone. Self-supervised training manager 1612 may perform process 800 to provide training image patch pairs and train image restoration network 708. In one embodiment, user equipment 1600 may perform process 900 to restore an image using the trained image restoration network 708.

[0144] Figure 17 This is a flowchart of one embodiment of a process 1700 for controlling the test environment during self-supervised training of an image restoration network. Process 1700 can be executed by system 1500, but is not limited thereto. For ease of explanation, the steps in process 1700 are described in a specific order. However, these steps can be executed in a different order. Step 1702 includes setting scene brightness, object depth, and camera motion control signals. In one embodiment, the self-supervised training manager 1512 determines the values ​​of each of these three parameters to control a test environment such as a laboratory.

[0145] Step 1704 includes: identifying a set of exposure settings with a variety of characteristics (e.g., aperture variety, shutter variety, exposure variety). Step 1706 includes: collecting training image patch pairs for the exposure settings. Steps 804 to 808 in process 800 can be used to collect training image patch pairs. Step 1708 includes: determining whether to collect training image patch pairs for another set of exposure settings. If yes, the above process repeats steps 1704 and 1706 for the other set of exposure settings. After collecting training image patches for different exposure settings, the above process continues to step 1710, determining whether all scene brightness settings have been traversed. If no, the above process can return to step 1702 to set different scene brightness settings. Then, by repeatedly executing steps 1704 to 1708, training image patch pairs can be collected for that scene brightness. After traversing all scene brightness settings, the above process continues to step 1712, determining whether all object depth settings have been traversed. If no, the above process can return to step 1702 to set different object depth settings. Then, by repeatedly executing steps 1704 to 1710, training image patch pairs can be collected for the object depth and different scene brightness.

[0146] After iterating through all object depth settings, the process continues to step 1714 to determine whether all camera motion settings have been iterated. If not, the process can return to step 1702 to set different camera motion settings. Then, by repeatedly executing steps 1704 to 1712, training image patch pairs can be collected for the camera motion and different scene brightness / object depth combinations. After iterating through all camera motion settings, the process ends. Process 1700 can collect the dense training image patch pairs described herein.

[0147] Figure 18 This is a flowchart of an embodiment of a self-supervised training process 1800 for an image restoration network based on a pair of training images. In describing process 1800, reference will be made to... Figure 11 The image restoration network 708 in the example. Process 1800 further details the training process. Figure 11 An embodiment of the image restoration network 708 in the image restoration network.

[0148] Step 1802 includes: determining a diversity strategy. Examples of diversity strategies may include, but are not limited to, aperture diversity, shutter diversity, and / or exposure diversity. Step 1804 includes: generating training image pairs for the diversity strategy. Step 1806 includes: generating training image patch pairs from the training image pairs. Step 1806 may include: detecting image differences caused by aperture diversity or shutter diversity based on an adaptive detection threshold to identify noisy (source) training image patches and clean (target) training image patches for self-supervised training of the blind denoising network. Step 1806 may include: using a multi-scale coarse-to-fine detector to detect image differences caused by aperture diversity or shutter diversity to identify blurred (source) training image patches and sharp (target) training image patches for self-supervised training of the blind deblurring network. Step 1806 may include: analyzing underexposed and overexposed pixel statistics caused by exposure diversity based on underexposure and overexposure conditions to identify source (underexposed, overexposed) training image patches and target (normally exposed) training image patches for self-supervised training of the exposure correction network. Figures 21A to 21C The identification of training image patch pairs is described in further detail. Process 1800 employs a multi-loop structure, with each loop processing one training image patch pair.

[0149] Step 1808 includes determining whether the training image patch pair should be used in the denoiser, deblurr, or exposure corrector in the image restoration network 708. This determination can be made by analyzing the training image patch pair. In one embodiment, if the diversity strategy is exposure diversity, the training image patch pair is used in the exposure corrector. However, if the diversity strategy is aperture diversity, the training image patch pair can be used in the denoiser or defocus deblurr, depending on the characteristics of the training image patch pair. If the diversity strategy is shutter diversity, the training image patch pair can be used in the denoiser or motion deblurr, depending on the characteristics of the training image patch pair. The following section discusses in further detail how to determine which type of image restoration training to perform on the training image patch pair.

[0150] Steps 1810 to 1814 can be performed when a training image patch pair is determined to be used for denoising the image denoiser. Steps 1810 to 1814 describe the process of updating the image denoiser based on a training image patch pair. These steps typically need to be performed multiple times to train the blind image denoiser 1110. Step 1810 includes: performing image denoising on the source training image patch. Step 1810 includes: providing the source image patch to the blind image denoiser 1110. This source image patch may be referred to herein as the noisy source image patch. The blind image denoiser 1110 outputs a denoised image patch, which is provided to the denoiser ground truth loss function module 1112. Step 1812 includes: calculating a loss value for training or optimizing the blind image denoiser 1110. Step 1812 may include: the denoiser ground truth loss function module 1112 comparing the denoised image patch with a target image patch. The target image patch may be referred to herein as the ground truth clean target image patch. Step 1814 includes updating the denoising network parameters. Step 1814 may include: the denoiser ground truth loss function module 1112 providing the calculated loss value to the blind image denoiser 1110, wherein the parameters of the blind image denoiser 1110 are updated through supervised training using a training algorithm or optimization algorithm to improve its denoising performance. Step 1814 can be used to update the denoiser model. In one embodiment, those skilled in the art will understand that the weights and / or biases used in the matrix of the neural network are updated through supervised training using a training algorithm or optimization algorithm.

[0151] Steps 1816 to 1820 can be performed when a training image patch pair is determined to be used for deblurring training. Steps 1816 to 1820 describe the process of updating the image deblurring based on a training image patch pair. These steps typically need to be performed multiple times to train the blind image deblurring 1114. Step 1816 includes: performing image deblurring on the source training image patch. Step 1816 includes: providing the source image patch to the blind image deblurring 1114. This source image patch may be referred to herein as the blurred source image patch. The blind image deblurring 1114 outputs a deblurred image patch, which is provided to the deblurring ground truth loss function module 1116. Step 1818 includes: calculating a loss value for training or optimizing the blind image deblurring 1114. Step 1818 may include: the deblurring ground truth loss function module 1116 comparing the deblurred image patch with a target image patch. The target image patch may be referred to herein as the ground truth sharp target image patch. Step 1820 includes updating the deblurring network parameters. Step 1820 may include: the deblurr ground truth loss function module 1116 providing the calculated loss value to the blind image deblurr 1114, wherein the parameters of the blind image deblurr 1114 are updated through supervised training using a training algorithm or an optimization algorithm to improve its deblurring performance. Step 1820 can be used to update the deblurr model.

[0152] Steps 1822 to 1826 can be performed when a training image patch pair is determined to be used for training the exposure corrector. Steps 1822 to 1826 describe the process of updating the exposure corrector based on a training image patch pair. These steps typically need to be performed multiple times to train the image exposure corrector 1118. Step 1822 includes: performing image exposure correction on the source training image patch. Step 1822 includes: providing the source image patch to the image exposure corrector 1118. This source image patch is referred to herein as the aberrant exposure source image patch. The image exposure corrector 1118 outputs an exposure-corrected image patch, which is provided to the exposure corrector ground truth loss function module 1120. Step 1824 includes: calculating a loss value for training or optimizing the image exposure corrector 1118. Step 1824 may include: the exposure corrector ground truth loss function module 1120 comparing the exposure-corrected image patch with a target image patch. The target image patch may be referred to herein as the ground truth normal exposure target image patch. Step 1826 includes: updating the exposure correction network parameters. Step 1826 may include: the exposure corrector ground truth loss function module 1120 providing the calculated loss value to the image exposure corrector 1118, wherein the parameters of the image exposure corrector 1118 are updated through supervised training using a training algorithm or an optimization algorithm to improve its exposure correction performance. Step 1826 may be used to update the exposure corrector model.

[0153] Step 1828 includes determining whether there are any other training image patch pairs to be generated from the training images. If yes, the process returns to step 1806. If no, the process ends.

[0154] Figure 19 This is a flowchart of an embodiment of a process 1900 for generating training image pairs using a binocular camera system or a monocular camera system. Process 1900 is further described in detail below. Figure 8 Step 806 and Figure 18 An embodiment of step 1804 in [the text]. Combined with [the text] Figure 10A and Figure 10B The process described in 1900, but not limited to... Figure 10A Binocular cameras, and not limited to Figure 10B A monocular camera. Process 1900 describes the various processes within a single frame.

[0155] Step 1902 is to determine whether the camera is a binocular camera or a monocular camera. For a binocular camera, steps 1904 to 1910 are performed. Step 1904 includes: synchronously scanning the two image sensors 1004(1) and 1004(2) in the binocular camera. Step 1904 is used to capture two raw images within a frame. Figure 4A , Figure 5A and Figure 6A Examples of how to synchronously scan two image sensors 1004(1) and 1004(2) such that each image sensor is simultaneously exposed within a frame are described. Step 1906 includes: synchronizing the two raw images. Step 1906 may include: inputting raw image 1 from image sensor 1004(1) and raw image 2 from image sensor 1004(2) to frame synchronizer 1006. Step 1908 includes image signal processing, which may be performed by image signal processor 1008. Step 1908 may also include automatic exposure control, which may be performed by automatic exposure module 1014. Step 1910 includes: removing image distortion and parallax correction. Step 1910 may include: aligning and registering the images from each image sensor 1004(1) and 1004(2) after processing by image signal processor 1008. Step 1910 may be performed by image distortion remover 1010 and parallax corrector 1012.

[0156] Steps 1912 to 1916 are performed for the monocular camera. Step 1912 includes performing multiple scans using the image sensor 1054 in the monocular camera. Step 1912 is used to capture the raw image within a frame. Figure 4B , Figure 5B and Figure 6BExamples of how to scan image sensor 1054, such that the image sensor performs multiple scans within a frame, are described. Step 1914 includes decomposing the original image into two original training images. Step 1914 may include inputting the original image from image sensor 1054 into frame decomposer 1056. Step 1916 includes image signal processing, which may be performed by image signal processor 1058. Step 1916 may also include automatic exposure control, which may be performed by automatic exposure module 1060.

[0157] Figure 20 This is a flowchart of an embodiment of a process 2000 for generating training image patch pairs from training image pairs. Process 2000 further describes in detail an embodiment of the operation of the training controller 710. In discussing process 2000, references are made to... Figure 11 . refer to Figure 11 The training controller 710 takes training image pairs and diversity indicators as input. The training controller 710 outputs training image patch pairs.

[0158] Step 2002 includes generating a difference image from the training image pair. The difference image may be based on the difference between a first pixel in the first training image and a second pixel in the second training image. The difference image may also be referred to as a residual image. Those skilled in the art will understand how to generate a difference image or residual image from two images of the same scene. Step 2002 may be performed by the image difference generator 1102. Step 2004 includes generating noise thresholds for the training image pair. These noise thresholds can be used for adaptive difference detection based on the noise distribution of the image sensor. Step 2004 may be performed by the noise threshold generator 1104. Step 2006 is a branch step, depending on the diversity type of the training image pair. If the diversity indicator specifies exposure diversity, steps 2008 through 2012 are performed. Step 2008 includes analyzing the training image pair based on the brightness distribution. Step 2008 may be performed by the training image patch collector 1108. Step 2010 includes collecting source / target image patch pairs based on the exposure diversity analysis. Step 2010 can be performed by training the image patch collector 1108. Figure 21A An embodiment of steps 2008 and 2010 is described in further detail. Step 2012 includes determining whether there are other image patches to be analyzed in the training image pair. If so, steps 2008 to 2012 are repeated for the next image patch. The processing of the training image pair ends when all image patches in the training image pair have been analyzed.

[0159] If the diversity indicator specifies shutter diversity, steps 2014 through 2018 are performed. Step 2014 includes analyzing the difference image based on difference detection. Step 2014 can be performed by a multi-scale difference detector 1106. Step 2016 includes collecting source / target image patch pairs based on shutter diversity analysis. Step 2106 can be performed by training an image patch collector 1108. Figure 21B An embodiment of steps 2014 and 2016 is described in further detail. Step 2018 includes determining whether there are other image patches to be analyzed in the training image pair. If so, steps 2014 to 2018 are repeated for the next image patch. The processing of the training image pair ends when all image patches in the training image pair have been analyzed.

[0160] If the diversity indicator specifies aperture diversity, then steps 2020 through 2024 are performed. Step 2020 includes analyzing the difference image based on difference detection. Step 2020 can be performed by the multi-scale difference detector 1106. Step 2022 includes collecting source / target image patch pairs based on aperture diversity analysis. Step 2022 can be performed by the training image patch collector 1108. Figure 21C An embodiment of steps 2020 and 2022 is described in further detail. Step 2024 includes determining whether there are other image patches to be analyzed in the training image pair. If so, steps 2020 to 2024 are repeated for the next image patch. The processing of the training image pair ends when all image patches in the training image pair have been analyzed.

[0161] Figure 21A This is a flowchart of one embodiment of a process 2100 for generating training image patch pairs from training image pairs with exposure diversity. In one embodiment, the first and second images within the training image pair are captured using different exposure values ​​(EVs), but with substantially the same aperture settings and substantially the same shutter speed. Process 2100 further describes in detail one embodiment of the exposure diversity path in process 2000 (steps 2008 to 2012). Step 2102 includes: performing a brightness distribution calculation per image patch. In one embodiment, a brightness distribution can be determined for each image patch in each training image. The brightness distribution of each image patch is used to evaluate the regional distribution of underexposed and overexposed pixels in each training image. In one embodiment, the brightness distribution of image patches in the training image may include image patches P The weighted count of underexposed and overexposed pixels within the area ( C UE , C OE Weighted count of underexposed pixels. C UEIt can be with image blocks P The brightness value of each pixel within L i All corresponding first scalar-scalar function values f UE ( L i The sum of ) First scalar-scalar function f UE It is the pixel brightness value. L i To the corresponding weight value f UE ( L i A predetermined mapping relationship is established, where the weight value represents the degree of underexposure (i.e., over-darkening) of a pixel. For example, the first scalar-scalar function... f UE It is possible when all pixel brightness values ​​are less than a first predetermined brightness value. V When 1 (determined to be underexposed) is selected, it is set to 1, while the brightness value of all pixels is greater than the second predetermined brightness value. V When 2 (determined to be no underexposure) is set to 0. For values ​​between the first predetermined brightness value... V 1 and second predetermined brightness value V Pixel brightness values ​​between 2, first scalar-scalar function f UE It can be a continuously decreasing function from 1 to 0. Similarly, the weighted count of overexposed pixels. C OE It can be with image blocks P The brightness value of each pixel within L i All corresponding second scalar-scalar function values f OE ( L i The sum of ) Second scalar-scalar function f OE It is the pixel brightness value. L i To the corresponding weight value f OE ( L i A predetermined mapping relationship, where the weight value represents the degree of overexposure (i.e., excessive brightening) of a pixel. For example, the second scalar-scalar function. f OE It is possible when all pixel brightness values ​​are less than the third predetermined brightness value. V When 3 (determined to be no overexposure) is set to 0, and all pixel brightness values ​​are greater than the fourth predetermined brightness value. VWhen 4 (determined to be overexposed), set it to 1. For values ​​between the third predetermined brightness value... V 3 and the fourth predetermined brightness value V Pixel brightness values ​​between 4, second scalar-scalar function f OE It can be a continuously increasing function from 0 to 1. In this example, the four predetermined brightness values ​​are arranged in ascending order of their assigned values, i.e. V 1< V 2< V 3< V 4. In other embodiments, the brightness distribution can be defined as a brightness histogram, where intervals are set to count overexposed and underexposed pixels within an image block; it can also be defined as percentile values, corresponding to the number of overexposed and underexposed pixels within an image block. Those skilled in the art should understand how to generate a brightness distribution based on pixel brightness values ​​within an image block and the total number of pixels within the image block, according to predetermined rules.

[0162] Step 2104 includes analyzing the brightness distribution of image patch pairs using a pixel threshold. This analysis determines whether to collect training image patch pairs for training an underexposure corrector (performing step 2106 at this point), whether to collect training image patch pairs for training an overexposure corrector (performing step 2108 at this point), or whether not to collect training image patch pairs. In one embodiment, if one image patch is severely underexposed while the other is more properly exposed, step 2106 is performed to collect training image patch pairs for training the underexposure corrector. The severely underexposed image patch can be used as an underexposed source image patch, while the more properly exposed image patch can be used as a ground truth normal exposure target image patch. In one embodiment, if one image patch is severely overexposed while the other is more properly exposed, step 2108 is performed to collect training image patch pairs for training the overexposure corrector. The severely overexposed image patch can be used as an overexposed source image patch, while the more properly exposed image patch can be used as a ground truth normal exposure target image patch. This analysis can compare the brightness distribution of image patches in the training image with a low pixel threshold. T LP and high pixel threshold T HP Comparisons can be made. This analysis can use a low pixel threshold. T LP To test for severe underexposure within corresponding image patches in the training images, a high pixel threshold is used. T HPThe test is performed to determine severe overexposure within corresponding image patches in the training images. In one embodiment, if the brightness distribution of an image patch satisfies a first normal exposure condition for underexposure testing and a second normal exposure condition for overexposure testing, then the image patch can be defined as a normal exposure image patch (and thus suitable as a ground truth normal exposure target image patch). In one embodiment, the conditions for underexposure training image patch pairs and overexposure training image patch pairs can be based on tests for severe underexposure and severe overexposure within image patches in the training images.

[0163] In one embodiment, the analysis in step 2104 is based on obtaining the ISO value of the training image containing the image patch. In the following text, the training image with a larger ISO value can be referred to as... High_ISO Training images with smaller ISO values ​​can be called Low_ ISO The following describes in detail one or more conditions for determining whether a training image patch pair satisfies an underexposed training image patch pair. In one embodiment, image patches in the training image with the lower ISO value are subjected to an underexposed test, while image patches in the training image with the higher ISO value are subjected to a normal exposure test. In one embodiment, the determination is made if the underexposed criteria are met for the image patch in the training image with the lower ISO value, and one or more normal exposure conditions are met for the image patch in the training image with the higher ISO value. In one embodiment, the determination is made if the weighted count of underexposed pixels in the brightness distribution of the image patch in the training image with the lower ISO value... C UE ( Low_ISO (greater than the low pixel threshold) T LP If the ISO value is high, then the underexposure criterion is met. In one embodiment, if the weighted count of underexposure pixels in the brightness distribution of image patches in training images with higher ISO values ​​is high, then the underexposure criterion is met. C UE ( High_ISO (Significantly smaller than the low pixel threshold) T LP Furthermore, the weighted count of overexposed pixels in the brightness distribution of image patches in training images with larger ISO values. C OE ( High_ISO (Significantly smaller than the high pixel threshold) T HP If the exposure is within acceptable limits, then the normal exposure standard is met. These conditions can be expressed as: C UE ( Low_ISO )> T LP ,at the same time C UE ( High_ISO )< k 1T LP and C OE ( High_ISO )< k 2 T HP ,in, k 1 and k 2 are two predetermined constants less than 1.

[0164] Step 2106 includes: collecting training image patch pairs for the underexposure corrector when it is determined that one image patch is an underexposed image patch and the other image patch is a normally exposed image patch (i.e., one or more conditions of the underexposed training image patch pair are met). Step 2106 includes: designating one image patch as a source image patch and the other image patch as a target image patch. This designation can be based on two different ISO values ​​of the two training images. The image patch in the training image with the smaller ISO value is designated as the source image patch. The image patch in the training image with the larger ISO value is designated as the target image patch. The image patch in the training image with the smaller ISO value can be defined as an underexposed image patch, while the image patch in the training image with the larger ISO value can be defined as a normally exposed image patch. In step 2106, a confidence score can also be assigned to the training image patch pairs. The confidence score can be based on the degree to which the weighted count of underexposed pixels in the underexposed image patch is greater than a low pixel threshold, etc. The confidence score can also be based on the degree to which the weighted counts of underexposed pixels and overexposed pixels in the non-underexposed image patch are less than a low pixel threshold and a high pixel threshold, respectively, etc.

[0165] The following describes in detail an embodiment of step 2108 for determining whether a training image patch pair satisfies one or more conditions for overexposure. In one embodiment, the above analysis is based on obtaining the ISO value of the training image containing the image patch. In one embodiment, image patches in the training image with a higher ISO value are subjected to an overexposure test, while image patches in the training image with a lower ISO value are subjected to a normal exposure test. In one embodiment, the determination is made if the overexposure criterion is met for the image patch in the training image with the higher ISO value, and one or more normal exposure conditions are met for the image patch in the training image with the lower ISO value. In one embodiment, the weighted count of overexposure pixels in the brightness distribution of the image patch in the training image with the higher ISO value is... C OE ( High_ISO (greater than the high pixel threshold) T HP If the ISO value is low, then the overexposure criterion is met. In one embodiment, if the weighted count of underexposed pixels in the brightness distribution of image patches in training images with lower ISO values ​​is... C UE( Low_ISO (Significantly smaller than the low pixel threshold) T LP Furthermore, the weighted count of overexposed pixels in the brightness distribution of image patches in training images with smaller ISO values. C OE ( Low_ISO (Significantly smaller than the high pixel threshold) T HP If the exposure is within acceptable limits, then the normal exposure standard is met. These conditions can be expressed as: C OE ( High_ISO )> T HP ,at the same time C UE ( Low_ISO )< k 3 T LP and C OE ( Low_ ISO )< k 4 T HP ,in, k 3 and k 4 are two predetermined constants less than 1.

[0166] Step 2108 includes: collecting training image patch pairs for the overexposure corrector when it is determined that one image patch is an overexposed image patch and the other image patch is a normally exposed image patch (i.e., satisfying one or more conditions of the overexposed training image patch pair). Step 2108 includes: designating one image patch as a source image patch and the other image patch as a target image patch. This designation can be based on two different ISO values ​​of the two training images. The image patch in the training image with the larger ISO value is designated as the source image patch. The image patch in the training image with the smaller ISO value is designated as the target image patch. The image patch in the training image with the larger ISO value can be defined as an overexposed image patch, while the image patch in the training image with the smaller ISO value can be defined as a normally exposed image patch. In step 2108, a confidence score can also be assigned to the training image patch pair. This confidence score can be based on the degree to which the weighted count of overexposed pixels in the overexposed image patch is greater than a high pixel threshold, etc. The confidence score can also be based on the degree to which the weighted counts of underexposed pixels and overexposed pixels in the unexposed image patch are less than a low pixel threshold and a high pixel threshold, respectively, etc.

[0167] As mentioned above, in some cases, neither step 2106 nor step 2108 is executed. This may occur when neither of the two image patches meets the underexposure standard nor the overexposure standard. This may also occur when neither of the two image patches meets the normal exposure standard.

[0168] Step 2110 includes determining whether there are any other image patches to be processed in the training image pair. If yes, the process returns to step 2104 to process the next image patch in each training image. If no, the process ends.

[0169] Figure 21B This is a flowchart of one embodiment of process 2140 for generating training image patch pairs from training image pairs with shutter diversity. In one embodiment, the first and second images within the training image pair are captured using different shutter speeds, but with the same aperture setting and the same EV. Process 2140 further describes in detail one embodiment of the shutter diversity path in process 2000 (steps 2014 to 2018). Step 2142 includes performing multi-scale per-image patch difference detection based on a noise threshold. In one embodiment, difference detection subtracts the value of a pixel in one image patch from the value of the corresponding pixel in another image patch. This subtraction operation can yield the difference image (or residual image) described herein. In one embodiment, difference detection is performed based on RGB values. However, difference detection does not necessarily have to be performed based on RGB values. For example, difference detection can be performed based on luminance values. The low and high detection thresholds used for difference detection for each pixel in the difference image should adaptively depend on a noise threshold generated based on the image sensor noise distribution of each corresponding pixel in each image within the training image pair. Figure 11 The backend 1100 of the system for self-supervised training of image restoration networks described herein has been detailed in detail, including an embodiment of multi-scale per-image patch difference detection based on a noise threshold and how to compute an adaptive detection threshold based on the noise threshold.

[0170] Step 2144 includes analyzing the difference values ​​of image patch pairs using a detection threshold. This analysis is used to determine whether to collect training image patch pairs for training the blind denoiser (in which case step 2146 is performed), whether to collect training image patch pairs for training the blind motion deblurr (in which case step 2148 is performed), or whether not to collect training image patch pairs. In one embodiment, if the difference value of an image patch pair is less than a low detection threshold, i.e., no motion blur is detected between image patch pairs, step 2146 is performed to collect training image patch pairs for training the blind denoiser. In one embodiment, if the difference value of an image patch pair is greater than a high detection threshold, i.e., severe motion blur is detected between image patch pairs, step 2148 is performed to collect training image patch pairs for training the blind motion deblurr. The above analysis can compare the difference values ​​of image patch pairs with a low detection threshold and a high detection threshold. The above analysis can use a low detection threshold to test for no motion blur between image patch pairs and a high detection threshold to test for severe motion blur between image patch pairs.

[0171] Step 2146 includes collecting training image patch pairs for the blind denoiser when the difference value of the image patch pair is determined to be less than a low detection threshold. Step 2146 designates one image patch as a source image patch and the other as a target image patch. This designation can be based on two different shutter speeds of the two training images. The image patch in the training image with the faster shutter speed is designated as the source image patch. The image patch in the training image with the slower shutter speed is designated as the target image patch. The image patch with the faster shutter speed can be defined as a noisy image patch, while the image patch with the slower shutter speed can be defined as a clean image patch. In step 2146, a confidence score can also be assigned to the training image patch pair. This confidence score can be based on the degree to which the difference value is less than the low detection threshold, etc.

[0172] Step 2148 includes collecting training image patch pairs for a blind motion deblurr when the difference value of the image patch pair is determined to be greater than a high detection threshold. Step 2148 designates one image patch as the source image patch and the other as the target image patch. This designation can be based on two different shutter speeds of the two training images. The image patch in the training image with the slower shutter speed is designated as the source image patch. The image patch in the training image with the faster shutter speed is designated as the target image patch. The slower shutter speed image patch can be defined as a motion-blurred image patch, while the faster shutter speed image patch can be defined as a sharp image patch. In step 2148, a confidence score can also be assigned to the training image patch pair. This confidence score can be based on the degree to which the difference value is greater than the high detection threshold, etc. In some cases, the image patches may be scaled. For example, the training image patch pair can be reduced in size to provide a sharper target image patch.

[0173] Step 2150 includes determining whether there are any other image patches to be processed in the training image pair. If yes, the process returns to step 2144 to process the next image patch in each training image. If no, the process ends.

[0174] Figure 21C This is a flowchart of one embodiment of a process 2180 for generating training image patch pairs from training image pairs with aperture diversity. In one embodiment, the first and second images within the training image pair are captured using different aperture settings, but with the same shutter speed and EV. Process 2180 further details one embodiment of the aperture diversity path (steps 2020 to 2024) in process 2000. Step 2182 includes performing multi-scale per-image patch difference detection based on a noise threshold. In one embodiment, difference detection subtracts the value of a pixel in one image patch from the value of the corresponding pixel in another image patch. This subtraction operation yields the difference image described herein. In one embodiment, difference detection is performed based on RGB values. However, difference detection is not necessarily performed based on RGB values. For example, difference detection can be performed based on luminance values. The low and high detection thresholds used for difference detection for each pixel in the difference image should adaptively depend on a noise threshold generated based on the image sensor noise distribution of each corresponding pixel in each image within the training image pair. Figure 11 The backend 1100 of the system for self-supervised training of image restoration networks described herein has been detailed in detail, including an embodiment of multi-scale per-image patch difference detection based on a noise threshold and how to compute an adaptive detection threshold based on the noise threshold.

[0175] Step 2184 includes analyzing the difference values ​​of image patch pairs using a detection threshold. This analysis determines whether to collect training image patch pairs for training the blind denoiser (in which case step 2186 is executed), whether to collect training image patch pairs for training the blind defocus deblurrer (in which case step 2188 is executed), or whether not to collect training image patch pairs. In one embodiment, if the difference value of an image patch pair is less than a low detection threshold, i.e., no defocus blur is detected between image patch pairs, step 2186 is executed to collect training image patch pairs for training the blind denoiser. In one embodiment, if the difference value of an image patch pair is greater than a high detection threshold, i.e., severe defocus blur is detected between image patch pairs, step 2188 is executed to collect training image patch pairs for training the blind defocus deblurrer. The above analysis compares the difference values ​​of image patch pairs with a low detection threshold and a high detection threshold. The above analysis can use a low detection threshold to test for no defocus blur between image patch pairs and a high detection threshold to test for severe defocus blur between image patch pairs.

[0176] Step 2186 includes collecting training image patch pairs for a blind denoiser when the difference value of the image patch pair is determined to be less than a low detection threshold. Step 2186 designates one image patch as a source image patch and the other as a target image patch. This designation can be based on two different aperture settings of the two training images. The image patch in the training image with the smaller aperture is designated as the source image patch. The image patch in the training image with the larger aperture is designated as the target image patch. The image patch with the smaller aperture can be defined as a noisy image patch, while the image patch with the larger aperture can be defined as a clean image patch. In step 2186, a confidence score can also be assigned to the training image patch pair. This confidence score can be based on the degree to which the difference value is less than the low detection threshold, etc.

[0177] Step 2188 includes collecting training image patch pairs for a blind defocus deblurrer when the difference value of an image patch is determined to be greater than a high detection threshold. Step 2188 designates one image patch as a source image patch and the other as a target image patch. This designation can be based on two different aperture settings of the two training images. The image patch in the training image with the larger aperture is designated as the source image patch. The image patch in the training image with the smaller aperture is designated as the target image patch. The image patch with the larger aperture can be defined as a blurred image patch due to defocus, while the image patch with the smaller aperture can be defined as a sharp image patch. In step 2188, a confidence score can also be assigned to the training image patch pair. This confidence score can be based on the degree to which the difference value is greater than the high detection threshold, etc. In some cases, the image patches may be scaled. For example, the training image patch pair can be reduced in size to provide a sharper target image patch.

[0178] Step 2190 includes determining whether there are any other image patches to be processed in the training image pair. If yes, the process returns to step 2184 to process the next image patch in each training image. If no, the process ends.

[0179] Figure 22This is a diagram of an embodiment of a federated learning system implemented through self-supervised training of an image restoration network. The system includes a central server 2200 and multiple edge devices 2202. Each edge device 2202 can be a user device, such as a camera (e.g., a surveillance camera, vehicle camera, consumer camera), smartphone, tablet, laptop, desktop computer, AR / VR device, drone (equipped with a camera), robot, etc. Each edge device 2202 is equipped with a self-supervised trained image restoration network 2210. The self-supervised trained image restoration network 2210 may include, but is not limited to, system 700 or backend 1100. Each edge device 2202 is equipped with a camera system, such as, but not limited to, any binocular or monocular camera system described in this specific embodiment and shown in the figures.

[0180] At a certain point in time, the initial global model of the image restoration network is loaded into edge device 2202. For example, the initial global model could be loaded in a factory and then sold to end users. However, the initial global model can also be provided to edge device 2202 from a server such as central server 2200. Each edge device 2202 is capable of determining updates to the image restoration network model. These updates can be determined through processes such as execution 800 or 1800. Updates by edge device 2202 are called local updates. Central server 2200 is equipped with a global image restoration model updater 2220, which is capable of updating the global model based on the local updates provided to central server 2200 from each edge device 2202.

[0181] Figure 23 This is a flowchart of an embodiment of a federated learning process 2300 implemented through self-supervised training of an image restoration network. Process 2300 can be used for... Figure 22In the depicted system, step 2302 includes: the central server 2200 sending a global model of the image restoration network to the edge devices 2202. Step 2304 includes: each edge device 2202 generating training image patch pairs locally. In one embodiment, each edge device 2202 performs steps 802 to 808 of process 800 to collect training image patch pairs locally. Step 2306 includes: each edge device 2202 using the locally collected training image patch pairs to compute a local update of the model. In one embodiment, a gradient descent training algorithm is employed, in which case the local update may be referred to as a local gradient update. In one embodiment, the local update is determined by the blind image denoiser 1110, the blind image deblurr 1114, and / or the image exposure corrector 1118 in the edge device 2202. In one embodiment, in process 2300, the image restoration network 708 in the edge device 2202 is not trained (updated) locally. Therefore, while these local updates can be used in process 1800 to train (update) the parameters of the image restoration network 708, in process 2300, these locally determined updates are not necessarily used to train the image restoration network 708 locally in the edge device 2202. Step 2308 includes: the edge device 2202 sending the local updates to the central server 2200. Step 2310 includes: the central server 2200 updating the global model of the image restoration network 708. The central server 2200 may aggregate the local updates and update the global model based on the aggregated updates. Then, process 2300 may return to step 2302 so that the central server 2200 sends the current global model to the edge device 2202.

[0182] Figure 22 The system and Figure 23 This method fully utilizes the diverse training data collected by the edge device 2202 while protecting the privacy of end users, and the computational burden is shared by the edge device 2202. To protect the privacy of end users, there is no need to send images from the edge device 2202 to the central server 2200.

[0183] Figure 24This is a block diagram of a processing device 2400 that can be used to implement various embodiments, including client processing devices such as system 700, front-end 1000, front-end 1050, back-end 1100, user equipment 1600, and / or edge device 2202. A particular device may utilize all or only a subset of the components shown, and the level of integration may vary from device to device. Furthermore, processing device 2400 may include multiple instances of components, such as multiple processing units, processors, memory, transmitters, receivers, etc. Processing device 2400 may include a central processing unit (CPU) 2410, a graphics processing unit (GPU) 2412, memory 2420, a mass storage device 2430, and an I / O interface 2460 connected to bus 2470. Bus 2470 may be one or more of several bus architectures of any type, including a memory bus, a memory controller, or a peripheral bus. Network interface 2450 enables the processing device to communicate with other processing devices, such as central server 2200, via network 2480. I / O interface 2460 is shown as being connected to display device 2465 and image capture device 2455.

[0184] CPU 2410 may include any type of electronic data processor. GPU 2412 is a processing unit capable of highly efficient data parallel processing. While GPUs can be used in graphics cards and the like for computer graphics processing, their applications extend far beyond this. In some embodiments, processing device 2400 includes CPU 2410 but not GPU 2412. Memory 2420 may include any type of system memory, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous DRAM (SDRAM), read-only memory (ROM), or combinations thereof. In one embodiment, memory 2420 may include ROM for use at startup and DRAM for storing programs and data for use during program execution. In embodiments, memory 2420 is non-transient memory. In one embodiment, memory 2420 includes computer-readable instructions executable by CPU 2410 and / or GPU 2412 to implement embodiments of the disclosed technology, including generating training image block pairs. The self-supervised training manager 1612 may include computer-readable instructions executed by the CPU 2410 and / or GPU 2412. The image restoration network 708 may include computer-readable instructions executed by the CPU 2410 and / or GPU 2412. The memory 2420 may also store the image restoration model 2424. In one embodiment, the processing device trains or updates the image restoration model 2424. For example, the self-supervised training manager 1612 may collect training image patch pairs, which the image restoration network 708 can use to train (or update) the image restoration model 2424.

[0185] Mass storage device 2430 may include any type of storage device for storing data, programs, and other information and making the data, programs, and other information accessible via bus 2470. Mass storage device 2430 may include, for example, one or more of solid-state drives, hard disk drives, disk drives, or optical disk drives. Instances of image restoration models 2424 may be stored in mass storage device 2430. It should also be noted that instances implementing the instructions of self-supervised training manager 1612 and image restoration network 708 may be stored in mass storage device 2430. When processing device 2400 is powered on, instances in mass storage device 2430 may be loaded into memory 2420.

[0186] Figure 25This is a block diagram of a service host 2500 that can be used to implement various embodiments of the central server 2200. A particular server may utilize all or only a subset of the components shown, and the level of integration may vary from device to device. Figure 25 In this context, similar numbers represent... Figure 24 Similar components. In one embodiment, memory 2420 includes computer-readable instructions that, when executed on CPU 2410 and / or GPU 2412, implement global image restoration model updater 2220. Mass storage 2430 may also include code comprising instructions for causing CPU 2410 and / or GPU 2412 to implement global image restoration model updater 2220.

[0187] The embodiments of the disclosed technologies described above may also be implemented at least in part using hardware logic components, such as, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), and special-purpose computers.

[0188] For the purposes of this document, it should be noted that the dimensions of the various features depicted in the figures may not necessarily be drawn to scale.

[0189] For the purposes of this document, references to "embodiment", "one embodiment", "some embodiments" or "another embodiment" in the specification may be used to describe different or the same embodiments.

[0190] For the purposes of this document, a connection can be a direct connection or an indirect connection (e.g., through one or more other components). In some cases, when one element is said to be connected to or coupled to another element, that element can be directly connected to the other element or indirectly connected to the other element through an intermediate element. When one element is said to be directly connected to another element, there is no intermediate element between that element and the other element. If two devices are directly or indirectly connected so that they can transmit electronic signals between them, then they are in “communication”.

[0191] While the invention has been described with reference to specific features and embodiments thereof, it will be apparent that various modifications and combinations thereof can be made without departing from the scope of the invention. Therefore, the specification and drawings are to be regarded merely as a description of the invention as defined by the appended claims, and are intended to cover any and all modifications, variations, combinations, or equivalents falling within the scope of the invention.

[0192] The techniques described herein can be implemented using hardware, software, or a combination of hardware and software. The software used is stored in one or more processor-readable storage devices described above to program one or more processors to perform the functions described herein. Processor-readable storage devices can include computer-readable media, such as volatile and non-volatile media, removable and non-removable media. For example, but not limited to, computer-readable media can include computer-readable storage media and communication media. Computer-readable storage media can be implemented using any method or technique to store computer-readable instructions, data structures, program modules, or other data and other information. Examples of computer-readable storage media include RAM, ROM, EEPROM, flash memory or other storage technologies, CD-ROM, digital versatile disk (DVD) or other optical disc storage, magnetic tape cassettes, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. One or more computer-readable media do not include propagating signals, modulated signals, or transient signals.

[0193] Communication media typically embody computer-readable instructions, data structures, program modules, or other data in the form of transmitted data signals, modulated data signals, or transient data signals (such as carrier waves or other transmission mechanisms), and include any information transmission medium. The term "modulated data signal" refers to a signal in which one or more characteristics are set or altered by means of information encoded in the signal. As an example, and not a limitation, communication media include wired media such as wired networks or direct wired connections, as well as wireless media such as RF and other wireless media. Combinations of the foregoing are also included within the scope of computer-readable media.

[0194] In alternative embodiments, some or all of the software may be replaced by dedicated hardware logic components. Illustrative types of hardware logic components that may be used include, for example, but not limited to, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SOCs), complex programmable logic devices (CPLDs), dedicated computers, etc. In one embodiment, software implementing one or more embodiments (stored in a storage device) is used to program one or more processors. These one or more processors may communicate with one or more computer-readable media / storage devices, peripheral devices, and / or communication interfaces.

[0195] It should be understood that the subject matter of this invention can be embodied in many different ways and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to make the subject matter thorough and complete and to fully convey the invention to those skilled in the art. In fact, the purpose of this subject matter is to cover alternatives, modifications, and equivalents to these embodiments, which are included within the scope and spirit of the subject matter as defined by the appended claims. Furthermore, in the foregoing detailed description of the subject matter, numerous specific details have been set forth to provide a thorough understanding of the subject matter. However, it will be apparent to those skilled in the art that the subject matter can be practiced without these specific details.

[0196] Various aspects of the invention are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via a processor of a computer or other programmable instruction execution apparatus, create a mechanism for implementing the functions / actions specified in one or more blocks of the flowchart illustrations and / or block diagrams.

[0197] The description of this invention is presented for illustrative and descriptive purposes only and is not intended to be exhaustive or to limit the invention in any way disclosed. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the invention. Various aspects of the invention were chosen and described in order to better explain the principles and practical application of the invention and to enable those skilled in the art to understand the invention and the various modifications suited to the intended particular use.

[0198] For the purposes of this document, each process associated with the disclosed technology may be executed sequentially by one or more computing devices. Each step in the process may be executed by the same or different computing devices used in the other steps, and each step need not be executed by a single computing device.

[0199] Although the subject matter has been described in language specific to structural features and / or methodological actions, it should be understood that the subject matter as defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are disclosed as exemplary ways of implementing the claims.

Claims

1. A system, characterized in that, include: An image capture system for capturing images, wherein the image capture system is used to control the exposure of each captured image by adjusting at least one of a plurality of exposure parameters; One or more control circuits are coupled to and communicate with the image capture system, wherein the one or more control circuits are used for: Control the plurality of exposure parameters of the image captured by the image capture system; Training image pairs are generated from the captured images, wherein each training image pair includes a first training image of the scene and a second training image of the scene, the first training image and the second training image having different values ​​on at least one of the plurality of exposure parameters; Training image patch pairs are collected from the training image pairs, wherein each training image patch pair includes a source image patch and a target image patch; The image restoration network is trained based on the training image patch pairs.

2. The system according to claim 1, characterized in that, The training image block pair includes a denoised training image block pair, and each denoised training image block pair includes a noisy source image block and a ground truth clean target image block. The image restoration network includes a blind denoising network; The one or more control circuits are used to train the blind denoising network based on the denoised training image patch pairs.

3. The system according to claim 1, characterized in that, The training image block pair includes a deblurred training image block pair, and each deblurred training image block pair includes a blurred source image block and a ground truth clear target image block; The image restoration network includes a blind deblurring network; The one or more control circuits are used to train the blind deblurring network based on the deblurring training image patch pairs.

4. The system according to claim 1, characterized in that, The training image block pair includes exposure training image block pairs, and each exposure training image block pair includes an abnormal exposure source image block and a ground truth normal exposure target image block; The image restoration network includes an exposure correction network; The one or more control circuits are used to train the exposure correction network based on the exposure training image patch pairs.

5. The system according to claim 4, characterized in that, The one or more control circuits are used for: Collect the first set of exposure training image patch pairs, wherein each exposure training image patch pair includes an underexposed source image patch and a ground truth normally exposed target image patch; Collect a second set of exposure training image patch pairs, wherein each exposure training image patch pair includes an overexposed source image patch and a ground truth normally exposed target image patch; The exposure correction network is trained based on the first set of exposure training image block pairs and the second set of exposure training image block pairs.

6. The system according to claim 1, characterized in that, The one or more control circuits are used for: A first aperture setting is set for the first training image, and a second aperture setting is set for the second training image, wherein a specific pair of training images has substantially the same exposure value (EV). Analyze the pixel differences between the first pixel in the first image block of the first training image and the second pixel in the second image block of the second training image within the specific training image pair; Based on the pixel differences, the first aperture setting, and the second aperture setting, one of the first image blocks or the second image blocks is classified as a source image block, and the other of the first image blocks or the second image blocks is classified as a target image block.

7. The system according to claim 6, characterized in that, The one or more control circuits are used for: The first training image and the second training image in the specific training image pair are set to substantially the same shutter speed, wherein the first training image and the second training image in the specific training image pair have different ISO levels.

8. The system according to claim 7, characterized in that, The first aperture setting is lower than the second aperture setting, and the one or more control circuits are used for: When the difference between the first pixel in the first image block and the second pixel in the second image block is less than a low detection threshold, the first image block is classified as the source image block, and the second image block is classified as the target image block. When the difference between the first pixel in the first image block and the second pixel in the second image block is greater than a high detection threshold, the first image block is classified as the target image block, and the second image block is classified as the source image block.

9. The system according to claim 8, characterized in that, The one or more control circuits are used for: When the difference between the first pixel in the first image block and the second pixel in the second image block is less than the low detection threshold, the image restoration network is trained based on the source image block and the target image block to achieve blind denoising; When the difference between the first pixel in the first image block and the second pixel in the second image block is greater than the high detection threshold, the image restoration network is trained based on the source image block and the target image block to achieve blind deblurring.

10. The system according to claim 1, characterized in that, The one or more control circuits are used for: A first shutter speed is set for the first training image, and a second shutter speed is set for the second training image, wherein a specific pair of training images has substantially the same exposure value (EV). Analyze the pixel differences between the first pixel in the first image block of the first training image and the second pixel in the second image block of the second training image within the specific training image pair; Based on the pixel difference, the first shutter speed, and the second shutter speed, one of the first image blocks or the second image blocks is classified as a source image block, and the other of the first image blocks or the second image blocks is classified as a target image block.

11. The system according to claim 10, characterized in that, The one or more control circuits are used for: The first training image and the second training image within the specific training image pair are given substantially the same aperture settings, wherein the first training image and the second training image within the specific training image pair have different ISO levels.

12. The system according to claim 11, characterized in that, The first shutter speed is higher than the second shutter speed, and the one or more control circuits are used to: When the difference between the first pixel in the first image block and the second pixel in the second image block is less than a low detection threshold, the first image block is classified as the source image block, and the second image block is classified as the target image block. When the difference between the first pixel in the first image block and the second pixel in the second image block is greater than a high detection threshold, the first image block is classified as the target image block, and the second image block is classified as the source image block.

13. The system according to claim 12, characterized in that, The one or more control circuits are used for: When the difference between the first pixel in the first image block and the second pixel in the second image block is less than the low detection threshold, the image restoration network is trained based on the source image block and the target image block to achieve blind denoising; When the difference between the first pixel in the first image block and the second pixel in the second image block is greater than the high detection threshold, the image restoration network is trained based on the source image block and the target image block to achieve blind deblurring.

14. The system according to claim 1, characterized in that, The one or more control circuits are used for: For a specific pair of training images, a first exposure value is set for the first training image, and a second exposure value is set for the second training image; For the specific training image pair, analyze the first brightness distribution generated based on the first pixel in the first image block of the first training image and the second brightness distribution generated based on the second pixel in the second image block of the second training image; Based on the first brightness distribution, the second brightness distribution, the first exposure value, and the second exposure value, determine whether to classify one of the first image blocks or the second image blocks as a source image block, and whether to classify the other of the first image blocks or the second image blocks as a target image block.

15. The system according to claim 14, characterized in that, The one or more control circuits are used for: For the specific training image pair, a first ISO level is set for the first training image, and a second ISO value is set for the second training image, wherein the aperture settings and shutter speeds are basically the same.

16. The system according to claim 15, characterized in that, The first ISO value is less than the second ISO value, and the one or more control circuits are used to: When the first brightness distribution meets the underexposure condition and the second brightness distribution meets the normal exposure condition, the first image block is classified as an underexposure source image block and the second image block is classified as a ground truth normal exposure target image block. When the first brightness distribution meets the normal exposure condition and the second brightness distribution meets the overexposure condition, the first image block is classified as a ground truth normally exposed target image block, and the second image block is classified as an overexposed source image block.

17. The system according to claim 16, characterized in that, The one or more control circuits are used for: When the first brightness distribution satisfies the underexposure condition and the second brightness distribution satisfies the normal exposure condition, the image restoration network is trained based on the underexposure source image block and the ground true normal exposure target image block to achieve underexposure correction; When the first brightness distribution satisfies the normal exposure condition and the second brightness distribution satisfies the overexposure condition, the image restoration network is trained based on the overexposure source image block and the ground truth normally exposed target image block to achieve overexposure correction.

18. The system according to claim 1, characterized in that, The one or more control circuits are used for: The system is run in training mode, wherein the training image patch pairs are used for self-supervised training of the image restoration network in the training mode; The system is operated in inference mode, wherein the first training image and the second training image within each training image pair are merged into a merged image and provided to the image restoration network in the inference mode.

19. The system according to claim 1, characterized in that, The image capture system includes a binocular camera, which includes a first image sensor and a second image sensor. The first image sensor and the second image sensor are used to capture frames, wherein each frame has one exposure. The one or more control circuits are used to generate training image pairs from a first frame of the first image sensor and a second frame of the second image sensor, wherein the first frame and the second frame are exposed within the same time interval.

20. The system according to claim 1, characterized in that, The image capture system includes a monocular camera for capturing frames, wherein each frame has a first exposure and a second exposure. The one or more control circuits are used to generate training image pairs from the first exposure and the second exposure within each frame.

21. The system according to claim 1, characterized in that, The one or more control circuits are used for: Based on the training image patch pairs, the local updates of the image restoration network are determined; The local update is sent to a central server, wherein the central server receives the local update from multiple edge devices, and the multiple edge devices determine the local update of the image restoration network based on training image patch pairs; The local updates are aggregated at the central server, and the global image restoration network is updated based on the aggregated updates; The updated global image restoration network is forwarded to the edge device.

22. A method for self-supervised training of image restoration networks, characterized in that, The method includes: Control multiple exposure parameters of the image captured by the image capture system; Training image pairs are generated from the captured images, wherein each training image pair includes a first training image of the scene and a second training image of the scene, the first training image and the second training image having different values ​​on at least one of the plurality of exposure parameters; Training image patch pairs are generated from the training image pairs, wherein each training image patch pair includes a source image patch and a target image patch; The training image patch pairs are provided to the image restoration network; The image restoration network is trained based on the training image patch pairs.

23. The method according to claim 22, characterized in that, Also includes: The trained image restoration network is used to perform image restoration on the images captured by the image capture system.

24. The method according to claim 22, characterized in that, Also includes: The global model of the image restoration network is received from the central server at the edge device; At the edge device, based on the training of the image restoration network using the training image patch pairs, a local update of the global model of the image restoration network is calculated; The local update is sent to the central server.

25. The method according to claim 24, characterized in that, Also includes: Local updates are received from multiple edge devices at the central server. Aggregate the local updates received from the plurality of edge devices; The global model of the image restoration network is updated based on the aggregated local updates from the multiple edge devices. The updated global model of the image restoration network is sent to at least one of the plurality of edge devices.

26. A user equipment, characterized in that, include: Image capture system, used to capture images; A training image generator is coupled to the image capture system and used to generate training image pairs, wherein each training image pair includes a first training image of the scene and a second training image of the scene, and the image capture system is used to control the exposure of each training image by adjusting at least one of a plurality of exposure parameters; An exposure controller, coupled to the image capture system, wherein the exposure controller is used to control the exposure of the training images such that the first training image and the second training image within each training image pair have different values ​​on at least one of the plurality of exposure parameters; An image restoration network is coupled to the image capture system, wherein the image restoration network is used to restore the image captured by the image capture system; A training controller is coupled to the training image generator, the exposure controller, and the image restoration network, wherein the training controller is configured to collect training image patch pairs from the training image pairs, each training image patch pair including a source image patch and a target image patch, and the training controller is further configured to provide the training image patch pairs to the image restoration network and control the supervised training of the image restoration network based on the training image patch pairs.

Images