Apparatus and method for image processing

Abstract

The present invention provides a selection device, a training device using the selection device, and an image restoration device trained using the training device, wherein the selection device is used to select a set of image segments for training the image restoration device from one or more candidate images. The selection device is used to implement a segmentation neural network model to identify image segments of one or more candidate images in the set of image segments.

Description

Technical Field

[0001] The present invention relates to restoring degraded image data using a machine learning model. Background Art

[0002] Image restoration is the process of improving the quality of degraded images. There are many common forms of image degradation, including noise, blur, limited contrast, or low resolution. Many modern image restoration methods use machine learning based on deep neural networks, where a deep neural network consists of an architecture and a set of parameters (also called weights). These parameters are found in a training process (also called learning), which mathematically optimizes the error calculated using the training data.

[0003] In the case of image restoration, the training data typically consists of a collection of image pairs, each pair consisting of a degraded image and a ground truth restored image. Each degraded image is passed to the network, which restores the image in a forward pass. The restored image is compared to the ground truth image, and the difference between the two is encoded as an error (also called loss). This error is then backpropagated through the network in a backward pass that updates the parameters. This process (a forward pass followed by a backward pass) is repeated multiple times on the collection of images until convergence. Training methods that update all parameters in the network from input to output are called end-to-end methods. The final trained deep network, including the architecture and learned parameters, is also called a model and can be used to restore new, unseen degraded images during inference. For computational efficiency reasons, patches or cropped segments of the image are often used for training.

[0004] The standard image restoration learning pipeline includes building a database, sampling training data, building a network and optimizing it. Existing training methods usually uniformly sample training data from the database. However, uniform sampling does not carefully consider data distribution and intrinsic characteristics. Image restoration tasks usually use small segments randomly cropped from the entire image as training samples. However, many segments cropped from the image are too simple, resulting in a long-tail training data distribution. This data distribution leads to poor model performance.

[0005] Existing image restoration methods use end-to-end schemes to generate high-quality images from low-quality images. In most methods, each pixel is sampled with the same probability and incurs the same loss, such as image denoising, image deblurring, joint denoising and demosaicing, super-resolution, and other image restoration tasks. However, different parts of real images have different characteristics, such as high-frequency and low-frequency patterns. Feature differences are crucial for some tasks with local degradation (e.g., local motion blur). Training models on images with different characteristics also fails to achieve the highest performance. To adjust the network to adapt to challenging samples, some studies have reweighted the training data and demonstrated that deep models can achieve better performance by mining difficult samples. Specifically, after training a deep neural network for demosaicing, existing methods manually select difficult samples to fine-tune the network. However, its data weighting method is done through a laborious two-stage process. In addition, there is no guarantee that the selected difficult samples can improve the universality of the network.

[0006] There is a need to develop an improved sampling process for training image restoration networks, thereby using images that are generally more useful for training the network to improve the performance of image restoration. Summary of the invention

[0007] According to one aspect, the present invention provides a selection device for selecting a set of image segments for training an image restoration device from one or more candidate images, and the selection device is used to implement a segmentation neural network model to identify image segments of one or more candidate images in the set of image segments.

[0008] According to a second aspect, the present invention provides a training device for training an image restoration device based on a set of image segments selected from one or more candidate images according to a segmentation neural network model, wherein the training device is used to implement the segmentation neural network model to select the set of image segments for training the image restoration device, thereby enhancing the ability of the image processing neural network model to instruct restoration processing of degraded image data.

[0009] The segmentation neural network model can be used to generate trainable scalar data based on the reconstructed image output by the image processing neural network model, and the trainable scalar data is combined with the reconstructed image to optimize the image processing neural network model.

[0010] The image processing neural network model can be trained according to the following steps: receiving the degraded image data and providing it as an initial input to the training device; passing the degraded image data to the image processing neural network model, the image processing neural network model is used to create reconstructed image data by performing the restoration process on the degraded image data; feeding the reconstructed image to the segmentation neural network model, the segmentation neural network model is used to select the set of image segments from one or more candidate images and generate trainable scalar data; applying the trainable scalar data to the reconstructed image data to determine weighted loss data; updating the image processing neural network model based on the weighted loss data.

[0011] The segmentation neural network model generating the trainable scalar data may include: dividing the reconstructed image into multiple segments; downsampling each segment to a single pixel value; and combining the multiple single pixel values ​​to form a scalar weight map.

[0012] The downsampling may include a plurality of convolutional blocks and a final binarization layer to provide a scalar value representing the trainability of each of the plurality of segments.

[0013] The binarization layer may include a binarization function. The binarization function may be:

[0014]

[0015] Here, φ(·) is a sigmoid function with temperature T, and x is any possible variable.

[0016] The segmentation neural network model may operate on the reconstructed image to generate the trainable scalar data according to the following equation:

[0017] t p =φ(f(I p ; W f )),

[0018] Among them, I p Denotes each segment at position p on the reconstructed image I, f and W f They respectively represent the function of the segmentation neural network model and the corresponding weights learned for the function f, and φ represents the binarization function.

[0019] The weighted loss data may be determined by applying the trainable scalar data to the reconstructed image data by the following calculation:

[0020]

[0021] in, is the loss function for each segment restoration process, N represents the number of segments in the degraded image data, and the loss weight Rescale the loss to its original size.

[0022] According to a third aspect, the present invention provides a method for selecting a set of image segments from one or more candidate images for training an image restoration device, the method comprising implementing a segmentation neural network model, which is used to identify image segments of the one or more candidate images to be included in the set of image segments, thereby providing optimized candidate images to train the image restoration device.

[0023] According to a fourth aspect, the present invention provides a method for training an image restoration device based on a set of image segments selected from the one or more candidate images according to a segmentation neural network model, the method comprising: implementing the segmentation neural network model to select the set of image segments for training the image restoration device from one or more candidate images; optimizing the image processing neural network model of the image restoration device by performing restoration processing on the selected set of image segments.

[0024] The method may include: the segmentation neural network model generates trainable scalar data based on the reconstructed image output by the image processing neural network model; combining the trainable scalar data with the reconstructed image output by the image processing neural network model; and optimizing the image processing neural network model based on the combined scalar data and image output.

[0025] The segmentation neural network model generating the trainable scalar data may include: dividing the reconstructed image into multiple segments; downsampling each segment to a single pixel value; and combining the multiple single pixel values ​​to form a scalar weight map.

[0026] The downsampling includes a plurality of convolutional blocks and a final binarization layer to provide a scalar value representing the trainability of each of the plurality of segments.

[0027] The binarization layer may include a binarization function. The binarization function may be:

[0028]

[0029] Here, φ(·) is a sigmoid function with temperature T, and x is any possible variable.

[0030] The segmentation neural network model may operate on the reconstructed image to generate the trainable scalar data according to the following equation:

[0031] t p =φ(f(Ip ; W f ))

[0032] Among them, I p Denotes each segment at position p on the reconstructed image I, f and W f They respectively represent the function of the segmentation neural network model and the corresponding weights learned for the function f, and φ represents the binarization function.

[0033] The weighted loss data may be determined by applying the trainable scalar data to the reconstructed image data by the following calculation:

[0034]

[0035] in, is the loss function for each segment restoration process, N represents the number of segments in the degraded image data, and the loss weight Rescale the loss to its original size.

[0036] According to a fifth aspect, the present invention provides an image restoration device for performing restoration processing on degraded image data based on a set of image segments, the image restoration device comprising an image processing neural network model trained by a training device, the training device being used to implement a segmentation neural network model to select the set of image segments from one or more candidate images.

[0037] The device can be used to train an image processing system according to the method described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] The present invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0039] Figure 1 A brief overview of the working principle of the proposed method is shown;

[0040] Figure 2 The architecture of the proposed method is shown;

[0041] Figure 3 An example of selected segmentations resulting from implementing the proposed segmentation neural network is shown;

[0042] Figure 4 An example of the result of the demosaicing restoration process is shown;

[0043] Figure 5 An example of a device for implementing the proposed image restoration processing method is shown. DETAILED DESCRIPTION

[0044] The proposed method focuses on training data to improve model performance. Specific aspects of the method include an end-to-end learning approach that considers each candidate image segmentation with different weights. The weight map for each candidate image is learned by a separate network based on active learning. The proposed method can be applied to multiple low-level vision tasks, including image demosaicing, denoising, and joint denoising and demosaicing.

[0045] Figure 1 A brief overview of the working principle of the proposed method is shown; during the active learning based training process, the proposed method learns different weights for different image parts or segments. The weights are encoded on a per-segment basis to form Figure 1 The weight map 106 is shown on the right. The weight map 106 is optimized based on the restored image 102. The proposed method does not operate directly on the data, but performs strong negative sampling at the loss level. The segmentation neural network model 104 evaluates the restored image 102 to identify image segments of one or more candidate images for training, thereby creating a set of image segments for training an image restoration device.

[0046] The goal is to automatically distinguish the hard samples in the entire dataset and select them as training samples. Generally speaking, methods designed for low-level vision tasks are image-to-image translation, which means that the difficulty of each input sample can be measured by the quality of the output image. Therefore, another network, such as denoising / demosaicing, is applied on top of the main task network to evaluate the quality of the output image. The network can also be called PatchNet because it is used to reweight the blocks or segments in the image; accordingly, the relay network responsible for restoring the original image can be called RestoreNet.

[0047] Therefore, a selection device is proposed, the selection device being used to select a set of image segments for training an image restoration device from one or more candidate images. Therefore, the selection device can be used to implement a segmentation neural network model to identify image segments of one or more candidate images in the set of image segments.

[0048] In addition, a training device is also proposed, the training device is used to train an image restoration device based on a first set of image segments selected from one or more candidate images according to a segmentation neural network model. The training device can be used to implement the segmentation neural network model to select the first set of image segments for training the image restoration device, thereby enhancing the ability of the image processing neural network model to instruct the restoration processing of degraded image data.

[0049] Specifically, the segmentation neural network model can be used to generate trainable scalar data based on the reconstructed image output by the image processing neural network model, and the trainable scalar data is combined with the reconstructed image to optimize the image processing neural network model.

[0050] The image processing neural network model can be trained according to the following steps: receiving the degraded image data and providing it as an initial input to the training device; passing the degraded image data to the image processing neural network model, the image processing neural network model is used to create reconstructed image data by performing the restoration process on the degraded image data; feeding the reconstructed image to the segmentation neural network model, the segmentation neural network model is used to select the set of image segments from one or more candidate images and generate trainable scalar data; applying the trainable scalar data to the reconstructed image data to determine weighted loss data; and then, updating the image processing neural network model based on the weighted loss data.

[0051] Figure 2 The architecture of the proposed method is shown in FIG. 1 . The segmentation neural network 104 for selecting the image segment can be a neural network with log 2 A feed-forward network of k levels. Each level may contain several convolution and downsampling operators 202, 204, 206, 208, so that each input segment can eventually be downsampled to only a single trainable scalar. The trainable scalar can then be applied back to the entire segment through an attention-like element-wise multiplication operation indicated by the weighted segment 210. The mask output can be supervised by a loss so that the entire framework can be trained in an end-to-end manner.

[0052] like Figure 2 As shown, the output of the image processing neural network (e.g., an RGB image) can be fed as an input into the segmentation neural network. The problem is formulated as a supervised regression problem by downsampling each segment of a predefined size to a single pixel representing the trainability of the segment. The segmentation neural network is divided into multiple stages, each of which gradually converts the image into a trainable scalar. Each stage can be composed of several convolution blocks and downsampling layers. That is, generating the trainable scalar may include: dividing the reconstructed image into multiple segments; downsampling each segment to a single pixel value; and combining the multiple single pixel values ​​to form a scalar weight map. The downsampling may include multiple convolution blocks and a final binarization layer to provide a scalar value representing the trainability of each of the multiple segments.

[0053] Each block at position p on the output restored image I from the image processing neural network is represented as I p, where the trainability scalar is represented by I p , the whole process can be described as:

[0054] t p =φ(f(I p ; W f )), (1)

[0055] Among them, f and W f represents the segmentation neural network and its corresponding learnable weights in the entire network, and φ represents the binarization function. Typically, p on the input image should be a square bounding box of size k×k.

[0056] To simulate the selection operation, the trainability scalar should be binarized. Here, an S-shaped function φ(·) with a temperature of T is used to control the function sharpness for binarization, which can be expressed as:

[0057] φ(x)= 1 / (1++e -Tx ) (2)

[0058] Where x represents any possible variable. The binary output of the function φ(·) will also be used as the final output of the segmentation neural network. In this way, the trainability value t of each input segment p can be obtained. p .

[0059] Intuitively, to use the trainability value to guide the image processing neural network, t can be used during the loss calculation. p Apply back the output of the image processing neural network I p .

[0060] Assume that the initial loss function of the image processing neural network for each block is L R , the total loss of the segmentation neural network and the image processing neural network is The loss can be calculated by the following formula:

[0061]

[0062] Where the constant N represents the number of blocks to be cropped from each image. Here, the loss weight The

[0063] The loss is rescaled to its original size. In this case, the original loss will be assigned to the blocks with greater trainability. The loss that penalizes the image processing neural network may be varied. The L2 loss is selected as the loss function of the image processing neural network, and the corresponding final loss is calculated based on this.

[0064] According to the above description of the proposed method, a method for selecting a set of image segments for training an image restoration device from one or more candidate images is also proposed. The method includes implementing a segmentation neural network model, which is used to identify image segments of the one or more candidate images to be included in the set of image segments, thereby providing optimized candidate images to train the image restoration device.

[0065] Similarly, a method for training the image restoration device is also proposed. The method for training the image restoration device is based on a set of image segments selected from the one or more candidate images according to a segmentation neural network model. The method comprises: implementing the segmentation neural network model to select the set of image segments for training the image restoration device from one or more candidate images; optimizing the image processing neural network model of the image restoration device by performing restoration processing on the selected set of image segments.

[0066] Due to the special needs of cross-domain verification, it is necessary to collect two datasets containing natural images and only difficult samples respectively. For natural images, the large dataset Vimeo-90K can be selected as a benchmark. The dataset was originally created to evaluate various video restoration tasks. The dataset contains 89,800 independent segments from 4,278 images with a resolution of 448×256, which is large and diverse enough to be selected as the natural image dataset. The 4th image of a total of 7 frames in each segment can be selected. The entire dataset is divided into 64,612 segments for training and 7,824 segments for evaluation.

[0067] For the dataset with only hard samples, the MIT Moire dataset can be selected. The dataset is carefully collected from a very large parent set (downloaded from the Web). The failure cases can be detected and retained by applying the network trained on Imagenet to millions of new segments to form the MIT Moire dataset. There are 2.6 million segments of size 128×128 in the dataset. Of course, other suitable datasets can be used to create a suitable training dataset for the proposed method.

[0068] Two different networks are constructed, which differ in the number of convolutional layers or depth. The segmentation neural network or PatchNet can be constructed by stacking Conv-BatchNorm-ReLU blocks 3 times at each level, and the number of levels can be calculated based on the segment size k×k by log 2k is calculated. The input of the segmentation neural network is the full-size image with padding. The size of each segment k×k in the image is 64×64, and the sum of horizontal and vertical padding can be set to 64. It should be noted that for each iteration, the padding size on each side of the image can be a random number less than 64. Random padding can be regarded as a data augmentation method to generate more segments with different patterns. It should be noted that the segmentation neural network is only applied during training, so there is no need to apply all the above processes in testing.

[0069] The image restoration device itself can be used to perform restoration processing on degraded image data based on a set of image segments. The image restoration device includes an image processing neural network model trained by a training device, and the training device is used to implement a segmentation neural network model to select the set of image segments from one or more candidate images.

[0070] The image processing neural network can be placed in any type of framework for demosaicing or JDD tasks, such as DeepJDD, SGNet, etc. DeepJDD was chosen as the baseline due to its simplicity. The network consists of 18 convolutions with a kernel size of 3×3. Each has a ReLU activation function as a nonlinear function.

[0071] During training, the image processing neural network can be optimized using the ADAM optimizer. The learning rate is initialized to 2.5×10 -4 , and it is adjusted at each epoch by a half-periodic cosine schedule. The batch size is set to 16 for all experiments. The network can be trained for 150 epochs, and the version with the best performance on the evaluation dataset can be selected as the final output. For joint denoising and demosaicing, the input can be perturbed with noise variance σ∈[0,16].

[0072] Figure 3 An example of the resulting segmentation learned from the segmentation neural network or PatchNet is shown. The darker the segment (here highlighted with a dotted line), the higher the trainability. Most of the higher frequency segments will be focused by the segmentation neural network. The set of selected segments 302 can then be used during optimization and training of the image processing neural network.

[0073] Figure 4 An example of the results of the demosaicing restoration process is shown. The image 402 on the left has been corrected by an image processing neural network trained on the image segments selected by the segmentation neural network. It can be seen that the mosaic processed by the trained image processing neural network is smoother around the characters than the baseline result on the right 406. The middle image 404 is the ground truth image.

[0074] The proposed method has the following advantages: No additional computation is required during inference to improve performance. Compared with traditional methods, the proposed method only requires additional computation during training.

[0075] Robust to imbalanced training data. In low-level vision tasks, it is difficult to balance the training data with respect to image features, since they are difficult to describe or quantify and are likely to be local.

[0076] Typical models can overfit on the essential patterns in the dataset, but ignore the rare hard patterns. The proposed method can reweight the training data, resulting in a more robust model.

[0077] End-to-end training: The proposed method learns the weight map in an end-to-end manner without any separate / pre-training process.

[0078] The proposed method is widely applicable to many low-level vision problems, including joint denoising and demosaicing, super-resolution, and deblurring.

[0079] Figure 5 An example of a camera is shown, which is used to implement the above method to perform image restoration processing on an image captured by an image sensor 1102 in a camera 1101. Such a camera 1101 includes some onboard processing capabilities. This can be provided by the processor 1104. The processor 1104 can also be used for basic functions of the device. The camera typically also includes a memory 1103.

[0080] The transceiver 1105 is capable of communicating with other entities 1110, 1111 via a network. These entities may be physically remote from the camera 1101. The network may be a publicly accessible network, such as the Internet. The entities 1110, 1111 may be cloud-based. In one example, entity 1110 is a computing entity and entity 1111 is a command and control entity. These entities are logical entities. In practice, each of them may be provided by one or more physical devices (e.g., a server and a data storage area), and the functions of two or more of the entities may be provided by a single physical device. Each physical device implementing the entity includes a processor and a memory. The device also includes a transceiver for sending data to and receiving data from the transceiver 1105 of the camera 1101. The memory stores code in a non-transient manner, which may be executed by the processor to implement the corresponding entity in the manner described herein.

[0081] The command and control entity 1111 can train the artificial intelligence models used in each module of the system. This is generally a computationally intensive task, even if the obtained models can be efficiently described, so the development of the algorithms can be efficiently performed in the cloud, where it is foreseeable that a large amount of energy and computing resources are available. It is foreseeable that this is more efficient than forming such models on a typical camera.

[0082] In one implementation, after developing a deep learning algorithm in the cloud, the command and control entity can automatically form a corresponding model and transmit it to the relevant camera device. In this example, the system is implemented at the camera 1101 by the processor 1104.

[0083] In another possible implementation, the image may be captured by the camera sensor 1102, and the image data may be sent to the cloud by the transceiver 1105 for processing in the system. Then, the generated target image may be sent back to the camera 1101, such as Figure 5 As shown in 1112.

[0084] Thus, the method can be deployed in a variety of ways, such as in the cloud, on the device, or in dedicated hardware. As described above, a cloud facility can perform training to develop new algorithms or improve existing algorithms. Depending on the computing power near the data corpus, the training can be performed close to the source data or in the cloud, such as using an inference engine. The system can also be implemented at the camera, in dedicated hardware, or in the cloud.

[0085] Applicants hereby disclose separately each individual feature described herein and any combination of two or more such features. With the common knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, without considering whether such features or combinations of features can solve any problem disclosed herein, and are not limited to the scope of the claims. This application shows that various aspects of the present invention can be composed of any such individual features or combinations of features. In view of the foregoing description, it will be obvious to those skilled in the art that various modifications can be made within the scope of the present invention.

Claims

1. A training device for training an image restoration device based on a set of image segments selected from one or more candidate images according to a segmentation neural network model, It is characterized in that The training device is used to implement the segmentation neural network model to select the set of image segments used to train the image restoration device, thereby enhancing the ability of the image processing neural network model to instruct the restoration processing of degraded image data; The segmentation neural network model is used to generate trainable scalar data based on the reconstructed image output by the image processing neural network model, and the trainable scalar data is combined with the reconstructed image to optimize the image processing neural network model.

2. The training device according to claim 1, It is characterized in that The image processing neural network model is trained according to the following steps: receiving the degraded image data and providing it as initial input to the training device; Passing the degraded image data to the image processing neural network model, the image processing neural network model is used to create reconstructed image data by performing the restoration process on the degraded image data; Feeding the reconstructed image to the segmentation neural network model, the segmentation neural network model is used to select the set of image segments from one or more candidate images and generate trainable scalar data; applying the trainable scalar data to the reconstructed image data to determine weighted loss data; The image processing neural network model is updated based on the weighted loss data.

3. The training device according to claim 2, It is characterized in that The segmentation neural network model generates the trainable scalar data including: dividing the reconstructed image into a plurality of segments; Downsample each segment to a single pixel value; The plurality of single pixel values ​​are combined to form a scalar weight map.

4. The training device according to claim 3, It is characterized in that The downsampling includes a plurality of convolutional blocks and a final binarization layer to provide a scalar value representing the trainability of each of the plurality of segments.

5. The training device according to claim 4, It is characterized in that The binarization layer includes a binarization function.

6. The training device according to claim 5, It is characterized in that The binarization function is: Here, φ(·) is a sigmoid function with temperature T, and x is any possible variable.

7. The device according to any one of claims 2 to 6, It is characterized in that The segmentation neural network model operates on the reconstructed image to generate the trainable scalar data according to the following equation: t p =φ(f(I p ;W f )), Among them, I p Denotes each segment at position p on the reconstructed image I, f and W f They respectively represent the function of the segmentation neural network model and the corresponding weights learned for the function f, and φ represents the binarization function.

8. The device according to claim 7, It is characterized in that The weighted loss data is determined by applying the trainable scalar data to the reconstructed image data by the following calculation: in, is the loss function for each segment restoration process, N represents the number of segments in the degraded image data, and the loss weight Rescale the loss to its original size.

9. A method for training an image restoration device based on a set of image segments selected from one or more candidate images according to a segmentation neural network model, It is characterized in that The method comprises: implementing the segmentation neural network model to select the set of image segments for training the image restoration device from one or more candidate images; Optimizing an image processing neural network model of the image restoration device by performing restoration processing on a selected set of image segments; The segmentation neural network model generates trainable scalar data based on the reconstructed image output by the image processing neural network model; combining the trainable scalar data with the reconstructed image output by the image processing neural network model; Based on the combined scalar data and image output, the image processing neural network model is optimized.

10. The method according to claim 9, It is characterized in that The segmentation neural network model generates the trainable scalar data including: dividing the reconstructed image into a plurality of segments; Downsample each segment to a single pixel value; The plurality of single pixel values ​​are combined to form a scalar weight map.

11. The method according to claim 10, It is characterized in that The downsampling includes a plurality of convolutional blocks and a final binarization layer to provide a scalar value representing the trainability of each of the plurality of segments.

12. The method according to claim 11, It is characterized in that The binarization layer includes a binarization function.

13. The method according to claim 12, It is characterized in that The binarization function is: Here, φ(·) is a sigmoid function with temperature T, and x is any possible variable.

14. The method according to any one of claims 9 to 13, It is characterized in that The segmentation neural network model operates on the reconstructed image to generate the trainable scalar data according to the following equation: t p =φ(f(I p ;W f )) Among them, I p Denotes each segment at position p on the reconstructed image I, f and W f They respectively represent the function of the segmentation neural network model and the corresponding weights learned for the function f, and φ represents the binarization function.

15. The method according to claim 14, It is characterized in that The weighted loss data is determined by applying the trainable scalar data to the reconstructed image data by the following calculation: in, is the loss function for each segment restoration process, N represents the number of segments in the degraded image data, and the loss weight Rescale the loss to its original size.

16. An image restoration device for performing restoration processing on degraded image data based on a set of image segments, It is characterized in that The image restoration device includes an image processing neural network model trained by a training device, the training device being used to implement a segmentation neural network model to select the set of image segments from one or more candidate images; The device is used for training an image processing system according to the method according to any one of claims 9-15.

Images