Apparatus and method for image processing

Abstract

An apparatus including an image processor, the image processor being configured to implement: a first machine learning model for performing restoration processing on degraded image data; a second machine learning model for identifying image regions that need to be enhanced during the restoration processing; wherein the output of the second machine learning model is the input of the first machine learning model for optimizing the restoration processing.

Description

Technical Field

[0001] This invention relates to apparatus and methods for implementing and training machine learning models for image processing. Background Technology

[0002] Image processing can be used to alter images for a variety of purposes. One such purpose is to restore degraded images. Image restoration processing aims to improve the quality of degraded images. Image degradation has many common forms, including noise, blurring, limited contrast, or low resolution. Similarly, there are many types of image restoration processing designed to mitigate these forms of degradation.

[0003] Figure 1 An example of an image restoration process is shown. The upper left is a low-resolution image 102 of a tiger. Low-resolution image 102 is passed forward for the image restoration process. The image restoration process is based on a machine learning model (represented as f in the figure). θ This is achieved by improving the image resolution. Therefore, the image restoration produces a higher resolution image 106, as shown in the upper right corner. Thus, the higher resolution image 106 has higher quality than the input image 102.

[0004] Many modern image restoration methods are based on machine learning, where f θ It is a deep neural network consisting of an architecture and a set of parameters θ (also called weights). These parameters are found during the training process (also called learning), which performs mathematical optimization on the error calculated using the training data. In the case of image restoration, the training data typically consists of a set of image pairs, each pair including a degraded image and a ground truth restored image or an optimized image. Each degraded image is fed into the network, which then forwards the restored image. Figure 1 The forward pass 108 is illustrated as a series of arrows from left to right, starting with the low-resolution image 102, proceeding to the restoration process 104, and continuing to the high-resolution image 106. The restored image 106 is compared with the ground truth image, and the difference between the two is encoded as an error, also known as a loss. This loss is then backpropagated through the network in the backward pass used to update the parameters. Figure 1 In the diagram, backward propagation is shown as a dashed arrow 110 from right to left. The process of forward propagation followed by backward propagation is repeated multiple times on a set of images until convergence. The training method that updates all parameters in a neural network from input to output is called end-to-end. A trained deep network, including its architecture and learned parameters, is also called a model or machine learning model. This model can be used to recover new, unseen degraded images in what is called inference. For machine learning models, inference is used in a manner similar to that performed in computer programs. To improve computational efficiency and reduce complexity, sample patches or slices of images are often used for training.

[0005] Figure 1 It also includes a standard image restoration learning pipeline. The pipeline shown includes: building a database of training data 112, sampling training data from the database 114, building a network model 116 by training using the sampled training data, and subsequent optimization 118.

[0006] Existing training methods uniformly sample training data from a database. There are industry standards for sampling data based on the task being performed to train the neural network. Typically, a set size of sample regions from the image is used, covering a pixel-wise defined area of ​​the training image. This uniform sampling does not consider the data distribution and intrinsic features of the training images. It is well known that some images, even some pixels, are often more useful for training the network, thereby improving the performance of image restoration.

[0007] Existing image restoration methods use Figure 1 The end-to-end scheme shown generates high-quality images from low-quality images. Typically, each pixel is sampled with the same probability and contributes equally to the loss, for example, in image denoising [Zhang, Kai et al., "Beyond agaussian denoiser: Residual learning of deep CNN for image denoising," IEEE Transactions on Image Processing, 26.7(2017):3142-3155], image deblurring [Nah, Seungjun, Tae Hyun Kim, and KyoungMu Lee, "Deep multi-scale convolutional neural network for dynamic scene deblurring," 2017 IEEE Conference on Computer Vision and Pattern Recognition Proceedings], and super-resolution [Lim, Bee et al., "Enhanced deep residual networks for single image super-resolution"]. [Super-resolution], Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition] and other image restoration tasks [Learning deep CNN denoiser prior for image restoration], Zhang, Kai et al., Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition.

[0008] However, different parts of a real image possess different features, such as high-frequency and low-frequency patterns. These feature differences are crucial for certain tasks where degradation is localized (e.g., local motion blur). Training machine learning models equally on images with different features does not achieve peak performance. To adapt the network to challenging samples, some studies have reweighted the training data and demonstrated that deep models can achieve better performance by mining hard samples [Shrivastava, Abhinav, Abhinav Gupta, and Ross Girshick, “Training region-based object detectors with online hard example mining,” Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition]. Specifically, after training a deep neural network for demosaicing, Gharbi et al. manually selected hard samples to fine-tune the network [Gharbi, [Deep joint demosaicking and denoising, ACM Transactions on Graphics (TOG) 35.6 (2016): 1-12]. However, their data weighting method is accomplished through a laborious two-stage process. Furthermore, there is no guarantee that selecting hard samples will improve the general application of neural networks in image processing tasks.

[0009] Therefore, it is desirable to develop a sampling process for training a machine learning model for restoration tasks that takes into account the non-uniformity of images. Summary of the Invention

[0010] According to a first aspect, an apparatus including an image processor is provided, the image processor being configured to implement: a first machine learning model for performing restoration processing on degraded image data; and a second machine learning model for identifying image regions that need to be enhanced during the restoration processing; wherein the output of the second machine learning model is the input of the first machine learning model for optimizing the restoration processing.

[0011] The first machine learning model can be trained according to the following steps: receiving training data including the degraded image data and the corresponding optimal image data, and providing the degraded image data as initial input to the system; passing the degraded image data to the first machine learning model, which is used to create reconstructed image data by performing the restoration process on the degraded image data; determining loss data by comparing the reconstructed image data with the corresponding optimal image data; combining the loss data with a weight map to form weighted loss data; and updating the first machine learning model based on the weighted loss data. This allows the restoration process to consider specific regions in the image that require more attention.

[0012] The second machine learning model can be trained according to the following steps: the second machine learning model receives the weighted loss data; the second machine learning model determines the spatial distribution of the loss based on the weighted loss data; and the weight graph is updated to take into account the spatial distribution of the loss derived from the weighted loss data. This allows the weight graph to be optimized based on the weighted loss data output during the training of the first machine learning model.

[0013] The second machine learning model can be trained to: identify spatially distributed regions in a degraded image that are more prone to degradation based on one or more image features; and generate a weight map for performing restoration processing on the degraded image, so as to apply greater weights to the identified regions. This allows the second machine learning model to infer the appropriate weight map directly from the image.

[0014] According to a second aspect, a method for training an image processing system is provided, the image processing system including a first machine learning model, the method comprising training the first machine learning model by performing the following steps: receiving training data including degraded image data and corresponding optimal image data, and providing the degraded image data as input to the system; passing the degraded image data to the first machine learning model, the first machine learning model being used to create restored image data by restoring the degraded image data; determining loss data by comparing the restored image data with the corresponding optimal image data; combining the loss data with a weight map to form weighted loss data including the spatial distribution of the loss data; and updating the first machine learning model according to the weighted loss data.

[0015] The image processing system may include a second machine learning model, and the method may include training the second machine learning model by implementing an update process that performs the following steps: the second machine learning model receives the weighted loss data; the second machine learning model determines the spatial distribution of the loss data based on the weighted loss data; and the weight map is updated to take into account the spatial distribution of the loss derived from the weighted loss data. This allows the weight map to be optimized based on the weighted loss data output during the training of the first machine learning model. This can improve the operation of the second machine learning model inferring the weight map from image data in the future.

[0016] The update process can be repeated to iteratively update the weight map based on the weighted loss data generated from the previous weight map and the first machine learning model. This allows the second machine learning model to infer a more detailed and optimized weight map.

[0017] In at least some iterations of the method, the training data may differ from the training data received in the previous iteration of the method. The training data may be changed between iterations, allowing the machine learning model to be trained to perform restoration processing on different sets of image data.

[0018] The method may include modifying the first machine learning model by combining the first machine learning model with a second machine learning model to create a modified first machine learning model, such that the modified first machine learning model is trained to focus on regions in the degraded image that are more prone to degradation. This enables the creation of a machine learning model that combines the learning ability to focus restoration processing on regions within an image with the learning ability to perform restoration processing on the same image.

[0019] The method may include: receiving test data comprising degraded image data and corresponding optimal image data, and providing the degraded image data as input to the modified first machine learning model; creating reconstructed image data by performing restoration processing on the degraded image data; determining loss data by comparing the reconstructed image data with the corresponding optimal image data; and optimizing the second machine learning model based on the loss data. This can provide efficient restoration processing by further optimizing the ability of the second machine learning model to infer weight maps from degraded image data.

[0020] The method may include training a first machine learning model updated as described above, wherein the weight map is generated by the optimized second machine learning model previously trained according to the method described above. This can provide efficient restoration processing by further training the first machine learning model to perform restoration processing using the weight map inferred by the already optimized second machine learning model.

[0021] The method may include updating the optimized second machine learning model by implementing an update process that performs the following steps: the optimized second machine learning model receives weighted loss data; the optimized second machine learning model determines the spatial distribution of the loss data based on the weighted loss data; and the optimized second machine learning model is updated to generate a weight map that takes into account the spatial distribution of the loss derived from the weighted loss data. This can provide efficient recovery processing by further training the second machine learning model based on the outputs of the already updated first machine learning model and the already optimized second machine learning model.

[0022] The method may include modifying the modified first machine learning model by combining the updated first machine learning model with the updated optimized second machine learning model to create a second modified first machine learning model, such that the second modified first machine learning model is trained to focus on regions in the degraded image that are more prone to degradation. This can provide a further optimized first machine learning model for restoration processing.

[0023] The restoration process can be a combination of denoising and desacrifice, and the received degraded image data is RAW image data including the red, green, or blue values ​​of each sampled pixel, allowing a first machine learning model to be trained to infer the denoised and desacrifice RGB image from the received RAW image data. This enables efficient denoising and desacrifice processing.

[0024] According to a third aspect, an apparatus is provided for training an image processing system according to any one of claims 5 to 14. Attached Figure Description

[0025] The invention will now be described by way of example with reference to the accompanying drawings. In the drawings:

[0026] Figure 1 This demonstrates a standard learning pipeline for image restoration tasks;

[0027] Figure 2 The proposed method is shown in comparison with existing methods based on the generated weight graph;

[0028] Figure 3The three main training structures of the overall framework for training the first machine learning model using the generated weight graph are shown;

[0029] Figure 4A A detailed schematic diagram of the provided method is shown;

[0030] Figure 4B Four main computations are shown as steps in a complete iterative process of the provided training procedure;

[0031] Figure 5A The first machine learning model f is shown. θ An exemplary network architecture;

[0032] Figure 5B The second machine learning model g is shown. ω An exemplary network architecture;

[0033] Figure 6A and Figure 6B The results of applying the provided method to the image demosaic task and the joint denoising and demosaicing (JDD) task are shown respectively. Detailed Implementation

[0034] The proposed method aims to highlight key features of the training data, thereby improving model performance.

[0035] A solution is proposed to improve image restoration performance by better sampling of training data. Specifically, an end-to-end learning method is used, which considers each training image pixel with different weights. The different weights are implemented as a weight map. The weight map of each training image is learned by a gradient-based meta-task, also referred to here as the second machine learning model g. ω .

[0036] The provided method involves an image processing machine learning model (or a first machine learning model) that learns different weights for different image samples during training, based on parallel meta-learning steps using a second machine learning model. The weights are encoded based on each pixel and can therefore be used to form a weight map. The first machine learning model can then be further optimized based on its performance on another independent dataset.

[0037] The proposed method involves training a machine learning model based on the weights required for different pixels in the training image. Existing image restoration methods calculate the network image sample pair f according to the following formula. θ Loss function:

[0038]

[0039] here, The training set {T} L ,T H The loss of} This is the pixel loss criterion, typically the L1 or L2 norm. H and W are the height and width of the image sample. T L (h,w) and T H (h, w) represent the intensities of the low-quality L and high-quality H images at pixel (h, w), respectively. This method aims to obtain weights for each pixel. Therefore, the modified loss function... It will become:

[0040]

[0041] Where W(h,w) is the pixel weight.

[0042] The norm is a function that measures the difference between inputs. In this case, the difference between the measured ground-based image and the image restored by the method is used.

[0043] The L1 norm is the sum of the absolute differences between each matching color of each matching pixel in the ground truth image and the restored image. The L2 norm is the sum of the squared differences between each matching color of each matching pixel in the ground truth image and the restored image. In either case, if the image is perfectly restored, it will match the ground truth for every pixel, so either the L1 or L2 norm will be zero.

[0044] The norm can be used as an error signal and can be backpropagated through the network during training to adjust the network weights.

[0045] Figure 2 The provided method is illustrated using weight maps. The left side shows the training image 102. The right side shows two different weight maps based on different methods. In the upper weight map 202, it can be seen that each pixel has the same weight. That is, each pixel contributes equally to the loss. In the map, all pixels have the same level of color filter to show uniform sampling. In the lower weight map 204, it can be seen that each pixel has its own weight. That is, each pixel can contribute differently to the loss. This can be shown by applying different levels of color filter (shown in grayscale) to each individual pixel. Different pixels have a certain weight, which can correspond to the amount of detail or texture in the image at that pixel location. The representation of the tiger in the image can be seen in the weight map, showing its individual relevance and importance to the loss function.

[0046] Multiple sample squares 206 are shown on weight maps 202 and 204. During the training of a machine learning model for an image processing task, samples of training images (and possibly also test images as described herein) are known to minimize processing costs during training. This makes training more computationally efficient and faster end-to-end. Samples 206 can be obtained from the training and testing image data based on a standard sample size defined for the specific image processing task for which the machine learning model is trained.

[0047] Figure 3 An overall framework 300 for training a machine learning model using a weight graph 204 is shown, which includes three main training structures.

[0048] The first structure, 302, is the weight generator structure. The weight generator model g... ω It is a neural network trained to reweight image pixels, also referred to in this paper as the second machine learning model. ω Optimize in the outer loop 302 of the training framework. Learn the parameter ω during training.

[0049] The second structure 304 is the restoration network or the first machine learning model f. θ A restoration network is a neural network that reconstructs a high-quality image from a corresponding low-quality image. θ Training is performed on the image restoration task within the inner loop 304.

[0050] The third structure 306 is a gradient-based meta-learning scheme 306 that guides the processes of the outer loop 302 and the inner loop 304. In the third structure 306, the first machine learning model 104 and the second machine learning model are combined to modify the first machine learning model 308. The second machine learning model g is updated using the training dataset. ω To improve the performance of the first model in the next stage, which involves processing previously unseen retained data, also known as the meta-test dataset, the weight map 204 created based on the loss from the modified first machine learning model 308 is also optimized in the meta-learning scheme of the third structure 306 by backpropagation to the second machine learning model. Training data that is likely to achieve good performance of the first model on the test data can be assigned high weights.

[0051] exist Figure 3In the diagram, the three structures 302, 304, and 306 represent various update and optimization loops separated into different structural processes. However, during the implementation of training the first machine learning model, these are processes that occur simultaneously or sequentially. For example, the first and second structures of the training process can be performed concurrently, such that the first machine learning model is trained to perform restoration processing on the degraded image 102, while the second machine learning model is also trained to identify the spatial distribution of the weighted loss and use this spatial distribution of the weighted loss to infer the updated weight map.

[0052] These processes can be initialized with a uniform weight map, which can then be iteratively updated in one implementation of the training process by repeating the update process to generate a gradually updated weight map each time until a sufficient level of convergence is reached.

[0053] The next step could then be the processing loop shown in the third structure 306, where the first and second machine learning models are combined to provide a modified first machine learning model, which is additionally trained to focus on regions in the degraded image that are more prone to degradation. This focusing capability comes from the modified model, which now incorporates some training directly from the second machine learning model. The modified first model can then be tested on test data, and the loss derived from the modified first model can be used to further fine-tune the second machine learning model.

[0054] In an alternative implementation, an iterative process for updating the weight graph can be performed such that each iteration of the weight graph update is executed only after the corresponding iteration of the process in the third structure 306. That is, the processes of the first and second structures are executed once, and then the process of the third structure is executed before the processes of the first and second structures are executed again.

[0055] Between iterations of any of the above implementation options, the training data may or may not be changed. For example, Figure 3 The tiger image in the example can be used to create a weight map and update the first model in one iteration. Then, in the next iteration, the tiger can be used again, and this process can be repeated in subsequent iterations to further update the first model until the weight map of the tiger image is updated to convergence. Alternatively, a different image, such as an image of a lion, can be used in the next iteration. Any of these options will ultimately produce the same result: a second machine learning model, which is trained to identify spatially distributed regions in a degraded image that are more prone to degradation based on one or more image features; and to generate a weight map for performing restoration processing on the degraded image so that greater weights are applied to the identified regions.

[0056] Figure 4AA detailed schematic diagram of the provided method is shown. Figure 4B Four main computations are shown as steps in a complete iteration of the provided training process. To help understand how these computations are related to each other, Figure 4B Each calculation in the process uses the same numbering and Figure 4A As shown in the diagram. Figure 4A The diagram illustrates the elements of the image restoration training process, such as the machine learning model, f, and g. ω and f θ′ The training and testing datasets and the weight graph 204; while the steps of a single iteration in 5B show how specific elements affect each other.

[0057] The first step of the provided training process is to use a weight map 204, also known as an image restoration network, during the training of the first machine learning model. The first iteration may include a weight map 204 with a predefined weight distribution, such as a uniform weight distribution, or a distribution with a specific shape or pattern. However, in subsequent iterations, training may use a weight map 204 derived from the training data 102.

[0058] Figure 4A left side and Figure 4B Steps (1) and (2) illustrate one iteration of the core training process. The two machine learning models are updated in a nested manner. First, in the central loop, the training data T... L Feeded into g ω To generate a weight graph, which is the meta-training set {T} L ,T H Assign different weights. Secondly, in the leftmost loop, assign T... L Input to the restoration network f θ And calculate the corresponding loss 402

[0059] The weight map is used to train the image restoration network using the standard loss function given below 404.

[0060]

[0061] Unlike the normal training procedure, the loss Weighted by weighted graph 204, and then transformed into As in equation (3) and Figure 4A As shown, loss data 402 is combined with weight graph 204 to produce weighted loss data 406.

[0062] Based on the weighted loss 406, the new state of the restored network can be calculated according to the virtual update steps, such as... Figure 4B Step (3) and Figure 4AAs shown by the corresponding arrow, the following f is obtained. θ′ Update parameters

[0063]

[0064] Here, α is f θ The learning rate. Note that the updated parameter θ′ is g. ω The function, therefore, can be obtained through g. ω Update θ′.

[0065] Third, V L Input into the updated restoration network f θ′ Then train the meta-learner g ω Based on minimizing the meta-test set {V} with respect to the second-order gradient with respect to ω L V H The loss. This is in Figure 4A The rightmost loop and corresponding Figure 4B Arrow (4) is shown in step (4).

[0066] To optimize g ω This provides a meta-learning scheme, which is based on the meta-test dataset {V}. L V H Gradient training g ω Specifically, in g ω Under the guidance of the meta-training dataset, the restoration network f was trained. θ Driven to perform better on the meta-test dataset. That is, a second machine learning model can be trained using the output loss 408 from the test dataset {VL,VH} processed by the modified restoration network.

[0067] Finally, in updating g ω After that, a new iteration of the training process can begin, and the restored network can then be further updated and modified using the optimized weight graph.

[0068] The training process can also be summarized in the following example code:

[0069] need:

[0070] {T L ,T H}: Meta-training data, {V L V H}: Meta-test data.

[0071] g ω (T L ): Training set weight generator, f θ (T L ): Restore the network.

[0072] 1: Initialize g ω and f θ

[0073] 2: In each iteration of the for loop, do

[0074] 3: Sample a batch of metadata {T L ,T H} and {V L V H}

[0075] 4: Calculate data T L Weighted graph g ω (T L ).

[0076] 5: θ′=θ. >Inner loop, one iteration may be sufficient.

[0077] 6: For a sufficient number of times, do

[0078] 7: Calculate the new θ′ using the gradient: θ′

[0079] 8: Regarding the meta-gradient update of g using θ′ for ω ω :> outer loop.

[0080]

[0081] 9: Update f using the updated weight graph θ :

[0082] Although calculating the second gradient requires significant computation, it can be approximated efficiently using finite difference finite differences. Specifically, g ω The parameters are updated to

[0083]

[0084] Here, β is Figure 4A The rightmost loop of the learning rate, f θ′ It is an updated restoration network, such as Figure 4B The steps are shown in step (3). It is the meta-test set {V L V H Losses on}

[0085]

[0086] According to the chain rule, the gradient in the second term of Equation 4 can be rewritten as follows.

[0087]

[0088] Using the finite difference approximation, the right side of Equation 7 can be rewritten as follows:

[0089]

[0090] here

[0091]

[0092] Small scalars are empirically chosen as

[0093]

[0094] As an approximation, the gradient in Equation 4 can be calculated using two forward and two backward propagations. The computational complexity can be reduced from O(θω) to O(θ+ω).

[0095] The following is similar to Figure 3 The structure shown describes the series of mathematical steps in the training process again in a structure-by-structure format.

[0096] The first step can be considered as training the first machine learning model f using training data that includes degraded image data and the corresponding best image data. θ In this process, degraded image data is provided as input to a first machine learning model. The degraded image data already provided to the first machine learning model is restored according to a restoration process configured by the first machine learning model to create restored image data. The image processing system then determines loss data by comparing the restored image data with the corresponding best image data. The loss data can then be combined with a weight map to form weighted loss data that includes the spatial distribution of the loss data. The first backpropagation of the training process updates the first machine learning model based on the calculated weighted loss data. This process... Figure 4A The diagram shows a left loop, including structure 302 and a reverse pass arrow labeled (1), while... Figure 4B The diagram is shown as step (1). Therefore, the first machine learning model is updated to take into account the weighting of the weight graph in its restoration process, but the weight graph is not directly merged. In later iterations of this step, the weight graph can be an updated weight graph with various different convergence levels. Each time the first machine learning model is implemented with a different weight graph, the restoration process implementation of the first machine learning model itself is changed until it also reaches a convergence point. In any case, convergence can be defined as the state where the effect of repeated training iterations on the trained model is negligible. The point where the effect is negligible will depend on the implementation choice.

[0097] The second machine learning model g ω Training can be achieved by implementing an update process. The update process is... Figure 4AThe central loop is marked as (2) and Figure 4B Step (2) is indicated. This step in the training process involves the second machine learning model receiving weighted loss data. For example, this is the result of the first step (1) above. The second machine learning model can now consider the spatial distribution of the loss data based on the weighted loss data. Based on this information, the second machine learning model can identify spatially distributed regions in the degraded image that are more prone to degradation based on the features of the specific training image used. Therefore, the weight map can be updated to take into account the spatial distribution of the loss derived from the weighted loss data. In future iterations of the training of the first machine learning model, this updated weight map can be used to generate the next instance of weighted loss data. Therefore, a weight map can be generated for performing restoration processing on the degraded image so that larger weights are applied to the identified regions. The second machine learning model has been trained to identify image regions that need to be enhanced during the restoration process.

[0098] As described elsewhere in this document, the weight graph update process can be repeated to iteratively update the weight graph based on the weighted loss data generated from the previous weight graph and the first machine learning model. In another iteration, the first machine model can be an updated first machine learning model that has been updated to take into account the previously updated weight graph. It should be understood that in at least some iterations of the above method, the training data may differ from the training data received in a previous iteration of the method. For example, in Figure 4A In the example, the training data is shown as images of tigers, but in subsequent iterations of the training process (where the weight map or the first machine learning model is updated, or both), the training data may include different images (e.g., lions). The frequency of image changes can be chosen based on the specific image restoration process performed. Similarly, the frequency of training image changes may depend on whether it is important to obtain an updated weight map to reach a convergence point before further testing the first machine learning model on retained test data, or whether it is preferable to slowly converge the weight map on a large number of different training images. This choice may depend on the final implementation of the first machine learning model with respect to the specific image restoration process performed, or the expected degree of variation in the images being processed.

[0099] The next step of this method involves modifying the first machine learning model by combining it with the second machine learning model. This step is in... Figure 4A The arrow marked (3) is shown in the middle and corresponds to Figure 4BStep (3) in the process is as follows. This creates a modified first machine learning model that directly incorporates aspects of the second machine learning model trained so far. This allows the modified first machine learning model to focus on regions in the degraded image that are more prone to degradation without directly computing the weight map. In other words, the first machine learning model is modified based on the output of the second machine learning model to identify regions in the image that need to be focused on or enhanced. Therefore, the weight map generated by the second machine learning model becomes the benchmark for the modified first machine learning model, enabling it to infer the image portions that need to be focused on during image restoration processing to obtain optimal output.

[0100] The modified first machine learning model can then be tested on unseen test data. In a process similar to the initial training of the first machine learning model, test data, including degraded image data, is provided as input to the modified first machine learning model. The modified first machine learning model is then implemented to create reconstructed image data through restoration processing of the degraded image data. The loss data can then be determined by comparing the reconstructed image data with the corresponding best image data. However, when training the modified first machine learning model using the test data, the loss data is not combined with the weight map data. This is because, as mentioned above, the weight map is now essentially part of the modified first model. The loss data from the test data can instead be used for a second backpropagation to optimize the second machine learning model. This backpropagation step... Figure 4A The arrow marked (4) indicates that in Figure 4B The corresponding step (4) is shown in the middle.

[0101] The updated first machine learning model can be further trained by generating a weight map for further training of image data using the now optimized second machine learning model. That is, the updated first machine learning model can be trained according to the method of step (1) above, but the weight map is generated by the optimized second machine learning model previously trained according to step (4) of the above method.

[0102] Similarly, another round of the above training cycle may follow, in which, by implementing the above-mentioned... Figure 4B Step (2) describes the update process to update the optimized second machine learning model. The optimized second machine learning model is updated to generate a weight map to take into account the spatial distribution of the loss derived from the weighted loss data.

[0103] Ultimately, the aforementioned training methods and their various loops can be combined to iteratively modify the first machine learning model, then combined with an updated and optimized second machine learning model to create a second modified first machine learning model. This second modified machine learning model is trained to focus its image restoration processing on regions of the degraded image that are more prone to degradation.

[0104] In the specific implementation of the above training method, the restoration process can be a combination of denoising and desacrifice. In this specific case, the received degraded image data can be RAW image data including the red, green, or blue values ​​of each sampled pixel. Therefore, a first machine learning model can be trained to infer denoised and desacrificeable red, green, blue (RGB) images from the received RAW image data.

[0105] Figure 5A and Figure 5B The restoration network or the first machine learning model f are shown respectively. θ and weight generator or second machine learning model g ω An exemplary network architecture.

[0106] In one example implementation, the restored network f θ It can be a convolutional neural network. In this implementation, the residual network can include 16 residual blocks with convolutional layers and rectified linear unit (ReLU) activation layers.

[0107] Machine learning model g ω This can also be described as a convolutional neural network in an encoder-decoder architecture, with four downsampling layers and four upsampling layers. To ensure that the generated weight map is always non-negative, the ReLU function can be applied to the machine learning model g. ω The output.

[0108] As mentioned above, during the training process, the training dataset can be divided into two subsets: the meta-training set {T}. L ,T H} and meta-test set {V L V H}. Set {T L ,T H} and {V L V H The variables can be swapped between iterations; in some implementations, they can be swapped between each iteration.

[0109] The method presented above may have several advantages compared to previous methods. For example, the presented method can provide improved image processing performance during inference without requiring additional computation. This is because, compared to conventional methods, the presented method only requires additional computation during training.

[0110] The proposed method also improves robustness to imbalanced training data. In low-level vision tasks, it is difficult to balance training data on image features because these features are difficult to describe or quantify, and they are likely to be local. Models may overfit basic patterns in the dataset while ignoring hard or rare patterns. The proposed method reweights the training data, resulting in a more robust model.

[0111] The proposed method learns how to infer weight graphs in an end-to-end manner without using a separate or pre-training process during training. Instead, training is performed in a nested loop configuration, with the loops located in parallel portions of the training structure.

[0112] The method of this invention is widely applicable to many low-level vision problems that can be solved by image restoration processing, which includes joint denoising and demosaicing, super-resolution, and deblurring.

[0113] The provided image restoration processing method has been applied to several low-level vision tasks, including image depigmentation, denoising, super-resolution, and deblurring.

[0114] Figure 6A and Figure 6B The results of applying the proposed method to image desamicing and joint denoising and desamicing (JDD) tasks are shown. Compared to baseline methods, the proposed method achieves significant improvements in these examples, providing more detail and sharper edges. More importantly, the performance improvement does not increase model complexity or require additional inference computation. Furthermore, the proposed method can be easily integrated into other image restoration tasks.

[0115] Figure 6A The results of performing an image restoration task, specifically image de-mosaicing, are shown using a machine learning model trained as described above and compared with other existing methods. Figure 6A Image (a) shows a ground-based image. Figure 6A(b) shows the result of a simple demosaic operation that converts each red, green, green, blue (RGGB) quad-pair arrangement in the color filter array (CFA) into RGB pixels by averaging the green (G) channel. Figure 6A (c) in the figure shows the ResNet demosaic results. Figure 6A (d) shows the demosaicing results of ResNet with weighted graphs according to the provided method. Figure 6A As can be seen in (d), implementing the weighted graph method to train the machine learning model reduces distortion and restores more details in the output of the restoration process. Specifically, the blurring of details in the brickwork is reduced and falls into the surrounding bricks 602, and the edges of shadows and highlights around the relief structure are sharper and have greater depth 604.

[0116] Figure 6B The results of performing an image restoration task using a machine learning model trained as described above and compared with other existing methods are shown. This task specifically requires joint denoising and demosaicing (JDD). Figure 6B (a) shows a ground-based image. (b) shows the result of a simple demosaic operation, which converts each RGGB quad-alter arrangement in the CFA to RGB pixels by averaging the G channels. (c) shows the JDD result of ResNet. (d) shows the JDD result of ResNet with the provided weight map. The weight map reduces artifacts and produces sharper edges. Specifically, details have less noise or are less blurred606, and fine details are sharper and have greater clarity608.

[0117] The applicant hereby discloses each individual feature described herein, as well as any combination of two or more such features. With ordinary knowledge of those skilled in the art, such features or combinations can be implemented as a whole based on this specification, regardless of whether such features or combinations of features solve any problem disclosed herein; and without limiting the scope of the claims. This application demonstrates that aspects of the invention can be constituted by any such individual features or combinations of features. In view of the foregoing description, it will be apparent to those skilled in the art that various modifications can be made within the scope of this invention.

Claims

1. A device including an image processor, characterized in that, The image processor is used to achieve: The first machine learning model is used to perform restoration processing on the degraded image data (102); A second machine learning model is used to identify image regions that need to be enhanced during the restoration process; Wherein, the output of the second machine learning model is the input of the first machine learning model, and is used to optimize the restoration process; The second machine learning model is used to: receive weighted loss data (406); determine the spatial distribution of loss data (402) based on the weighted loss data; and update the weight graph (204) to take into account the spatial distribution of the loss data derived from the weighted loss data.

2. The device according to claim 1, characterized in that, The first machine learning model is used for: Receive training data including the degraded image data and the corresponding optimal image data; The restoration process is performed on the degraded image data to create reconstructed image data (106); The loss data is determined by comparing the reconstructed image data with the corresponding optimal image data; The loss data is combined with the weight map (204) (404) to form the weighted loss data; The first machine learning model is updated based on the weighted loss data.

3. The device according to claim 1 or 2, characterized in that, The second machine learning model is used for: Based on one or more image features, identify spatially distributed regions in a degraded image that are more prone to degradation; Generate a weight map for performing the restoration process on the degraded image, so as to apply greater weight to the identified regions.

4. A method for training an image processing system, characterized in that, The image processing system includes a first machine learning model, and the method includes: The system receives training data including degraded image data (102) and corresponding optimal image data, and provides the degraded image data as input. The degraded image data is passed to a first machine learning model, which is used to create restored image data by restoring the degraded image data (106); The loss data is determined by comparing the restored image data with the corresponding best image data (402); The loss data is combined with the weight map (204) to form weighted loss data (406) that includes the spatial distribution of the loss data; Update the first machine learning model based on the weighted loss data; The image processing system includes a second machine learning model, and the method includes: the second machine learning model receiving the weighted loss data; the second machine learning model determining the spatial distribution of the loss data based on the weighted loss data; and updating the weight map to take into account the spatial distribution of the loss data derived from the weighted loss data.

5. The method according to claim 4, characterized in that, The update process is repeated to iteratively update the weight map based on the weighted loss data generated from the previous weight map and the first machine learning model.

6. The method according to claim 5, characterized in that, In at least some iterations of the method, the training data is different from the training data received in the previous iteration of the method.

7. The method according to any one of claims 4 to 6, characterized in that, The method includes modifying the first machine learning model by combining the first machine learning model with the second machine learning model to create a modified first machine learning model, such that the modified first machine learning model is trained to focus on regions in the degraded image that are more prone to degradation.

8. The method according to claim 7, characterized in that, The method further includes: Receive test data including the degraded image data and the corresponding best image data, and provide the degraded image data as input to the modified first machine learning model; Reconstructed image data is created by performing restoration processing on the degraded image data; The loss data is determined by comparing the reconstructed image data with the corresponding optimal image data; The second machine learning model is optimized based on the loss data.

9. The method according to claim 8, characterized in that, The method includes training the updated first machine learning model according to claim 4, wherein the weight map is generated by the optimized second machine learning model previously trained by the method according to any one of claims 5 to 7.

10. The method according to claim 9, characterized in that, The method includes: The optimized second machine learning model receives the weighted loss data; The optimized second machine learning model determines the spatial distribution of the loss data based on the weighted loss data; The optimized second machine learning model is updated to generate the weight map, taking into account the spatial distribution of the loss data derived from the weighted loss data.

11. The method according to claim 10, characterized in that, The method includes modifying the modified first machine learning model by combining the updated first machine learning model with the updated optimized second machine learning model to create a second modified first machine learning model, such that the second modified first machine learning model is trained to focus on regions in the degraded image that are more prone to degradation.

12. The method according to any one of claims 4 to 11, characterized in that, The restoration process is a combined denoising and de-mosaic process. The received degraded image data is RAW image data including the red, green, or blue values ​​of each sampled pixel, such that the first machine learning model is trained to infer a denoised and de-mosaiced RGB image from the received RAW image data.

13. A training image processing system, characterized in that, A method for performing the training image processing system according to any one of claims 4 to 12.

Images