Method for training a machine learning model and related products

Abstract

The disclosure discloses a training method of a machine learning model and corresponding computing device, computer readable storage medium and computer program product. A new loss function is introduced in the training method, so that the blurred pixels in image restoration can be processed more effectively. The scheme of the embodiment of the disclosure can improve the accuracy of the machine learning model related to image restoration.

Description

Technical Field

[0001] This disclosure generally relates to the field of data processing, and more particularly to the field of image processing. More specifically, this disclosure relates to a method for training a machine learning model, a computing device, a computer-readable storage medium, and a computer program product. Background Technology

[0002] Image restoration refers to methods for recovering high-quality images from degraded images, including super-resolution, denoising, and deblurring. In recent years, methods based on neural networks (NNs) have achieved great success in image restoration tasks.

[0003] Image restoration methods benefit not only from the architecture of neural networks but also from loss functions. Besides developing carefully designed network architectures, the loss function, as the optimization objective, has also attracted considerable attention in neural network-based image restoration methods.

[0004] Neural network-based image restoration methods typically use a degraded image as input and train a model by minimizing the error between the output image and the corresponding reference image, as defined by a loss function. Common loss functions can be categorized into two types: pixel-level loss and perceptually driven loss. Pixel-level loss minimizes the absolute distance between the output pixel value and the reference pixel value, such as mean squared error (MSE or L2) and mean absolute error (MAE or L1). Perceptually driven loss compares the differences between the output image and the reference image in the feature space, such as perceptual loss and adversarial loss. To generate high-quality, realistic output images with high accuracy, perceptually driven loss is often used in conjunction with pixel-level loss.

[0005] The optimization objective of pixel-level loss is to maximize the peak signal-to-noise ratio (PSNR). However, pixel-level loss yields somewhat blurry results and cannot recover fine structures. Therefore, this presents new requirements and challenges for the design of loss functions used in image restoration tasks. Summary of the Invention

[0006] To address at least one or more of the technical problems mentioned above, this disclosure proposes a novel loss function for training a machine learning model to perform an image restoration task. This loss function preserves structural information in the image, enabling the trained machine learning model to retain fine image structure while restoring the image.

[0007] In a first aspect, this disclosure discloses a method for training a machine learning model for image restoration processing, the method comprising: in forward training, calculating the value of a loss function based on a reference image and an output image processed by the machine learning model, wherein the loss function preserves structural information in the reference image; and in backward training, updating the parameters of the machine learning model based on the loss function.

[0008] In a second aspect, this disclosure discloses a computing apparatus for training a machine learning model, comprising: a processor configured to execute program instructions; and a memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform a training method according to a first aspect of this disclosure.

[0009] In a third aspect, this disclosure discloses a computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the training method according to the first aspect of this disclosure.

[0010] In a fourth aspect, this disclosure discloses a computer program product, including a computer program or instructions that, when executed by a processor, implement the training method of the first aspect of this disclosure.

[0011] By utilizing the training method of the machine learning model disclosed above, the trained machine learning model can accurately restore the image while preserving the fine structure of the image and improving the image quality. Attached Figure Description

[0012] The above and other objects, features, and advantages of exemplary embodiments of this disclosure will become readily apparent upon reading the following detailed description with reference to the accompanying drawings. In the drawings, several embodiments of this disclosure are illustrated by way of example and not limitation, and like or corresponding reference numerals denote like or corresponding parts, wherein:

[0013] Figure 1 An example of the output of a neural network model trained using an existing loss function is shown.

[0014] Figure 2 An exemplary flowchart of a training method for a machine learning model according to an embodiment of this disclosure is shown;

[0015] Figure 3 An exemplary flowchart illustrating the calculation of a loss function according to some embodiments of this disclosure is shown;

[0016] Figure 4 An exemplary illustration of calculating the loss function during the training process of this disclosure embodiment is shown;

[0017] Figure 5 A block diagram illustrating the hardware configuration of a computing device that can implement the embodiments of this disclosure;

[0018] Figure 6 A structural diagram of a combined processing apparatus according to an embodiment of the present disclosure is shown; and

[0019] Figure 7 A schematic diagram of the structure of a board card according to an embodiment of this disclosure is shown. Detailed Implementation

[0020] The technical solutions in the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this disclosure, not all of them. Based on the embodiments in this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the solutions disclosed in this disclosure.

[0021] It should be understood that the terms "first," "second," "third," and "fourth," etc., that may appear in the claims, specification, and drawings of this disclosure are used to distinguish different objects, rather than to describe a specific order. The terms "comprising" and "including" as used in the specification and claims of this disclosure indicate the presence of the described features, integrals, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components, and / or collections thereof.

[0022] It should also be understood that the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this disclosure. As used in this disclosure and claims, the singular forms “a,” “an,” and “the” are intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the term “and / or” as used in this disclosure and claims refers to any combination and all possible combinations of one or more of the associated listed items, and includes such combinations.

[0023] As used in this specification and claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0024] The specific embodiments disclosed herein will now be described in detail with reference to the accompanying drawings.

[0025] To address the problem that existing loss functions often fail to recover structural information from images, this disclosure provides a novel loss function. It is understood that this loss function can be used not only to train neural network models but also to train other machine learning models for image restoration tasks. Therefore, although neural network models are frequently used as examples in the following descriptions, those skilled in the art will understand that these descriptions are also applicable to other machine learning models.

[0026] The following section uses a neural network model as an example to illustrate the training process of a machine learning model.

[0027] In practical applications, an initial neural network is trained with a large amount of sample data (including input and output). After training, a trained neural network is obtained. This neural network can provide a correct output for inputs from real-world environments in the future.

[0028] Before training a neural network, a loss function needs to be defined. The loss function is a measure of the neural network's performance on a specific task. In some contexts, the loss function can be derived as follows: during the training of a neural network, for each sample data point, the output value is obtained by passing it along the network. The difference between this output value and the expected value is then squared. The resulting loss function is the distance between the predicted and the true values, and the goal of training the neural network is to reduce this distance, or the value of the loss function. In these contexts, the loss function can be expressed as:

[0029]

[0030] In the above formula, y represents the expected value. This refers to the actual result obtained by processing each sample data in the sample dataset through a neural network, where i is the index of each sample data in the sample dataset. This represents the expected value y and the actual result. The error value between them. m is the number of sample data in the sample dataset.

[0031] To calculate the loss function, it is necessary to iterate through every sample image in the sample dataset to obtain the actual result for each sample image. Then calculate the loss function according to the definition above. If the loss function is large, for example, exceeding a predetermined threshold, it means that the neural network has not been trained well, and the weights need to be further adjusted.

[0032] When starting to train a neural network, the weights need to be randomly initialized. In most cases, the initialized neural network will not provide good training results. During training, assuming you start with a very poor neural network, you can train it to obtain a network with high accuracy.

[0033] The training process of a neural network is divided into two stages. The first stage is the forward signal processing operation (referred to as forward training or forward propagation in this disclosure), where training proceeds from the input layer through the hidden layers and finally to the output layer. The second stage is the backpropagation gradient operation (referred to as backpropagation training or backward propagation in this disclosure), where training proceeds from the output layer to the hidden layers and finally to the input layer, adjusting the weights and biases of each layer in the neural network according to the gradient.

[0034] During the forward propagation process, the input value is fed into the input layer of the neural network. After the corresponding operations performed by relevant operators in multiple hidden layers, the so-called predicted value is obtained from the output layer of the neural network. When the input value is provided to the input layer of the neural network, it may not perform any operations or may perform some necessary preprocessing depending on the application scenario. In the hidden layers, the second hidden layer obtains the intermediate predicted value from the first hidden layer, performs calculation and activation operations, and then passes the obtained intermediate predicted value to the next hidden layer. The same operation is performed in subsequent layers, and finally the output value is obtained at the output layer of the neural network. After the forward processing of the forward propagation process, an output value called the predicted value is usually obtained. To calculate the error, the predicted value can be compared with the actual output value to obtain the corresponding error value.

[0035] During backpropagation, the chain rule of differential calculus can be used to update the weights of each layer, aiming to obtain a lower error value in the next forward propagation compared to the previous one. In the chain rule, the derivative of the error value of the weights of the last layer of the neural network is first calculated. These derivatives are called gradients, and then these gradients are used to calculate the gradient of the penultimate layer in the neural network. This process is repeated until the gradient corresponding to each weight in the neural network is obtained. Finally, the corresponding gradient is subtracted from each weight in the neural network, thus updating the weights to reduce the error value. Similar to the various operators used in forward propagation (referred to as forward operators in this disclosure), there are also corresponding backward operators in the backpropagation process. For example, for convolution operators in convolutional layers, there are forward convolution operators in the forward propagation process and deconvolution operators in the backpropagation process.

[0036] During the training of a neural network, each forward propagation of the signal and each corresponding backward propagation of the error results in an update of the network's weights using the gradient; this is called an iteration. To obtain a neural network with the expected accuracy, a large sample dataset is needed during training, but inputting the entire dataset into a computing device (such as a computer) at once is nearly impossible. Therefore, to solve this problem, the sample dataset needs to be divided into multiple batches and passed to the computer in batches. Each batch of the dataset undergoes forward propagation and then a corresponding backward propagation to update the neural network's weights. When a complete sample dataset has passed through the neural network once and returned a corresponding weight update, this process is called an epoch. In practice, passing the complete dataset once to the neural network is insufficient; the complete dataset needs to be passed through the same neural network multiple times, i.e., multiple epochs, to ultimately obtain a neural network with the expected accuracy.

[0037] Figure 1 An example of the output of a neural network model trained using an existing loss function is shown. In this example, the loss function used is the known pixel-level loss function L1. Other pixel-level loss functions will produce similar results.

[0038] The figure shows a baseline image (a), a blurred image (b) output by a trained neural network model, and a sharp image (c). It is clear from the figure that the sharp image (c) has better visual quality and preserves image structure better than the blurred image (b). The figure also shows that the L1 loss function value for the blurred image (b) is 0.0579, and the L1 loss function value for the sharp image (c) is 0.0578. However, the L1 loss values ​​show that the L1 losses for the blurred image (b) and the sharp image (c) are very close, almost indistinguishable. That is, existing pixel-level loss functions cannot distinguish whether edges in an image are blurred or sharp.

[0039] To further analyze the differences between the blurred image b and the sharp image c, Figure 1 The diagram also shows the pixel values ​​of the reference image a, the blurred image b, and the sharp image c, especially the pixel values ​​of the edge regions. Curve a1 indicates the pixel values ​​of the edge regions of the reference image a, curve b1 indicates the pixel values ​​of the edge regions of the blurred image b, and curve c1 indicates the pixel values ​​of the edge regions of the sharp image c.

[0040] To analyze the difference between the blurred image b and the sharp image c, we can calculate the pixel difference between the blurred image b and the reference image a, called the blur error; similarly, we can calculate the pixel difference between the sharp image c and the reference image a, called the sharpness error. Calculations show that the blur error and sharpness error have the same numerical value but different directions. However, the L1 loss functions of the blurred image b and the sharp image c are very similar.

[0041] Based on the above analysis, this disclosure provides a loss function that preserves sharp edges and fine structures in an image, thereby enabling a machine learning model trained with this loss function to output higher quality images.

[0042] Figure 2 An exemplary flowchart of a training method for a machine learning model according to an embodiment of this disclosure is shown. The machine learning model using this training method is used to process an image restoration task. In this embodiment of the disclosure, the input to the machine learning model for the image restoration task may be an image obtained by degrading a reference image (such as a bicubic interpolation method), which is provided to the machine learning model as the input image to be restored, while the ideal or desired output is the reference image.

[0043] As shown in the figure, in step 210 of method 200, during forward training, the value of the loss function is calculated based on the benchmark image and the output image processed by the machine learning model.

[0044] Image blurring is a significant problem in image restoration tasks, as it destroys the image's structure. As mentioned in the background section, existing loss functions cannot preserve image structure, resulting in accurate but blurry images. Therefore, in this disclosed embodiment, to improve image quality, the provided loss function preserves the structural information in the reference image.

[0045] Image blurring tends to reduce the difference between the current pixel and its surrounding pixels, especially at edges in the image. The inventors noted that existing loss functions cannot distinguish between blurred and sharp edges in an image. Therefore, in some embodiments of this disclosure, the structural information preserved by the loss function can be edge information in the image, thereby allowing for specific optimization targeting edges.

[0046] In this paper, an edge refers to the junction between two attribute regions in an image. It is where the attribute of a region changes abruptly, representing the area with the greatest uncertainty and the most concentrated image information. For example, in an image containing green grass and a soccer ball on the grass, the image region of the soccer ball and the image region of the green grass change abruptly in color, and the circular outline of the soccer ball is an edge in the image. Furthermore, edges also exist in the black and white pattern of the soccer ball; the junction between black and white blocks changes color and is considered an edge.

[0047] The edges of an image contain a wealth of information. Edge information mainly includes the direction and magnitude of the edge. Generally, the magnitude changes more gently along the direction of the edge, while the magnitude changes more sharply perpendicular to the direction of the edge. For example, in the image of the green grass and the soccer ball mentioned above, the magnitude changes more gently along the circular outline of the soccer ball, while the magnitude changes more sharply perpendicular to the circular outline.

[0048] Next, in step 220, during reverse training, the parameters of the machine learning model are updated based on the aforementioned loss function.

[0049] Through repeated adjustments, a trained machine learning model can be obtained, whose output image can better reflect the structural features of the image, especially the edge features in the image.

[0050] Figure 3 An exemplary flowchart illustrating the calculation of a loss function according to some embodiments of this disclosure is shown.

[0051] As shown in the figure, in step 310, blurry pixels at the edges in the output image are first detected.

[0052] This disclosure aims to improve image blurring in image restoration. Therefore, blurred pixels at edges in the output image can be detected first, allowing for targeted adjustments. As previously mentioned, the loss function provided in this disclosure preserves structural information, particularly edge information in the image. Therefore, this edge information can be used to help detect blurred pixels at edges in the output image.

[0053] The inventors discovered through observation of blurred and sharp pixels that whether a pixel in the output image is blurred is related to its relative position to a corresponding pixel in a reference image and the direction of change of adjacent pixels in the reference image. The inventors also observed that when a pixel in the output image is blurred, the direction of the error between that pixel and its corresponding pixel in the reference image (referred to here as the first direction) is consistent with the direction of change of adjacent pixels at the corresponding pixel position in the reference image (referred to here as the second direction). Therefore, this directional determination can be used to detect blurred pixels at edges in the output image.

[0054] Specifically, in some embodiments, step 310 of detecting blurred pixels may include the following sub-steps: determining a first direction of pixel error between the output image and the reference image (sub-step 311), determining a second direction of change of adjacent pixels at the corresponding pixel position in the reference image (sub-step 312), and identifying pixels whose first direction and second direction are consistent as blurred pixels (sub-step 313).

[0055] Various methods can be used to determine changes in adjacent pixels in an image. In some embodiments, for example, the second-order gradient of pixel grayscale values ​​can be used to reflect the direction and intensity of changes in adjacent pixels. Specifically, the sign of the second-order gradient value between a pixel location in the reference image and its adjacent pixels represents the direction of change, i.e., the second direction; the magnitude of the second-order gradient value represents the intensity of the change. In some implementations, the Laplacian operator can be used to calculate the second-order gradient value. Of course, other methods can also be used to calculate the second-order gradient value, and the embodiments disclosed herein are not limited to these.

[0056] Therefore, by comparing whether the first and second directions are consistent, it is possible to determine which pixels in the output image are blurred pixels. The above detection method can be specifically expressed as the following formula:

[0057]

[0058]

[0059] Where m i,j The fuzzy factor map is a matrix of the same size as the output image, where a value of 0 indicates that pixel (i, j) is a non-fuzzy pixel, i.e., a normal pixel, and a value of non-0 indicates that pixel (i, j) is a fuzzy pixel. This represents the pixel value at pixel (i, j) in the output image; Represents the pixel value at pixel (i, j) in the reference image; e i,j Represents the pixel error at pixel (i, j); and This represents the second-order gradient value at pixel (i, j) on the reference image.

[0060] continue Figure 3 Next, in step 320, based on the detected blurred pixels, a weighting factor is determined to weight the pixel error between the output image and the reference image for calculating the loss function.

[0061] In this disclosed embodiment, for detected blurry pixels, the weighting factor of the pixel error corresponding to the blurry pixel can be increased, thereby increasing the penalty for the blurry pixel, so as to train the machine learning model to make targeted adjustments to the blurry pixel.

[0062] In some examples, the weighting factor can be expressed as the following formula:

[0063] ω i,j =1+αm i,j (4)

[0064] Where ω i,j Let α represent the weighting factor at pixel (i, j), and m represent the penalty factor. i,j This is the previously determined fuzzy factor diagram. Because m i,j The corresponding value for non-fuzzy pixels is 0, so the penalty for non-fuzzy pixels will not be increased, but the penalty will only be increased for fuzzy pixels.

[0065] Optionally or additionally, the weighting factor can be different for blurred pixels at different locations. In one implementation, the corresponding weighting factor can be determined based on the magnitude of the change in adjacent pixels at the corresponding pixel location in the reference image.

[0066] In one implementation, the fuzzy factor graph can be represented as follows:

[0067]

[0068] As can be seen from formula (5), for the fuzzy factor, its position in the fuzzy factor graph m i,j The value in the reference image is the magnitude of the change between adjacent pixels at the corresponding pixel position, such as the absolute value of the second-order gradient.

[0069] In some embodiments, the penalty factor α is a positive real number, thereby increasing the weighting factor. The penalty factor is adjustable, and an appropriate value can be selected, for example, according to different machine learning models, application scenarios, etc. It can be understood that the larger the penalty factor, the heavier the penalty on blurred pixels, and the sharper the edges in the resulting output image. Preferably, in some embodiments, the penalty factor α ranges from approximately 1.0 to 5.0; more preferably, the penalty factor α is approximately 2.

[0070] After determining the weighting factors for each pixel, the loss function can be calculated.

[0071] In one example, the loss function can be calculated using the following formula:

[0072]

[0073] Among them, L SP The loss function is represented by W and H, which represent the width and height in pixels of the reference image and the output image, respectively.

[0074] It is understandable that other forms of loss functions can be designed, in which a weighting factor ω is introduced. i,j This allows for special processing of blurred pixels, thus optimizing image blurring issues.

[0075] Figure 4 An exemplary illustration is shown of the calculation of the loss function during the training process of this disclosure embodiment.

[0076] As shown in the figure, the output image 410 includes a sharp region 411 and a blurred region 412. The pixel value curves corresponding to the sharp region 411 and the blurred region 412 are shown below the output image 410. The reference image 420 shows regions 421 and 422 corresponding to the sharp region 411 and the blurred region 412 in the output image, respectively, and their respective pixel value curves.

[0077] When calculating the loss function, blurry pixels in the output image are first detected in stage 1.

[0078] The figure shows pixel error curve 430, and more specifically, pixel error curve 431 between the sharp region 411 in the output image 410 and the corresponding region 431 in the reference image 430, and pixel error curve 432 between the blurred region 412 in the output image 410 and the corresponding region 422 in the reference image 420. As can be seen from the figure, the pixel error 431 in the sharp region is initially negative and then positive, meaning its direction is initially small and then large; while the pixel error 432 in the blurred region is initially positive and then negative, meaning its direction is initially large and then small.

[0079] The figure also shows the second-order gradient map 440 of the reference image, more specifically showing the second-order gradient value curve 441 corresponding to the sharp region 411 and the second-order gradient value curve 442 corresponding to the blurred region 412. In the example in the figure, the second-order gradient is calculated using the Laplacian operator, which can, for example, process the grayscale value of a pixel. The Laplacian operator can use the property of zero crossing to locate edges, and can also determine whether a pixel is on the dark side or the bright side of the edge. The second-order gradient value is positive on the bright side and negative on the dark side. As can be seen from the figure, the second-order gradient value 441 of the sharp region is positive first and then negative, opposite to the direction of the pixel error 431; while the second-order gradient value 442 of the blurred region is also positive first and then negative, consistent with the direction of the pixel error 432.

[0080] By comparing the pixel error curve with the second-order gradient value curve, it is possible to determine which pixels in the output image are blurred pixels. Specifically, the figure shows that a NOR operation can be performed on the pixel error and the second-order gradient to identify pixels whose directions are consistent as blurred pixels and those whose directions are inconsistent as unblurred pixels. These detection results can form a blur factor graph 450.

[0081] As described in the example above, in the blur factor graph 450, blurry pixels can be assigned the absolute value of the corresponding second-order gradient, while non-blurry pixels can be assigned 0. This distinguishes the weighting of blurry and non-blurry pixels, which means increasing the penalty in the loss function later.

[0082] Next, in the loss function calculation of stage 2, blurred pixels are penalized.

[0083] As shown in the figure, when calculating the loss function, the weighting factors can be adjusted based on the fuzziness factor map determined in Stage 1, thereby differentiating fuzzy pixels. More specifically, compared to non-fuzzy pixels, the weighting factor corresponding to fuzzy pixels can be increased, thus penalizing fuzzy pixels more severely.

[0084] The figure shows the weighting operation performed on the pixel error 430 after determining the weighting factors of each pixel based on the fuzzy factor map. For example, a matrix multiplication operation such as the Hadamard product is performed to obtain the loss function 460.

[0085] The training method for a machine learning model according to embodiments of this disclosure has been described above with reference to the accompanying drawings. This disclosure also provides a computing apparatus that can be used to train machine learning processing models, particularly neural network models.

[0086] Figure 5 A block diagram illustrating the hardware configuration of a computing device 500 capable of implementing the training scheme of embodiments of the present disclosure is shown. As shown, the computing device 500 may include a processor 510 and a memory 520. Figure 5 In the computing device 500, only the components relevant to this embodiment are shown. Therefore, it will be apparent to those skilled in the art that the computing device 500 may also include components related to... Figure 5 The following are common components with different constituent elements, such as: display.

[0087] The computing device 500 can correspond to a computing device with various processing functions, such as functions for generating neural network models, training or learning neural network models, or generating training image sets. For example, the computing device 500 can be implemented as various types of devices, such as personal computers (PCs), server devices, mobile devices, etc.

[0088] Processor 510 is configured to execute program instructions to control all functions of computing device 500. For example, processor 510 controls all functions of computing device 500 by executing programs stored in memory 520 on computing device 500. Processor 510 may be implemented by a central processing unit (CPU), graphics processing unit (GPU), application processor (AP), artificial intelligence processor chip (IPU), etc., provided in computing device 500. However, this disclosure is not limited thereto.

[0089] In some embodiments, processor 510 may include an input / output (I / O) unit 511 and a computing unit 512. I / O unit 511 may be used to receive various types of data, such as a training image set. Computing unit 512 may be used to train a machine learning model based on the training image set. The trained machine learning model may be output by I / O unit 511, for example. The trained machine learning model may be provided to memory 520 for use by other devices (not shown), or it may be directly provided to other devices.

[0090] Memory 520 is hardware used to store various data processed in computing device 500. For example, memory 520 can store processed data and data to be processed in computing device 500. Memory 520 can store data that processor 510 has processed or is about to process, such as training images, or datasets involved in neural network operations, such as training image sets, data of an untrained initial neural network, intermediate data of a neural network generated during training, and data of a neural network that has completed all training. Furthermore, memory 520 can store program instructions such as applications and drivers to be driven by computing device 500. For example, memory 520 can store various programs related to training algorithms for machine learning models to be executed by processor 510. Memory 520 can be DRAM, but this disclosure is not limited thereto. Memory 520 can include at least one of volatile memory and non-volatile memory. Non-volatile memory may include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, phase-change RAM (PRAM), magnetic RAM (MRAM), resistive RAM (RRAM), ferroelectric RAM (FRAM), etc. Volatile memory may include dynamic RAM (DRAM), static RAM (SRAM), synchronous DRAM (SDRAM), PRAM, MRAM, RRAM, ferroelectric RAM (FeRAM), etc. In an embodiment, memory 820 may include at least one of hard disk drive (HDD), solid-state drive (SSD), high-density flash memory (CF), secure digital card (SD), micro-secure digital card (Micro-SD), mini-secure digital card (Mini-SD), extreme digital card (xD), cache, or memory stick.

[0091] In summary, the specific functions implemented by the memory 520 and processor 510 of the computing device 500 provided in the embodiments of this specification can be explained in comparison with the foregoing embodiments in this specification, and can achieve the technical effects of the foregoing embodiments. Therefore, they will not be repeated here.

[0092] In this disclosure embodiment, a computer-readable storage medium is also provided, wherein program instructions are stored, which, when loaded and executed by a processor, cause the processor to perform the training method of the machine learning model described in this disclosure embodiment.

[0093] In this disclosure embodiment, a computer program product is also provided, including a computer program or instructions that, when executed by a processor, implement a training method for a machine learning model according to this disclosure embodiment.

[0094] Figure 6This is a structural diagram illustrating a combined processing apparatus 600 according to an embodiment of the present disclosure. As shown, the combined processing apparatus 600 includes a computing processing unit 602, an interface unit 604, other processing units 606, and a storage unit 608. Depending on different application scenarios, the computing processing unit may include one or more computing units 610, which can be configured to... Figure 5 The computing device 500 shown is used to perform the operations described herein in conjunction with the accompanying drawings.

[0095] In different embodiments, the computing processing apparatus disclosed herein can be configured to perform user-specified operations. In exemplary applications, the computing processing apparatus can be implemented as a single-core artificial intelligence processor or a multi-core artificial intelligence processor. Similarly, one or more computing devices included within the computing processing apparatus can be implemented as an artificial intelligence processor core or a portion of the hardware structure of an artificial intelligence processor core. When multiple computing devices are implemented as artificial intelligence processor cores or portions of the hardware structure of artificial intelligence processor cores, the computing processing apparatus disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure.

[0096] In exemplary operation, the computing processing device disclosed herein can interact with other processing devices through an interface device to jointly complete user-specified operations. Depending on the implementation, the other processing devices disclosed herein may include one or more types of processors such as a Central Processing Unit (CPU), a Graphics Processing Unit (GPU), and an artificial intelligence processor, both general-purpose and / or special-purpose processors. These processors may include, but are not limited to, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., and their number can be determined according to actual needs. As mentioned above, the computing processing device disclosed herein can be considered to have a single-core structure or a homogeneous multi-core structure. However, when the computing processing device and other processing devices are considered together, they can be considered to form a heterogeneous multi-core structure.

[0097] In one or more embodiments, the other processing device may serve as an interface between the computing processing device disclosed herein (which may be specifically embodied in artificial intelligence, such as neural network operations) and external data and control, performing basic controls including but not limited to data transfer, starting and / or stopping the computing device. In another embodiment, the other processing device may also cooperate with the computing processing device to jointly complete computational tasks.

[0098] In one or more embodiments, the interface device can be used to transfer data and control commands between a computing processing device and other processing devices. For example, the computing processing device can obtain input data from other processing devices via the interface device and write it to on-chip storage (or memory) of the computing processing device. Further, the computing processing device can obtain control commands from other processing devices via the interface device and write them to on-chip control cache of the computing processing device. Alternatively or optionally, the interface device can also read data from the storage device of the computing processing device and transmit it to other processing devices.

[0099] Additionally or optionally, the combined processing apparatus disclosed herein may further include a storage device. As shown in the figures, the storage device is connected to both the computing processing device and the other processing device. In one or more embodiments, the storage device may be used to store data from the computing processing device and / or the other processing device. For example, the data may be data that cannot be fully stored in the internal or on-chip storage of the computing processing device or other processing device.

[0100] In some embodiments, this disclosure also discloses a chip (e.g. Figure 7 The chip shown is 702. In one implementation, the chip is a system-on-chip (SoC) and integrates one or more such... Figure 6 The combined processing unit shown is illustrated. This chip can be connected to external interface devices (such as...). Figure 7 The external interface device 706 shown is connected to other related components. These related components may be, for example, a camera, monitor, mouse, keyboard, network card, or Wi-Fi interface. In some applications, the chip may integrate other processing units (e.g., video codecs) and / or interface modules (e.g., DRAM interfaces). In some embodiments, this disclosure also discloses a chip package structure that includes the aforementioned chip. In some embodiments, this disclosure also discloses a board that includes the aforementioned chip package structure. The following will be combined with… Figure 7 This board is described in detail.

[0101] Figure 7This is a schematic diagram illustrating the structure of a board 700 according to an embodiment of this disclosure. As shown, the board includes a storage device 704 for storing data, which includes one or more storage cells 710. This storage device can be connected and transmit data with a controller 708 and the aforementioned chip 702 via, for example, a bus. Furthermore, the board also includes an external interface device 706 configured for data relay or switching between the chip (or a chip in a chip package structure) and an external device 712 (e.g., a server or computer). For example, data to be processed can be transmitted from the external device to the chip via the external interface device. Alternatively, the calculation results of the chip can be transmitted back to the external device via the external interface device. Depending on the application scenario, the external interface device can have different interface forms, such as a standard PCIe interface.

[0102] In one or more embodiments, the controller in the disclosed board can be configured to regulate the state of the chip. Therefore, in one application scenario, the controller may include a microcontroller (MCU) for regulating the operating state of the chip.

[0103] Based on the above combination Figure 6 and Figure 7 Based on the description, those skilled in the art will understand that this disclosure also discloses an electronic device or apparatus that may include one or more of the aforementioned boards, one or more of the aforementioned chips, and / or one or more of the aforementioned combined processing apparatus.

[0104] Depending on the application scenario, the electronic devices or apparatus disclosed herein may include servers, cloud servers, server clusters, data processing devices, robots, computers, printers, scanners, tablets, smart terminals, PC devices, IoT terminals, mobile terminals, mobile phones, dashcams, navigators, sensors, cameras, video cameras, projectors, watches, headphones, mobile storage, wearable devices, visual terminals, autonomous driving terminals, vehicles, home appliances, and / or medical devices. The vehicles include airplanes, ships, and / or vehicles; the home appliances include televisions, air conditioners, microwave ovens, refrigerators, rice cookers, humidifiers, washing machines, lights, gas stoves, and range hoods; the medical devices include MRI scanners, ultrasound machines, and / or electrocardiographs. The electronic devices or apparatus disclosed herein can also be applied in fields such as the Internet, IoT, data centers, energy, transportation, public management, manufacturing, education, power grids, telecommunications, finance, retail, construction sites, and healthcare. Furthermore, the electronic devices or apparatus disclosed herein can also be used in application scenarios related to artificial intelligence, big data, and / or cloud computing, such as cloud computing, edge computing, and terminal applications. In one or more embodiments, the high-computing-power electronic devices or apparatuses according to the present disclosure can be applied to cloud devices (e.g., cloud servers), while the low-power electronic devices or apparatuses can be applied to terminal devices and / or edge devices (e.g., smartphones or cameras). In one or more embodiments, the hardware information of the cloud devices and the hardware information of the terminal devices and / or edge devices are compatible with each other, so that suitable hardware resources can be matched from the hardware resources of the cloud devices to simulate the hardware resources of the terminal devices and / or edge devices based on the hardware information of the terminal devices and / or edge devices, so as to complete the unified management, scheduling and collaborative work of end-to-cloud or cloud-edge-end integration.

[0105] It should be noted that, for the sake of brevity, this disclosure describes some methods and their embodiments as a series of actions and combinations thereof. However, those skilled in the art will understand that the solutions disclosed herein are not limited by the order of the described actions. Therefore, based on the disclosure or teachings of this document, those skilled in the art will understand that some steps can be performed in a different order or simultaneously. Furthermore, those skilled in the art will understand that the embodiments described in this disclosure can be considered optional embodiments, that is, the actions or modules involved are not necessarily essential for the implementation of one or more solutions disclosed herein. In addition, depending on the solution, the description of some embodiments in this disclosure may have different emphases. In view of this, those skilled in the art will understand that parts not described in detail in a certain embodiment of this disclosure can also be referred to the relevant descriptions of other embodiments.

[0106] In terms of specific implementation, based on the disclosure and teachings of this document, those skilled in the art will understand that several embodiments disclosed herein can also be implemented in other ways not disclosed herein. For example, regarding the various units in the electronic device or apparatus embodiments described above, this document divides them based on logical functions, but in actual implementation, there may be other division methods. As another example, multiple units or components can be combined or integrated into another system, or some features or functions in a unit or component can be selectively disabled. Regarding the connection relationships between different units or components, the connections discussed above in conjunction with the accompanying drawings can be direct or indirect couplings between units or components. In some scenarios, the aforementioned direct or indirect couplings involve communication connections utilizing interfaces, where the communication interface can support electrical, optical, acoustic, magnetic, or other forms of signal transmission.

[0107] In this disclosure, the units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units. The aforementioned components or units may be located in the same location or distributed across multiple network units. Furthermore, depending on actual needs, some or all of the units can be selected to achieve the purpose of the solution described in the embodiments of this disclosure. Additionally, in some scenarios, multiple units in the embodiments of this disclosure may be integrated into one unit or each unit may exist physically independently.

[0108] In other implementation scenarios, the integrated units described above can also be implemented in hardware, i.e., as specific hardware circuits, which may include digital circuits and / or analog circuits. The physical implementation of the circuit's hardware structure may include, but is not limited to, physical devices, which may include, but are not limited to, transistors or memristors. Therefore, the various devices described herein (e.g., computing devices or other processing devices) can be implemented using appropriate hardware processors, such as CPUs, GPUs, FPGAs, DSPs, and ASICs. Furthermore, the aforementioned storage units or storage devices can be any suitable storage medium (including magnetic storage media or magneto-optical storage media), such as resistive random access memory (RRAM), dynamic random access memory (DRAM), static random access memory (SRAM), enhanced dynamic random access memory (EDRAM), high-bandwidth memory (HBM), hybrid memory cube (HMC), ROM, and RAM.

[0109] The foregoing can be better understood in accordance with the following terms:

[0110] Clause 1. A method for training a machine learning model, said machine learning model being used for image restoration processing, said method comprising:

[0111] In forward training, a loss function is calculated based on a baseline image and the output image processed by the machine learning model, wherein the loss function preserves the structural information in the baseline image; and

[0112] In reverse training, the parameters of the machine learning model are updated based on the loss function.

[0113] Clause 2, the training method according to Clause 1, wherein the structural information is edge information in the image.

[0114] Clause 3. The training method described in Clause 2, wherein calculating the value of the loss function includes:

[0115] Detecting blurred pixels at edges in the output image; and

[0116] Based on the detected blurred pixels, a weighting factor is determined to weight the pixel error between the output image and the reference image for calculating the loss function.

[0117] Clause 4. The training method according to Clause 3, wherein detecting blurred pixels at edges in the output image includes:

[0118] Determine a first direction of the pixel error between the output image and the reference image;

[0119] Determine the second direction of the change of adjacent pixels at the corresponding pixel position in the reference image; and

[0120] Pixels whose first direction and second direction are consistent are defined as blurred pixels.

[0121] Clause 5. The training method according to Clause 4, wherein determining the second direction of the change of adjacent pixels at the corresponding pixel position in the reference image includes:

[0122] Calculate the second-order gradient value of the corresponding pixel position and its adjacent pixels in the reference image, where the sign of the second-order gradient value represents the second direction.

[0123] Clause 6. The training method according to any one of Clauses 3-5, wherein determining the weighting factor for weighting the pixel error between the output image and the reference image based on the detected blurred pixels includes:

[0124] For detected blurry pixels, increase the weighting factor of the pixel error corresponding to the blurry pixel.

[0125] Clause 7. The training method according to any one of Clauses 3-6, wherein determining the weighting factor for weighting the pixel error between the output image and the reference image based on the detected blurred pixels further includes:

[0126] For blurred pixels at different locations, the corresponding weighting factor is determined based on the magnitude of the change in adjacent pixels at the corresponding pixel location in the reference image.

[0127] Clause 8. The training method according to any one of Clauses 3-7, wherein the weighting factor ω is determined according to the following formula. i,j :

[0128] ω i,j =1+αm i,j

[0129]

[0130]

[0131] Where ω i,j Let α represent the weighting factor at pixel (i, j), and m represent the penalty factor. i,jThis represents a blur factor map, where 0 values ​​represent non-blurred pixels and non-zero values ​​represent blurred pixels. This represents the pixel value at pixel (i, j) in the output image. e represents the pixel value of the reference image at pixel (i, j). i,j This represents the pixel error at pixel (i, j). This represents the second-order gradient value at pixel (i, j) on the reference image.

[0132] Clause 9. The training method according to Clause 8, wherein the loss function L SP Calculate as follows:

[0133]

[0134] Among them, L SP The loss function is represented by W and H, which represent the width and height (in pixels) of the reference image and the output image, respectively.

[0135] Clause 10. The training method according to any one of Clauses 8-9, wherein the penalty factor is adjustable, and the larger the penalty factor, the sharper the edges in the resulting output image.

[0136] Clause 11. The training method according to Clause 10, wherein the penalty factor is a positive real number, preferably ranging from about 1.0 to 5.0, more preferably 2.

[0137] Clause 12. The training method according to any one of Clauses 1-11, wherein the machine learning model is a neural network-based model.

[0138] Clause 13. A computing device for training a machine learning model, comprising:

[0139] A processor, configured to execute program instructions; and

[0140] A memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform the training method described in any one of clauses 1-12.

[0141] Clause 14. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform a training method according to any one of Clauses 1-12.

[0142] Clause 15. A computer program product comprising a computer program or instructions that, when executed by a processor, implement any of the training methods described in Clauses 1-12.

[0143] While numerous embodiments of this disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Many modifications, alterations, and alternatives will occur to those skilled in the art without departing from the spirit and intent of this disclosure. It should be understood that various alternatives to the embodiments of this disclosure described herein may be employed in the practice of this disclosure. The appended claims are intended to define the scope of this disclosure and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for training a machine learning model, said machine learning model being used for image restoration processing, the method comprising: In forward training, the value of a loss function is calculated based on a baseline image and the output image processed by the machine learning model, wherein the loss function preserves the structural information in the baseline image; as well as In reverse training, the parameters of the machine learning model are updated based on the loss function; The structural information mentioned above refers to edge information in the image; The calculation of the loss function includes: Detect blurred pixels at the edges in the output image; as well as Based on the detected blurred pixels, a weighting factor is determined to weight the pixel error between the output image and the reference image for calculating the loss function.

2. The training method according to claim 1, wherein detecting blurred pixels at edges in the output image comprises: Determine a first direction of the pixel error between the output image and the reference image; Determine the second direction of change of adjacent pixels at the corresponding pixel position in the reference image; as well as Pixels whose first direction and second direction are consistent are defined as blurred pixels.

3. The training method according to claim 2, wherein determining the second direction of the change of adjacent pixels at the corresponding pixel position in the reference image includes: Calculate the second-order gradient value of the corresponding pixel position and its adjacent pixels in the reference image, where the sign of the second-order gradient value represents the second direction.

4. The training method according to claim 1, wherein determining the weighting factor for weighting the pixel error between the output image and the reference image based on the detected blurred pixels includes: For detected blurry pixels, increase the weighting factor of the pixel error corresponding to the blurry pixel.

5. The training method according to claim 1, wherein determining the weighting factor for weighting the pixel error between the output image and the reference image based on the detected blurred pixels further includes: For blurred pixels at different locations, the corresponding weighting factor is determined based on the magnitude of the change in adjacent pixels at the corresponding pixel location in the reference image.

6. The training method according to claim 1, wherein the weighting factor is determined according to the following formula. : in This represents the weighting factor at pixel (i, j). Indicates the penalty factor. This represents a blur factor map, where 0 values ​​represent non-blurred pixels and non-zero values ​​represent blurred pixels. This represents the pixel value at pixel (i, j) in the output image. This represents the pixel value of the reference image at pixel (i, j). This represents the pixel error at pixel (i, j). This represents the second-order gradient value at pixel (i, j) on the reference image.

7. The training method according to claim 6, wherein the loss function Calculate as follows: in, The loss function is represented by W and H, which represent the width and height (in pixels) of the reference image and the output image, respectively.

8. The training method according to claim 6, wherein the penalty factor is adjustable, and the larger the penalty factor, the sharper the edges in the resulting output image.

9. The training method according to claim 8, wherein the penalty factor is a positive real number and its range is 1.0 to 5.

0.

10. The training method according to claim 9, wherein the penalty factor is 2.

11. The training method according to any one of claims 1-10, wherein the machine learning model is a neural network-based model.

12. A computing device for training a machine learning model, comprising: A processor, configured to execute program instructions; as well as A memory configured to store the program instructions, which, when loaded and executed by the processor, cause the processor to perform the training method according to any one of claims 1-11.

13. A computer-readable storage medium storing program instructions that, when loaded and executed by a processor, cause the processor to perform the training method according to any one of claims 1-11.

14. A computer program product comprising a computer program or instructions which, when executed by a processor, implement the training method according to any one of claims 1-11.

Images