A training method and related device for noise reduction model

Abstract

The present application discloses a method for training a denoising model, which is applied to the field of artificial intelligence and belongs to computer vision technology. The method includes: obtaining a sample of an image to be denoised from a sample set, the sample set including a plurality of sample images to be denoised; performing a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively obtaining a first sub-image and a second sub-image; inputting the first sub-image into the denoising model to obtain a first target image; obtaining a first loss function according to the first target image and the second sub-image, the first loss function being used to indicate the difference between the first target image and the second sub-image; training the denoising model at least according to the first loss function to obtain a target denoising model. In this solution, the denoising model can be trained based on the noise image without obtaining a clean image corresponding to the noise image, thereby reducing the difficulty of training the denoising model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a training method for a noise reduction model and related devices. Background Art

[0002] Artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is also the study of the design principles and implementation methods of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] At present, in the process of image generation or transmission, it is often easy to be disturbed by the imaging equipment or external environment noise, and produce noise that affects the image quality. Such images that include noise due to interference are usually called noisy images or noisy images. In order to improve the quality of such images, image denoising methods have emerged. Image denoising methods refer to the application of algorithms to remove noise from observed noisy images, retain image details, and reconstruct the corresponding clean images. At present, image denoising methods have important application value in the fields of mobile phone photography, high-definition television, monitoring equipment, satellite images and medical imaging.

[0004] In the related art, the image is denoised mainly by a learning-based denoising model (such as a convolutional neural network). The denoising model is trained with a large amount of training data to obtain a denoising model with good denoising effect, thereby achieving denoising of the image.

[0005] Generally speaking, the denoising effect of a denoising model depends largely on the training data used to train the denoising model, i.e., the noisy image-clean image pairs. However, in the field of image processing, it is often very difficult to obtain the noisy image-clean image pairs. Therefore, there is an urgent need for a method that can train a denoising model without the need for noisy image-clean image pairs. Summary of the invention

[0006] The embodiment of the present application provides a method and a related device for training a denoising model, which performs two random downsampling processes on denoised image samples respectively to obtain sub-image sample pairs, and uses one sub-image in the sub-image sample pair as the input value of the denoising model, and the other sub-image in the sub-image sample pair as the output expected value of the denoising model, thereby realizing the training of the denoising model. The denoising model can be trained based on the noise image without obtaining the clean image corresponding to the noise image, which reduces the difficulty of training the denoising model.

[0007] In a first aspect, the present application provides a method for training a denoising model, which can be applied to scenes such as terminal photography, medical images or surveillance videos, and is used to achieve image denoising. The method includes: the terminal obtains a sample of an image to be denoised from a sample set, and the sample set includes a plurality of sample of an image to be denoised. The terminal performs a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, and obtains a first sub-image and a second sub-image respectively, and the resolution of the first sub-image and the second sub-image are the same. That is, the terminal performs two random downsampling processes on the same sample of the image to be denoised, and obtains a first sub-image and a second sub-image. The first sub-image and the second sub-image are different images, and the resolution of the first sub-image and the second sub-image is the same, and the resolution of the first sub-image and the second sub-image is lower than the resolution of the sample of the image to be denoised. The random downsampling process refers to randomly sampling pixels in the sample of the image to be denoised based on a set sampling method, and obtaining a sub-image with a resolution less than the sample of the image to be denoised based on the pixels obtained by sampling. The above-mentioned first random downsampling process and the second random downsampling process perform pixel sampling in the same way. However, since the first random downsampling process and the second random downsampling process are two independent pixel random sampling processes, the first sub-image obtained by the first random downsampling process and the second sub-image obtained by the second random downsampling process are highly likely to be different.

[0008] The terminal inputs the first sub-image into a denoising model to obtain a first target image, and the denoising model includes but is not limited to a learning-based model such as a convolutional neural network or a denoising model based on sparse feature expression. The terminal obtains a first loss function based on the first target image and the second sub-image, and the first loss function is used to indicate the difference between the first target image and the second sub-image. Finally, the terminal trains the denoising model at least according to the first loss function until a model training condition is met to obtain a target denoising model. The model training condition refers to that the first loss function obtained by the terminal is less than a preset threshold.

[0009] In this scheme, the denoising model is trained by performing two random downsampling processes on the denoising image samples, obtaining sub-image sample pairs, and using one sub-image in the sub-image sample pair as the input value of the denoising model, and the other sub-image in the sub-image sample pair as the output expected value of the denoising model. The denoising model can be trained based on the noisy image without obtaining the clean image corresponding to the noisy image, which reduces the difficulty of training the denoising model.

[0010] Optionally, in a possible implementation, the terminal performs a first random downsampling process and a second random downsampling process on the image sample to be denoised to obtain a first sub-image and a second sub-image respectively, including: the terminal divides the image sample to be denoised into M image units, each of the M image units includes n*n pixels. The terminal performs a first random selection of pixels in each of the M image units to obtain M first pixels. Based on the M first pixels obtained by random sampling, the terminal pieces together the M first pixels according to the positions of the image units corresponding to the M first pixels in the image to be denoised to obtain the first sub-image.

[0011] Similarly, the terminal performs a second random selection of pixels in each of the M image units to obtain M second pixels. Based on the M second pixels obtained by random sampling, the terminal pieces together the M second pixels according to the positions of the image units corresponding to the M second pixels in the image to be denoised to obtain the second sub-image.

[0012] Optionally, in a possible implementation, the terminal performs a second random selection of pixels in each of the M image units, including: the terminal obtains n*n-1 target pixels in each of the M image units, and the n*n-1 target pixels are pixels in each image unit that have not been selected when performing the first random selection of pixels. The terminal performs random selection of pixels in the n*n-1 target pixels in each of the M image units to obtain M second pixels, and the M second pixels are all different from the M first pixels.

[0013] That is to say, when the terminal performs the second random selection of pixels for each image unit, the terminal needs to first determine the pixels in each image unit that have not been selected as the first pixel, that is, the n*n-1 target pixels in each image unit. Then, the terminal randomly selects a pixel from these n*n-1 target pixels as the second pixel. In this way, the first pixel and the second pixel selected by the terminal in each image unit must be different pixels. For example, Figure 5 Each pixel in the first sub-image and the second sub-image shown is a different pixel.

[0014] By ensuring that the same pixels as those in the first sub-image are not sampled during the downsampling process of the second sub-image, it can be ensured that the first sub-image and the second sub-image obtained by the random downsampling process are two completely independent images, that is, there is no overly strong correlation between the first sub-image and the second sub-image, so that the two images with random noise in the same scene can be better simulated based on the first sub-image and the second sub-image, thereby improving the training effect of the denoising model.

[0015] Optionally, in a possible implementation, the terminal performs random selection of pixels among the n*n-1 target pixels in each of the M image units, including: the terminal randomly selects a second pixel adjacent to the pixel selected in the first random selection among the n*n-1 target pixels in each of the M image units, to obtain M second pixels, each of the M second pixels being adjacent to the corresponding first pixel.

[0016] That is, for the n*n-1 target pixels in each image unit, the terminal can first determine the target pixels adjacent to the first pixel selected when performing the first random selection of pixels among the n*n-1 target pixels. Then, the terminal randomly selects a pixel as the second pixel among the determined target pixels adjacent to the first pixel. In this way, the first pixel and the second pixel selected by the terminal in each image unit must be adjacent pixels.

[0017] By selecting a target pixel adjacent to the first pixel as the second pixel, it can be ensured that the obtained second sub-image has a higher similarity with the first sub-image, that is, based on the first sub-image and the second sub-image, two images with random noise in the same scene are better simulated, thereby improving the training effect of the denoising model.

[0018] Optionally, in a possible implementation, the method further includes: the terminal inputs the sample of the image to be denoised into the denoising model to obtain a second target image. The terminal performs downsampling processing on the second target image based on the sampling position of the pixel in the first random downsampling process to obtain a first sub-target image. That is, the positions of the first sub-target image and each group of corresponding pixels in the first sub-image are the same in the source image before downsampling. The terminal performs downsampling processing on the second target image based on the sampling position of the pixel in the second random downsampling process to obtain a second sub-target image. The terminal obtains a second loss function based on the first target image, the second sub-image, the first sub-target image and the second sub-target image. The terminal trains the denoising model based on at least the first loss function and the second loss function.

[0019] In this scheme, by introducing the second loss function, the denoising model can be further constrained so that the denoising model will not generate an over-smoothed image due to the inconsistent positions of the pixels corresponding to the first sub-image and the second sub-image in the image sample to be denoised, that is, the high-frequency detail information in the noisy image is protected from being removed by the denoising model, ensuring that the denoised image still retains the high-frequency detail information.

[0020] Optionally, in a possible implementation, the terminal trains the denoising model at least according to the first loss function and the second loss function, including: the terminal trains the denoising model at least according to the first loss function, a first weight coefficient, the second loss function, and a second weight coefficient. The first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0021] The larger the second weight coefficient is, the weaker the noise reduction strength is. The weaker the noise reduction strength is, the more noise there is, but the loss of high-frequency details of the image will be less. The smaller the second weight coefficient is, the stronger the noise reduction strength is. The stronger the noise reduction strength is, the less noise there is, but the loss of high-frequency details of the image will be more. Therefore, in practical applications, the balance between the noise reduction strength and the loss of high-frequency details can be achieved by adjusting the first weight coefficient and / or the second weight coefficient.

[0022] Optionally, in a possible implementation, the denoising model includes a learning-based denoising model such as a convolutional neural network or a denoising model based on sparse feature expression.

[0023] The second aspect of the present application provides an image denoising method, comprising: obtaining an image to be denoised; inputting the image to be denoised into a target denoising model to obtain a denoised image. The target denoising model is obtained by training the denoising model based on at least a first loss function, the first loss function is obtained based on a first target image and a second sub-image, the first loss function is used to indicate the difference between the first target image and the second sub-image, the first target image is obtained after the first sub-image is input into the denoising model, the first sub-image and the second sub-image are obtained after performing a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively, and the resolution of the first sub-image is the same as that of the second sub-image.

[0024] Optionally, in a possible implementation, the first sub-image is obtained based on M first pixels, and the M first pixels are obtained by performing a first random selection of pixels in each of the M image units after the image sample to be denoised is divided into M image units; the second sub-image is obtained based on M second pixels, and the M second pixels are obtained by performing a second random selection of pixels in each of the M image units after the image sample to be denoised is divided into the M image units.

[0025] Optionally, in a possible implementation, the M second pixels are obtained by performing random selection of pixels among n*n-1 target pixels in each of the M image units, the n*n-1 target pixels are pixels in each image unit that are not selected when performing the first random selection of pixels, and the M second pixels are different from the M first pixels.

[0026] Optionally, in a possible implementation, the M second pixels are obtained by randomly selecting a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, and each of the M second pixels is adjacent to the corresponding first pixel.

[0027] Optionally, in a possible implementation, the target denoising model is obtained by training the denoising model based on at least the first loss function and the second loss function, the second loss function is obtained according to the first target image, the second sub-image, the first sub-target image and the second sub-target image, the first sub-target image is obtained by downsampling the second target image based on the sampling position of the pixel in the first random downsampling process, the second sub-target image is obtained by downsampling the second target image based on the sampling position of the pixel in the second random downsampling process, and the second target image is obtained by inputting the sample of the image to be denoised into the denoising model.

[0028] Optionally, in a possible implementation, the target denoising model is obtained by training the denoising model based on at least a first weight coefficient of the first loss function, the second loss function, and a second weight coefficient, wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0029] Optionally, in a possible implementation, the denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

[0030] The third aspect of the present application provides a model training device, including: an acquisition unit and a processing unit. The acquisition unit is used to acquire a sample of an image to be denoised from a sample set, wherein the sample set includes a plurality of sample of an image to be denoised; the processing unit is used to perform a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively obtaining a first sub-image and a second sub-image, wherein the first sub-image has the same resolution as the second sub-image; the processing unit is also used to input the first sub-image into a denoising model to obtain a first target image; the processing unit is also used to acquire a first loss function based on the first target image and the second sub-image, wherein the first loss function is used to indicate the difference between the first target image and the second sub-image; the processing unit is also used to train the denoising model at least based on the first loss function to obtain a target denoising model.

[0031] Optionally, in a possible implementation, the processing unit is further used to: divide the image sample to be denoised into M image units, each of the M image units including n*n pixels; perform a first random selection of pixels in each of the M image units to obtain M first pixels, and obtain the first sub-image based on the M first pixels; perform a second random selection of pixels in each of the M image units to obtain M second pixels, and obtain the second sub-image based on the M second pixels.

[0032] Optionally, in a possible implementation, the acquisition unit is further used to acquire n*n-1 target pixels in each of the M image units, where the n*n-1 target pixels are pixels in each image unit that have not been selected when the first random selection of pixels is performed; the processing unit is further used to perform random selection of pixels among the n*n-1 target pixels in each of the M image units to obtain M second pixels, and the M second pixels are all different from the M first pixels.

[0033] Optionally, in a possible implementation, the processing unit is also used to randomly select a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, to obtain M second pixels, each of the M second pixels being adjacent to the corresponding first pixel.

[0034] Optionally, in a possible implementation, the processing unit is further used to: input the image sample to be denoised into the denoising model to obtain a second target image; based on the sampling position of the pixel in the first random downsampling process, downsample the second target image to obtain a first sub-target image; based on the sampling position of the pixel in the second random downsampling process, downsample the second target image to obtain a second sub-target image; obtain a second loss function based on the first target image, the second sub-image, the first sub-target image and the second sub-target image; and train the denoising model at least based on the first loss function and the second loss function.

[0035] Optionally, in a possible implementation, the processing unit is also used to train the denoising model based on at least the first loss function, the first weight coefficient, the second loss function and the second weight coefficient; wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0036] Optionally, in a possible implementation, the denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

[0037] In a fourth aspect, the present application provides an image denoising device, comprising: an acquisition unit and a processing unit. The acquisition unit is used to acquire the image to be denoised; the processing unit is used to input the image to be denoised into a target denoising model to obtain a denoised image; wherein the target denoising model is obtained by training the denoising model based on at least a first loss function, the first loss function is obtained based on a first target image and a second sub-image, the first loss function is used to indicate the difference between the first target image and the second sub-image, the first target image is obtained after the first sub-image is input into the denoising model, the first sub-image and the second sub-image are obtained after performing a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively, and the resolution of the first sub-image is the same as that of the second sub-image.

[0038] Optionally, in a possible implementation, the first sub-image is obtained based on M first pixels, and the M first pixels are obtained by performing a first random selection of pixels in each of the M image units after the image sample to be denoised is divided into M image units; the second sub-image is obtained based on M second pixels, and the M second pixels are obtained by performing a second random selection of pixels in each of the M image units after the image sample to be denoised is divided into the M image units.

[0039] Optionally, in a possible implementation, the M second pixels are obtained by performing random selection of pixels among n*n-1 target pixels in each of the M image units, the n*n-1 target pixels are pixels in each image unit that are not selected when performing the first random selection of pixels, and the M second pixels are different from the M first pixels.

[0040] Optionally, in a possible implementation, the M second pixels are obtained by randomly selecting a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, and each of the M second pixels is adjacent to the corresponding first pixel.

[0041] Optionally, in a possible implementation, the target denoising model is obtained by training the denoising model based on at least the first loss function and the second loss function, the second loss function is obtained according to the first target image, the second sub-image, the first sub-target image and the second sub-target image, the first sub-target image is obtained by downsampling the second target image based on the sampling position of the pixel in the first random downsampling process, the second sub-target image is obtained by downsampling the second target image based on the sampling position of the pixel in the second random downsampling process, and the second target image is obtained by inputting the sample of the image to be denoised into the denoising model.

[0042] Optionally, in a possible implementation, the target denoising model is obtained by training the denoising model based on at least a first weight coefficient of the first loss function, the second loss function, and a second weight coefficient; wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0043] Optionally, in a possible implementation, the denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

[0044] In a fifth aspect, the present application provides a model training device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the first aspect is implemented. For the steps in each possible implementation method of the processor executing the first aspect, the details can be referred to the first aspect, and no further description is given here.

[0045] In a sixth aspect, the present application provides an image noise reduction device, which may include a processor, the processor and a memory are coupled, the memory stores program instructions, and when the program instructions stored in the memory are executed by the processor, the method described in the second aspect is implemented. For the steps in each possible implementation method of the processor executing the second aspect, the details can be referred to the second aspect, and will not be repeated here.

[0046] In a seventh aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the first aspect.

[0047] In an eighth aspect, the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer-readable storage medium is run on a computer, the computer executes the method described in the second aspect.

[0048] A ninth aspect of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the method described in the first aspect or the second aspect.

[0049] In a tenth aspect, the present application provides a computer program, which, when executed on a computer, enables the computer to execute the method described in the first or second aspect above.

[0050] In the eleventh aspect of the present application, a chip system is provided, which includes a processor for supporting a server or a threshold value acquisition device to implement the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above methods. In one possible design, the chip system also includes a memory, which is used to store program instructions and data necessary for the server or communication device. The chip system can be composed of chips, or it can include chips and other discrete devices. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 A schematic diagram of the structure of the artificial intelligence main framework provided in the embodiment of the present application;

[0052] Figure 2a An image processing system provided in an embodiment of the present application;

[0053] Figure 2b Another image processing system provided in an embodiment of the present application;

[0054] Figure 2c A schematic diagram of an image processing device provided in an embodiment of the present application;

[0055] Figure 3a A schematic diagram of a system 100 architecture provided in an embodiment of the present application;

[0056] Figure 3b A schematic diagram of image noise reduction provided in an embodiment of the present application;

[0057] Figure 4 A flowchart of a method for training a noise reduction model provided in an embodiment of the present application;

[0058] Figure 5 A schematic diagram of a random downsampling process provided in an embodiment of the present application;

[0059] Figure 6 A schematic diagram of another random downsampling process provided in an embodiment of the present application;

[0060] Figure 7 A schematic diagram of another random downsampling process provided in an embodiment of the present application;

[0061] Figure 8 A schematic diagram of a training process of a noise reduction model provided in an embodiment of the present application;

[0062] Fig. 9 A schematic diagram of the training process of another noise reduction model provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of an experimental process for determining the noise reduction effect of a noise reduction model provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram comparing the noise reduction effects of various noise reduction methods provided in the embodiments of the present application;

[0065] Fig.12 A schematic diagram of another experimental process for determining the noise reduction effect of a noise reduction model provided in an embodiment of the present application;

[0066] Fig.13 A schematic diagram comparing noise reduction indicators of various noise reduction methods provided in the embodiments of the present application;

[0067] Fig.14 A schematic diagram comparing the noise reduction effects of various noise reduction methods provided in the embodiments of the present application;

[0068] Fig.15 A schematic diagram of a process of an image noise reduction method provided in an embodiment of the present application;

[0069] Fig.16 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0070] Fig.17 A schematic diagram of the structure of an image noise reduction device provided in an embodiment of the present application;

[0071] Fig.18 A schematic diagram of the structure of an execution device provided in an embodiment of the present application;

[0072] Fig.19 A schematic diagram of the structure of a training device provided in an embodiment of the present application;

[0073] Fig. 20 A schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0074] The embodiments of the present invention are described below in conjunction with the accompanying drawings in the embodiments of the present invention. The terms used in the implementation mode of the present invention are only used to explain the specific embodiments of the present invention, and are not intended to limit the present invention.

[0075] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0076] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0077] First, the overall workflow of the artificial intelligence system is described. Figure 1 , Figure 1 The figure shows a structural diagram of the main framework of artificial intelligence. The following is an explanation of the above artificial intelligence theme framework from the two dimensions of "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis). Among them, the "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, the data has undergone a condensation process of "data-information-knowledge-wisdom". The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the industrial ecology process of the system.

[0078] (1) Infrastructure

[0079] The infrastructure provides computing power support for the AI ​​system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips (CPU, NPU, GPU, ASIC, FPGA and other hardware acceleration chips); the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0080] (2) Data

[0081] The data on the upper layer of the infrastructure is used to represent the data sources in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and perception data such as force, displacement, liquid level, temperature, and humidity.

[0082] (3) Data processing

[0083] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0084] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0085] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0086] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0087] (4) General ability

[0088] After the data has undergone the data processing mentioned above, some general capabilities can be further formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc.

[0089] (5) Smart products and industry applications

[0090] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical applications. Its application areas mainly include: smart terminals, smart transportation, smart medical care, autonomous driving, safe cities, etc.

[0091] Next, several application scenarios of this application are introduced.

[0092] Figure 2a An image processing system is provided in an embodiment of the present application, and the image processing system includes a user device and a data processing device. The user device includes a smart terminal such as a mobile phone, a personal computer or an information processing center. The user device is the initiator of image processing, and as the initiator of the image enhancement request, the request is usually initiated by the user through the user device.

[0093] The above-mentioned data processing device can be a device or server with data processing function such as a cloud server, a network server, an application server and a management server. The data processing device receives the image enhancement request from the intelligent terminal through the interactive interface, and then performs image processing in the form of machine learning, deep learning, search, reasoning, decision-making, etc. through the memory for storing data and the processor for data processing. The memory in the data processing device can be a general term, including local storage and a database for storing historical data. The database can be on the data processing device or on other network servers.

[0094] exist Figure 2a In the image processing system shown, the user device can receive the user's instruction. For example, the user device can obtain an image input / selected by the user, and then initiate a request to the data processing device, so that the data processing device performs an image noise reduction application on the image obtained by the user device, thereby obtaining a corresponding processing result for the image. Exemplarily, the user device can obtain an image input by the user, and then initiate an image noise reduction request to the data processing device, so that the data processing device performs image noise reduction on the image, thereby obtaining a noise-reduced image.

[0095] exist Figure 2a In the embodiment of the present application, the data processing device can execute the training method of the noise reduction model.

[0096] Figure 2b Another image processing system provided in an embodiment of the present application includes: Figure 2b In the process, the user device directly acts as a data processing device. The user device can directly obtain input from the user and directly process it by the hardware of the user device itself. The specific process is similar to Figure 2a Similarly, please refer to the above description and will not repeat them here.

[0097] exist Figure 2b In the image processing system shown, the user device can receive instructions from the user. For example, the user device can obtain an image selected by the user in the user device, and then the user device itself executes an image processing application on the image to obtain a corresponding processing result for the image.

[0098] exist Figure 2b In the embodiment of the present application, the user equipment itself can execute the training method of the noise reduction model.

[0099] Figure 2c It is a schematic diagram of the image processing related equipment provided in the embodiment of the present application.

[0100] Above Figure 2a and Figure 2b The user equipment in the example may specifically be Figure 2c The local device 301 or the local device 302 in Figure 2a The data processing device in the embodiment may be Figure 2c The execution device 210 in the embodiment, wherein the data storage system 250 can store the data to be processed by the execution device 210, and the data storage system 250 can be integrated on the execution device 210, or can be set on the cloud or other network servers.

[0101] Figure 2a and Figure 2b The processor in the image processing apparatus can perform data training / machine learning / deep learning through a neural network model or other models (for example, a model based on a support vector machine), and use the model finally trained or learned from the data to execute image processing applications on the image, thereby obtaining corresponding processing results.

[0102] Figure 3a is a schematic diagram of a system 100 architecture provided in an embodiment of the present application. Figure 3a In the embodiment, the execution device 110 is configured with an input / output (I / O) interface 112 for data interaction with an external device. The user can input data to the I / O interface 112 through the client device 140. The input data may include: various tasks to be scheduled, callable resources and other parameters in the embodiment of the application.

[0103] When the execution device 110 preprocesses the input data, or when the computing module 111 of the execution device 110 performs calculation and other related processing (such as implementing the function of the neural network in the present application), the execution device 110 can call the data, code, etc. in the data storage system 150 for the corresponding processing, and can also store the data, instructions, etc. obtained by the corresponding processing in the data storage system 150.

[0104] Finally, the I / O interface 112 returns the processing result to the client device 140 so as to provide it to the user.

[0105] It is worth noting that the training device 120 can generate corresponding target models / rules based on different training data for different goals or tasks, and the corresponding target models / rules can be used to achieve the above goals or complete the above tasks, thereby providing the user with the desired results. The training data can be stored in the database 130 and come from the training samples collected by the data collection device 160.

[0106] exist Figure 3a In the case shown in the figure, the user can manually give input data, and the manual giving can be operated through the interface provided by the I / O interface 112. In another case, the client device 140 can automatically send input data to the I / O interface 112. If the client device 140 is required to automatically send input data and needs to obtain the user's authorization, the user can set the corresponding authority in the client device 140. The user can view the results output by the execution device 110 on the client device 140, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 140 can also be used as a data acquisition terminal to collect the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as shown in the figure as new sample data, and store them in the database 130. Of course, it is also possible not to collect through the client device 140, but the I / O interface 112 directly stores the input data of the input I / O interface 112 and the output results of the output I / O interface 112 as new sample data in the database 130.

[0107] It is worth noting that Figure 3a This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. For example, in Figure 3a In the embodiment, the data storage system 150 is an external memory relative to the execution device 110. In other cases, the data storage system 150 may also be placed in the execution device 110. Figure 3a As shown, a neural network can be obtained by training according to the training device 120.

[0108] The present application also provides a chip including a neural network processor NPU. The chip can be arranged in Figure 3a The execution device 110 shown in FIG. 1 is used to complete the calculation work of the calculation module 111. The chip can also be set in Figure 3a The training device 120 shown is used to complete the training work of the training device 120 and output the target model / rule.

[0109] Neural network processor NPU, NPU is mounted on the main central processing unit (CPU) (host CPU) as a coprocessor, and the main CPU assigns tasks. The core part of NPU is the operation circuit, and the controller controls the operation circuit to extract data from the memory (weight memory or input memory) and perform operations.

[0110] In some implementations, the arithmetic circuit includes multiple processing units (process engines, PEs) inside. In some implementations, the arithmetic circuit is a two-dimensional systolic array. The arithmetic circuit can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuit is a general-purpose matrix processor.

[0111] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory and performs matrix operations with matrix B. The partial results or final results of the matrix are stored in the accumulator.

[0112] The vector computing unit can further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. For example, the vector computing unit can be used for network calculations of non-convolutional / non-FC layers in neural networks, such as pooling, batch normalization, local response normalization, etc.

[0113] In some implementations, the vector computation unit can store the processed output vector to a unified buffer. For example, the vector computation unit can apply a nonlinear function to the output of the computation circuit, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector computation unit generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the computation circuit, such as for use in a subsequent layer in a neural network.

[0114] The unified memory is used to store input data and output data.

[0115] The weight data is directly transferred from the external memory to the input memory and / or the unified memory through the direct memory access controller (DMAC), the weight data in the external memory is stored in the weight memory, and the data in the unified memory is stored in the external memory.

[0116] The bus interface unit (BIU) is used to implement interaction between the main CPU, DMAC and instruction fetch memory through the bus.

[0117] An instruction fetch buffer connected to the controller, used to store instructions used by the controller;

[0118] The controller is used to call the instructions cached in the memory to control the working process of the computing accelerator.

[0119] Generally, the unified memory, input memory, weight memory and instruction fetch memory are all on-chip memories, and the external memory is a memory outside the NPU, which can be a double data rate synchronous dynamic random access memory (DDRSDRAM), a high bandwidth memory (HBM) or other readable and writable memory.

[0120] Since the embodiments of the present application involve the application of a large number of neural networks, in order to facilitate understanding, the relevant terms and related concepts such as neural networks involved in the embodiments of the present application are first introduced below.

[0121] (1) Neural Network

[0122] A neural network may be composed of neural units, and a neural unit may refer to an operation unit with xs and intercept 1 as input, and the output of the operation unit may be:

[0123]

[0124] Where s=1, 2, ...n, n is a natural number greater than 1, Ws is the weight of xs, and b is the bias of the neural unit. f is the activation function of the neural unit, which is used to introduce nonlinear characteristics into the neural network to convert the input signal in the neural unit into the output signal. The output signal of the activation function can be used as the input of the next convolution layer. The activation function can be a sigmoid function. A neural network is a network formed by connecting many of the above-mentioned single neural units together, that is, the output of one neural unit can be the input of another neural unit. The input of each neural unit can be connected to the local receptive field of the previous layer to extract the characteristics of the local receptive field. The local receptive field can be an area composed of several neural units.

[0125] The work of each layer in the neural network can be expressed mathematically as To describe: From a physical level, the work of each layer in a neural network can be understood as completing the transformation from input space to output space (i.e., from row space to column space of a matrix) through five operations on the input space (a set of input vectors). These five operations include: 1. Dimension increase / reduction; 2. Enlargement / reduction; 3. Rotation; 4. Translation; 5. "Bending". Operations 1, 2, and 3 are represented by Completed, operation 4 is completed by +b, and operation 5 is implemented by a(). The word "space" is used here because the classified object is not a single thing, but a class of things, and space refers to the set of all individuals of this class of things. Among them, W is a weight vector, and each value in the vector represents the weight value of a neuron in this layer of the neural network. The vector W determines the spatial transformation from the input space to the output space described above, that is, the weight W of each layer controls how to transform the space. The purpose of training a neural network is to eventually obtain the weight matrix of all layers of the trained neural network (the weight matrix formed by many layers of vectors W). Therefore, the training process of a neural network is essentially to learn how to control spatial transformation, and more specifically to learn the weight matrix.

[0126] Because we want the output of the neural network to be as close as possible to the value we really want to predict, we can compare the current network's predicted value with the target value we really want, and then update the weight vector of each layer of the neural network based on the difference between the two (of course, there is usually an initialization process before the first update, that is, pre-configuring parameters for each layer in the neural network). For example, if the network's predicted value is high, adjust the weight vector to make it predict a lower value, and keep adjusting until the neural network can predict the target value we really want. Therefore, it is necessary to predefine "how to compare the difference between the predicted value and the target value", which is the loss function or objective function, which are important equations used to measure the difference between the predicted value and the target value. Among them, taking the loss function as an example, the higher the output value (loss) of the loss function, the greater the difference, so the training of the neural network becomes a process of minimizing this loss as much as possible.

[0127] (2) Back propagation algorithm

[0128] Neural networks can use the error back propagation (BP) algorithm to correct the size of the parameters in the initial neural network model during the training process, so that the reconstruction error loss of the neural network model becomes smaller and smaller. Specifically, the forward transmission of the input signal to the output will generate error loss, and the parameters in the initial neural network model are updated by back propagating the error loss information, so that the error loss converges. The back propagation algorithm is a back propagation movement dominated by error loss, which aims to obtain the optimal parameters of the neural network model, such as the weight matrix.

[0129] (3) Image enhancement

[0130] Image enhancement refers to processing the brightness, color, contrast, saturation, dynamic range, etc. of an image to meet certain specific indicators. In simple terms, by purposefully emphasizing the overall or local characteristics of the image during the image processing process, the originally unclear image becomes clear or emphasizes certain features of interest, expands the differences between the features of different objects in the image, and suppresses features of no interest, thereby improving the image quality and enriching the image information. It can enhance the image interpretation and recognition effect and meet the needs of certain special analyses. Exemplarily, image enhancement can include but is not limited to image super-resolution reconstruction, image denoising, image defogging, image deblurring, and image contrast enhancement.

[0131] (4) Image Noise Reduction

[0132] Image denoising method refers to the application of algorithms to remove noise from observed noisy images, retain the more important details in the image, and reconstruct the corresponding clean image. The reconstructed image looks clear and clean. Image denoising can improve the image quality, which is conducive to the subsequent image processing processes such as image classification or object recognition. In the image processing process, image denoising is a very important step. At present, many image denoising methods have been applied in academia and industry. Image denoising is one of the current research hotspots in the field of image processing technology. You can refer to Figure 3b , Figure 3b A schematic diagram of image noise reduction provided in an embodiment of the present application. Figure 3b As shown, image denoising can eliminate noise in the image as much as possible and improve the image quality.

[0133] The method provided in the present application is described below from the training side of the neural network and the application side of the neural network.

[0134] The training method of the neural network provided in the embodiment of the present application involves the processing of images, and can be specifically applied to data processing methods such as data training, machine learning, and deep learning, to perform symbolic and formalized intelligent information modeling, extraction, preprocessing, and training on the training data (such as the image in the present application), and finally obtain a trained image processing model; and the image denoising method provided in the embodiment of the present application can use the above-mentioned trained denoising model to input the input data (such as the image to be processed in the present application) into the trained image processing model to obtain output data (such as the target image in the present application). It should be noted that the training method of the denoising model and the image denoising method provided in the embodiment of the present application are inventions based on the same concept, and can also be understood as two parts in a system, or two stages of an overall process: such as the model training stage and the model application stage.

[0135] At present, in the process of image generation or transmission, it is often easy to be disturbed by the imaging equipment or external environmental noise, and generate noise that affects the image quality. Such images that include noise due to interference are usually called noisy images or noisy images. In order to improve the quality of such images, image denoising methods have emerged. Image denoising methods refer to the application of algorithms to remove noise from observed noisy images, retain image details, and reconstruct corresponding clean images. By extracting features from noisy images, removing noise and filling details with the help of image prior knowledge, image self-similarity, and complementary information of multiple frames of images, a common idea in image denoising research is to generate corresponding high-quality images by removing noise and filling details. In the industrial world, image denoising technology has important application value in the fields of mobile phone photography, high-definition television, monitoring equipment, satellite images, and medical imaging.

[0136] Generally speaking, image denoising algorithms are mainly divided into traditional filtering methods and learning-based methods. Traditional filtering methods include mean filtering, Gaussian filtering, bilateral filtering, and three-dimensional block matching filtering (Block-Matching and 3D filtering, BM3D). Traditional filtering methods mainly use the similarity of images to reduce the random noise in the image through filtering and smoothing, while retaining the high-frequency signal of the image itself. Generally, image denoising methods with better effects are mostly a combination of multiple methods, which can not only well maintain edge information, but also remove noise in the image. For example, the median filtering method and the wavelet filtering method are combined for image filtering to achieve a better image denoising effect. In general, traditional denoising algorithms all find patterns from the noise image and then perform corresponding denoising processing. When the pattern cannot be found from the noise image itself, the traditional filtering method performs poorly and it is difficult to achieve a good image denoising effect, thereby limiting the further improvement of image denoising performance.

[0137] In this situation, learning-based image denoising methods came into being. Learning-based image denoising methods are data-driven methods. They achieve image denoising by learning the rules of large-scale noisy image-clean image pairs through the denoising model, so as to achieve good image denoising effects. In recent years, deep neural networks (DNNs) have rapidly surpassed traditional image denoising methods with their powerful learning capabilities and have achieved great success. Image denoising methods based on deep neural networks can generate cleaner, clearer, and less artifact-prone images, further promoting the development of image denoising technology.

[0138] Thanks to the continuous improvement of processor computing power and the rapid development of deep convolutional neural networks, the effect of image denoising networks has been greatly improved, which has further promoted the application of image denoising networks based on deep learning. The denoising effect of the current image denoising network based on supervised learning depends largely on the training data used to train the network, that is, the noisy image-clean image pair. Among them, the noisy image-clean image pair refers to a noisy image and the clean noise-free image corresponding to the noisy image. Simply put, the noisy image-clean image pair is a pair of images in the same scene. The scene information included in the noisy image and the clean image is the same, and the difference is that the clean image does not include noise.

[0139] However, in the field of image processing, it is often very difficult to obtain a noisy image-clean image pair. For example, in the field of photography, due to the interference of imaging equipment and external environmental noise, the images taken by the camera are often noisy. Although relatively clean images (i.e., images with less noise) can be obtained by increasing exposure time, multi-frame smoothing and other technical means, their limitations are also very obvious. Especially for dynamic scenes, the content of images taken at different times itself changes, so it is often difficult to obtain a noisy image-clean image pair in dynamic scenes.

[0140] For another example, in the field of medical imaging, medical images are generated by instruments generating radiation or electromagnetic waves and acting on the human body. Due to the instrument itself, the images obtained by the instrument often contain a large amount of random noise, that is, it is often difficult to obtain a clean and noise-free image through the instrument. In addition, due to the particularity of medical images, the process of taking medical images is likely to have a certain negative impact on the human body. Therefore, in actual application, it is often difficult to obtain multiple medical images of the same part.

[0141] In view of the problem that it is difficult to obtain a noise image-clean image pair during the denoising model training process, this embodiment proposes a method for training the denoising model using only a noise image. By performing two random downsampling processes on the denoised image samples, a sub-image sample pair is obtained, and one sub-image in the sub-image sample pair is used as the input value of the denoising model, and the other sub-image in the sub-image sample pair is used as the output expected value of the denoising model, thereby realizing the training of the denoising model. The denoising model can be trained based on the noise image without obtaining the clean image corresponding to the noise image, which reduces the difficulty of training the denoising model.

[0142] For ease of understanding, the devices and scenarios to which the training method for the denoising model provided in this embodiment is applied are introduced below.

[0143] The training method of the noise reduction model provided in the embodiment of the present application can be applied to a terminal, which is a device capable of performing model training. Exemplarily, the terminal can be, for example, a personal computer (PC), a laptop, a server, a mobile phone, a tablet computer, a mobile internet device (MID), a wearable device, a virtual reality (VR) device, an augmented reality (AR) device, a wireless terminal in industrial control, a wireless terminal in self-driving, a wireless terminal in remote medical surgery, a wireless terminal in smart grid, a wireless terminal in transportation safety, a wireless terminal in smart city, a wireless terminal in smart home, etc. The terminal can be a device running Android system, IOS system, Windows system and other systems.

[0144] The denoising model training method provided in this embodiment can be applied to scenarios such as terminal device photography, video surveillance, and medical image processing where noisy images need to be converted into clean images. Through the denoising model training method provided in this embodiment, a denoising model for image denoising can be trained to achieve image denoising.

[0145] With the popularity of portable terminal devices such as smart phones and tablets, taking pictures has become an indispensable function of smart devices. As the importance of the camera function in terminal devices continues to increase, people have put forward more requirements for improving the quality of photographing in terminal devices. At present, the shortcomings of terminal devices in taking pictures mainly lie in the image quality in dark light scenes. In scenes such as dark light or at night, due to the weak ambient light, the photos taken by the terminal device will have very obvious noise, which greatly affects the image quality. It should be understood that since the noise in the image captured by the terminal device is caused by the poor ambient light, it is difficult to obtain a clean image in this scene through the terminal device in this scene, that is, it is difficult to obtain a noise image-clean image pair.

[0146] Medical images refer to images of human organs and other parts obtained by imaging instruments, such as magnetic resonance imaging (MRI) and computed tomography (CT). Imaging instruments will introduce noise during the imaging process, and large noise will reduce the signal-to-noise ratio and seriously affect subsequent image processing. Therefore, in medical image processing, denoising the noisy images generated by imaging instruments is an essential step.

[0147] Due to the particularity of medical images, many medical images can only be taken once. It is very difficult to take multiple noisy images of the same scene at the same time, and it is even more difficult to obtain clean images. Therefore, in the field of medical image denoising, it is also difficult to obtain noisy image-clean image pairs, which can easily affect the normal training of the denoising model.

[0148] See also Figure 4 , Figure 4 A flowchart of a method for training a noise reduction model provided in an embodiment of the present application is shown below. Figure 4 As shown, a denoising model training method provided in an embodiment of the present application includes the following steps 401-405.

[0149] Step 401: Acquire a sample of an image to be denoised from a sample set, where the sample set includes a plurality of samples of an image to be denoised.

[0150] When the training method of the denoising model provided in this embodiment is applied to a terminal photographing scenario, that is, when a model for denoising an image obtained by photographing needs to be trained, the terminal can obtain a sample set including a plurality of image samples to be denoised. The image samples to be denoised in the sample set can be images collected by the terminal in different scenarios, and all of the images collected by the terminal include noise.

[0151] When the training method of the denoising model provided in this embodiment is applied to a medical image scenario, the image samples to be denoised included in the sample set are medical images of the same type. For example, multiple image samples to be denoised in the sample set are all MRI images, or multiple image samples to be denoised in the sample set are all CT images.

[0152] In general, the multiple image samples to be denoised included in the sample set are images of the same type, and all of the image samples to be denoised include noise. The terminal can obtain the corresponding sample set according to the scenario to which the training method of the denoising model is applied, and train the denoising model based on the image samples to be denoised in the sample set.

[0153] Step 402 : performing a first random downsampling process and a second random downsampling process on the image sample to be denoised, to obtain a first sub-image and a second sub-image respectively, wherein the first sub-image and the second sub-image have the same resolution.

[0154] After obtaining the image sample to be denoised, the terminal performs two random downsampling processes on the same image sample to be denoised to obtain a first sub-image and a second sub-image, wherein the first sub-image and the second sub-image are different images, the resolutions of the first sub-image and the second sub-image are the same, and the resolutions of the first sub-image and the second sub-image are lower than the resolution of the image sample to be denoised.

[0155] Among them, the random downsampling process refers to randomly sampling pixels in the sample of the image to be denoised based on a set sampling method, and piecing together the pixels obtained by sampling to obtain a sub-image with a resolution smaller than the sample of the image to be denoised. The above-mentioned first random downsampling process and the second random downsampling process perform pixel sampling in the same way. However, since the first random downsampling process and the second random downsampling process are two independent pixel random sampling processes, the first sub-image obtained by the first random downsampling process and the second sub-image obtained by the second random downsampling process are most likely different.

[0156] For ease of understanding, two methods of performing random downsampling processing provided by this embodiment are described below. It should be understood that in practical applications, other random downsampling processing methods may also be used, and the random downsampling processing methods are not specifically limited here.

[0157] Method 1: Divide the image sample to be denoised into multiple image units, and randomly sample a pixel in each image unit, so as to obtain a sub-image composed of multiple sampled pixels.

[0158] Exemplarily, the terminal divides the sample of the image to be denoised into M image units on average, each of the M image units includes n*n pixels. Then, the terminal performs a first random selection of pixels in each of the M image units, that is, randomly selects a pixel as the first pixel from the n*n pixels of each image unit, thereby obtaining M first pixels. Based on the M first pixels obtained by random sampling, the terminal pieces together the M first pixels according to the positions of the image units corresponding to the M first pixels in the image to be denoised, thereby obtaining the first sub-image.

[0159] Similarly, the terminal performs a second random selection of pixels in each of the M image units, that is, randomly selects a pixel from the n*n pixels of each image unit as the second pixel, thereby obtaining M second pixels. Based on the M second pixels obtained by random sampling, the terminal pieces together the M second pixels according to the positions of the image units corresponding to the M second pixels in the image to be denoised, thereby obtaining the second sub-image.

[0160] See also Figure 5 , Figure 5 A schematic diagram of a random downsampling process provided in an embodiment of the present application. Figure 5 As shown, it is assumed that the resolution of the image to be denoised is 4*4, that is, the image to be denoised is composed of 4*4 pixels. Figure 5 The image to be denoised is divided into 4 image units, each of which includes 2*2 pixels.

[0161] In the process of performing the first random downsampling process on the image to be denoised, for the first image unit (i.e., the image unit at the upper left corner of the image to be denoised), the pixel at the upper left corner of the first image unit (i.e., pixel 1A) is randomly sampled as the first pixel. For the second image unit (i.e., the image unit at the upper right corner of the image to be denoised), the pixel at the upper right corner of the second image unit (i.e., pixel 1B) is randomly sampled as the first pixel; for the third image unit (i.e., the image unit at the lower left corner of the image to be denoised), the pixel at the lower right corner of the third image unit (i.e., pixel 1C) is randomly sampled as the first pixel; for the fourth image unit (i.e., the image unit at the lower right corner of the image to be denoised), the pixel at the lower right corner of the fourth image unit (i.e., pixel 1D) is randomly sampled as the first pixel. Based on the pixels obtained by sampling the four image units, the positions of the image units corresponding to the sampled pixels (i.e., pixel 1A, pixel 1B, pixel 1C, pixel 1D) in the image to be denoised can be pieced together to obtain the first sub-image.

[0162] In the process of performing the second random downsampling process on the image to be denoised, for the first image unit, the pixel at the lower right corner of the first image unit (i.e., pixel 2A) is randomly sampled as the second pixel. For the second image unit, the pixel at the lower left corner of the second image unit (i.e., pixel 2B) is randomly sampled as the second pixel; for the third image unit, the pixel at the upper left corner of the third image unit (i.e., pixel 2C) is randomly sampled as the second pixel; for the fourth image unit, the pixel at the upper right corner of the fourth image unit (i.e., pixel 2D) is randomly sampled as the second pixel. Based on the pixels sampled from the four image units, the positions of the image units corresponding to the sampled pixels (i.e., pixel 2A, pixel 2B, pixel 2C, pixel 2D) in the image to be denoised can be pieced together to obtain the second sub-image.

[0163] Optionally, when the terminal performs a second random selection of pixels in each of the M image units, the terminal may obtain n*n-1 target pixels in each of the M image units, where the n*n-1 target pixels are pixels in each image unit that were not selected when the first random selection of pixels was performed. Then, the terminal performs a random selection of pixels in the n*n-1 target pixels in each of the M image units to obtain M second pixels, where the M second pixels are all different from the M first pixels.

[0164] That is to say, when the terminal performs the second random selection of pixels for each image unit, the terminal needs to first determine the pixels in each image unit that have not been selected as the first pixel, that is, the n*n-1 target pixels in each image unit. Then, the terminal randomly selects a pixel from these n*n-1 target pixels as the second pixel. In this way, the first pixel and the second pixel selected by the terminal in each image unit must be different pixels. For example, Figure 5 Each pixel in the first sub-image and the second sub-image shown is a different pixel.

[0165] By ensuring that the same pixels as those in the first sub-image are not sampled during the downsampling process of the second sub-image, it can be ensured that the first sub-image and the second sub-image obtained by the random downsampling process are two completely independent images, that is, there is no overly strong correlation between the first sub-image and the second sub-image, so that two images with random noise in the same scene can be better simulated based on the first sub-image and the second sub-image, thereby improving the training effect of the denoising model. For example, when it is not ensured that the same pixels as those in the first sub-image are not sampled during the downsampling process of the second sub-image, in some more extreme cases, each second pixel in the second sub-image may be the same as the first pixel, that is, the second sub-image is the same as the first sub-image. In this case, it is difficult to obtain a good training effect by training the denoising model based on the same two sub-images.

[0166] Optionally, when the terminal performs random selection of pixels among the n*n-1 target pixels in each of the M image units, the terminal can randomly select a second pixel adjacent to the pixel selected in the first random selection among the n*n-1 target pixels in each of the M image units, and obtain M second pixels, each of the M second pixels being adjacent to the corresponding first pixel.

[0167] That is, for the n*n-1 target pixels in each image unit, the terminal can first determine the target pixels adjacent to the first pixel selected when performing the first random selection of pixels among the n*n-1 target pixels. Then, the terminal randomly selects a pixel as the second pixel among the determined target pixels adjacent to the first pixel. In this way, the first pixel and the second pixel selected by the terminal in each image unit must be adjacent pixels.

[0168] For example, see Figure 6 , Figure 6 A schematic diagram of another random downsampling process provided in an embodiment of the present application. Figure 6 As shown, each second pixel in the second sub-image (i.e., pixel 2A, pixel 2B, pixel 2C, pixel 2D) is adjacent to its corresponding first pixel (i.e., pixel 1A, pixel 1B, pixel 1C, pixel 1D), and there is no situation where the second pixel is located diagonally to the first pixel.

[0169] In this embodiment, for each image unit, it can be considered that adjacent pixels in the image unit have a higher similarity. Therefore, by selecting a target pixel adjacent to the first pixel as the second pixel, it can be ensured that the obtained second sub-image has a higher similarity with the first sub-image, that is, based on the first sub-image and the second sub-image, two images with random noise in the same scene are better simulated, thereby improving the training effect of the denoising model.

[0170] Method 2: Based on the downsampling ratio, determine the pixels that need to be sampled in the image sample to be denoised, and randomly sample a pixel from multiple pixels around the pixel, so as to obtain a sub-image composed of multiple sampled pixels.

[0171] Exemplarily, for an image to be denoised with a resolution of W*H, it can be predetermined that the image to be denoised is downsampled into a sub-image with a resolution of (Wx)*(Hx), that is, the resolutions of the first sub-image and the second sub-image are both (Wx)*(Hx). Based on the downsampling ratio, that is, W / (Wx) and H / (Hx), the terminal determines the pixels in the image to be denoised that need to be sampled, and the pixels need to be used as pixels in the sub-image after the sampling process is performed. Then, the terminal performs sampling on the pixels that need to be sampled, that is, randomly samples a pixel from multiple pixels around the pixel or multiple pixels around the pixel and the pixel, and the sampled pixel is used as a pixel in the sub-image, and finally a sub-image consisting of multiple sampled pixels is obtained. For example, the terminal can randomly select a row from the 1st to the xth row as the starting row for performing pixel sampling processing, and continuously perform sampling processing on the pixels of the Wx row starting from this row; similarly, the terminal can randomly select a column from the 1st to the xth column as the starting column for performing pixel sampling processing, and continuously perform column sampling processing on the pixels of the Hx column starting from this column.

[0172] See also Figure 7 , Figure 7 A schematic diagram of another random downsampling process provided in an embodiment of the present application. Figure 7 As shown in the figure, it is assumed that the image to be denoised is an image with a resolution of 4*4, and the image to be denoised includes 16 pixels, namely pixel 1 to pixel 16. The image to be denoised needs to be downsampled to a sub-image with a resolution of (4-2)*(4-2), that is, the resolution of the sub-image is 2*2. Based on the downsampling ratio, it is determined to take pixels 6, 7, 10 and 11 in the image to be denoised as the target, and perform random downsampling on these four pixels.

[0173] In the process of performing random downsampling processing on pixel 6, multiple pixels around pixel 6 are determined, namely pixel 1, pixel 2, pixel 3, pixel 5, pixel 7, pixel 9, pixel 10 and pixel 11. Then, a pixel is randomly selected from the multiple pixels around pixel 6 as a pixel in the sub-image. For example, Figure 7As shown, when pixel 6 is used as the target for random downsampling, pixel 2 is selected as the pixel in the sub-image. Optionally, a pixel may be randomly selected from a plurality of pixels around pixel 6 and pixel 6 as the pixel in the sub-image, that is, pixel 6 itself may also be selected as the pixel in the sub-image.

[0174] In the process of performing random downsampling processing on pixel 7, multiple pixels around pixel 7 are determined, namely pixel 2, pixel 3, pixel 4, pixel 6, pixel 8, pixel 10, pixel 11 and pixel 12. Then, a pixel is randomly selected from the multiple pixels around pixel 7 as a pixel in the sub-image. For example, Figure 7 As shown, when pixel 7 is targeted for random downsampling, pixel 8 is selected as a pixel in the sub-image.

[0175] In the process of performing random downsampling processing on pixel 10, multiple pixels around pixel 10 are determined, namely pixel 5, pixel 6, pixel 7, pixel 9, pixel 11, pixel 13, pixel 14 and pixel 15. Then, a pixel is randomly selected from the multiple pixels around pixel 10 as a pixel in the sub-image. For example, Figure 7 As shown, when the pixel 10 is used as the target for performing the random downsampling process, the pixel 9 is selected as the pixel in the sub-image.

[0176] In the process of performing random downsampling processing on pixel 11, multiple pixels around pixel 11 are determined, namely pixel 6, pixel 7, pixel 8, pixel 10, pixel 12, pixel 14, pixel 15 and pixel 16. Then, a pixel is randomly selected from the multiple pixels around pixel 11 as a pixel in the sub-image. Figure 7 As shown, when pixel 11 is targeted for random downsampling, pixel 12 is selected as a pixel in the sub-image.

[0177] Step 403: input the first sub-image into a denoising model to obtain a first target image.

[0178] After obtaining the first sub-image, the terminal can input the first sub-image into the denoising model to obtain the first target image output by the denoising model. The denoising model is used to perform denoising on the input image and output a clean image after denoising. The denoising model is a learnable model. Before the denoising model is trained, the denoising ability of the denoising model is poor. During the training process, the denoising module inside the denoising model is continuously optimized, and the denoising ability is continuously enhanced. After the training is completed, the denoising model can be used to achieve image denoising.

[0179] The denoising model includes but is not limited to a learning-based model such as a convolutional neural network or a denoising model based on sparse feature expression. This embodiment does not specifically limit the specific structure of the denoising model.

[0180] Step 404: Acquire a first loss function according to the first target image and the second sub-image, where the first loss function is used to indicate a difference between the first target image and the second sub-image.

[0181] In this embodiment, after randomly downsampling the image to be denoised to obtain the first sub-image and the second sub-image, the first sub-image and the second sub-image can be used as a sample pair for training. The first sub-image is used as the input value of the denoising model, and the second sub-image is used as the expected output value of the denoising model. Based on the actual output value of the denoising model (i.e., the first target image) and the expected output value of the denoising model (i.e., the second sub-image), a first loss function can be obtained, and the first loss function is used to indicate the difference between the first target image and the second sub-image. Based on the first loss function, the denoising model can be guided to learn the denoising capability.

[0182] Exemplarily, assume that the first sub-image is g1(y), the second sub-image is g2(y), and the denoising model is f. After the first sub-image g1(y) is input into the denoising model f, the image output by the denoising model f is f(g1(y)). A possible example of the first loss function is shown in Formula 1.

[0183] loss1=||f(g1(y))-g2(y)|| p Formula 1

[0184] Among them, loss1 is the first loss function, f(g1(y)) is the image output after the denoising model f inputs the first sub-image g1(y), g2(y) is the second sub-image, p is a power, and the value of p can be 1 or 2. It should be understood that f(g1(y))-g2(y) can be understood as subtracting the value of the pixel in the f(g1(y)) image from the value of the corresponding pixel in the g2(y) image. Generally speaking, the value of the pixel in the image can be 0-255 or 0-4095, and different values ​​are used to represent different colors.

[0185] Step 405: Train the denoising model at least according to the first loss function to obtain a target denoising model.

[0186] After obtaining the first loss function, the terminal can train the denoising model based on the value of the first loss function. The process of the terminal training the denoising model based on the first loss function includes: the terminal adjusts the parameters in the denoising model based on the value of the first loss function, and repeatedly executes steps 401-405, so as to continuously adjust the parameters of the denoising model terminal until the first loss function obtained is less than the preset threshold, and it can be determined that the model training conditions have been met, and the target denoising model is obtained. The target denoising model is a trained denoising model, which can be used for subsequent image denoising.

[0187] Optionally, in a possible embodiment, the terminal may also train the noise reduction model based on the first loss function and the second loss function.

[0188] Exemplarily, before training the denoising model, the terminal may input the sample of the image to be denoised into the denoising model to obtain a second target image output by the denoising model. Based on the sampling position of the pixel in the first random downsampling process, the terminal performs downsampling processing on the second target image to obtain a first sub-target image. Based on the sampling position of the pixel in the second random downsampling process, the terminal performs downsampling processing on the second target image to obtain a second sub-target image. Wherein, the terminal performs downsampling processing on the second target image based on the sampling position of the pixel in the first random downsampling process means that when the terminal performs downsampling processing on the second target image, the terminal adopts the same method as the first random downsampling processing to perform downsampling processing.

[0189] by Figure 5 For example, the first random downsampling process is to collect pixels at the upper left corner of the first image unit, collect pixels at the upper right corner of the second image unit, collect pixels at the lower right corner of the third image unit, and collect pixels at the lower left corner of the fourth image unit. Then, based on the same method as the first random downsampling process, the terminal can divide the second target image into four image units based on the same method, and collect corresponding pixels in the four image units of the second target image according to the position of the pixels collected in each image unit during the first random downsampling process, so as to obtain the first sub-target image. In other words, the first sub-target image and each group of corresponding pixels in the first sub-image have the same position in the source image before downsampling.

[0190] In practical applications, after the terminal performs the first random downsampling process on the sample of the image to be denoised, the terminal can record the position of each first pixel collected when performing the first random downsampling process. Then, based on the position of each first pixel, the terminal determines the corresponding position in the second target image, and collects pixels at the corresponding position in the second target image as pixels on the first sub-target image. In addition, the terminal can also generate a first sampler for performing the first random downsampling process and a second sampler for performing the second random downsampling process, respectively, wherein the way the first sampler and the second sampler perform downsampling is fixed. In this way, the terminal can perform downsampling processing on the second target image based on the first sampler to obtain the first sub-target image; and the terminal can perform downsampling processing on the second target image based on the second sampler to obtain the second sub-target image.

[0191] The terminal obtains a second loss function according to the first target image, the second sub-image, the first sub-target image and the second sub-target image. The second loss function is mainly used to indicate the difference between the first sub-target image and the second sub-target image.

[0192] Exemplarily, assume that the first sub-image is g1(y), the second sub-image is g2(y), and the denoising model is f. After the first sub-image g1(y) is input into the denoising model f, the image output by the denoising model f is f(g1(y)). The first sub-image is g1(f(y)), and the second sub-image is g2(f(y)). A possible example of the second loss function is shown in Formula 2.

[0193] loss2=||f(g1(y))-g2(y)-(g1(f(y))-g2(f(y)))|| p Formula 2

[0194] Wherein, loss2 is the second loss function, f(g1(y)) is the image output by the denoising model f after inputting the first sub-image g1(y), g2(y) is the second sub-image, g1(f(y)) is the first sub-image, g2(f(y)) is the second sub-image, and p is the power.

[0195] It can be understood that the second loss function is a correction term for the inconsistent positions of the two sub-images during the downsampling process. Its purpose is to constrain the denoising model so that the denoising model does not generate an overly smoothed image due to the inconsistent positions of the pixels corresponding to the first sub-image and the second sub-image in the sample of the image to be denoised. In short, the clean images corresponding to the first sub-image and the second sub-image obtained after two random downsampling processes are not exactly the same. The clean images corresponding to the first sub-image and the second sub-image are only in a neighbor-like relationship. On the basis of using only these two sub-images to construct the loss function, the denoising ability learned by the denoising model will not only erase the noise in the image, but also the difference between the true value images (i.e., clean images) corresponding to the two sub-images will be erased. That is, the denoising ability of the denoising model is too strong, resulting in the high-frequency detail information in the noisy image being treated as noise information and processed. Therefore, on the basis of the first loss function, the introduction of the second loss function can protect the high-frequency detail information in the noisy image from being removed by the denoising model, thereby ensuring the resolution of the denoised image.

[0196] Finally, after calculating the first loss function and the second loss function, the terminal trains the denoising model at least according to the first loss function and the second loss function until the denoising model meets the model training condition. Exemplarily, the terminal may add the first loss function and the second loss function to obtain a total loss function, and then train the denoising model based on the total loss function.

[0197] Optionally, the first loss function and the second loss function also have corresponding weight coefficients, that is, the terminal calculates the total loss function based on the weight coefficients corresponding to the first loss function and the second loss function. In other words, the terminal trains the denoising model at least according to the first loss function, the first weight coefficient, the second loss function, and the second weight coefficient; wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0198] Exemplarily, the process of calculating the total loss function based on the first loss function, the first weight coefficient, the second loss function and the second weight coefficient can be as shown in Formula 3.

[0199] Total loss = a*loss1*b*loss2 Formula 3

[0200] Among them, loss is the total loss function, loss1 is the first loss function, loss2 is the second loss function, a is the first weight coefficient, and b is the second weight coefficient. Optionally, the ratio between a and b can be a value such as 1 or 1 / 2. In actual application, the first weight coefficient and / or the second weight coefficient can be adjusted according to the actual noise reduction requirements. Among them, the larger the second weight coefficient, the weaker the noise reduction strength, the weaker the noise reduction strength, the more noise, but the loss of high-frequency details of the image will also be less; the smaller the second weight coefficient, the stronger the noise reduction strength, the stronger the noise reduction strength, the smaller the noise, but the loss of high-frequency details of the image will also be more. Therefore, in actual applications, the first weight coefficient and / or the second weight coefficient can be adjusted to achieve a balance between the noise reduction strength and the degree of loss of high-frequency details.

[0201] The above describes the process of training the denoising model. For ease of understanding, the following will introduce the principle of training the denoising model based on the sub-image pairs obtained by random downsampling.

[0202] When two noisy images are used to form a sample pair, if the clean images corresponding to the two noisy images are consistent, a denoising model with denoising ability can be trained based on a large number of such sample pairs. It can be understood that when there are few samples used to train the denoising model, the denoising model actually learns the conversion relationship between the two noise modes. When there are enough samples used to train the denoising model, since the noise in the sample is random and fluctuates around the true value, then from the perspective of minimizing the loss function, it can be found that the denoising model can learn a clean and noise-free image. Because it is impossible for the denoising model to learn a certain noise conversion law to minimize the loss function, because the noise is always random. The way the denoising model minimizes the loss function is to convert the randomly fluctuating noise in the sample into an intermediate value, and the intermediate value of a large number of noises is actually the true value corresponding to the noise. Therefore, based on the sample pairs composed of noisy images, a denoising model with denoising ability can be trained.

[0203] For ease of understanding, the specific derivation process is introduced below.

[0204] For the indoor temperature estimation problem: Assume that a series of observed temperatures (y1, y2, y3, y4…) are obtained in a certain way. Then, the real temperature z can be obtained based on a series of observed temperatures, which can be modeled as Formula 4.

[0205]

[0206] Among them, argmin represents the expectation of minimizing the loss function L, which is a function of z. For all observed temperatures y, the loss must be minimized. Therefore, L can be regarded as a probability distribution with y as the variable. Minimizing the loss of all samples is actually minimizing the mean of the loss of all samples. If the distance metric is the L2 norm, then the true temperature z is actually the mean of y. Therefore, the true temperature z is actually the mean of y. As for what the observed temperature y(i) is each time, it does not matter at all. The goal of optimization is to consider the mean of all observations.

[0207] Then, for the image denoising problem, assuming that the input noisy image is x(i) and the output clear image is y(i), then the image denoising problem can be modeled as Formula 5.

[0208]

[0209] In Formula 5, θ is actually the weight parameter of the denoising model. Also, x and y are not independent of each other, so Formula 5 can be transformed into Formula 6.

[0210]

[0211] Similarly, if the distribution of p(y / x) is changed, as long as the conditional expectation remains unchanged, the final θ will not change. Therefore, if a Gaussian noise with a mean of 0 is added to the label y as a disturbance to obtain y', y' can be considered as the expected value with noise. In other words, by using the noisy image as the output expected value of the denoising model, a denoising model with denoising ability can be trained.

[0212] Based on similar ideas, in this embodiment, the same noise image is subjected to two random downsampling processes to obtain two sub-images. These two sub-images can be used to simulate two images with random noise in the same scene. Therefore, based on the sub-images obtained by the downsampling process, the training of the denoising model can also be realized. Compared with using two noise images corresponding to the same clean image as sample pairs to train the denoising model, this embodiment obtains sub-images as sample pairs by performing two random downsampling processes on the same noise image, which makes it easier to obtain sample pairs. Because in some special scenes or fields, two noise images corresponding to the same clean image are actually difficult to obtain. For example, in the field of terminal photography, since objects in dynamic scenes are constantly moving, it is actually very difficult to continuously obtain two noise images in dynamic scenes and require that the clean images corresponding to the two noise images are the same. For another example, in the field of medical images, since obtaining medical images corresponding to human body parts based on medical instruments will have certain negative effects on the human body, it is also very difficult to continuously obtain two noise images through medical instruments.

[0213] For ease of understanding, the training method of the denoising model provided in this embodiment will be introduced below with reference to specific examples.

[0214] See also Figure 8 and Fig. 9 , Figure 8 A schematic diagram of a training process of a noise reduction model provided in an embodiment of the present application; Fig. 9 A schematic diagram of another training process of a noise reduction model provided in an embodiment of the present application. Figure 8 As shown, the training process of the denoising model includes steps 801-809.

[0215] Step 801: Select a noise image y from a sample set.

[0216] First, the terminal obtains a sample set including a large number of noise images, and selects a noise image y that is not used for training from the sample set.

[0217] Step 802, construct sampler g1 and sampler g2.

[0218] In this embodiment, the terminal may pre-construct a sampler g1 and a sampler g2. Sampler g1 is used to perform a first random downsampling process to downsample the noise image y. Sampler g2 is used to perform a second random downsampling process to downsample the noise image y for a second time.

[0219] Step 803: down-sample the noise image y through the sampler g1 and the sampler g2 to obtain the noise sub-image g1(y) and the noise sub-image g2(y).

[0220] Specifically, the process of performing random down-sampling processing on the noise image y by the sampler g1 and the sampler g2 can refer to the above step 402, which will not be repeated here.

[0221] Step 804: reduce the noise of the noise sub-image g1(y) by using a noise reduction module to obtain a noise-reduced image f(g1(y)).

[0222] After obtaining the noise sub-image g1(y), the noise sub-image g1(y) can be input into a denoising module, and the denoising module performs denoising on the noise sub-image g1(y) to obtain a denoised image f(g1(y)). The denoising module can be the above-mentioned learning-based denoising model, such as a convolutional neural network or a denoising model based on sparse feature expression.

[0223] Step 805: reduce the noise image y using a noise reduction module to obtain a denoised image f(y).

[0224] Step 806: down-sample the denoised image f(y) by using samplers g1 and g2 to obtain denoised sub-image g1(f(y)) and denoised sub-image g2(f(y)).

[0225] After obtaining the denoised image f(y), samplers g1 and g2 are used to perform downsampling processing on the denoised image f(y) to obtain denoised sub-image g1(f(y)) and denoised sub-image g2(f(y)), respectively.

[0226] Step 807, calculate the loss function.

[0227] The loss function is calculated by the first loss function, the second loss function and the weight coefficients corresponding to the first loss function and the second loss function. The first loss function can be calculated based on the denoised image f(g1(y)) and the noise sub-image g2(y); the second loss function can be calculated based on the denoised image f(g1(y)), the noise sub-image g2(y), the denoised sub-image g1(f(y)) and the denoised sub-image g2(f(y)). The calculation process of the first loss function and the second loss function can refer to the above-mentioned step 404, which will not be repeated here.

[0228] Step 808: Update the parameters of the noise reduction module.

[0229] After the loss function is calculated, the parameters of the denoising module can be adaptively updated based on the value of the loss function.

[0230] Step 809, determining whether the noise reduction module has converged.

[0231] Based on the value of the loss function, it is determined whether the denoising module has converged, that is, whether the denoising module has met the training conditions. If the denoising module has converged, it is considered that the denoising module has been trained, and the denoising module can be output as a trained denoising module; if the denoising module has not converged, it is turned to continue to execute steps 801-808 and continue to train the denoising module until the denoising module converges.

[0232] In order to verify the effect of the training method of the denoising model provided in this embodiment, this embodiment provides a plurality of experiments to evaluate the denoising effect of the denoising model trained by the training method of the denoising model.

[0233] See also Fig.10 , Fig.10 A schematic diagram of an experimental process for determining the noise reduction effect of a noise reduction model provided in an embodiment of the present application. Fig.10As shown, first obtain a batch of clean images without noise, then randomly synthesize noise in the clean images to obtain a batch of noisy images. Then input the noisy images into the denoising network trained based on the above training method to obtain the denoised images, i.e., denoised images. Finally, based on the denoised images and the clean images corresponding to the denoised images, calculate the indicators between the two, determine the gap between the denoised images and the clean images, and thus determine the denoising effect of the denoising network.

[0234] Among them, synthetic noise is a means commonly used to measure noise reduction methods, and noisy images are synthesized by artificially adding Gaussian noise, Poisson noise, etc. In this embodiment, the ImageNet data set is used as training data, which contains 50,000 high-definition pictures, covering most scenes of daily life. At the same time, the Kodak image (Kodak) data set, the (Set14) data set of low-complexity single image super-resolution embedded in non-negative neighborhoods, and the Berkeley image segmentation (BSD300) data set are also selected as test sets to facilitate horizontal comparison with other noise reduction methods. For the noise reduction network, a U-Net network is built on the PyTorch platform as a noise reduction network. In addition, in order to evaluate the quality of the output results, the image without noise is used as a clean image, and the peak signal-to-noise ratio (Peak signal-to-noiseRatio, PSNR) of each test image is calculated respectively, and finally the average PSNR of the entire test set is calculated.

[0235] Specifically, the specific implementation steps are as follows.

[0236] 1. Construct a training set by extracting 50,000 high-definition images from the ImageNet validation set. Construct a test set by adding noise to the Kodak dataset, Set14 dataset, and BSD300 dataset.

[0237] 2. Construct a denoising network. In this embodiment, the UNet network is used as the denoising network.

[0238] 3. Construct a downsampler, a first loss function, and a second loss function. This embodiment uses a random downsampler. At the same time, the pixels at the same position of the two sampled sub-images are adjacent to each other in the original image. The weight coefficient of the first loss function is set to 1, and the weight coefficient of the second loss function is set to 2.

[0239] 4. Training the network: Based on the constructed training set, denoising network, sampler and loss function, the denoising network is trained to convergence using the denoising model training method provided in this embodiment.

[0240] 5. Use the trained denoising network to denoise the test set and generate a denoised image.

[0241] 6. Calculate the PSNR between the denoised images and the clean images in the test set.

[0242] In order to verify the effectiveness of the training method of the denoising model provided in the embodiment of the present application, the present method is compared with several current mainstream image denoising methods. Multiple models are trained based on the training set, and then tested on the Kodak, BSD300 and Set14 datasets to calculate their PSNR. The comparison results of various methods are shown in Table 1 and Fig.11 As shown, the higher the PSNR, the better the noise reduction effect. Fig.11 A schematic diagram comparing the noise reduction effects of various noise reduction methods provided in the embodiments of the present application.

[0243] Table 1

[0244]

[0245]

[0246] As can be seen from Table 1, the denoising effect of the denoising model training method provided in the embodiment of the present application is slightly lower than that of the Noise2Noise method, but is higher than that of other methods. Among them, the Noise2Noise method is the method mentioned above that needs to be trained based on multiple noisy images with the same clean image. However, in practical applications, multiple noisy images with the same clean image are actually very difficult. Therefore, compared with the denoising model training method provided in the embodiment of the present application, the Noise2Noise method is difficult to implement in most scenarios.

[0247] The above experiment synthesizes noise on a clean image to obtain a noisy image and a clean image corresponding to the noisy image. The following will take the terminal photo taking scene as an example to experiment with the training method of the noise reduction model provided in the embodiment of the present application.

[0248] See also Fig.12 , Fig.12 A schematic diagram of another experimental process for determining the noise reduction effect of a noise reduction model provided in an embodiment of the present application. Fig.12 As shown, firstly, multiple noisy images of the same scene are acquired through the camera sensor, namely Fig.12Noise image 1, noise image 2...noise image N shown in . These multiple noise images can be collected by the camera in a scene with poor light, so these multiple noise images all include obvious noise. Then, a noise image is randomly selected from these multiple noise images and input into the denoising network trained based on the above training method to obtain the denoised image, that is, the denoised image. Finally, based on the denoised image and the clean image corresponding to the denoised image, the index between the two is calculated to determine the difference between the denoised image and the clean image, thereby determining the denoising effect of the denoising network. Among them, the clean image can be obtained by averaging these multiple noise images. Since the camera sensor cannot obtain a clean image, by averaging multiple noise images, an image as close to the real clean image as possible can be obtained.

[0249] Specifically, the specific implementation steps are as follows.

[0250] 1. Construct a training set and a test set. In this embodiment, a Smartphone Image Denoising Dataset (SIDD) dataset constructed based on a real mobile phone sensor is used. The SIDD dataset is a real scene dataset collected using a mobile phone camera sensor in a dark scene, and its corresponding clean image is obtained by weighted averaging of multiple noisy images.

[0251] 2. Construct a denoising network. This embodiment uses the UNet network model as the denoising network.

[0252] 3. Construct a downsampler, a first loss function, and a second loss function. This embodiment uses a random downsampler. At the same time, the pixels at the same position of the two sampled sub-images are adjacent to each other in the original image. The weight coefficients of the first loss function and the second loss function are both set to 1.

[0253] 4. Training the network: Based on the constructed training set, denoising network, sampler and loss function, the denoising network is trained to convergence using the denoising model training method provided in this embodiment.

[0254] 5. Use the trained denoising network to denoise the test set and generate a denoised image.

[0255] 6. Calculate the PSNR between the denoised images and the clean images in the test set.

[0256] In order to verify the effectiveness of the training method of the denoising model provided in the embodiment of the present application, this method was compared horizontally with several current mainstream image denoising methods. Multiple models were trained based on the SIDD dataset, and the average PSNR and structural similarity index measurement (SSIM) on the SIDD verification data and the benchmark data were calculated. Finally, the comparison of indicators and the actual denoising effect was performed. The comparison results of various methods are shown in Figure 2. Fig.13 and Fig.14 As shown in the figure, the higher the PSNR and SSIM, the better the noise reduction effect. Fig.13 A schematic diagram comparing noise reduction indicators of various noise reduction methods provided in the embodiments of the present application; Fig.14 A schematic diagram comparing the noise reduction effects of various noise reduction methods provided in the embodiments of the present application. Fig.13 and Fig.14 It can be seen that the PSNR and SSIM of the denoising model training method provided in the embodiment of the present application are higher than those of other methods, that is, the denoising effect is better than that of other methods.

[0257] See also Fig.15 , Fig.15 The following is a flow chart of an image noise reduction method provided in an embodiment of the present application. Fig.15 As shown, the image noise reduction method includes steps 1501-1502.

[0258] Step 1501: Obtain an image to be denoised.

[0259] The image to be denoised is a noisy image that contains noise and needs to be denoised during actual application. For example, the image to be denoised can be an image taken by a terminal, a medical image, or a monitoring image. This embodiment does not specifically limit the type of the image to be denoised.

[0260] Step 1502: input the image to be denoised into a target denoising model to obtain a denoised image.

[0261] Among them, the target denoising model is obtained by training the denoising model based on at least a first loss function, the first loss function is obtained based on the first target image and the second sub-image, the first loss function is used to indicate the difference between the first target image and the second sub-image, the first target image is obtained after the first sub-image is input into the denoising model, the first sub-image and the second sub-image are obtained after performing the first random downsampling process and the second random downsampling process on the denoised image sample, respectively, and the first sub-image has the same resolution as the second sub-image. In short, the target denoising model is obtained by training based on the denoising model training method described in the above embodiment, and the specific training process can refer to the description of steps 401-405, which will not be repeated here.

[0262] Optionally, the first sub-image is obtained according to M first pixels, and the M first pixels are obtained by performing a first random selection of pixels in each of the M image units after dividing the image sample to be denoised into M image units;

[0263] The second sub-image is obtained based on M second pixels, and the M second pixels are obtained by performing a second random selection of pixels in each of the M image units after dividing the image sample to be denoised into the M image units.

[0264] Optionally, the M second pixels are obtained by performing random selection of pixels among n*n-1 target pixels in each of the M image units, the n*n-1 target pixels are pixels in each image unit that are not selected when performing the first random selection of pixels, and the M second pixels are different from the M first pixels.

[0265] Optionally, the M second pixels are obtained by randomly selecting a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, and each of the M second pixels is adjacent to the corresponding first pixel.

[0266] Optionally, the target denoising model is obtained by training the denoising model based on at least the first loss function and the second loss function, the second loss function is obtained according to the first target image, the second sub-image, the first sub-target image and the second sub-target image, the first sub-target image is obtained by downsampling the second target image based on the sampling positions of pixels in the first random downsampling process, the second sub-target image is obtained by downsampling the second target image based on the sampling positions of pixels in the second random downsampling process, and the second target image is obtained by inputting the sample of the image to be denoised into the denoising model.

[0267] Optionally, the target denoising model is obtained by training the denoising model based on at least a first weight coefficient of the first loss function, the second loss function and a second weight coefficient; wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0268] Optionally, the denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

[0269] See also Fig.16 , Fig.16 This is a schematic diagram of the structure of a model training device provided in an embodiment of the present application. Fig.16 As shown, the model training device includes: an acquisition unit 1601 and a processing unit 1602. The acquisition unit 1601 is used to acquire a sample of an image to be denoised from a sample set, wherein the sample set includes a plurality of sample of an image to be denoised; the processing unit 1602 is used to perform a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively obtaining a first sub-image and a second sub-image, wherein the first sub-image has the same resolution as the second sub-image; the processing unit 1602 is also used to input the first sub-image into a denoising model to obtain a first target image; the processing unit 1602 is also used to acquire a first loss function according to the first target image and the second sub-image, wherein the first loss function is used to indicate the difference between the first target image and the second sub-image; the processing unit 1602 is also used to train the denoising model at least according to the first loss function to obtain a target denoising model.

[0270] Optionally, in a possible implementation, the processing unit 1602 is further used to: divide the image sample to be denoised into M image units, each of the M image units including n*n pixels; perform a first random selection of pixels in each of the M image units to obtain M first pixels, and obtain the first sub-image based on the M first pixels; perform a second random selection of pixels in each of the M image units to obtain M second pixels, and obtain the second sub-image based on the M second pixels.

[0271] Optionally, in a possible implementation, the acquisition unit 1601 is further used to acquire n*n-1 target pixels in each of the M image units, where the n*n-1 target pixels are pixels in each image unit that have not been selected when the first random selection of pixels is performed; the processing unit 1602 is further used to perform random selection of pixels in the n*n-1 target pixels in each of the M image units to obtain M second pixels, and the M second pixels are all different from the M first pixels.

[0272] Optionally, in a possible implementation, the processing unit 1602 is further used to randomly select a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, to obtain M second pixels, each of the M second pixels being adjacent to the corresponding first pixel.

[0273] Optionally, in a possible implementation, the processing unit 1602 is further used to: input the image sample to be denoised into the denoising model to obtain a second target image; based on the sampling position of the pixel in the first random downsampling process, downsample the second target image to obtain a first sub-target image; based on the sampling position of the pixel in the second random downsampling process, downsample the second target image to obtain a second sub-target image; obtain a second loss function based on the first target image, the second sub-image, the first sub-target image and the second sub-target image; and train the denoising model at least based on the first loss function and the second loss function.

[0274] Optionally, in a possible implementation, the processing unit 1602 is also used to train the denoising model based on at least the first loss function, the first weight coefficient, the second loss function and the second weight coefficient; wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0275] Optionally, in a possible implementation, the denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

[0276] See also Fig.17 , Fig.17 Schematic diagram of the structure of an image noise reduction device provided in an embodiment of the present application. Fig.17 As shown, the image denoising device includes: an acquisition unit 1701 and a processing unit 1702. The acquisition unit 1701 is used to acquire the image to be denoised; the processing unit 1702 is used to input the image to be denoised into a target denoising model to obtain a denoised image; wherein the target denoising model is obtained by training the denoising model based on at least a first loss function, the first loss function is obtained based on a first target image and a second sub-image, the first loss function is used to indicate the difference between the first target image and the second sub-image, the first target image is obtained after the first sub-image is input into the denoising model, the first sub-image and the second sub-image are obtained after performing a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively, and the resolution of the first sub-image is the same as that of the second sub-image.

[0277] Optionally, in a possible implementation, the first sub-image is obtained based on M first pixels, and the M first pixels are obtained by performing a first random selection of pixels in each of the M image units after the image sample to be denoised is divided into M image units; the second sub-image is obtained based on M second pixels, and the M second pixels are obtained by performing a second random selection of pixels in each of the M image units after the image sample to be denoised is divided into the M image units.

[0278] Optionally, in a possible implementation, the M second pixels are obtained by performing random selection of pixels among n*n-1 target pixels in each of the M image units, the n*n-1 target pixels are pixels in each image unit that are not selected when performing the first random selection of pixels, and the M second pixels are different from the M first pixels.

[0279] Optionally, in a possible implementation, the M second pixels are obtained by randomly selecting a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, and each of the M second pixels is adjacent to the corresponding first pixel.

[0280] Optionally, in a possible implementation, the target denoising model is obtained by training the denoising model based on at least the first loss function and the second loss function, the second loss function is obtained according to the first target image, the second sub-image, the first sub-target image and the second sub-target image, the first sub-target image is obtained by downsampling the second target image based on the sampling position of the pixel in the first random downsampling process, the second sub-target image is obtained by downsampling the second target image based on the sampling position of the pixel in the second random downsampling process, and the second target image is obtained by inputting the sample of the image to be denoised into the denoising model.

[0281] Optionally, in a possible implementation, the target denoising model is obtained by training the denoising model based on at least a first weight coefficient of the first loss function, the second loss function, and a second weight coefficient; wherein the first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

[0282] Optionally, in a possible implementation, the denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

[0283] Next, an execution device provided by an embodiment of the present application is introduced. Fig.18 , Fig.18 This is a schematic diagram of the structure of the execution device provided in the embodiment of the present application. The execution device 1800 can be specifically a mobile phone, a tablet, a laptop, a smart wearable device, a server, etc., which is not limited here. Among them, the execution device 1800 can be deployed with Fig.18 The data processing device described in the corresponding embodiment is used to implement Fig.18 The data processing function in the corresponding embodiment. Specifically, the execution device 1800 includes: a receiver 1801, a transmitter 1802, a processor 1803 and a memory 1804 (wherein the number of the processor 1803 in the execution device 1800 can be one or more, Fig.18 In the example of FIG. 1803 , the processor 1803 may include an application processor 18031 and a communication processor 18032. In some embodiments of the present application, the receiver 1801, the transmitter 1802, the processor 1803 and the memory 1804 may be connected via a bus or other means.

[0284] The memory 1804 may include a read-only memory and a random access memory, and provides instructions and data to the processor 1803. A portion of the memory 1804 may also include a non-volatile random access memory (NVRAM). The memory 1804 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or extended sets thereof, wherein the operation instructions may include various operation instructions for implementing various operations.

[0285] The processor 1803 controls the operation of the execution device. In a specific application, the various components of the execution device are coupled together through a bus system, wherein the bus system includes not only a data bus but also a power bus, a control bus, and a status signal bus, etc. However, for the sake of clarity, various buses are referred to as bus systems in the figure.

[0286] The method disclosed in the above embodiment of the present application can be applied to the processor 1803, or implemented by the processor 1803. The processor 1803 can be an integrated circuit chip with signal processing capabilities. In the implementation process, each step of the above method can be completed by the hardware integrated logic circuit or software instructions in the processor 1803. The above processor 1803 can be a general processor, a digital signal processor (digital signal processing, DSP), a microprocessor or a microcontroller, and can further include an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field-programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The processor 1803 can implement or execute the various methods, steps and logic block diagrams disclosed in the embodiments of the present application. The general processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiment of the present application can be directly embodied as a hardware decoding processor to execute, or the hardware and software modules in the decoding processor can be combined and executed. The software module may be located in a storage medium mature in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory 1804, and the processor 1803 reads the information in the memory 1804 and completes the steps of the above method in combination with its hardware.

[0287] The receiver 1801 can be used to receive input digital or character information and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1802 can be used to output digital or character information through the first interface; the transmitter 1802 can also be used to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1802 can also include a display device such as a display screen.

[0288] In one embodiment of the present application, the processor 1803 is configured to execute Figure 4 The training method of the denoising model is performed by the execution device in the corresponding embodiment.

[0289] The present application also provides a training device. Fig.19 , Fig.19 A structural diagram of a training device provided in an embodiment of the present application, specifically, the training device 1900 is implemented by one or more servers, and the training device 1900 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPU) 1919 (for example, one or more processors) and memory 1932, one or more storage media 1930 (for example, one or more mass storage devices) storing application programs 1942 or data 1944. Among them, the memory 1932 and the storage medium 1930 can be short-term storage or permanent storage. The program stored in the storage medium 1930 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations in the training device. Furthermore, the central processing unit 1919 can be configured to communicate with the storage medium 1930 to execute a series of instruction operations in the storage medium 1930 on the training device 1900.

[0290] The training device 1900 may also include one or more power supplies 1926, one or more wired or wireless network interfaces 1950, one or more input and output interfaces 1958; or one or more operating systems 1941, such as Windows Server 2003 or Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0291] Specifically, the training device can perform Figure 4 The steps in the corresponding embodiments.

[0292] Also provided in an embodiment of the present application is a computer program product which, when executed on a computer, enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0293] A computer-readable storage medium is also provided in an embodiment of the present application, which stores a program for signal processing. When the computer-readable storage medium is run on a computer, it enables the computer to execute the steps executed by the aforementioned execution device, or enables the computer to execute the steps executed by the aforementioned training device.

[0294] The execution device, training device or terminal device provided in the embodiments of the present application may specifically be a chip, and the chip includes: a processing unit and a communication unit, wherein the processing unit may be, for example, a processor, and the communication unit may be, for example, an input / output interface, a pin or a circuit, etc. The processing unit may execute the computer execution instructions stored in the storage unit so that the chip in the execution device executes the data processing method described in the above embodiment, or so that the chip in the training device executes the data processing method described in the above embodiment. Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit may also be a storage unit located outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0295] For details, please refer to Fig. 20 , Fig. 20 A schematic diagram of the structure of a chip provided in an embodiment of the present application, the chip can be expressed as a neural network processor NPU 2000, NPU 2000 is mounted on the host CPU (Host CPU) as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 2003, which is controlled by the controller 2004 to extract matrix data from the memory and perform multiplication operations.

[0296] In some implementations, the operation circuit 2003 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 2003 is a two-dimensional systolic array. The operation circuit 2003 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 2003 is a general-purpose matrix processor.

[0297] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 2002 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 2001 and performs matrix operation with matrix B. The partial result or final result of the matrix is ​​stored in the accumulator 2008.

[0298] The unified memory 2006 is used to store input data and output data. The weight data is directly transferred to the weight memory 2002 through the direct memory access controller (DMAC) 2005. The input data is also transferred to the unified memory 2006 through the DMAC.

[0299] BIU stands for Bus Interface Unit 2010 , which is used for the interaction between the AXI bus and the DMAC and the instruction fetch buffer (IFB) 2009 .

[0300] The bus interface unit 2010 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 2009 to obtain instructions from the external memory, and is also used for the storage unit access controller 2005 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0301] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 2006 or to transfer weight data to the weight memory 2002 or to transfer input data to the input memory 2001.

[0302] The vector calculation unit 2007 includes multiple operation processing units, and further processes the output of the operation circuit 2003 when necessary, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc. It is mainly used for non-convolutional / fully connected layer network calculations in neural networks, such as Batch Normalization, pixel-level summation, upsampling of feature planes, etc.

[0303] In some implementations, the vector calculation unit 2007 can store the processed output vector to the unified memory 2006. For example, the vector calculation unit 2007 can apply a linear function; or a nonlinear function to the output of the operation circuit 2003, such as linear interpolation of the feature plane extracted by the convolution layer, and then for example, a vector of accumulated values ​​to generate an activation value. In some implementations, the vector calculation unit 2007 generates a normalized value, a pixel-level summed value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 2003, for example, for use in a subsequent layer in a neural network.

[0304] An instruction fetch buffer 2009 connected to the controller 2004, for storing instructions used by the controller 2004;

[0305] The unified memory 2006, the input memory 2001, the weight memory 2002 and the instruction fetch memory 2009 are all on-chip memories. The external memory is private to the NPU hardware architecture.

[0306] The processor mentioned in any of the above places may be a general-purpose central processing unit, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the above program.

[0307] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0308] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0309] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0310] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A method for training a denoising model, characterized in that: include: Acquire a sample of an image to be denoised from a sample set, wherein the sample set includes a plurality of samples of an image to be denoised; Performing a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, to obtain a first sub-image and a second sub-image, respectively, wherein the first sub-image has the same resolution as the second sub-image; Inputting the first sub-image into a denoising model to obtain a first target image; Acquire a first loss function according to the first target image and the second sub-image, where the first loss function is used to indicate a difference between the first target image and the second sub-image; The denoising model is trained at least according to the first loss function to obtain a target denoising model.

2. The method according to claim 1, characterized in that The performing a first random downsampling process and a second random downsampling process on the sample of the image to be denoised, respectively obtaining a first sub-image and a second sub-image, comprises: Divide the image sample to be denoised into M image units, each of the M image units includes n*n pixels; Performing a first random selection of pixels in each of the M image units to obtain M first pixels, and obtaining the first sub-image according to the M first pixels; A second random selection of pixels is performed in each of the M image units to obtain M second pixels, and the second sub-image is obtained according to the M second pixels.

3. The method according to claim 2, characterized in that The performing a second random selection of pixels in each of the M image units comprises: Acquire n*n-1 target pixels in each of the M image units, where the n*n-1 target pixels are pixels in each image unit that are not selected when performing the first random selection of pixels; Random pixel selection is performed in n*n-1 target pixels in each of the M image units to obtain M second pixels, where the M second pixels are all different from the M first pixels.

4. The method according to claim 3, characterized in that Performing random selection of pixels among n*n-1 target pixels in each of the M image units comprises: Among the n*n-1 target pixels in each of the M image units, a second pixel adjacent to the pixel selected in the first random selection is randomly selected to obtain M second pixels, each of the M second pixels being adjacent to the corresponding first pixel.

5. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: Inputting the image sample to be denoised into the denoising model to obtain a second target image; Based on the sampling positions of the pixels in the first random downsampling process, downsampling the second target image to obtain a first sub-target image; Based on the sampling positions of the pixels in the second random downsampling process, downsampling the second target image to obtain a second sub-target image; Acquire a second loss function according to the first target image, the second sub-image, the first sub-target image, and the second sub-target image; The step of training the denoising model at least according to the first loss function comprises: The denoising model is trained at least according to the first loss function and the second loss function.

6. The method according to claim 5, characterized in that The training of the denoising model at least according to the first loss function and the second loss function comprises: Training the denoising model at least according to the first loss function, the first weight coefficient, the second loss function and the second weight coefficient; The first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

7. The method according to any one of claims 1 to 4, characterized in that: The denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

8. An image denoising method, characterized in that: include: Acquire an image to be denoised; Inputting the image to be denoised into a target denoising model to obtain a denoised image; Among them, the target denoising model is obtained by training the denoising model based on at least a first loss function, the first loss function is obtained based on the first target image and the second sub-image, the first loss function is used to indicate the difference between the first target image and the second sub-image, the first target image is obtained after the first sub-image is input into the denoising model, the first sub-image and the second sub-image are obtained after performing a first random downsampling process and a second random downsampling process on the denoised image sample respectively, and the resolution of the first sub-image is the same as that of the second sub-image.

9. The method according to claim 8, characterized in that The first sub-image is obtained according to M first pixels, and the M first pixels are obtained by performing a first random selection of pixels in each of the M image units after dividing the image sample to be denoised into M image units; The second sub-image is obtained based on M second pixels, and the M second pixels are obtained by performing a second random selection of pixels in each of the M image units after dividing the image sample to be denoised into the M image units.

10. The method according to claim 9, characterized in that The M second pixels are obtained by performing random selection of pixels among n*n-1 target pixels in each of the M image units, the n*n-1 target pixels are pixels in each image unit that are not selected when performing the first random selection of pixels, and the M second pixels are different from the M first pixels.

11. The method according to claim 10, characterized in that The M second pixels are obtained by randomly selecting a second pixel adjacent to the pixel selected in the first random selection from the n*n-1 target pixels in each of the M image units, and each of the M second pixels is adjacent to the corresponding first pixel.

12. The method according to any one of claims 8 to 11, characterized in that: The target denoising model is obtained by training the denoising model based on at least the first loss function and the second loss function, the second loss function is obtained according to the first target image, the second sub-image, the first sub-target image and the second sub-target image, the first sub-target image is obtained by downsampling the second target image based on the sampling positions of the pixels in the first random downsampling process, the second sub-target image is obtained by downsampling the second target image based on the sampling positions of the pixels in the second random downsampling process, and the second target image is obtained by inputting the sample of the image to be denoised into the denoising model.

13. The method according to claim 12, characterized in that The target denoising model is obtained by training the denoising model based on at least the first weight coefficient of the first loss function, the second loss function and the second weight coefficient; The first weight coefficient is used to indicate the weight of the first loss function, and the second weight coefficient is used to indicate the weight of the second loss function.

14. The method according to any one of claims 8 to 11, characterized in that: The denoising model includes a convolutional neural network or a denoising model based on sparse feature expression.

15. A terminal, characterized in that: The terminal comprises a memory and a processor; the memory stores codes, the processor is configured to execute the codes, and when the codes are executed, the terminal executes the method according to any one of claims 1 to 14.

16. A computer-readable storage medium, characterized in that: The method comprises computer-readable instructions, which, when executed on a computer, cause the computer to perform the method according to any one of claims 1 to 14.

17. A computer program product, characterized in that The method comprises computer-readable instructions, which, when executed on a computer, cause the computer to execute the method according to any one of claims 1 to 14.

Images