Image noise reduction processing method, apparatus, device, storage medium, and program product

Abstract

The present application relates to an image noise reduction processing method, apparatus, device, storage medium, and program product. The method includes inputting target image data including pixel values of each channel of a target image into an image noise reduction model to obtain noise reduction image data. The image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n upsampling modules that are cascaded and correspond one-to-one to the n downsampling modules. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a first downsampling layer and a first convolutional layer. The second downsampling module includes a second downsampling layer. By adopting this method, it is possible to meet the real-time requirement of the ISP chip and be used for noise reduction processing of video images.

Description

Technical Field

[0001] (Cross - reference to Related Applications) This application claims the priority of a Chinese patent application filed with the State Intellectual Property Office of China on September 16, 2022, with the application number 2022111286666 and the application title "Image Noise Reduction Processing Method, Apparatus, Device, Storage Medium and Program Product", and all of its contents are incorporated herein by reference.

[0002] This application relates to the technical field of image processing, and particularly to an image noise reduction processing method, apparatus, device, storage medium and program product.

Background Art

[0003] The technology related to image noise reduction is an important task in the field of image processing. On the other hand, an ISP (Image Signal Processor) chip is mainly used to process the real - time video images captured by a terminal. In the aspect of image noise reduction processing, the real - time performance requirement for the image noise reduction algorithm is high.

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, the current image noise reduction algorithms are not capable of achieving both an excellent noise reduction effect and meeting the real - time performance requirements of the ISP chip.

Means for Solving the Problems

[0005] In view of this, in order to address the above - mentioned technical problems, it is necessary to provide an image noise reduction processing method, apparatus, device, storage medium and program product that can meet the real - time performance requirements of the ISP chip and perform noise reduction processing on video images.

[0006] In a first aspect, this application provides an image noise reduction processing method. The method includes: Including inputting target image data including pixel values of each channel of a target image into an image noise reduction model to obtain noise reduction image data output from the image noise reduction model, where the image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer, the downsampling model includes n cascaded downsampling modules, the upsampling model includes n upsampling modules that are cascaded and correspond one-to-one to the n downsampling modules, the downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module, the first downsampling module includes a cascaded first downsampling layer and a first convolutional layer, and the second downsampling module includes a second downsampling layer.

[0007] In one embodiment, inputting the target image data into the image noise reduction model to obtain the noise reduction image data output from the image noise reduction model includes inputting the target image data into the downsampling model, each downsampling module in the downsampling model performing downsampling processing on the target image data to obtain downsampling feature data, inputting the downsampling feature data into the upsampling model, each upsampling module in the upsampling model performing upsampling processing on the downsampling feature data to obtain upsampling feature data, and the output layer obtaining the noise reduction image data based on the upsampling feature data and the target image data.

[0008] In one embodiment, the resolution of the image data for each channel of the target image is the same, and each downsampling module in the downsampling model performs downsampling processing on the target image data to obtain downsampling feature data, which means that in the i-th downsampling module, downsampling processing is performed on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module. When i = 1, the input data of the i-th downsampling module is the target image data; when i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module, and the intermediate downsampling feature data output from the last downsampling module is used as the downsampling feature data.

[0009] In one embodiment, for each upsampling module in the upsampling model to perform upsampling processing on the downsampling feature data to obtain upsampling feature data, in the i-th upsampling module, upsampling processing is performed on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is the set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module, and taking the intermediate upsampling feature data output from the last upsampling module as the upsampling feature data.

[0010] In one embodiment, for the output layer to obtain the noise reduction image data based on the upsampling feature data and the target image data, it includes inputting the upsampling feature data and the target image data into the output layer for fusion processing, and obtaining the noise reduction image data output from the output layer.

[0011] In one embodiment, the resolution of the image data for each channel of the target image is different, and the downsampling model further includes an additional downsampling module. Inputting the target image data into the downsampling model, for each downsampling module in the downsampling model to perform downsampling processing on the target image data to obtain downsampling feature data, Input the first-channel pixel values of the target image included in the target image data into the additional downsampling module to obtain channel feature data output from the additional downsampling module; fuse the channel feature data with the second-channel pixel values of the target image included in the target image data to obtain candidate target image data; in the i-th downsampling module, perform downsampling processing on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module, where when i = 1, the input data of the i-th downsampling module is the candidate target image data, and when i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module; and use the intermediate downsampling feature data output from the last downsampling module as the downsampling feature data.

[0012] In one embodiment, the upsampling model further includes an additional upsampling module. Inputting the downsampling feature data into the upsampling model, and each upsampling module in the upsampling model performing upsampling processing on the downsampling feature data to obtain upsampling feature data, In the i-th upsampling module, perform upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is the set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module. Input the first intermediate channel feature data included in the intermediate upsampling feature data output from the last upsampling module corresponding to the first channel pixel value into the additional upsampling module to obtain the upsampling feature data output from the additional upsampling module.

[0013] In one embodiment, the output layer obtaining the noise reduction image data based on the upsampling feature data and the target image data includes: inputting the first channel pixel values in the upsampling feature data and the target image data into the output layer to perform fusion processing to obtain candidate noise reduction image data output from the output layer; and obtaining the noise reduction image data based on the candidate noise reduction image data and the second intermediate channel feature data corresponding to the second channel pixel value included in the intermediate upsampling feature data output from the last upsampling module.

[0014] In one embodiment, the downsampling processing of the input data of the i-th downsampling module to obtain the intermediate downsampling feature data output from the i-th downsampling module includes: Performing downsampling processing on the input data of the $i$-th downsampling module using the first downsampling layer to obtain first downsampling feature data output from the first downsampling layer; performing convolution processing on the first downsampling feature data using the first convolutional layer to obtain first convolutional feature data output from the first convolutional layer; performing downsampling processing on the input data of the $i$-th downsampling module using the second downsampling layer to obtain second downsampling feature data output from the second downsampling layer; and performing fusion processing on the first convolutional feature data and the second downsampling feature data using the fusion module to obtain the intermediate downsampling feature data output from the fusion module.

[0015] In one embodiment, the upsampling module includes a second convolutional layer and an upsampling layer connected in cascade. Performing upsampling processing on the input data of the $i$-th upsampling module to obtain intermediate upsampling feature data output from the $i$-th upsampling module includes performing convolution processing on the input data of the $i$-th upsampling module using the second convolutional layer to obtain second convolutional feature data output from the second convolutional layer; and performing upsampling processing on the second convolutional feature data using the upsampling layer to obtain the intermediate upsampling feature data output from the upsampling layer.

[0016] In one embodiment, the image noise reduction model is used in a RAW image noise reduction module, an RGB image noise reduction module, or a YUV image noise reduction module in an ISP chip. Correspondingly, the format of the target image is the RAW format, the RGB format, or the YUV format.

[0017]

[0017] In one embodiment, the upsampling layer upsamples the input data of the upsampling layer by convolution processing, inverse pooling processing, or interpolation processing.

[0018] As a second aspect, the present application further provides an image noise reduction processing device. The device includes a noise reduction module, The noise reduction module is used to input target image data including pixel values of each channel of a target image into an image noise reduction model and obtain noise reduction image data output from the image noise reduction model. The image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n upsampling modules that are cascaded and correspond one-to-one to the n downsampling modules. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a cascaded first downsampling layer and a first convolutional layer. The second downsampling module includes a second downsampling layer.

[0019] In one embodiment, the noise reduction module specifically, The target image data is input into the downsampling model, and each downsampling module in the downsampling model performs downsampling processing on the target image data to obtain downsampled feature data. The downsampled feature data is input into the upsampling model, and each upsampling module in the upsampling model performs upsampling processing on the downsampled feature data to obtain upsampled feature data. The output layer is used to obtain the noise-reduced image data based on the upsampled feature data and the target image data.

[0020] In one embodiment, the resolution of the image data for each channel of the target image is the same, and the noise reduction module specifically In the i-th downsampling module, downsampling processing is performed on the input data of the i-th downsampling module to obtain intermediate downsampled feature data output from the i-th downsampling module. When i = 1, the input data of the i-th downsampling module is the target image data. When i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampled feature data output from the (i - 1)-th downsampling module. The intermediate downsampled feature data output from the last downsampling module is used as the downsampled feature data.

[0021] In one embodiment, the noise reduction module specifically In the i-th upsampling module, upsampling processing is performed on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module. The intermediate upsampling feature data output from the last upsampling module is used as the upsampling feature data.

[0022] In one embodiment, the noise reduction module specifically is used to input the upsampling feature data and the target image data into the output layer to perform a fusion process, and obtain the noise reduction image data output from the output layer.

[0023] In one embodiment, the resolution of the image data for each channel of the target image is different, and the downsampling model further includes an additional downsampling module. The noise reduction module specifically Input the first-channel pixel values of the target image included in the target image data into the additional downsampling module to obtain channel feature data output from the additional downsampling module, fuse the channel feature data with the second-channel pixel values of the target image included in the target image data to obtain candidate target image data. In the i-th downsampling module, perform downsampling processing on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module. When i = 1, the input data of the i-th downsampling module is the candidate target image data. When i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module, and the intermediate downsampling feature data output from the last downsampling module is used as the downsampling feature data.

[0024] In one embodiment, the upsampling model further includes an additional upsampling module, and specifically, the noise reduction module In the i-th upsampling module, perform upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is set feature data obtained by fusing intermediate upsampling feature data output from the (i - 1)-th upsampling module and intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module. The first intermediate channel feature data included in the intermediate upsampling feature data output from the last upsampling module corresponding to the first channel pixel value is input to the additional upsampling module to obtain the upsampling feature data output from the additional upsampling module.

[0025] In one embodiment, the noise reduction module specifically Input the upsampling feature data and the first channel pixel value in the target image data into the output layer to perform fusion processing, obtain candidate noise reduction image data output from the output layer, and based on the candidate noise reduction image data and the second intermediate channel feature data corresponding to the second channel pixel value included in the intermediate upsampling feature data output from the last upsampling module, it is used to obtain the noise reduction image data.

[0026] In one embodiment, the noise reduction module specifically When the first downsampling layer is used to perform downsampling processing on the input data of the i-th downsampling module to obtain first downsampling feature data output from the first downsampling layer, the first convolutional layer is used to perform convolutional processing on the first downsampling feature data to obtain first convolutional feature data output from the first convolutional layer, the second downsampling layer is used to perform downsampling processing on the input data of the i-th downsampling module to obtain second downsampling feature data output from the second downsampling layer, and the fusion module is used to perform fusion processing on the first convolutional feature data and the second downsampling feature data to obtain the intermediate downsampling feature data output from the fusion module.

[0027] In one embodiment, the upsampling module includes a second convolutional layer and an upsampling layer connected in cascade, and specifically, the noise reduction module is used to perform convolutional processing on the input data of the i-th upsampling module using the second convolutional layer to obtain second convolutional feature data output from the second convolutional layer, and to perform upsampling processing on the second convolutional feature data using the upsampling layer to obtain the intermediate upsampling feature data output from the upsampling layer.

[0028] In one embodiment, the image noise reduction model is used for a RAW image noise reduction module, an RGB image noise reduction module or a YUV image noise reduction module in an ISP chip. Correspondingly, the format of the target image is a RAW format, an RGB format or a YUV format.

[0029] In one embodiment, the upsampling layer up-samples the input data of the upsampling layer by convolution processing, inverse pooling processing, or interpolation processing.

[0030] As a third aspect, the present application further provides an electronic device including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of the above first aspects is implemented.

[0031] As a fourth aspect, the present application further provides a computer-readable storage medium storing a computer program, and when the computer program is executed by a processor, the method according to any one of the above first aspects is implemented.

[0032] As a fifth aspect, the present application further provides a computer program product, which includes a computer program, and when the computer program is executed by a processor, the method according to any one of the above first aspects is implemented.

Advantages of the Invention

[0033] The above image noise reduction processing method, apparatus, device, storage medium, and program product can directly input target image data including pixel values of each channel of a target image into an image noise reduction model, that is, can obtain noise reduction image data output from the image noise reduction model, and can implement noise reduction processing on the target image. Generally, in an ISP chip, it is necessary to perform noise reduction processing based on the pixel values of the Y channel and the pixel values of the UV channel of a YUV image respectively to obtain image data after noise reduction processing. Since the Y channel and the UV channel are processed simultaneously, it is necessary to repeatedly call the image data, and its processing efficiency is low, and the real-time requirement of the ISP chip cannot be satisfied. On the other hand, in the present application, the target image data including the pixel values of each channel of the target image can be directly input into the image noise reduction model for noise reduction processing, that is, since the data of each channel of the target image is subjected to noise reduction processing simultaneously, compared with performing noise reduction processing separately for each channel, the data volume and the calculation amount are significantly reduced, the data processing efficiency can be effectively improved, and the real-time requirement can be satisfied. Furthermore, in the process of noise reduction processing, the information between each channel in the target image can be referred to each other, so that a better noise reduction processing effect can be achieved.In addition, the network structure of the image noise reduction model is simplified and includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n cascaded upsampling modules that correspond one-to-one to the n downsampling modules. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a cascaded first downsampling layer and a first convolutional layer. The second downsampling module includes a second downsampling layer. The simplified image noise reduction model effectively realizes noise reduction processing on target image data, and the image noise reduction model can be fully adapted to perform real-time noise reduction processing on video images in an ISP chip.

[0034] To more clearly illustrate the embodiments of the present invention or the technical solutions according to the prior art, the drawings necessary for describing the embodiments or the prior art will be briefly described. The drawings described below are only embodiments of the present invention, and it is obvious that those skilled in the art can obtain the drawings of other embodiments based on these drawings without creative efforts.

Brief Description of the Drawings

[0035]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Figure 13

Embodiments for Carrying Out the Invention

[0036] Hereinafter, with reference to the drawings in the embodiments of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described. It should be noted that the described embodiments are only some embodiments of the present application, not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0037] To make the objectives, technical solutions and advantages of the present application more clear, the present application will be described in detail below with reference to the appended drawings and embodiments. It should be noted that the specific embodiments described in this specification are only for interpreting the present application, not for limiting the present application.

[0038] In the field of image processing, image noise reduction is an image processing operation that cannot be perfectly processed. Image noise reduction is one of the image restoration techniques, and its purpose is to accurately find the signal value or noise value in the image, or to separate the signal part and the noise part in the image.

[0039] Image noise reduction algorithms are currently mainly divided into conventional algorithms and algorithms based on neural networks. The basic situation of the conventional ones is that the noise reduction effect of the conventional algorithms is low and cannot meet the needs of the noise reduction effect. Neural network algorithms have a very large amount of calculation and are not sufficiently compatible with conventional chips, and cannot meet the needs of the real-time performance of ISP chips.

[0040] In addition, conventional image noise reduction can be divided into spatial domain noise reduction, frequency domain noise reduction, and combined spatial frequency domain noise reduction according to the feature space for separating signal and noise, and can be divided into local area noise reduction and non-local area noise reduction according to the image range used for the noise reduction process. Specific conventional noise reduction methods include mean value filtering, median value filtering, Gaussian filtering, bilateral filtering, non-local area mean value filtering, guided filtering, discrete cosine domain filtering, wavelet transform domain filtering, etc. Conventional noise reduction methods are all based on the simple assumption that the characteristics of signals and noise are different statistically, and use a fixed method to separate signals and noise. Because the assumption about the characteristics of noise is very simple, some signals are mixed in at the same time when separating noise, or noise cannot be completely separated, and noise remains. When the noise is particularly prominent in the actual scene (for example, imaging in a low illumination environment), the noise reduction effect is low.

[0041] In recent years, in addition to conventional image noise reduction algorithms, various neural network-based image noise reduction algorithms have significantly improved the effect of image noise reduction. Some representative networks of such neural networks include the linear DnCNN (Denoising Convolutional Neural Network), CBDNet (Convolutional Blind Denoising Network) including a subnetwork for evaluating the level of noise, RIDNet (Real Image Denoising Based on Feature Attention) based on an attention mechanism, and others. The advantage of neural network-based image noise reduction algorithms is that the effect is significantly improved compared to conventional algorithms. Correspondingly, their disadvantage is that the computational complexity is much higher than that of conventional algorithms, and it is difficult to implement on an ISP (Image Signal Processor) chip with high real-time requirements in actual applications.

[0042] In view of this, in the embodiments of the present application, an image noise reduction processing method that meets the real-time requirements of the ISP chip and can successfully perform noise reduction processing on video images is provided.

[0043] In one embodiment, an image noise reduction processing method is provided. The embodiments of the present application will be described by taking the application of the method to a terminal equipped with an ISP chip as an example. For better understanding, the method may also be applied to a server, or to a system including a terminal and a server, and may be realized through the interaction between the terminal and the server. Specifically, the execution subject of the method may be the ISP chip in the terminal. The terminal may be various computer devices or imaging devices, etc., but is not limited thereto. The server may be realized by an independent server or a server cluster composed of multiple servers.

[0044] In the embodiments of the present application, the method includes inputting target image data into an image noise reduction model to obtain noise reduction image data output from the image noise reduction model, where the target image data includes pixel values of each channel of the target image. Further, as shown in FIG. 1, there is a schematic diagram of the configuration of the image noise reduction model provided by the embodiments of the present application. The image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules, and the upsampling model includes n cascaded upsampling modules that correspond one-to-one to the n downsampling modules. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded together with the first downsampling module and the second downsampling module. The first downsampling module includes a cascaded first downsampling layer and a first convolutional layer, and the second downsampling module includes a second downsampling layer.

[0045] In addition, an ISP (Image Signal Processor) chip is used to obtain an image captured by an image sensor at the front end of a terminal, perform a series of image processing, and output a processed image. Generally, the steps for an ISP chip to process an image transmitted in RAW format from an image sensor include performing defect point correction, dark current correction, lens shading correction, RAW image noise reduction, white balance, color interpolation, etc. on the RAW image to obtain an RGB image, and then obtaining a YUV image through Gamma correction (gamma correction), color correction, conversion from the RGB image to the YUV image, etc. After the YUV image undergoes processing such as noise reduction, edge enhancement, brightness / contrast / hue / saturation adjustment, etc., finally, the image data is encoded to obtain a final output video image. As an option, the image processed by the ISP chip may be a single image or a video image composed of consecutive frames. In the ISP chip, various image processing algorithms are integrated to implement the above-described steps of image processing by the ISP chip. In the embodiments of the present application, the image noise reduction model is an algorithm applied to the ISP chip to implement the steps of image noise reduction processing.

[0046] Note that in FIG. 1, only three downsampling modules and three upsampling modules are taken as examples, but the present application is not limited thereto.

[0047] In the embodiments of the present application, the target image is an image that requires noise reduction processing in the ISP chip, the target image data is the pixel value of each channel of the target image, and also, if the image obtained by the front-end image sensor is a single image, the target image is a single image, and if the image obtained by the front-end image processor is a real-time video image, the target image is a single frame image in the real-time video image.

[0048] As an option, the target image may be in a RAW format, an RGB format, a YUV format, etc., and the embodiments of the present application do not specifically limit this.

[0049] The image noise reduction model can perform different processes according to whether the resolutions of the image data for each channel of the target image to be processed are the same. Specifically, for a target image in which the resolutions of the image data for each channel are the same, the image noise reduction model has single input and output, and can directly perform noise reduction processing on the input target image data. For a target image in which the resolutions of the image data for each channel are not the same, the image noise reduction model has multi-input and output, and the image data of different channels in the target image data can be respectively input into the image noise reduction model for noise reduction processing. Thereby, the image noise reduction model can adapt to various types of target images and perform noise reduction processing.

[0050] Currently, image noise reduction is classified into types such as single-image noise reduction and multi-frame image joint noise reduction. Considering the computational load and data cache of the ISP chip, single-image noise reduction is more suitable for real-time processing scenarios compared to multi-frame image joint noise reduction. In accordance with the actual noise reduction effect, network calculation complexity, data read / write volume, etc., the embodiments of the present application optimize the network configuration and calculation unit according to the U-Net network, and propose a very simplified single-image noise reduction network configuration, that is, the image noise reduction model. The main network configuration of the main body of the image noise reduction model is the U-Net network. The downsampling model of the image noise reduction model is used to perform downsampling processing on the target image data, the upsampling model is used to perform upsampling processing on the feature data obtained after upsampling processing, and the output layer is used to output the noise reduction image data corresponding to the target image based on the data output from the upsampling model and the target image data. The input of each upsampling module is the output data of the previous upsampling module and the output data of the downsampling module corresponding to the upsampling module. Thereby, the deep features and shallow features of the image can be fused to improve the noise reduction effect.

[0051] When the downsampling model is composed of n cascaded downsampling modules, each downsampling module is a multi-convolution parallel module and is used to extract more image features. To more clearly recognize the multi-convolution parallel module, reference is made to FIG. 2, which is a schematic diagram of the structure of the multi-convolution parallel module provided in the embodiments of the present application. Exemplarily, the multi-convolution parallel module includes two downsampling layers 201, one convolutional layer 202, and a fusion layer 203. Optionally, the multi-convolution parallel module 200 may include other numbers of downsampling layers and convolutional layers, and in the embodiments of the present application, this is not specifically limited. In other words, correspondingly, the downsampling module may include other numbers of convolutional layers and downsampling layers in addition to including a first convolutional layer, a first downsampling layer, and a second downsampling layer, and specifically, it can be determined based on parameters such as the computing power and bandwidth of the ISP chip itself, and the embodiments of the present application do not specifically limit this. Note that by including the downsampling module with the above structure provided in the embodiments of the present application, the image noise reduction model can achieve a good downsampling effect, ensure the noise reduction effect, and at the same time meet the real-time requirements of the ISP chip.

[0052] Correspondingly, the number of downsampling modules and upsampling modules in the image noise reduction model can be determined according to parameters such as the computing power and bandwidth of the ISP chip itself, and the embodiments of the present application do not specifically limit this.

[0053] Optionally, the channel fusion method of the fusion module in the downsampling module may select methods such as element-wise addition or channel-wise concatenation, and the embodiments of the present application do not specifically limit this.

[0054] The above image noise reduction processing method can directly input target image data including pixel values of each channel of a target image into an image noise reduction model, that is, noise reduction image data output from the image noise reduction model can be obtained, and noise reduction processing of the target image can be realized. Generally, in an ISP chip, it is necessary to perform noise reduction processing based on the pixel values of the Y channel and the pixel values of the UV channel of a YUV image respectively to obtain image data after noise reduction processing. The Y channel and the UV channel are processed simultaneously, and it is necessary to repeatedly call the image data, and its processing efficiency is low, and the real-time requirement of the ISP chip cannot be satisfied. In this application, target image data including pixel values of each channel of a target image is directly input into an image noise reduction model for noise reduction processing, that is, the data of each channel of the target image is simultaneously subjected to noise reduction processing. Compared with the case where each channel is separately subjected to noise reduction processing, its data volume and calculation amount are significantly reduced, the data processing efficiency is effectively improved, the real-time requirement is satisfied, and in the noise reduction processing process, information between each channel in the target image can refer to each other, thereby achieving a better noise reduction processing effect.In addition, the network structure of the image noise reduction model is simplified and includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n cascaded upsampling modules that correspond one-to-one to the n downsampling modules. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a first convolutional layer and a first downsampling layer. The second downsampling module includes a second downsampling layer. The simplified image noise reduction model can effectively realize noise reduction processing for target image data, and the image noise reduction model is fully adapted to perform real-time noise reduction processing on video images in the ISP chip.

[0055] In one embodiment, the image noise reduction model is used in a RAW image noise reduction module, an RGB image noise reduction module, or a YUV image noise reduction module in the ISP chip. Correspondingly, the format of the target image is the RAW format, the RGB format, or the YUV format.

[0056] Please refer to FIG. 3 showing a schematic diagram of the image processing process of the conventional ISP chip provided by the embodiment of the present application. The processing process of the conventional ISP chip is to obtain a RAW image corresponding to each frame in the video image transmitted by the front-end image sensor, and through defect point correction, dark current correction, lens shading correction, RAW image noise reduction, white balance, color interpolation, etc., to obtain an RGB image, and further through Gamma correction, color correction, conversion from RGB to YUV, etc., to obtain a YUV image. The Y (brightness) channel data of the YUV image is subjected to noise reduction processing to perform edge enhancement and brightness / contrast adjustment, and the data of the UV (color) channel of the YUV image is subjected to hue / saturation adjustment to realize noise reduction processing of the YUV image, and then the YUV image after noise reduction processing is encoded to obtain the finally output video image.

[0057] The RAW image is the collection format of the image sensor and is essentially a special RGB format. After a series of processes, the RAW image is color-interpolated to obtain a general RGB image, and after further a series of processes, it is converted to obtain a YUV image. The YUV image format is a format that separates the brightness and color of the image. Also, the brightness is indicated by the Y channel, and the color is indicated by two channels of UV. In the processing of the YUV image, the Y channel and the UV channel are processed separately, and a noise reduction algorithm is used for the Y (brightness) channel and the UV (color) channel respectively, and edge enhancement, brightness and contrast adjustment are performed on the brightness component, and hue and saturation adjustment are performed on the color component.

[0058] In the conventional ISP chip processing, since it is necessary to separate the YUV image into channels for processing, in this process, it is necessary to repeatedly read and write the image data for each channel for processing, resulting in a low noise reduction effect and low processing efficiency. In view of this, in the embodiments of the present application, the image noise reduction model can be applied to the ISP chip and directly used for noise reduction processing of RGB images, RAW images or YUV images. For example, when the image noise reduction model is applied to the RAW image noise reduction module in the ISP chip, the format of the corresponding target image is the RAW format; when the image noise reduction model is applied to the RGB image noise reduction module in the ISP chip, the format of the corresponding target image is the RGB format; when the image noise reduction model is applied to the YUV image noise reduction module in the ISP chip, the format of the corresponding target image is the YUV format.

[0059] Thereby, instead of the process of separately performing noise reduction processing on channels for the conventional YUV image, the noise reduction effect and processing efficiency are improved.

[0060] In some cases, please refer to FIG. 4 showing the process schematic diagram of the image processing of the improved ISP chip in the first form provided by the embodiments of the present application. The image noise reduction model is applied to the YUV image noise reduction module in the ISP chip to directly perform noise reduction processing on the data of each channel of the YUV image. In this case, the target image data input to the image noise reduction model is the pixel value of each channel of the YUV image. Thereby, the noise reduction processing effect and processing efficiency of the YUV image are greatly improved, and in the scene of processing real-time video images, the processing efficiency results of the image noise reduction model are more remarkable.

[0061] In other cases, please refer to FIG. 5, which shows a schematic diagram of the image processing process of the improved ISP chip in the second form provided by the embodiments of the present application. The image noise reduction model is applied to the RGB image noise reduction module in the ISP chip to perform noise reduction processing on the data of each channel of the RGB image. Then, the ISP chip can directly convert the RGB image obtained after the noise reduction processing to obtain a YUV image, and subsequent processing can be performed. There is no need to divide the YUV image into channels again for noise reduction processing, which improves the efficiency of the image noise reduction processing. In this case, the target image data input into the image noise reduction model is the pixel value of each channel of the RGB image.

[0062] Optionally, the image noise reduction model can be further applied to the RAW image noise reduction module in the ISP chip to directly perform noise reduction processing on the RAW image. Correspondingly, in this case, the target image data input into the image noise reduction model is the pixel value of each channel of the RAW image.

[0063] In the embodiments of the present application, the noise reduction module in the conventional ISP chip is slightly adjusted to integrate the brightness noise reduction and color noise reduction, and the overall noise reduction processing is realized by the image noise reduction model. Thereby, the extraction ability of the image signal in the noise reduction processing can be improved, and the signal-to-noise ratio and the subjective noise reduction effect can be improved. The YUV image can achieve a good noise reduction effect by referring to the information between channels while reducing the noise of each channel (Y, U, V) of the image. In addition, processing by integrating channels has less actual calculation amount and data read / write amount than the sum of the actual calculation amount and data read / write amount when processing by dividing channels. Therefore, processing by integrating the channels of the image has positive merits in terms of both performance and effect. Therefore, applying the image noise reduction model to the ISP chip for image noise reduction processing can meet the requirements of the ISP chip in terms of both noise reduction effect and real-time performance.

[0064] The following describes the specific processing process of the target image data by the image noise reduction model.

[0065] In one embodiment, as disclosed in FIG. 6 showing a schematic diagram of the noise reduction processing provided by the embodiment of the present application, inputting the target image data into the image noise reduction model to obtain the noise reduction image data output from the image noise reduction model includes the following steps. In step 601, the target image data is input into the downsampling model, and each downsampling module in the downsampling model performs downsampling processing on the target image data to obtain downsampling feature data.

[0066] In step 602, the downsampling feature data is input into the upsampling model, and each upsampling module in the upsampling model performs upsampling processing on the downsampling feature data to obtain upsampling feature data.

[0067] In step 603, the output layer obtains the noise reduction image data based on the upsampling feature data and the target image data.

[0068] It should be noted that the downsampling model is composed of n cascaded downsampling modules. As an option, the value taken by n can be specified according to the actual operation situation and memory situation of the chip, etc., thereby specifying the configurations of the downsampling model and the upsampling model. In the embodiments of the present application, the number of upsampling modules and downsampling modules is not specifically limited, but it is natural that the number of upsampling modules should be the same as the number of downsampling modules.

[0069] Each downsampling module is used to perform downsampling processing on the input data and extract more image features. After the downsampling processing, the image is shrunk, and correspondingly, the upsampling module is used to perform upsampling processing to restore the size of the image.

[0070] What the last module among the n cascaded downsampling modules outputs is the downsampling feature data, and what the last module among the n cascaded upsampling modules outputs is the upsampling feature data.

[0071] Both the upsampling feature data and the target image data are input to the output layer for fusion processing, and based on the data output from the output layer, the noise-reduced image data can be obtained.

[0072] In the embodiments of the present application, currently, most single image denoising algorithms are difficult to execute in real time on an ISP chip. The present application simplifies the network structure, constructs a lightweight neural network model adapted to execution on a chip, obtains an image noise reduction model, so that the single image denoising algorithm can be applied to real-time noise reduction on an ISP chip, and at the same time meets the needs of image noise reduction effect and algorithm real-time performance, solves the problems of neural network placement and real-time execution on a chip, and significantly improves the noise reduction effect for conventional chips.

[0073] Specifically, the process of inputting and processing the target image data corresponding to different types of target images by the image noise reduction model is also different. Hereinafter, the processes of processing the target image data corresponding to two types of target images will be described respectively.

[0074] In this case, the target image is an image in a format where the resolution of the image data for each channel is the same, such as a RAW format, RGB format, or YUV444 format target image. At this time, for each downsampling module in the downsampling model to perform downsampling processing on the target image data to obtain downsampling feature data, in the i-th downsampling module, it is to perform downsampling processing on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module, and to use the intermediate downsampling feature data output from the last downsampling module as the downsampling feature data.

[0075] Note that when i = 1, the input data of the i-th downsampling module is the target image data, and when i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module.

[0076] In the ISP chip, images are stored in different data formats (such as RAW, RGB, YUV444, YUV420) in different modules. When the input and output of the image noise reduction model are in a format where the resolution of each channel such as RAW, RGB, YUV444 is the same, for the first downsampling module, the entire target image data is used as the input of the first downsampling module, and for other downsampling modules, the intermediate downsampling feature data output from the previous downsampling module is used as the input data of the downsampling module. Each downsampling module is used to extract image features to obtain intermediate downsampling feature data.

[0077] Correspondingly, in one embodiment, for each upsampling module in the upsampling model to perform upsampling processing on the downsampling feature data to obtain upsampling feature data includes, in the i-th upsampling module, performing upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module, and using the intermediate upsampling feature data output from the last upsampling module as the upsampling feature data. Note that when i = 1, the input data of the i-th upsampling module is the downsampling feature data; when i is greater than 1, the input data of the i-th upsampling module is the set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module.

[0078] Also, for the first upsampling module, the upsampling feature data is used as the input of the first upsampling module. For other upsampling modules, the intermediate upsampling feature data output from the previous upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the upsampling module are fused to obtain set feature data, and the set feature data is used as the input data of the upsampling module. Thereby, the deep features and shallow features in the target image data and the features with different resolutions are fully fused, improving the effect of noise reduction processing.

[0079] Correspondingly, in one embodiment, the output layer obtaining the noise-reduced image data based on the upsampled feature data and the target image data includes inputting the upsampled feature data and the target image data into the output layer to perform a fusion process, and obtaining the noise-reduced image data output from the output layer.

[0080] Specifically, the output layer is mainly used to perform a fusion process on the input data. In other words, both the output layer and the fusion module in the downsampling module are used to perform a feature fusion process. As an option, the output layer can perform a simple element-wise addition process or a channel superposition process on the input data to realize the fusion process.

[0081] After a series of processes, the upsampled feature data output from the last upsampling module includes the noise feature data of the target image. When the target image data and the upsampled feature data are fused, the noise features can be removed from the target image data. What is output from the output layer is the noise-reduced image data with the noise feature data removed, realizing the noise reduction process for the target image. That is, the entire image noise reduction model actually outputs the noise residue, and after superimposing the noise residue and the target image in the output layer, the noise-reduced image output from the output layer is obtained.

[0082] As an option, the upsampling module can be composed of a cascaded convolutional layer and an upsampling layer.

[0083] Based on the above content, the single input / output image noise reduction model provided by the embodiments of the present application can be obtained. Please refer to FIG. 7 showing a schematic diagram of the configuration of the noise reduction neural network provided by the embodiments of the present application. Exemplarily, the image noise reduction model shown in FIG. 7 includes three downsampling modules and three upsampling modules. The fusion modules and other fusion processes in the output layer and the downsampling modules all take the fusion method by element-wise addition as an example. The downsampling layer and the upsampling layer can realize downsampling or upsampling through convolution processing.

[0084] As can be seen from the above, the image noise reduction model based on the U-net network structure provided by the embodiments of the present application has a simple configuration, directly performs noise reduction processing on the entire data of each channel of the image, guarantees a good noise reduction effect, and has high processing efficiency.

[0085] When the structure of the neural network framework and the basic computing unit does not change, the number of downsampling modules and upsampling modules can be changed based on the limitations of the actual computing power and bandwidth, and the noise reduction effect can be appropriately improved. In FIG. 7, taking three times of downsampling and three times of upsampling as an example, the number can actually be increased (for example, 4 times, 5 times, etc.) or decreased (for example, 2 times). For the channel fusion method, element-wise addition or channel-wise concatenation can be selected.

[0086] In other cases, the target image is an image in a format where the resolution of the image data for each channel is different, for example, a target image in a format such as YUV420. At this time, the downsampling model in the image noise reduction model further includes an additional downsampling module. The upsampling model in the image noise reduction model further includes an additional upsampling module. Refer to FIG. 8 showing a schematic diagram of the configuration of the image noise reduction model in other forms provided by the embodiments of the present application.

[0087] Correspondingly, inputting the target image data into the downsampling model, and each downsampling module in the downsampling model performing a downsampling process on the target image data to obtain downsampling feature data, includes inputting the pixel values of the first channel of the target image included in the target image data into the additional downsampling module to obtain the channel feature data output from the additional downsampling module. Fusing the channel feature data with the pixel values of the second channel of the target image included in the target image data to obtain the candidate target image data as the input data of the first downsampling module. In the i-th downsampling module, performing a downsampling process on the input data of the i-th downsampling module to obtain the intermediate downsampling feature data output from the i-th downsampling module. The intermediate downsampling feature data output from the last downsampling module is used as the downsampling feature data.

[0088] Note that when i = 1, the input data of the i-th downsampling module is the candidate target image data, and when i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module.

[0089] In an ISP chip, images are stored in different data formats (e.g., RAW, RGB, YUV444, YUV420) in different modules. When the resolution of each channel of the input and output images, for which the image noise reduction model is embedded in the chip, is not the same, such as in the case of a YUV420 format, a multi-input multi-output network configuration, i.e., the configuration shown in FIG. 8, needs to be used.

[0090] Specifically, for a target image with each channel resolution, the target image data corresponding to the target image is composed of a first channel pixel value and a second channel pixel value. Taking a target image in YUV420 format as an example, the first channel pixel value is the Y channel pixel value, and the second channel pixel value is the UV channel pixel value. Since the resolutions are different, it is necessary to perform downsampling processing in advance on the first channel pixel value, i.e., the Y channel pixel value, using an additional downsampling module. The channel feature data output from the additional downsampling module may have the same resolution as the second channel pixel value, i.e., the UV channel pixel value. At this time, the channel feature data and the second channel pixel value can be directly fused, and normal downsampling processing can be performed using each downsampling module.

[0091] Correspondingly, in one embodiment, inputting the downsampling feature data into an upsampling model and having each upsampling module in the upsampling model perform upsampling processing on the downsampling feature data to obtain upsampling feature data In the i-th upsampling module, it includes performing upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. Input the first intermediate channel feature data corresponding to the first channel pixel values included in the intermediate upsampling feature data output from the last upsampling module into the additional upsampling module, and obtain the upsampling feature data output from the additional upsampling module.

[0092] Note that when i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is the set feature data in which the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module are fusion-processed. Thereby, both the deep features and shallow features in the target image data and the features with different resolutions are fully fused, and the effect of noise reduction processing is improved.

[0093] Corresponding to the target image data including the first channel pixel values and the second channel pixel values, the noise reduction image data output from the image noise reduction model should include noise reduction image data of different channels having different resolutions.

[0094] The upsampling module, similar to the above-described image noise reduction model, has each upsampling module perform upsampling processing on the input data to obtain intermediate upsampling feature data output from the last upsampling module. At this time, the intermediate upsampling feature data output from the last upsampling module includes first intermediate channel feature data and second intermediate channel feature data. The first intermediate channel feature data corresponds to the first channel pixel values, that is, the Y channel image data. In other words, the first intermediate channel feature data is obtained after performing noise reduction processing on the first channel pixel values. Correspondingly, the second intermediate channel feature data is obtained after performing noise reduction processing on the second channel pixel values.

[0095] Since the first channel pixel values are downsampled by the additional downsampling module, correspondingly, the first intermediate channel feature data is further upsampled by the additional upsampling module, and thereby, the output of the additional upsampling module is used as the upsampling feature data finally output by the upsampling model. Thereby, the output layer performs further fusion processing based on the upsampling feature data and the first channel pixel values in the target image data.

[0096] Correspondingly, in one embodiment, the output layer obtaining noise reduction image data based on the upsampling feature data and the target image data includes inputting the first channel pixel values in the upsampling feature data and the target image data into the output layer to perform fusion processing, and obtaining candidate noise reduction image data output from the output layer. Based on the candidate noise reduction image data and the second intermediate channel feature data corresponding to the second channel pixel values included in the intermediate upsampling feature data output from the last upsampling module, noise reduction image data is obtained.

[0097] Specifically, through a series of processes, the upsampling feature data includes noise features corresponding to the first-channel pixel values, that is, the noise residuals of the target image are extracted. The output layer fuses the upsampling feature data and the first-channel pixel values, that is, when the noise residuals and the target image are superimposed in the output layer, the noise features in the first-channel pixel values can be removed, and candidate noise reduction image data output from the output layer is obtained.

[0098] Based on the candidate image noise reduction data and the second intermediate channel feature data included in the intermediate upsampling feature data output from the last upsampling module, the noise reduction image data can be obtained.

[0099] Thereby, for images with different resolutions in each channel, the multi-input / output image noise reduction model can realize noise reduction processing, and the application range of the ISP chip can be expanded.

[0100] As an option, the additional downsampling module is composed of a cascaded downsampling layer and a convolutional layer, and the additional upsampling module is composed of a cascaded convolutional layer and an upsampling layer.

[0101] Based on the above content, a schematic diagram of the structure of the multi-input / output image noise reduction model provided by the embodiments of the present application can be obtained. Please refer to FIG. 9 showing a schematic diagram of the structure of other forms of noise reduction neural networks provided by the embodiments of the present application. Exemplarily, the image noise reduction model shown in FIG. 9 includes two downsampling modules and two upsampling modules. The fusion modules and other fusion processes in the output layer and the downsampling module all take the fusion method by adding elements as an example. The downsampling layer and the upsampling layer can realize downsampling through convolutional processing.

[0102] In the embodiments of the present application, as an important module in the ISP chip, the image noise reduction processing module needs to have the input and output formats and data array methods all consistent with the noise reduction processing module in the conventional ISP chip, so as to reduce the changes to the original layout of the ISP chip and accelerate the application process. Therefore, in the embodiments of the present application, a noise reduction neural network with multi-input and output formats is provided, and it is not necessary to significantly change the layout of the ISP chip instead of the conventional noise reduction module. Compared with the conventional neural network, in the embodiments of the present application, a multi-input and output network structure is used and embedded in the ISP chip, which can well solve the problem that the single-image noise reduction neural network cannot adapt to the layout and real-time performance of the ISP chip.

[0103] Compared with the image processing algorithms in the conventional ISP chip, in the embodiments of the present application, the image noise reduction model can fully fuse the information of each channel pixel value in the target image data, better mine the information in the image, and remove the noise in the image. In addition, the ISP noise reduction algorithm with channel fusion can effectively improve the signal-to-noise ratio of the image. Compared with the conventional noise reduction algorithm, by using a lightweight noise reduction neural network, the resolution of the image can be significantly improved and the noise can be reduced. While reducing the noise and improving the quality of the image, it can also improve the smear noise of the moving object in the image, improve the accuracy when performing subsequent image tasks such as target detection or face recognition on the target image, and expand the application range of the target image after noise reduction processing.

[0104] As described above, each downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascade-connected with the first downsampling module and the second downsampling module. The first downsampling module includes a first convolutional layer and a first downsampling layer, and the second downsampling module includes a second downsampling layer. Hereinafter, the processing process of the downsampling module will be described.

[0105] In one embodiment, obtaining intermediate downsampling feature data output from the i-th downsampling module by performing downsampling processing on the input data of the i-th downsampling module includes: performing downsampling processing on the input data of the i-th downsampling module using the first downsampling layer to obtain first downsampling feature data output from the first downsampling layer; performing convolutional processing on the first downsampling feature data using the first convolutional layer to obtain first convolutional feature data output from the first convolutional layer; performing downsampling processing on the input data of the i-th downsampling module using the second downsampling layer to obtain second downsampling feature data output from the second downsampling layer; and performing fusion processing on the first convolutional feature data and the second downsampling feature data using the fusion module to obtain intermediate downsampling feature data output from the fusion module.

[0106] Alternatively, the first downsampling layer and the second downsampling layer can achieve downsampling processing by convolutional processing.

[0107] As an option, each downsampling module can select an appropriate convolutional structure. For example, the first downsampling layer and the first convolutional layer can select a convolutional process with a 5x5 convolutional kernel, and the second convolutional layer can select a convolutional process with a 3x3 convolutional kernel, etc. Of course, the convolutional processes of each layer in the downsampling module can use any combination such as direct connection, 1x1 convolution, 3x3 convolution, 5x5 convolution, 7x7 convolution, etc., and the embodiments of the present application do not specifically limit this.

[0108] Hereinafter, the processing process of the upsampling module will be described.

[0109] In one embodiment, the upsampling module includes a second convolutional layer and an upsampling layer connected in cascade. Performing upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module includes using the second convolutional layer to perform convolutional processing on the input data of the i-th upsampling module to obtain second convolutional feature data output from the second convolutional layer, and using the upsampling layer to perform upsampling processing on the second convolutional feature data to obtain intermediate upsampling feature data output from the upsampling layer.

[0110] As an option, the upsampling layer performs upsampling processing on the input data of the upsampling layer by means of convolutional processing, inverse pooling processing or interpolation processing.

[0111] As an option, each upsampling module can further include other numbers of convolutional layers and upsampling layers, which can be specifically determined based on parameters such as the computing power, bandwidth or memory capacity of the ISP chip.

[0112] In addition, the second convolutional layer and the upsampling layer can select an appropriate convolutional structure, and the convolutional process of each layer can use any combination such as direct connection, 1x1 convolution, 3x3 convolution, 5x5 convolution, 7x7 convolution, etc. The embodiments of the present application do not specifically limit this.

[0113] In one embodiment, in the fusion process of the above-described fusion module, the fusion process of the output layer, and other fusion processes in the image noise reduction model, the fusion process can fuse the deep features and shallow features of the image, and any of them can be realized by using the addition of element units or the combination method of channel units. Please refer to FIG. 10 showing the schematic diagram of the fusion process by the addition of element units provided by the embodiments of the present application. Please refer to FIG. 11 showing the schematic diagram of the fusion process by the combination of channel units provided by the embodiments of the present application.

[0114] In addition, the method of fusing feature channels by the addition method of element units can significantly reduce the data reading amount and calculation amount, but there is a possibility of losing some of the already extracted features. When the computing power and cache of the chip are sufficient, the combination method of channel units can be selected. The channel addition method needs to ensure that the resolutions and channel numbers of the two groups of features to be fused are exactly the same, and the combination method of channel units does not require the channel numbers of the two groups of features to be fused to be the same. Based on this, for different ISP chips, different channel fusion methods and upsampling methods are selected, and there is a slight difference in the final noise reduction effect, but for some chips, the difference in calculation time and efficiency is very large. Therefore, based on the image processing needs specifically determined in advance based on the ISP chip, the fusion process method can be specifically determined as the addition of element units or the combination of channel units, thereby sufficiently improving the efficiency of chip image processing.

[0115] In one embodiment, the embodiments of the present application provide a neural network noise reduction algorithm arranged on an ISP chip, and the main arrangement process is as follows: In step 1, construct the basic structure of the U-Net neural network. Specifically, it includes specifying the network framework, the input / output format of the network (such as RAW, RGB, YUV444, etc.), the number of downsampling and upsampling layers in the network, and the sampling rate of each layer.

[0116] In step 2, specify the structure of the downsampling module in the network structure.

[0117] In step 3, select the channel fusion method and the upsampling method. For example, for channel fusion, use a processing method based on element-wise addition, and for upsampling, use transposed convolution processing to achieve upsampling.

[0118] In step 4, specify the specific position where the network is embedded in the image processing process of the ISP chip, such as the RAW image noise reduction module, the RGB image noise reduction module, or the YUV image noise reduction module.

[0119] In step 5, execute and debug the complete neural network noise reduction algorithm.

[0120] Furthermore, as described in step 1, illustratively, the entire network can include three multi-convolution parallel sub-modules used for downsampling processing, three ordinary convolution layers corresponding thereto and used for upsampling processing, and three upsampling layers. Before each downsampling is performed, the features of the original resolution are retained and fused after the corresponding upsampling layer, so that the deep features and shallow features in the image, and the features of different resolutions can be fully fused. The entire network is a total of 15 convolution and upsampling layers. After each layer of convolution, an activation layer is added. Taking as an example an input of a YUV format image with a size of 256x256x3, the data sizes output by the 15-layer network are 128x128x16, 128x128x16, 128x128x16, 64x64x32, 64x64x32, 64x64x32, 32x32x64, 32x32x64, 32x32x64, 32x32x16, 64x64x16, 64x64x16, 128x128x16, 128x128x16, 256x256x3 (output in YUV format), respectively.

[0121] Alternatively, in step 1, the number of downsampling layers may be N layers, where N is 1 or more. When the computing power and cache of the ISP chip are sufficient, N can be set to 4 or an integer greater than that.

[0122] Alternatively, in step 1, the downsampling rate may be N:1, where N is greater than 1. N can be set to 3, 4, etc. Different downsampling layers can use different sampling rates.

[0123] Alternatively, in step 3, the upsampling method may be inverse pooling or an interpolation algorithm, etc. The inverse pooling or interpolation method can effectively reduce the amount of parameters of the upsampling layer.

[0124] Compared with the image processing algorithm in the conventional ISP chip, in the embodiments of the present application, in the noise reduction process, the information of each channel of the image is sufficiently fused, so that the information in the image can be better mined and the noise in the image can be removed. The ISP noise reduction algorithm with channel fusion can effectively improve the signal-to-noise ratio of the image. Compared with the conventional noise reduction algorithm, by using a lightweight noise reduction neural network, the resolution of the image can be greatly improved and the noise can be reduced. In order to reduce the noise and improve the quality of the image, at the same time, the smear noise of the moving object in the image can be improved, and the accuracy when performing image tasks such as target detection and face recognition can be improved. Compared with the general noise reduction neural network, the neural network used in the embodiments of the present application improves the network structure and basic operators and can be executed in real time on the ISP chip, thereby solving the problem that the general neural network cannot be executed in real time on mobile devices. Compared with the general neural network, in the embodiments of the present application, a multi-input and output network structure is used, which is embedded in the image processing flow of the ISP chip and can well solve the problem that the single-image noise reduction neural network is not suitable for the image processing flow of the ISP chip.

[0125] In addition, although each step of the flowchart according to each of the above-described embodiments is sequentially shown in the order indicated by the arrows, it is not necessarily required to be sequentially executed in the order indicated by the arrows. The execution of these steps is not limited to a strict order unless explicitly described in this specification, and they may be executed in other orders. Also, at least a part of the steps of the flowchart according to the above-described embodiments may include a plurality of steps or a plurality of stages. These steps or stages do not necessarily need to be executed simultaneously, and may be executed at different times. The execution order of these steps or stages does not necessarily need to be continuously executed, and may be executed alternately with at least a part of other steps or steps or stages of other steps.

[0126] Based on the same inventive concept, the embodiments of the present application further provide an image noise reduction processing apparatus for implementing the above-described image noise reduction processing method. Since the technical means for solving the problems provided by the apparatus are similar to the technical means described in the above method, the specific limitations in one or more of the following embodiments of the image noise reduction processing apparatus can refer to the limitations on the above-described image noise reduction processing method, and will not be repeated here.

[0127] In one embodiment, as shown in FIG. 12, an image noise reduction processing apparatus is provided, and the image noise reduction processing apparatus 1200 includes a noise reduction module 1201.

[0128] The noise reduction module 1201 is used to input target image data into an image noise reduction model and obtain noise-reduced image data output from the image noise reduction model. The target image data includes pixel values of each channel of the target image. The image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n cascaded upsampling modules that correspond one-to-one to the n downsampling modules. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a cascaded first downsampling layer and a first convolutional layer. The second downsampling module includes a second downsampling layer.

[0129] In one embodiment, specifically, the noise reduction module 1201 inputs target image data into the downsampling model. Each downsampling module in the downsampling model performs downsampling processing on the target image data to obtain downsampled feature data. The downsampled feature data is input into the upsampling model. Each upsampling module in the upsampling model performs upsampling processing on the downsampled feature data to obtain upsampled feature data. The output layer is used to obtain noise-reduced image data based on the upsampled feature data and the target image data.

[0130] In one embodiment, the resolution of the image data for each channel of the target image is the same. Specifically, in the i-th downsampling module, the noise reduction module 1201 performs downsampling processing on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module. When i = 1, the input data of the i-th downsampling module is the target image data. When i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module. The intermediate downsampling feature data output from the last downsampling module is used as the downsampling feature data.

[0131] In one embodiment, specifically, in the i-th upsampling module, the noise reduction module 1201 performs upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is the set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module. The intermediate upsampling feature data output from the last upsampling module is used as the upsampling feature data.

[0132] In one embodiment, specifically, the noise reduction module 1201 inputs the upsampling feature data and the target image data into the output layer to perform fusion processing, and obtains the noise-reduced image data output from the output layer.

[0133] In one embodiment, the resolution of the image data for each channel of the target image is different, the downsampling model further includes an additional downsampling module, and specifically, the noise reduction module 1201 inputs the pixel values of the first channel of the target image included in the target image data into the additional downsampling module to obtain the channel feature data output from the additional downsampling module, and fuses the channel feature data with the pixel values of the second channel of the target image included in the target image data to obtain candidate target image data. In the i-th downsampling module, downsampling processing is performed on the input data of the i-th downsampling module to obtain the intermediate downsampling feature data output from the i-th downsampling module. When i = 1, the input data of the i-th downsampling module is the candidate target image data. When i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module, and the intermediate downsampling feature data output from the last downsampling module is used as the downsampling feature data.

[0134] In one embodiment, the upsampling model further includes an additional upsampling module. Specifically, in the i-th upsampling module, the noise reduction module 1201 performs upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is set feature data obtained by fusing intermediate upsampling feature data output from the (i - 1)-th upsampling module and intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module. The additional upsampling module is used to input the first intermediate channel feature data corresponding to the first channel pixel values included in the intermediate upsampling feature data output from the last upsampling module to obtain upsampling feature data output from the additional upsampling module.

[0135] In one embodiment, specifically, the noise reduction module 1201 inputs the first channel pixel values in the upsampling feature data and the target image data to an output layer to perform fusion processing, and obtains candidate noise reduction image data output from the output layer. Based on the candidate noise reduction image data and the second intermediate channel feature data corresponding to the second channel pixel values included in the intermediate upsampling feature data output from the last upsampling module, it is used to obtain noise reduction image data.

[0136] In one embodiment, the noise reduction module 1201 specifically performs downsampling processing on the input data of the i-th downsampling module by using a first downsampling layer to obtain first downsampling feature data output from the first downsampling layer, performs convolution processing on the first downsampling feature data by using a first convolutional layer to obtain first convolutional feature data output from the first convolutional layer, performs downsampling processing on the input data of the i-th downsampling module by using a second downsampling layer to obtain second downsampling feature data output from the second downsampling layer, and performs fusion processing on the first convolutional feature data and the second downsampling feature data by using a fusion module to obtain intermediate downsampling feature data output from the fusion module.

[0137] In one embodiment, the upsampling module includes a second convolutional layer and an upsampling layer connected in cascade. Specifically, the noise reduction module 1201 performs convolution processing on the input data of the i-th upsampling module by using the second convolutional layer to obtain second convolutional feature data output from the second convolutional layer, and performs upsampling processing on the second convolutional feature data by using the upsampling layer to obtain intermediate upsampling feature data output from the upsampling layer.

[0138] In one embodiment, the image noise reduction model is applied to the RAW image noise reduction module 1201, the RGB image noise reduction module 1201, or the YUV image noise reduction module 1201 in the ISP chip. Correspondingly, the format of the target image is the RAW format, the RGB format, or the YUV format.

[0139] In one embodiment, the upsampling layer performs upsampling processing on the input data of the upsampling layer by convolution processing, inverse pooling processing, or interpolation processing.

[0140] Each module in the above image noise reduction processing device can be implemented in whole or in part by software, hardware, and combinations thereof. Each of the above modules may be built into a processor in a computer device in the form of hardware or may be independent, and may be stored in a memory in the computer device in the form of software. Thereby, the processor calls and executes operations corresponding to each of the above modules.

[0141] In one embodiment, a computer device is provided. The computer device may be a terminal, and its internal structure diagram may be as shown in FIG. 13. The computer device includes a processor, a memory, a communication interface, a display, and an input device connected via a system bus. Also, the processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The operating system and computer program are stored in the non-volatile storage medium. The internal memory provides an environment for executing the operating system and computer program in the non-volatile storage medium. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner is implemented by WIFI, a mobile cellular network, NFC (Near Field Communication), or other technologies. When the computer program is executed by the processor, an image noise reduction processing method is implemented. The display of the computer device may be a liquid crystal display or an electronic ink display, and the input device of the computer device may be a touch layer covered on the display, a key provided on the housing of the computer device, a trackball, or a touch panel, or may further be an external keyboard, touch panel, mouse, etc.

[0142] As would be understood by those skilled in the art, the configuration shown in FIG. 13 is a block diagram of only a part of the structure related to the technical means of the present application, and does not limit the computer device to which the technical means of the present application is applied. A specific computer device may include more or fewer components than those shown in the figure, some components may be combined, or may have different arrangements of components.

[0143] In one embodiment, an electronic device is further provided. The electronic device includes a memory and a processor. A computer program is stored in the memory. When the processor executes the computer program, the steps in the embodiments of the above methods are implemented.

[0144] In one embodiment, a computer-readable storage medium is provided. A computer program is stored in the computer-readable storage medium. When the computer program is executed by a processor, the steps in the embodiments of the above methods are implemented.

[0145] In one embodiment, a computer program product is provided. The computer program product includes a computer program. When the computer program is executed by a processor, the steps in the embodiments of the above methods are implemented.

[0146] As can be understood by those skilled in the art, to implement all or part of the processes in the above-described embodiments, it can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-described various methods. Also, any reference to the memory, database, or other media used in each embodiment provided in this application can include at least one of non-volatile and volatile memories. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, and the like. Volatile memory can include random access memory (RAM) or external cache memory, and the like. By way of example and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), and the like. The database according to each embodiment provided in this application can include at least one of relational databases and non-relational databases. Non-relational databases can include, but are not limited to, distributed databases based on blockchain. The processor according to each embodiment provided in this application can be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, programmable logic, data processing logic based on quantum computing, and the like, and is not limited thereto.

[0147] Each of the technical features of the above-described embodiments can be arbitrarily combined. For the sake of brevity of description, not all combinations of the technical features in the above-described embodiments are described, but these combinations of technical features should be considered to be within the scope described in this specification as long as they do not conflict.

[0148] The above-described embodiments merely show some embodiments of the present application, and although the description is specific and detailed, it should not be construed as limiting the protection scope of the invention. It should be noted that those skilled in the art can make some modifications and improvements without departing from the spirit of the present application, and all of these also belong to the protection scope of the present invention. Therefore, the protection scope of the present invention should conform to the scope of the claims.

Claims

1. A method for reducing image noise, comprising: The method includes: inputting target image data including pixel values of each channel of a target image into an image noise reduction model to obtain noise reduction image data output from the image noise reduction model, The image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n upsampling modules that are in one-to-one correspondence with the n downsampling modules and are cascaded. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a cascaded first downsampling layer and a first convolutional layer. The second downsampling module includes a second downsampling layer. A method for reducing image noise, characterized by the above.

2. Inputting the target image data into the image noise reduction model to obtain the noise reduction image data output from the image noise reduction model includes: inputting the target image data into the downsampling model, and each downsampling module in the downsampling model performing downsampling processing on the target image data to obtain downsampling feature data; inputting the downsampling feature data into the upsampling model, and each upsampling module in the upsampling model performing upsampling processing on the downsampling feature data to obtain upsampling feature data; the output layer obtaining the noise reduction image data based on the upsampling feature data and the target image data; The method according to claim 1, characterized by including the above.

3. The resolution of the image data for each channel of the target image is the same. In the downsampling model, each of the downsampling modules performing downsampling processing on the target image data to obtain downsampling feature data is in the i-th downsampling module, performing downsampling processing on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module, and when i = 1, the input data of the i-th downsampling module is the target image data, and when i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module, and using the intermediate downsampling feature data output from the last downsampling module as the downsampling feature data, including The method according to claim 2, characterized in that.

4. In the upsampling model, each of the upsampling modules performing upsampling processing on the downsampling feature data to obtain upsampling feature data is in the i-th upsampling module, performing upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module, and when i = 1, the input data of the i-th upsampling module is the downsampling feature data, and when i is greater than 1, the input data of the i-th upsampling module is set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module, and using the intermediate upsampling feature data output from the last upsampling module as the upsampling feature data, including The method according to claim 3, characterized in that.

5. The output layer obtaining the noise-reduced image data based on the upsampled feature data and the target image data means that the method according to claim 4, characterized in that it includes inputting the upsampled feature data and the target image data into the output layer to perform a fusion process, and obtaining the noise-reduced image data output from the output layer. **Claim 6** the resolution of the image data for each channel of the target image is different, and the downsampling model further includes an additional downsampling module, inputting the target image data into the downsampling model, and each downsampling module in the downsampling model performing a downsampling process on the target image data to obtain downsampled feature data means that inputting the pixel values of the first channel of the target image included in the target image data into the additional downsampling module to obtain channel feature data output from the additional downsampling module, fusing the channel feature data and the pixel values of the second channel of the target image included in the target image data to obtain candidate target image data, in the i-th downsampling module, performing a downsampling process on the input data of the i-th downsampling module to obtain intermediate downsampled feature data output from the i-th downsampling module, where when i = 1, the input data of the i-th downsampling module is the candidate target image data, and when i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampled feature data output from the (i - 1)-th downsampling module, and using the intermediate downsampled feature data output from the last downsampling module as the downsampled feature data. This is the method according to claim 2, characterized in that. **Claim 7** the upsampling model further includes an additional upsampling module, Inputting the downsampling feature data into the upsampling model, and each upsampling module in the upsampling model performing upsampling processing on the downsampling feature data to obtain upsampling feature data, in the i-th upsampling module, performing upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is the set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module, inputting the first intermediate channel feature data included in the intermediate upsampling feature data output from the last upsampling module corresponding to the first channel pixel value into the additional upsampling module to obtain the upsampling feature data output from the additional upsampling module, The method according to claim 6, characterized in that.

8. The output layer obtaining the noise reduction image data based on the upsampling feature data and the target image data, inputting the first channel pixel values in the upsampling feature data and the target image data into the output layer to perform fusion processing to obtain candidate noise reduction image data output from the output layer, obtaining the noise reduction image data based on the candidate noise reduction image data and the second intermediate channel feature data corresponding to the second channel pixel values included in the intermediate upsampling feature data output from the last upsampling module, The method according to claim 7, characterized in that.

9. Performing downsampling processing on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module, Performing downsampling processing on the input data of the i-th downsampling module using the first downsampling layer to obtain first downsampling feature data output from the first downsampling layer, Performing convolution processing on the first downsampling feature data using the first convolutional layer to obtain first convolutional feature data output from the first convolutional layer, Performing downsampling processing on the input data of the i-th downsampling module using the second downsampling layer to obtain second downsampling feature data output from the second downsampling layer, Including performing fusion processing on the first convolutional feature data and the second downsampling feature data using the fusion module to obtain the intermediate downsampling feature data output from the fusion module. The method according to claim 3 or 6, characterized in that.

10. The upsampling module includes a second convolutional layer and an upsampling layer connected in cascade. Performing upsampling processing on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module, Performing convolution processing on the input data of the i-th upsampling module using the second convolutional layer to obtain second convolutional feature data output from the second convolutional layer, Including performing upsampling processing on the second convolutional feature data using the upsampling layer to obtain the intermediate upsampling feature data output from the upsampling layer. The method according to claim 4 or 7, characterized in that.

11. The image noise reduction model is used in a RAW image noise reduction module, an RGB image noise reduction module, or a YUV image noise reduction module in an ISP chip. Correspondingly, the format of the target image is a RAW format, an RGB format, or a YUV format. The method according to claim 1 is characterized by this.

12. The upsampling layer upsamples the input data of the upsampling layer by convolution processing, inverse pooling processing, or interpolation processing. The method according to claim 10 is characterized by this.

13. An image noise reduction processing device, The device includes a noise reduction module, The noise reduction module is used to input target image data including pixel values of each channel of a target image into an image noise reduction model and obtain noise reduction image data output from the image noise reduction model. The image noise reduction model includes a cascaded downsampling model, an upsampling model, and an output layer. The downsampling model includes n cascaded downsampling modules. The upsampling model includes n upsampling modules that correspond one-to-one to the n downsampling modules connected in cascade. The downsampling module includes a first downsampling module, a second downsampling module, and a fusion module cascaded with the first downsampling module and the second downsampling module. The first downsampling module includes a first downsampling layer and a first convolutional layer connected in cascade. The second downsampling module includes a second downsampling layer. An image noise reduction processing device characterized by this.

14. The noise reduction module specifically The target image data is input into the downsampling model, and each downsampling module in the downsampling model performs downsampling processing on the target image data to obtain downsampling feature data. The downsampling feature data is input into the upsampling model, and each upsampling module in the upsampling model performs upsampling processing on the downsampling feature data to obtain upsampling feature data. The output layer is used to obtain the noise-reduced image data based on the upsampling feature data and the target image data. The apparatus according to claim 13, characterized in that.

15. The resolution of the image data for each channel of the target image is the same, The noise reduction module specifically, In the i-th downsampling module, downsampling processing is performed on the input data of the i-th downsampling module to obtain intermediate downsampling feature data output from the i-th downsampling module. When i = 1, the input data of the i-th downsampling module is the target image data. When i is greater than 1, the input data of the i-th downsampling module is the intermediate downsampling feature data output from the (i - 1)-th downsampling module. The intermediate downsampling feature data output from the last downsampling module is used as the downsampling feature data. The apparatus according to claim 14, characterized in that.

16. The noise reduction module specifically, In the i-th upsampling module, upsampling processing is performed on the input data of the i-th upsampling module to obtain intermediate upsampling feature data output from the i-th upsampling module. When i = 1, the input data of the i-th upsampling module is the downsampling feature data. When i is greater than 1, the input data of the i-th upsampling module is set feature data obtained by fusing the intermediate upsampling feature data output from the (i - 1)-th upsampling module and the intermediate downsampling feature data output from the downsampling module corresponding to the i-th upsampling module. The intermediate upsampling feature data output from the last upsampling module is used as the upsampling feature data. The apparatus according to claim 15, characterized in that

17. Specifically, the noise reduction module is used to input the upsampling feature data and the target image data into the output layer to perform a fusion process, and obtain the noise reduction image data output from the output layer. The apparatus according to claim 16, characterized in that

18. An electronic device including a memory and a processor, wherein a computer program is stored in the memory, and when the processor executes the computer program, the method according to any one of claims 1 to 12 is implemented. An electronic device, characterized in that

19. A computer-readable storage medium storing a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented. A computer-readable storage medium, characterized in that

20. A computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 12 is implemented. A computer program product, characterized in that

Images