Information processing device and method

Abstract

The present application relates to an information processing device and method, and a computer-readable storage medium, which includes a processing circuit configured to generate a synthetic noise image based on an input original image, process a black level in the synthetic noise image, and train a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model.

Description

[Technical field]

[0001] This application claims priority to a Chinese patent application bearing application number 202010408019.5 and entitled "Information Processing Apparatus and Method, and Computer-Readable Storage Medium" filed with the China Patent Office on May 14, 2020, the entire contents of which are incorporated herein by reference.

[0002] The present disclosure relates to the field of information processing technology, and in particular to processing black levels in images, and more particularly to an information processing apparatus and method, and a computer-readable storage medium. [Background technology]

[0003] In the field of image denoising, synthetic data is commonly used to train denoising models based on deep learning, since synthetic data is easy to obtain and analyze. However, research has shown that when the trained denoising model is tested in real scenarios, the denoising model trained based on synthetic noise images cannot effectively denoise images in real scenarios, affecting the quality of the denoised images. This phenomenon is more serious in dim environments, because the black level in the synthetic noise images is inaccurate. Summary of the Invention [Means for solving the problem]

[0004] The following provides a brief overview of the present invention to provide a basic understanding of certain aspects of the present invention. It should be understood that this overview is not an exhaustive overview of the present invention. It is not intended to identify key or critical parts of the present invention, nor is it intended to limit the scope of the present invention. Its purpose is merely to provide some concepts in a simplified form as a prelude to the more detailed description that follows.

[0005] According to one aspect of the present disclosure, there is provided an information processing device including a processing circuit configured to generate a synthetic noise image based on an input original image, process a black level in the synthetic noise image, and train a machine learning model for image noise removal based on the processed synthetic noise image to obtain a trained machine learning model.

[0006] In an information processing device according to an embodiment of the present disclosure, by processing the black level in a synthetic noise image, the impact of inaccuracies in the black level on the training of a machine learning model can be reduced, thereby improving the noise removal performance of a trained machine learning model trained based on the processed synthetic noise image.

[0007] According to another aspect of the present disclosure, there is provided an information processing device including a processing circuit configured to input an image to be processed into a trained machine learning model obtained by the information processing device and perform noise removal processing on the image to be processed.

[0008] An information processing apparatus according to an embodiment of the present disclosure can more accurately remove noise from an image to be processed.

[0009] According to another aspect of the present disclosure, there is provided an information processing method including: generating a synthetic noise image based on an input original image; processing a black level in the synthetic noise image; and training a machine learning model for image noise removal based on the processed synthetic noise image to obtain a trained machine learning model.

[0010] According to another aspect of the present disclosure, there is provided an information processing method including inputting an image to be processed into a trained machine learning model obtained by the information processing device, and performing noise removal processing on the image to be processed.

[0011] According to another aspect of the present disclosure, there is further provided a computer program code for realizing the above-described information processing method, a computer program product, and a computer-readable storage medium having computer program code recorded thereon for realizing the above-described information processing method. [Brief description of the drawings]

[0012] In order to further explain the above and other advantages and features of the present invention, the following will describe in more detail specific embodiments of the present invention in conjunction with the drawings. The drawings are incorporated in the present specification together with the following detailed description and form a part of the specification. Elements having the same functions and configurations are indicated by the same reference numerals. It should be noted that these drawings are merely typical examples of the present invention and should not be considered as limiting the scope of the present invention. In the drawings,

[0013] [Figure 1] FIG. 1 is a schematic diagram showing a conventional image sensor. [Diagram 2] FIG. 2 is a schematic diagram showing the black reference column. [Diagram 3] FIG. 3 is a functional module block diagram of an information processing device according to an embodiment of the present disclosure. [Figure 4] FIG. 4 shows an example of a color filter array of a sensor. [Diagram 5] FIG. 5 is a schematic diagram illustrating the image signal processing performed by a conventional digital camera. [Figure 6] FIG. 6 is a schematic diagram illustrating training a machine learning model by perturbing the black level according to an embodiment of the present disclosure. [Figure 7] FIG. 7 is a flowchart illustrating an example of a procedure of an information processing method according to an embodiment of the present disclosure. [Figure 8] FIG. 8 is a functional module block diagram of an information processing device according to an embodiment of the present disclosure. [Figure 9] FIG. 9 is a flowchart illustrating an example of a procedure of an information processing method according to an embodiment of the present disclosure. [Figure 10]FIG. 10 is a block diagram showing a schematic configuration of a personal computer that can be used in an embodiment of the present disclosure. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] First, the black level will be described below by combining a conventional image sensor and the like.

[0015] Fig. 1 is a schematic diagram showing a conventional image sensor. As shown in Fig. 1, photons (light signals) are irradiated onto a photodiode (can be equivalent to a sensor) array through a lens and an RGB color filter, the photodiode converts the light signals into photocurrent based on the photoelectric effect, and the photocurrent generated by each photodiode is amplified by an amplifier (not shown in Fig. 1) and sent to an ADC (analog-to-digital converter), and finally converted into a digital signal in the ADC.

[0016] Even in the absence of light, a photodiode generates a reverse DC current (i.e., dark current). Therefore, a photodiode is a device that generates dark current. To solve this problem, the conventional technique cancels the dark current by setting a black level.

[0017] The black level refers to the signal level when image data is 0 (i.e., no light is output) in a display device that has been calibrated to a certain extent. Display devices include, for example, digital photographing devices (including digital cameras and digital video cameras), televisions, mobile phones, displays, etc. Below, a method for obtaining the black level in the conventional technology will be described using a digital camera as an example. The black level is first obtained by calibrating a black reference column. Later, with the development of electronic devices, the variation of the dark current is not so large, so the black reference column was removed from the conventional camera and the black level was set manually (the calculation method of the black level varies depending on the camera manufacturer, and is usually set to a scalar such as 512).

[0018] FIG. 2 is a schematic diagram showing the black reference column.

[0019] As shown in Figure 2, the black reference column is designed next to the photodiode array and is an area that is not exposed to light. Generally, the size of the black reference column is 8 rows, and the black reference column also implements the same photodiodes as the photodiode array, but it is not exposed to light. The signal of the black reference column is also amplified by the amplifier, read out and ADC converted. Finally, the camera calculates the black level based on the digital signal map acquired by the black reference column through the ADC.

[0020] As mentioned above, a manually set or calculated black level may not accurately represent the dark current and may result in a black level error ε, which is more noticeable under dim light conditions.

[0021] In the following, exemplary embodiments of the present disclosure will be described in conjunction with the drawings. For the sake of clarity and conciseness, not all features of actual embodiments will be described in the specification. It should be understood that embodiment-specific decisions must be made in the course of developing such actual embodiments to achieve the developer's specific goals, for example, subject to system and business constraints, which may vary from embodiment to embodiment. It should also be understood that while the development work may be very complex and time-consuming, such development work would be a routine task for those of ordinary skill in the art having the benefit of this disclosure.

[0022] It should also be noted here that in order to avoid obscuring the present disclosure with unnecessary details, the drawings only show device configurations and / or processing steps closely related to the solutions of the present disclosure, and omit other details that are largely unrelated to the present disclosure.

[0023] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings.

[0024] FIG. 3 shows a functional module block diagram of an information processing device 300 according to an embodiment of the present disclosure. As shown in FIG. 3, the information processing device 300 includes a first processing unit 302 configured to generate a synthetic noise image based on an input original image, and a second processing unit 304 configured to process a black level in the synthetic noise image, and train a machine learning model for image noise removal based on the processed synthetic noise image to obtain a trained machine learning model.

[0025] As an example, the original image includes RGB data in a raw format (also called a raw-RGB image or raw image). A raw-RGB image directly reflects how photons generate electrons after passing through a sensor such as a CCD / CMOS (Charge-coupled Device / Complementary Metal Oxide Semiconductor). Figure 4 shows an example of a Color Filter Array (CFA) of a sensor (e.g., CCD / CMOS). Usually, raw format sensor data exists in the form of a CFA. A CFA is a pixel array with size HxW (H and W are the dimensions of the sensor, H indicates the height and W indicates the width), where each pixel has information about one color channel: red, green, or blue. The light that hits any particular photodiode in a CMOS is recorded as the number of electrons in a capacitor, so it can only be stored as a scalar and a single pixel cannot hold the three-dimensional characteristics of the light that can be observed. A CFA offers a compromise by capturing information about each of the three color channels at different locations through selectable filters via a spectrum set at each pixel. Hereinafter, the size of the original image will be expressed as H × W.

[0026] By way of example, the machine learning model is a neural network, which may be, for example, a convolutional neural network, such as a U-net neural network, a Senet neural network, or a SKnet neural network.

[0027] In the information processing device 300 according to an embodiment of the present disclosure, by processing the black level in the synthetic noise image, the impact of inaccuracies in the black level on the training of the machine learning model can be reduced, thereby improving the noise removal performance of the trained machine learning model trained based on the processed synthetic noise image.

[0028] Training a machine learning model based on synthetic noise images also saves time and effort. When training a machine learning model, collecting paired image data including an original image and a real noise image requires a collector to take two shots with a fixed camera and ensure that the object does not move between the two shots. However, because the synthetic noise image exists, there is no need to collect paired data including an original image and a real noise image in the field, and it is only necessary to synthesize a synthetic noise image based on the original image. Also, training a machine learning model based on synthetic noise images instead of paired images including an original image and a real noise image incurs less manual annotation costs. The manipulability of synthetic noise images allows for training a more robust machine learning model using less data.

[0029] As an example, the original image is a long-exposure image taken by an image capture device under a long exposure condition, and the synthetic noise image is used to simulate a short-exposure image taken by the image capture device in a dim environment.

[0030] For example, the image acquisition device is a digital photographing device (including a digital camera, a digital video camera, etc.). Those skilled in the art should understand that the image acquisition device may be a device capable of acquiring RGB data in RAW format other than a digital photographing device.

[0031] For example, the image acquisition device may be the information processing device 300, or the image acquisition device may be different from the information processing device 300 (for example, the image acquisition device may be a digital camera or a digital video camera, and the information processing device 300 may be a mobile phone or a computer, etc.). For convenience of explanation, the image acquisition device and the information processing device 300 are hereinafter assumed to be the same digital camera unless otherwise specified.

[0032] For example, the long exposure image and the short exposure image are images actually captured by an image capture device, the long exposure image is a clean, noise-free image captured under long exposure conditions, the short exposure image is a noisy image captured in a dim environment, and the synthetic noise image is generated based on an original image as the long exposure image and is used to simulate the short exposure image.

[0033] When the synthetic noise image is used to simulate a short-exposure image taken in a dim environment by an image capture device, the trained machine learning model can improve its denoising performance for the short-exposure image taken in a dim environment. The trained machine learning model can also be applied to tasks such as all-weather autonomous driving, nighttime face recognition, etc.

[0034] For example, the first processing unit 302 may be configured to perform a noise processing on the original image based on the noise parameters obtained from the original image to generate a synthetic noise image.

[0035] For example, the noise of the original image captured by a digital photographing device is mainly shot noise and read noise. The above noise parameters may be a parameter related to read noise (abbreviated as read noise parameter) α and a parameter related to shot noise (abbreviated as shot noise parameter) β obtained from the original image. Those skilled in the art can easily think of other examples of noise parameters, which will not be repeated here.

[0036] As an example, the first processing unit 302 can be configured to perform a linearization process on the original image based on the black level and the white level obtained from the original image to obtain a linearized image of the original image, and the first processing unit 302 can be configured to perform a noise process on the linearized image to generate a synthetic noise image.

[0037] For example, the white level refers to the signal level when the image data is at the peak white level on a display device that has been calibrated to some extent. As mentioned above, the display device may be, for example, a digital photographing device (including a digital camera and a digital video camera), a television, a mobile phone, a display, etc.

[0038] If the pixel value of a pixel in the original image is RawImage, the white level is n, and the black level is m, then the pixel value y of the corresponding pixel in the linearized image of the original image (hereinafter, for convenience of explanation, the linearized image of the original image may be simply referred to as image y) can be expressed as follows:

[0039]

number

[0040] In formula (1), max{} indicates taking the maximum value, and min{} indicates taking the minimum value. A linearization process is performed based on formula (1) to limit each pixel in the original image to a value in the range from 0 to 1, thereby obtaining a linearized image y of the original image.

[0041] For example, the gain factor of the linearized image y may be changed, eg, by dividing the pixels in the sub-image of the linearized image y by a predetermined value (eg, 100 or 300).

[0042] For example, by adding at least one of Gaussian noise, Poisson noise, Gaussian-Poisson noise, Heteroscedastic-Gaussian noise (wherein this noise n can be expressed as n to N(0, (β×y+α))) to image y based on a read noise parameter α and a shot noise parameter β obtained from the original image, a synthetic noise image with pixel values ​​z can be generated (hereinafter, the synthetic noise image will be simply referred to as synthetic noise image z).

[0043] For example, the second processing unit 304 can be configured to acquire predetermined image acquisition parameters from the original image, acquire a corresponding offset image based on a correspondence relationship between the predetermined image acquisition parameters and the offset image for processing the black level, and process the black level in the synthetic noise image based on the acquired offset image.

[0044] The information processing device 300 according to an embodiment of the present disclosure processes the black level in the synthetic noise image based on the acquired offset image, so that the black level in the processed synthetic noise image more accurately represents the dark current, thereby reducing the black level error ε and making the processed synthetic noise image closer to the actual noise image.

[0045] For example, the correspondence is generated based on a black image captured for each predetermined image acquisition parameter of the image capture device in a dim environment to obtain an offset image corresponding to each predetermined image acquisition parameter. The correspondence generated in this way is advantageous for processing the black level in the composite noise image in a dim environment.

[0046] For example, a black image can be an image captured when an image capture component in an image capture device is covered by a black object. Those skilled in the art can easily conceive of other ways to capture a black image, which will not be duplicated here.

[0047] For example, the image capture device is a digital photographing device, the predetermined image capture parameters include an exposure time and a light sensitivity (ISO) of the digital photographing device, and the correspondence relationship is generated by closing a lid of the digital photographing device, capturing a black image for each parameter pair of the exposure time and the light sensitivity of the digital photographing device, and pre-processing the black image to obtain an offset image corresponding to each parameter pair.

[0048] As described above, the image acquisition device may be a device capable of acquiring RGB data in RAW format other than a digital photographing device, and correspondingly, the predetermined image acquisition parameters may be parameters other than the exposure time and light sensitivity, which are not overlapped here. In the following, an example in which the image acquisition device is a digital photographing device will be described. In the following, for convenience of explanation, the digital photographing device may be called a digital camera or a camera.

[0049] The reason for using the exposure time and light sensitivity of the digital photographing device as the predetermined image acquisition parameters is that the black level error ε is affected by the exposure time and light sensitivity: on the one hand, the black level error ε is affected by the light sensitivity because the dark current is amplified by the amplifier and the black level error ε is also amplified, and on the other hand, the black level error ε is affected by the exposure time because the longer the exposure time, the more photocurrent and dark current are accumulated in the photodiode before the switch is closed.

[0050] For example, take a digital camera with or without a lens, close the camera lid, set the camera output format to RAW format file, set the remaining camera parameters to default, and take a picture to get one black image for each pair of ISO value and exposure value. For example, if the camera's exposure time can be selected from 0.1s and 1s, and the ISO value can be selected from 100 and 200, there are four pairs of ISO value and exposure value: (0.1s, 100), (1s, 100), (0.1s, 200), and (1s, 200). Therefore, you can take a picture to get a total of four black images.

[0051] As an example, pre-processing the black image includes performing a linearization process on the black image based on the black and white levels of the digital capture device.

[0052] If the black level of the digital photographing device is m', the white level is n', a pixel in the black image is BlackFrame, and the corresponding pixel in the offset image is BiasFrame, then the linearization process for the black image can be expressed as follows:

[0053]

number

[0054] In formula (2), max{} indicates taking the maximum value, and min{} indicates taking the minimum value. A linearization process is performed based on formula (2) to limit each pixel in the black image to a value in the range from 0 to 1, thereby obtaining an offset image.

[0055] As an example, the second processing unit 304 can be configured to add the acquired offset image to a synthetic noise image.

[0056] For example, the second processing unit 304 can add the value of each pixel in the acquired offset image to the corresponding pixel in the synthetic noise image z, and process the black level in the synthetic noise image z to generate a processed synthetic noise image.

[0057] As an example, the second processing unit 304 can be configured to select an image block from a predetermined position in the acquired offset image and add an offset value calculated based on the selected image block to the synthetic noise image.

[0058] For example, the offset image has a height H and a width W, and the predetermined position is ps. For example, pixels ranging from H / 2 to H / 2+ps along the height direction and from W / 2 to W / 2+ps along the width direction can be selected as the selected image block from the acquired offset image. Those skilled in the art should understand that any image block can be selected from the acquired offset image as the selected image block. For example, ps can be determined according to experience, application scenarios, experiments, etc., and for example, ps can be 128. Those skilled in the art should understand that ps can be determined so that a value corresponding to any image block can be selected from the acquired offset image. For example, an average value of pixels in the selected image block is calculated, and the average value is added to the value of each pixel of the synthetic noise image z to process the black level in the synthetic noise image z, thereby generating a processed synthetic noise image. Those skilled in the art should further understand that other values ​​than the average value can be calculated based on the values ​​of pixels in the selected image block and used to process the black level in the synthetic noise image z.

[0059] As an example, the second processing unit 304 may be configured to select a pixel from a predetermined location in the acquired offset image and add the value of the selected pixel to the synthetic noise image.

[0060] For example, a pixel value at [H / 2, W / 2] can be selected from the acquired offset image and added to the value of each pixel in the synthetic noise image z to process the black level in the synthetic noise image z to generate a processed synthetic noise image. Those skilled in the art should understand that a pixel value at any position can be selected from the acquired offset image to process the black level in the synthetic noise image z.

[0061] For example, the R, G1, B, and G2 values ​​of the processed synthetic noise image can each constitute a single channel, and these four channels can be cascaded. The image data of each channel (the size of the data of each channel is H / 2 × W / 2) can be input to the machine learning model as a sample.

[0062] In the above, the linearization process is performed on the original image, noise is added to the obtained linearized image y to obtain a synthetic noise image z, and black level processing is performed on the synthetic noise image z to obtain a processed synthetic noise image. Instead, the R, G1, B, and G2 values ​​of the linearized image y each constitute a single channel, and then noise is added to each of the R, G1, B, and G2 channels of the linearized image y, respectively, and then black level processing is performed on the R, G1, B, and G2 channels, respectively, and the image data of each processed channel is input into the machine learning model as a sample.

[0063] When training a machine learning model using the above processed synthetic noise image as a sample, the display image captured by the image capture device, which corresponds to the original image, is used as a label image for training the machine learning model. When training the machine learning model, the R, G1, B, and G2 values ​​of the label image each constitute a single channel, and correspond to the R, G1, B, and G2 channels of the processed synthetic noise image.

[0064] The above label image is described below by taking the image capture device as an example of a digital camera. The digital camera can be regarded as a signal conversion tool, which converts the optical signal in the actual scenario into a digital signal (optical signal->electrical signal->digital signal), and this conversion process is completed by the hardware of the digital camera. The digital signal acquired from the digital camera is the original image. Then, the digital camera performs image signal processing to convert this digital signal (original image) into an image (display image) that can be used for display, for example, the original image is a RAW format image taken by the digital camera, and the display image is an s-RGB format image obtained by converting the original image and displayed on the screen of the digital camera.

[0065] FIG. 5 is a schematic diagram illustrating the image signal processing performed by a conventional digital camera.

[0066] In FIG. 5, first, a linearization process is performed on the original image as shown in equation (1).

[0067] In Figure 5, white balancing is performed after the linearization process, and white balancing functions to eliminate the effect of colored light sources on the color of an object. For example, a red apple is photographed under red light and green light, and the photographing results may be completely different. However, the "color" of a red apple (the distribution property of the spectral reflectance on the surface of the apple) is an objective fact and should not change with the light source. In order to obtain the original color of the object, white balancing solves this problem. In fact, it is completed by multiplying the image by three multipliers [R, G, B] with a dot product.

[0068] In Fig. 5, demosaicing is performed after white balancing, which is essentially a linear interpolation operation aimed at converting a previous one-channel original image into a three-channel image that can be displayed on a display device, i.e., converting an original image of size H x W into three images of size H x W (images in s-RGB format).

[0069] In FIG. 5, color space conversion is performed after demosaicing. In a digital camera, a three-dimensional vector of (R, G, B) represents the value of a pixel, i.e., (R, G, B) is the color base used by the digital camera. However, the color base used by the digital camera is different from the color base of the display device, and the color space conversion operation aims to convert the color base of the digital camera into the color base usable by the display device. The color space conversion is realized by multiplying the image after demosaicing by a 3×3 conversion matrix, which is a fixed value. An image in s-RGB format is obtained by color space conversion. That is, the image obtained by image signal processing is an image (display image) in s-RGB format.

[0070] For example, the parameters of the machine learning model can be updated based on the display image (label image) and the processed synthetic noise image. For example, the average distance is calculated in the second normal form for the display image and the processed synthetic noise image, and the parameters of the machine learning model are updated according to backpropagation to obtain a trained machine learning model.

[0071] As an example, the second processing unit 304 can be configured to extract an offset value from a numerical range of offset values ​​following a predetermined distribution, process a black level of an image block in the synthetic noise image based on the extracted offset value, and train a machine learning model based on the processed image block.

[0072] For example, a person skilled in the art can predetermine the size of the image block through experience, application scenarios and experiments, for example, the image block may be a 1024x1024 image block.

[0073] Manipulating the black level of the image block in the synthetic noise image based on the extracted offset value essentially perturbs the black level of the image block.

[0074] The information processing device 300 according to the embodiment of the present disclosure can increase the robustness of the machine learning model with respect to the black level by perturbing the black level.

[0075] For example, the second processing unit 304 can add the extracted offset value to each pixel of an image block in the synthetic noise image z to process the black level in the image block.

[0076] For example, the predetermined distribution can be a Gaussian distribution or a uniform distribution. For example, the parameters of the distribution function of the above-mentioned predetermined distribution are determined based on experience, application scenarios, experiments, etc. For example, the uniform distribution can be a uniform distribution according to (-10, 10).

[0077] As an example, the second processing unit 304 can be configured to randomly draw the offset value from the above numerical range.

[0078] 6 is a schematic diagram illustrating training a machine learning model by perturbing the black level according to an embodiment of the present disclosure. In FIG. 6, an example is taken of an image block of size 1024×1024.

[0079] As shown in FIG. 6, in (1), an image block is extracted from the synthetic noise image z generated after performing linearization processing on the original image and adding noise. In (2), a black level in the image block is processed by perturbation. In (3), the R, G1, B, and G2 values ​​of the processed image block are each configured as a single channel to obtain four data blocks of size (1024 / 2)×(1024 / 2) (i.e., the size of each data block is 512×512), and these four channels are cascaded. In (4), the above four data blocks of size 512×512 are input to the machine learning model as samples, and the corresponding image blocks in the display image in s-RGB format obtained by performing image signal processing on the original image in (5) are used as training labels to train the machine learning model. For example, the average distance is calculated in the second normal form for the image blocks as samples and the image blocks as labels, and the parameters of the machine learning model are updated according to backpropagation to obtain a trained machine learning model. The image signal processing in (5) is the same as that in FIG.

[0080] While the above describes perturbing the black level of an image block in the synthetic noise image, those skilled in the art should understand that the black level of the synthetic noise image may further be perturbed based on an extracted offset value (e.g., adding an extracted offset value to each pixel of the synthetic noise image), which will not be repeated here.

[0081] Corresponding to the above-described embodiment of the information processing device, the present disclosure further provides an embodiment of an information processing method.

[0082] FIG. 7 is a flowchart showing an example of a procedure of an information processing method S700 according to an embodiment of the present disclosure.

[0083] An information processing method S700 according to an embodiment of the present disclosure begins at S702.

[0084] In S704, a synthetic noise image is generated based on the input original image.

[0085] For specific examples of the original image, the description of the corresponding part (eg, the first processing unit 302) of the information processing device 300 can be referred to, and therefore will not be repeated here.

[0086] At S706, the black level in the synthetic noise image is processed, and a machine learning model for image denoising is trained based on the processed synthetic noise image to obtain a trained machine learning model.

[0087] For a specific example of the black level processing, the description of the corresponding part (eg, the second processing unit 304) of the information processing device 300 can be referred to, and therefore will not be repeated here.

[0088] The information processing method S700 ends at S708.

[0089] In the information processing method S700 according to an embodiment of the present disclosure, by processing the black level in the synthetic noise image, the impact of inaccuracy in the black level on the training of the machine learning model can be reduced, thereby improving the noise removal performance of the trained machine learning model trained based on the processed synthetic noise image.

[0090] According to another embodiment of the present disclosure, an information processing device 800 is further provided.

[0091] FIG. 8 shows a functional module block diagram of an information processing device 800 according to an embodiment of the present disclosure. As shown in FIG. 8, the information processing device 800 includes a third processing unit 802 configured to input an image to be processed into a trained machine learning model obtained by the information processing device 300 or the information processing method S700, and perform noise removal processing on the image to be processed.

[0092] Since the information processing device 300 or the information processing method S700 has obtained a trained machine learning model with improved noise removal performance, the information processing device 800 according to an embodiment of the present disclosure can more accurately remove noise in the image to be processed.

[0093] For example, the image to be processed is a short-exposure image taken in a dim environment. The information processing device 800 can more accurately remove noise from a short-exposure image taken in a dim environment.

[0094] Corresponding to the above-described embodiment of the information processing device, the present disclosure further provides an embodiment of an information processing method.

[0095] FIG. 9 is a flowchart showing an example of a procedure of an information processing method S900 according to an embodiment of the present disclosure.

[0096] An information processing method S900 according to an embodiment of the present disclosure begins at S902.

[0097] In S904, the image to be processed is input to the information processing device 300 or the trained machine learning model obtained by the information processing method S700, and noise removal processing is performed on the image to be processed.

[0098] The information processing method S900 ends at S906.

[0099] Since the information processing device 300 or the information processing method S700 has obtained a trained machine learning model with improved noise removal performance, the information processing method S900 according to an embodiment of the present disclosure can more accurately remove noise in the image to be processed.

[0100] The above describes the basic principles of the present invention by combining specific embodiments. However, it will be understood by those skilled in the art that all or any of the steps or components of the method and apparatus of the present invention can be realized in the form of hardware, firmware, software, or a combination thereof in any computer device (including a processor, a storage medium, etc.) or a network of computer devices, and this can be realized by those skilled in the art after reading the description of the present invention and utilizing their basic circuit design knowledge or basic programming skills.

[0101] And the present invention further proposes a program product having machine-readable instruction codes stored therein, which, when read and executed by a machine, can perform the above method according to the embodiment of the present invention.

[0102] Correspondingly, the disclosure of the present invention also includes a storage medium carrying a program product having the above-mentioned machine-readable instruction code stored therein, the storage medium including, but not limited to, a flexible disk, an optical disk, a magneto-optical disk, a memory card, a memory stick, etc.

[0103] When realizing the present invention by software or firmware, the programs that constitute the software are installed from a storage medium or network into a computer having a dedicated hardware configuration (e.g., general-purpose computer 1000 shown in FIG. 10), and when various programs are installed, the computer can perform various functions.

[0104] 10, a central processing unit (CPU) 1001 executes various processes based on a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage unit 1008 to a random access memory (RAM) 1003. The RAM 1003 stores data required when the CPU 1001 executes various processes, etc., as necessary. The CPU 1001, the ROM 1002, and the RAM 1003 are connected to one another via a bus 1004. An input / output interface 1005 is also connected to the bus 1004.

[0105] An input section 1006 (including a keyboard, a mouse, etc.), an output section 1007 (including a display such as a cathode ray tube (CRT) or a liquid crystal display (LCD), and a speaker, etc.), a storage section 1008 (including a hard disk, etc.), and a communication section 1009 (including a network interface card such as a LAN card, a modulator / demodulator, etc.) are connected to the input / output interface 1005. The communication section 1009 executes communication processing via a network such as the Internet. If necessary, a drive 1010 may be connected to the input / output interface 1005. A removable medium 1011 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc. is mounted in the drive 1010 as necessary, and a computer program read from the removable medium 1011 is installed in the storage section 1008 as necessary.

[0106] When the above series of processes are realized by software, the programs constituting the software are installed from a network such as the Internet, or from a storage medium such as the removable medium 1011.

[0107] Those skilled in the art should understand that such a storage medium is not limited to the removable medium 101 in which the program is stored and which is distributed separately to the device to provide the program to the user as shown in Fig. 10. Examples of the removable medium 1011 include magnetic disks (including flexible disks (registered trademark)), optical disks (including optical disk read only memory (CD-ROM) and digital versatile disk (DVD)), magneto-optical disks (including mini disks (MD) (registered trademark)), and semiconductor memories. Alternatively, the storage medium may be a ROM 1002, a hard disk included in the storage portion 1008, etc., in which the program is stored and which is distributed to the user together with the device containing them.

[0108] In the device, method, and system of the present invention, each component or each step can be disassembled and / or recombined. Such disassembly and / or recombination should also be considered as an equivalent solution of the present invention. Note that the execution steps of the above series of processes can be performed in the order of description or chronological order, but are not necessarily required to be performed in chronological order. Some steps may be performed in parallel or independently of each other.

[0109] Finally, the terms "comprise," "including," or any other variation thereof, are intended to include a non-exclusive inclusion, whereby a process, method, article, or device comprising a set of elements not only includes those elements, but also includes other elements not expressly listed, or includes the inherent elements of such process, method, article, or device. Also, unless specifically limited, an element qualified by the phrase "comprises a..." does not exclude the presence of additional identical elements in the process, method, article, or device that includes the elements listed.

[0110] Although the above describes the embodiments of the present disclosure in detail with reference to the drawings, it should be understood that the above-described embodiments are only for explaining the present disclosure and do not limit the present disclosure. Those skilled in the art can make various modifications and changes to the above-described embodiments without departing from the spirit and scope of the present disclosure. Therefore, the scope of the present disclosure is limited only by the appended claims and their equivalents.

[0111] The present technology can be further implemented as follows. Appendix 1. An information processing device, Generate a synthetic noise image based on the input original image; 11. An information processing apparatus comprising: a processing circuit configured to process a black level in the composite noise image; and train a machine learning model for image denoising based on the processed composite noise image to obtain a trained machine learning model. Appendix 2. An information processing device as described in Appendix 1, wherein the processing circuit is configured to acquire predetermined image acquisition parameters from the original image, acquire a corresponding offset image based on a correspondence between the predetermined image acquisition parameters and an offset image for processing a black level, and process the black level in the synthetic noise image based on the acquired offset image. Appendix 3. The above correspondence is as follows: The information processing device according to Appendix 2, which generates an offset image corresponding to each predetermined image acquisition parameter based on a black image captured for each predetermined image acquisition parameter of the image acquisition device in a dim environment. Appendix 4. The image capture device is a digital photographing device, and the predetermined image capture parameters include an exposure time and a light sensitivity of the digital photographing device; The correspondence relationship is The information processing device described in Appendix 3 is generated by closing a lid of the digital photographing device, capturing a black image for each parameter pair of exposure time and light sensitivity of the digital photographing device, and pre-processing the black image to obtain an offset image corresponding to each of the parameter pairs. Appendix 5. An information processing device according to Appendix 4, wherein pre-processing the black image includes performing a linearization process on the black image based on a black level and a white level of the digital photographing device. Appendix 6. An information processing device according to any one of Appendixes 2 to 5, wherein the processing circuitry is configured to add an acquired offset image to the synthetic noise image. Appendix 7. An information processing device described in any one of Appendices 2 to 5, wherein the processing circuit is configured to select an image block from a predetermined position in the acquired offset image and add an offset value calculated based on the selected image block to the synthetic noise image. Appendix 8. An information processing device according to any one of Appendixes 2 to 5, wherein the processing circuitry is configured to select a pixel from a predetermined position in the acquired offset image and add the value of the selected pixel to the synthetic noise image. Appendix 9. The processing circuit comprises: Extracting an offset value from a range of offset values ​​that follow a predetermined distribution; The information processing device of Appendix 1, configured to process black levels of image blocks in the synthetic noise image based on the extracted offset values, and to train the machine learning model based on the processed image blocks. Appendix 10. An information processing device according to Appendix 9, wherein the predetermined distribution is a Gaussian distribution or a uniform distribution. Appendix 11. An information processing device according to Appendix 9 or 10, wherein the processing circuitry is configured to randomly extract the offset value from the numerical range. Appendix 12. An information processing device described in any one of Appendices 1 to 11, wherein the processing circuit is configured to perform a noise generation process on the original image based on noise parameters obtained from the original image, to generate the synthetic noise image. Appendix 13. The processing circuit comprises: performing a linearization process on the original image based on the black level and the white level obtained from the original image to obtain a linearized image of the original image; 13. An information processing device according to Appendix 12, configured to perform the noise generation process on the linearized image to generate the synthetic noise image. Appendix 14. An information processing device described in any one of Appendices 1 to 13, wherein the original image is a long-exposure image captured under a long-exposure condition by an image capture device, and the synthetic noise image is used to simulate a short-exposure image captured in a dim environment by the image capture device. Appendix 15. An information processing device according to any one of Appendixes 1 to 14, wherein the original image includes RGB data in a RAW format. Appendix 16. An information processing device described in any one of Appendices 1 to 15, wherein the machine learning model is a neural network. Appendix 17. An information processing device, An information processing device including a processing circuit configured to input an image to be processed into a trained machine learning model obtained by an information processing device described in any one of Appendices 1 to 16, and perform noise removal processing on the image to be processed. Appendix 18. An information processing device according to Appendix 17, wherein the image to be processed is a short-exposure image taken in a dim environment. Appendix 19. An information processing method comprising: generating a synthetic noise image based on an input original image; processing a black level in the synthetic noise image; and training a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model. Appendix 20. An information processing method comprising: An information processing method comprising: inputting an image to be processed into a trained machine learning model obtained by an information processing device described in any one of Appendices 1 to 16, and performing noise removal processing on the image to be processed. Appendix 21. A computer readable storage medium having stored thereon computer executable instructions which, when executed, cause the information processing method of Appendix 19 or 20 to be performed.

Claims

1. An information processing device, Generate a synthetic noise image based on the input original image; a processing circuit configured to process a black level in the synthetic noise image and train a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model; The processing circuit is configured to acquire predetermined image acquisition parameters from the original image, acquire a corresponding offset image based on a correspondence relationship between the predetermined image acquisition parameters and an offset image for processing a black level, and process the black level in the synthetic noise image based on the acquired offset image.

2. The correspondence relationship is The information processing apparatus according to claim 1 , wherein the offset images corresponding to the predetermined image acquisition parameters are generated based on black images captured in a dim environment for each predetermined image acquisition parameter of the image acquisition device.

3. the image capture device is a digital photographic device, and the predetermined image capture parameters include an exposure time and a light sensitivity of the digital photographic device; The correspondence relationship is The information processing device of claim 2, wherein the information processing device is generated by closing a lid of the digital photographing device, capturing a black image for each parameter pair of exposure time and light sensitivity of the digital photographing device, and pre-processing the black image to obtain an offset image corresponding to each of the parameter pairs.

4. The information processing apparatus of claim 3 , wherein pre-processing the black image includes performing a linearization process on the black image based on a black level and a white level of the digital photographing device.

5. An information processing apparatus according to any one of claims 1 to 4, wherein the processing circuitry is configured to add an acquired offset image to the synthetic noise image.

6. 5. The information processing device according to claim 1, wherein the processing circuitry is configured to select an image block from a predetermined position in the acquired offset image, and to add an offset value calculated based on the selected image block to the synthetic noise image.

7. 5. The information processing apparatus of claim 1, wherein the processing circuitry is configured to select pixels from predetermined positions in the acquired offset image and add values ​​of the selected pixels to the synthetic noise image.

8. The processing circuitry includes: Extracting an offset value from a range of offset values ​​that follow a predetermined distribution; The information processing device of claim 1 , configured to process black levels of image blocks in the synthetic noise image based on the extracted offset values, and to train the machine learning model based on the processed image blocks.

9. The information processing device according to claim 8 , wherein the predetermined distribution is a Gaussian distribution or a uniform distribution.

10. The information processing device according to claim 8 or 9, wherein the processing circuitry is configured to randomly extract the offset value from the range of values.

11. The information processing device according to claim 1 , wherein the processing circuit is configured to perform a noise generation process on the original image based on a noise parameter obtained from the original image, to generate the synthetic noise image.

12. The processing circuitry includes: performing a linearization process on the original image based on the black level and the white level obtained from the original image to obtain a linearized image of the original image; The information processing apparatus according to claim 11 , configured to perform the noise generation process on the linearized image to generate the synthetic noise image.

13. The information processing device according to any one of claims 1 to 12, wherein the original image is a long-exposure image captured by an image capture device under a long-exposure condition, and the synthetic noise image is used to simulate a short-exposure image captured by the image capture device in a dim environment.

14. The information processing device according to any one of claims 1 to 13, wherein the original image includes RGB data in a RAW format.

15. The information processing device according to claim 1 , wherein the machine learning model is a neural network.

16. An information processing device, An information processing device including a processing circuit configured to input an image to be processed into a trained machine learning model obtained by the information processing device according to any one of claims 1 to 15, and to perform noise removal processing on the image to be processed.

17. The information processing apparatus according to claim 16 , wherein the image to be processed is a short-exposure image taken in a dim environment.

18. 1. An information processing method, comprising: generating a synthetic noise image based on an input original image; processing a black level in the synthetic noise image and training a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model; An information processing method comprising: acquiring predetermined image acquisition parameters from the original image; acquiring a corresponding offset image based on a correspondence relationship between the predetermined image acquisition parameters and an offset image for processing a black level; and processing the black level in the synthetic noise image based on the acquired offset image.

Images