Information processing apparatus and method, computer readable storage medium

Abstract

The present application relates to an information processing device and method, a computer readable storage medium. Wherein, the information processing device comprises processing circuitry configured to: generate a synthetic noise image based on an input original image; and process a black level in the synthetic noise image, and train a machine learning model for denoising an image based on the processed synthetic noise image to obtain a trained machine learning model.

Description

[0001] This application claims priority to Chinese Patent Application No. 202010408019.5, filed on May 14, 2020, entitled "Information Processing Apparatus and Method, Computer-Readable Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This disclosure relates to the field of information processing technology, specifically to processing black levels in an image. More specifically, it relates to an information processing apparatus and method, and a computer-readable storage medium. Background Technology

[0003] In the field of image denoising, synthetic data is often used to train deep learning-based denoising models because it is easy to acquire and analyze. However, research shows that when testing these trained denoising models in real-world scenes, models trained on synthetically noisy images fail to denoise images effectively, thus affecting the quality of the denoised images. This phenomenon is exacerbated under low-light conditions. This is due to the inaccurate black levels in synthetically noisy images. Summary of the Invention

[0004] A brief overview of the invention is given below to provide a basic understanding of certain aspects of it. It should be understood that this overview is not an exhaustive summary of the invention. It is not intended to identify key or essential parts of the invention, nor is it intended to limit the scope of the invention. Its purpose is merely to present certain concepts in a simplified form as a prelude to the more detailed description that follows.

[0005] According to one aspect of this disclosure, an information processing apparatus is provided, comprising processing circuitry configured to: generate a synthetic noise image based on an input original image; process the black level in the synthetic noise image; and train a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model.

[0006] In the information processing apparatus according to embodiments of the present disclosure, processing the black level in a synthetic noise image can mitigate the impact of inaccurate black levels on training a machine learning model, thereby improving the denoising performance of a trained machine learning model trained based on the processed synthetic noise image.

[0007] According to another aspect of this disclosure, an information processing apparatus is provided, comprising a processing circuit configured to input an image to be processed into a trained machine learning model obtained by the aforementioned information processing apparatus to perform denoising processing on the image to be processed.

[0008] The information processing apparatus according to embodiments of the present disclosure can remove noise from the image to be processed more accurately.

[0009] According to another aspect of this disclosure, an information processing method is provided, comprising: generating a synthetic noise image based on an input original image; processing the black level in the synthetic noise image, and training a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model.

[0010] According to another aspect of this disclosure, an information processing method is provided, comprising: inputting an image to be processed into a trained machine learning model obtained by the aforementioned information processing device to perform denoising processing on the image to be processed.

[0011] According to other aspects of the present invention, computer program code and computer program product for implementing the above-described information processing method, as well as a computer-readable storage medium having the computer program code for implementing the above-described information processing method recorded thereon, are also provided. Attached Figure Description

[0012] To further illustrate the above and other advantages and features of the present invention, specific embodiments of the present invention will be described in more detail below with reference to the accompanying drawings. The accompanying drawings, together with the following detailed description, are included in and form a part of this specification. Elements having the same function and structure are indicated by the same reference numerals. It should be understood that these drawings only depict typical examples of the invention and should not be construed as limiting the scope of the invention. In the drawings:

[0013] Figure 1 This is a schematic diagram showing an existing image sensor.

[0014] Figure 2 This is a summary diagram showing the black baseline column.

[0015] Figure 3 A functional block diagram of an information processing device according to an embodiment of the present disclosure is shown.

[0016] Figure 4 This is a diagram showing an example of a color filter array for a sensor.

[0017] Figure 5 This is a schematic diagram illustrating the image signal processing performed by existing digital cameras.

[0018] Figure 6 This is a schematic diagram illustrating a machine learning model trained by perturbing the black level according to an embodiment of the present disclosure.

[0019] Figure 7 This is a flowchart illustrating an example of an information processing method according to an embodiment of the present disclosure.

[0020] Figure 8 A functional block diagram of an information processing apparatus according to an embodiment of the present disclosure is shown.

[0021] Figure 9 This is a flowchart illustrating an example of an information processing method according to an embodiment of the present disclosure.

[0022] Figure 10 This is a block diagram illustrating an example structure of a personal computer that may be employed as an embodiment of this disclosure. Detailed Implementation

[0023] The following text will first introduce black level in conjunction with existing image sensors and other technologies.

[0024] Figure 1 This is a schematic diagram showing existing image sensors. (Example) Figure 1 As shown, photons (light signals) pass through a lens and an RGB color filter and illuminate an array of photodiodes (which can be equivalent to sensors). The photodiodes convert the light signals into photocurrents through the photoelectric effect. The photocurrent generated by each photodiode is amplified (in...) Figure 1 (Not shown in the image) The signal is amplified and transmitted to the ADC (Analog-to-Digital Converter), where it is finally converted into a digital signal.

[0025] Even in the absence of light, photodiodes generate a reverse direct current (i.e., dark current), thus making them devices that produce dark current. To address this issue, existing technologies use a black level to counteract the dark current.

[0026] Black level refers to the signal level corresponding to zero image data (i.e., no light output) on a calibrated display device. Display devices include, for example, digital video recording devices (including digital cameras and digital camcorders), televisions, mobile phones, and monitors. The following uses a digital camera as an example to introduce existing methods for obtaining black level. Initially, black level was obtained through calibration using black reference columns. Later, due to advancements in electronic devices, changes in dark current became less significant, and existing cameras eliminated the black reference columns, adopting a manually set black level (the calculation method for black level varies depending on the camera manufacturer, but it is usually set to a scalar, such as 512).

[0027] Figure 2 This is a summary diagram showing the black baseline column.

[0028] like Figure 2As shown, the black reference column is a region designed next to the photodiode array that cannot be exposed to light. Typically, the black reference column is 8 rows in size and contains photodiodes similar to those in the photodiode array, but these are not illuminated. The signal on the black reference column is amplified, read out, and converted by an ADC. Finally, the camera calculates the black level based on the digital signal obtained from the black reference column through the ADC.

[0029] As mentioned above, the black level set or calculated by humans cannot accurately represent the dark current, but will produce a black level error ε, and this error becomes more obvious in low light.

[0030] Exemplary embodiments of the present disclosure will be described below with reference to the accompanying drawings. For clarity and brevity, not all features of actual implementations are described in the specification. However, it should be understood that many implementation-specific decisions must be made in the development of any such actual embodiment to achieve the developer’s specific goals, such as complying with constraints related to the system and business, and these constraints may vary from implementation to implementation. Furthermore, it should be understood that while development work can be very complex and time-consuming, such development work is merely a routine task for those skilled in the art who benefit from the present disclosure.

[0031] It should also be noted that, in order to avoid obscuring this disclosure with unnecessary details, only the equipment structure and / or processing steps closely related to the solution according to this disclosure are shown in the accompanying drawings, while other details that are not closely related to this disclosure are omitted.

[0032] The embodiments according to this disclosure are described in detail below with reference to the accompanying drawings.

[0033] Figure 3 A functional block diagram of an information processing device 300 according to an embodiment of the present disclosure is shown, such as... Figure 3 As shown, the information processing device 300 includes: a first processing unit 302, which can be configured to generate a synthetic noise image based on an input original image; and a second processing unit 304, which can be configured to process the black level in the synthetic noise image and train a machine learning model for image denoising based on the processed synthetic noise image to obtain a trained machine learning model.

[0034] As an example, the raw image includes RGB data in RAW format (also known as a raw-RGB image or raw image). A raw-RGB image can directly reflect how photons generate electrons after passing through a sensor such as a CCD / CMOS (charge-coupled device / complementary metal-oxide-semiconductor). Figure 4This is a diagram illustrating an example of a color filter array (CFA) for a sensor (e.g., CCD / CMOS). Sensor data in RAW format is typically stored in the form of a CFA. A CFA is an H x W pixel array (where H and W are the dimensions of the sensor, where H represents height and W represents width), where each pixel carries information about a single color channel, either red, green, or blue. Since light falling on any given photodiode in a CMOS sensor is recorded as the number of electrons in a capacitor, it can only be stored as a scalar value; a single pixel cannot preserve the three-dimensional characteristics of the observable light. A CFA offers a trade-off: by means of a spectrally selectable filter placed on each pixel, information about each of the three color channels is captured at different locations. In the following text, the size of the original image is denoted as H x W.

[0035] As an example, a machine learning model is a neural network. For instance, a neural network can be a convolutional neural network, such as a U-net neural network, a Senet neural network, or an SKnet neural network.

[0036] In the information processing apparatus 300 according to an embodiment of the present disclosure, processing the black level in the synthetic noise image can mitigate the impact of inaccurate black level on the training of the machine learning model, thereby improving the denoising performance of the trained machine learning model trained based on the processed synthetic noise image.

[0037] Furthermore, training machine learning models based on synthetic noise images is more time-efficient and labor-saving. When training a machine learning model, collecting paired image data including the original image and real noise images means that the photographer needs to hold the camera still for two shots, ensuring the object doesn't move between them. However, with synthetic noise images, it's unnecessary to physically collect paired data including the original image and real noise images; the synthetic noise image can be synthesized directly from the original image. Additionally, training a machine learning model based on synthetic noise images, rather than paired images including the original image and real noise images, requires less manual annotation. The ease of using synthetic noise images allows for training more robust machine learning models with less data.

[0038] As an example, the original image is a long-exposure image captured by the image acquisition device under long-exposure conditions, and the synthesized noise image is used to simulate a short-exposure image captured by the image acquisition device in a low-light environment.

[0039] For example, the image acquisition device can be a digital photography device (including a digital camera or digital video camera, etc.). Those skilled in the art will understand that the image acquisition device can also be any device other than a digital photography device capable of acquiring RGB data in RAW format.

[0040] For example, the image acquisition device may be the same as the information processing device 300, or the image acquisition device may be different from the information processing device 300 (for example, the image acquisition device may be a digital camera or a digital video camera, and the information processing device 300 may be a mobile phone or a computer, etc.). In the following text, for the sake of convenience, unless otherwise specified, the image acquisition device and the information processing device 300 will be referred to as the same digital camera.

[0041] For example, long exposure images and short exposure images are images actually captured by the image acquisition device. A long exposure image is a clean, noise-free image captured under long exposure conditions, while a short exposure image is a noisy image captured in low light conditions. A synthetic noisy image is generated based on the original image used as a long exposure image and is used to simulate a short exposure image.

[0042] When synthesizing noisy images is used to simulate short-exposure images captured by an image acquisition device in low-light environments, the trained machine learning model improves the denoising performance of short-exposure images captured in low-light environments. This trained machine learning model can also be applied to tasks such as all-weather autonomous driving and nighttime face recognition.

[0043] As an example, the first processing unit 302 can be configured to noise-encode the original image based on noise parameters obtained from the original image, thereby generating a synthetic noise image.

[0044] For example, the noise in a raw image captured by a digital camera device is mainly shot noise and readout noise. The aforementioned noise parameters can be a parameter related to readout noise (hereinafter referred to as the readout noise parameter) α obtained from the raw image and a parameter related to shot noise (hereinafter referred to as the shot noise parameter) β. Other examples of noise parameters may be conceived by those skilled in the art, and will not be elaborated here.

[0045] As an example, the first processing unit 302 can be configured to linearize the original image based on the black and white levels obtained from the original image to obtain a linearized image of the original image; and the first processing unit 302 can be configured to noise-encode the linearized image to generate a synthetic noise image.

[0046] For example, white level refers to the signal level corresponding to the peak white value of image data on a calibrated display device. As mentioned above, display devices include, for example, digital camera devices (including digital cameras and digital video cameras), televisions, mobile phones, monitors, etc.

[0047] Let the pixel value of a pixel in the original image be RawImage, the white level be n, and the black level be m. Then the pixel value y of the corresponding pixel in the linearized image of the original image (in the following text, for convenience, the linearized image of the original image is sometimes simply referred to as image y) can be expressed as:

[0048]

[0049] In expression (1), max{} represents taking the maximum value and min{} represents taking the minimum value. By performing linearization based on expression (1), each pixel in the original image is restricted to a value in the range of 0-1, thus obtaining the linearized image y of the original image.

[0050] For example, the gain factor of the linearized image y can be changed, for example, by dividing the pixels in a local image of the linearized image y by a predetermined value (e.g., 100 or 300).

[0051] For example, based on the readout noise parameter α and shot noise parameter β obtained from the original image, at least one of Gaussian noise, Poisson noise, Gaussian-Poisson noise, and Heteroscedastic-Gaussian noise (the noise n can be represented as n~N(0,(β×y+α))) can be added to the image y to generate a synthetic noise image with pixel value z (hereinafter, the synthetic noise image is simply referred to as the synthetic noise image z).

[0052] As an example, the second processing unit 304 can be configured to obtain predetermined image acquisition parameters from the original image, obtain a corresponding bias image based on the correspondence between the predetermined image acquisition parameters and the bias image used for processing black levels, and process the black levels in the synthetic noise image based on the obtained bias image.

[0053] According to an embodiment of the present disclosure, the information processing device 300 processes the black level in the synthetic noise image based on the obtained bias image, so that the black level in the processed synthetic noise image more accurately represents the dark current, thereby reducing the black level error ε, and making the processed synthetic noise image closer to the real noise image.

[0054] As an example, the correspondence is generated as follows: In a low-light environment, based on black images captured for each predetermined image acquisition parameter of the image acquisition device, a bias image corresponding to each predetermined image acquisition parameter is obtained. The correspondence generated in this way is beneficial for processing black levels in synthetic noise images in low-light environments.

[0055] For example, a black image can be an image captured when the image acquisition component in an image acquisition device is obscured by a black object. Other methods for capturing black images will also be conceived by those skilled in the art, and will not be elaborated upon here.

[0056] As an example, the image acquisition device is a digital photography device and the predetermined image acquisition parameters include the exposure time and ISO of the digital photography device; and the correspondence is generated by closing the cover of the digital photography device, capturing a black image for each parameter pair of the exposure time and ISO of the digital photography device, and preprocessing the black image to obtain an offset image corresponding to each parameter pair.

[0057] As described above, the image acquisition device can be any device capable of acquiring RAW format RGB data, other than a digital photography device, and correspondingly, the predetermined image acquisition parameters can be parameters other than exposure time and ISO sensitivity, which will not be elaborated here. In the following description, a digital photography device will be used as an example of the image acquisition device. For ease of description, the digital photography device will sometimes be referred to as a digital camera or simply a camera.

[0058] The reason for using the exposure time and ISO sensitivity of a digital camera as predetermined image acquisition parameters is that the black level error ε is affected by both exposure time and ISO sensitivity. On one hand, dark current is amplified by the amplifier, and thus the black level error ε is also amplified, therefore the black level error ε is affected by ISO sensitivity. On the other hand, when the exposure time increases, both the photocurrent and dark current accumulated by the photodiode before the switch is closed increase, therefore the black level error ε is affected by the exposure time.

[0059] For example, take a digital camera, with or without a lens, close the camera cap, set the camera output format to .RAW file, and set the other camera parameters to default values; for each pair of ISO and exposure values, capture a black image. For example, assuming the camera has exposure times of 0.1s and 1s, and ISO values ​​of 100 and 200, there are four pairs of ISO and exposure values: (0.1s, 100), (1s, 100), (0.1s, 200), and (1s, 200). Therefore, a total of four black images can be captured.

[0060] As an example, preprocessing a black image includes linearizing the black image based on the black and white levels of the digital camera device.

[0061] Let the black level of the digital camera device be m', the white level be n', the pixel in the black image be BlackFrame, and the corresponding pixel in the bias image be BiasFrame. Then the linearization process performed on the black image above can be expressed as:

[0062]

[0063] In expression (2), max{} represents taking the maximum value and min{} represents taking the minimum value. By performing linearization based on expression (2), each pixel in the black image is restricted to the range of 0-1, thereby obtaining the bias image.

[0064] As an example, the second processing unit 304 can be configured to add the obtained bias image to the synthetic noise image.

[0065] For example, the second processing unit 304 can add the value of each pixel of the obtained bias image to the corresponding pixel in the synthetic noise image z to process the black level in the synthetic noise image z, thereby generating a processed synthetic noise image.

[0066] As an example, the second processing unit 304 may be configured to select an image patch from a predetermined position in the obtained bias image and add a bias value calculated based on the selected image patch to the synthetic noise image.

[0067] For example, the bias image has a height of H and a width of W, with a predetermined position of ps. For example, pixels located in the range of H / 2 to H / 2+ps along the height direction and W / 2 to W / 2+ps along the width direction can be selected from the obtained bias image as selected image blocks. Those skilled in the art will understand that any image block can be selected from the obtained bias image. For example, ps can be determined based on experience, application scenarios, or experiments; for example, ps can be 128. Those skilled in the art will understand that ps can be determined as a value corresponding to the selection of any image block from the obtained bias image. For example, the mean value of the pixels in the selected image block can be calculated and added to the value of each pixel in the synthesized noise image z to process the black level in the synthesized noise image z, thereby generating a processed synthesized noise image. Those skilled in the art will understand that other values ​​besides the mean value can also be calculated based on the pixel values ​​in the selected image block for processing the black level in the synthesized noise image z.

[0068] As an example, the second processing unit 304 can be configured to select pixels from predetermined positions in the obtained bias image and add the values ​​of the selected pixels to the synthesized noise image.

[0069] For example, the value of a pixel located at [H / 2, W / 2] can be selected from the obtained bias image and added to the value of each pixel in the synthesized noise image z to process the black level in the synthesized noise image z, thereby generating a processed synthesized noise image. Those skilled in the art will understand that the value of a pixel located at any position can be selected from the obtained bias image to process the black level in the synthesized noise image z.

[0070] As an example, the R, G1, B, and G2 values ​​of the processed synthetic noise image can be grouped into separate channels, and these four channels can be cascaded together. The image data of each channel (each channel's data size is H / 2 × W / 2) can then be input as samples into the machine learning model.

[0071] In the above, the original image was linearized, and noise was added to the linearized image y to obtain a synthetic noise image z. Black level processing was then applied to the synthetic noise image z to obtain the processed synthetic noise image. Alternatively, the R, G1, B, and G2 values ​​of the linearized image y can be grouped into separate channels. Noise can then be added to each of these channels, followed by black level processing on the R, G1, B, and G2 channels. Finally, the processed image data from each channel can be input as samples into a machine learning model.

[0072] When using the processed synthetic noise image as a sample to train the machine learning model, the display image captured by the image acquisition device, corresponding to the original image, is used as the label image for training the machine learning model. During machine learning model training, the R, G1, B, and G2 values ​​of the label image can be grouped into separate channels to correspond to the R, G1, B, and G2 channels of the processed synthetic noise image.

[0073] The following description of the aforementioned label image uses a digital camera as an example of an image acquisition device. A digital camera can be viewed as a signal conversion tool: it converts light signals from a real scene into digital signals (light signal → electrical signal → digital signal), a process performed by the camera's hardware. The digital signal obtained from the digital camera is the original image described above. Then, the digital camera performs image signal processing to convert this digital signal (original image) into an image suitable for display (display image). For example, the original image is a RAW format image captured by the digital camera, and the display image is an s-RGB format image obtained after the original image is converted and displayed on the digital camera's screen.

[0074] Figure 5This is a schematic diagram illustrating the image signal processing performed by existing digital cameras.

[0075] exist Figure 5 First, the original image is linearized as shown in expression (1).

[0076] exist Figure 5 In this process, after linearization comes white balance, which aims to eliminate the influence of colored light sources on the color of objects. For example, photographing a red apple under red light and green light respectively might produce drastically different results. However, the "color" of the red apple (the distribution of spectral reflectance on the apple's surface) should be an objective fact and should not change with the light source. To obtain the object's true color, white balance addresses this issue. In practice, this is achieved by multiplying the three multipliers [R, G, B] onto the image using a dot product.

[0077] exist Figure 5 In the process of white balance, the next step is depigmentation. The purpose of depigmentation is to transform the original 1-channel image into a 3-channel image that can be displayed on a display device. Essentially, it is a linear interpolation operation. That is, the original image of size H×W is transformed into three images of size H×W (s-RGB format images).

[0078] exist Figure 5 In the image processing, after depixelation comes color space conversion. In a digital camera, the value of a pixel is represented by a three-dimensional vector (R, G, B), which is the color base used by the digital camera. However, the color base used by the digital camera is different from the color base of the display device. The purpose of color space conversion is to transform the color base of the digital camera into a base usable on the display device. Color space conversion is achieved by multiplying the depixelated image by a 3×3 transformation matrix, which is a fixed value. Color space conversion yields an s-RGB format image. In other words, the image obtained through image signal processing is an s-RGB format image (display image).

[0079] For example, the parameters of a machine learning model can be updated based on a displayed image (labeled image) and a processed synthetic noise image. For instance, the average distance between the displayed image and the processed synthetic noise image can be calculated in the second normal form, and the parameters of the machine learning model can be updated based on backpropagation to obtain a trained machine learning model.

[0080] As an example, the second processing unit 304 can be configured to extract bias values ​​from a range of bias values ​​that follow a predetermined distribution, process the black level of image patches in a synthetic noise image based on the extracted bias values, and train a machine learning model based on the processed image patches.

[0081] For example, those skilled in the art can predetermine the size of the image block based on experience, application scenarios, or experiments. For example, the image block can be a 1024x1024 image block.

[0082] Processing the black level of an image patch in a synthesized noisy image based on the extracted bias value is essentially perturbing the black level of the image patch.

[0083] The information processing device 300 according to an embodiment of the present disclosure can enhance the robustness of a machine learning model to black levels by perturbing the black level.

[0084] For example, the second processing unit 304 can add the extracted bias value to each pixel of the image block in the synthesized noise image z to process the black level in the image block.

[0085] As an example, the predetermined distribution can be a Gaussian distribution or a uniform distribution. For instance, the parameters of the distribution function of the predetermined distribution are determined based on experience, application scenarios, or experiments. For example, a uniform distribution can be a uniform distribution following the range (-10, 10).

[0086] As an example, the second processing unit 304 can be configured to randomly sample bias values ​​from the aforementioned numerical range.

[0087] Figure 6 This is a schematic diagram illustrating training a machine learning model based on perturbing the black level according to an embodiment of the present disclosure. Figure 6 In this example, a 1024x1024 image patch is used as a sample for description.

[0088] like Figure 6 As shown, in (1), image patches are extracted from the synthetic noise image z generated by linearizing the original image and adding noise. In (2), the black level in the image patch is processed by perturbation. In (3), the R, G1, B, and G2 values ​​of the processed image patch are respectively grouped into separate channels to obtain four data blocks of size (1024 / 2) x (1024 / 2) (i.e., each data block is 512 x 512) and these four channels are concatenated together. In (4), the above four 512 x 512 data blocks are respectively used as samples to input into the machine learning model, and the corresponding image patches in the s-RGB format display image obtained by image signal processing of the original image in (5) are used as training labels to train the machine learning model. For example, the average distance between the image patch as a sample and the image patch as a label can be calculated under the second normal form, and the parameters of the machine learning model can be updated according to backpropagation to obtain the trained machine learning model. Among them, the image signal processing in (5) and Figure 5 The image signal processing is the same.

[0089] Although the above description describes perturbing the black level of image blocks in a synthetic noise image, those skilled in the art will understand that the black level of a synthetic noise image can also be perturbed based on the extracted bias value (e.g., by adding the extracted bias value to each pixel of the synthetic noise image), which will not be elaborated here.

[0090] Corresponding to the above-described information processing device embodiments, this disclosure also provides embodiments of information processing methods.

[0091] Figure 7 This is a flowchart illustrating a process example of an information processing method S700 according to an embodiment of the present disclosure.

[0092] The information processing method S700 according to an embodiment of this disclosure begins at S702.

[0093] In S704, a synthetic noise image is generated based on the input original image.

[0094] Specific examples of the original image can be found in the description of the relevant part of the information processing device 300 (e.g., the first processing unit 302), which will not be repeated here.

[0095] In S706, the black level in the synthetic noise image is processed, and a machine learning model for image denoising is trained based on the processed synthetic noise image to obtain the trained machine learning model.

[0096] For specific examples of processing black levels, please refer to the description of the relevant part of the information processing device 300 (e.g., the second processing unit 304), which will not be repeated here.

[0097] Information processing method S700 ends in S708.

[0098] In the information processing method S700 according to an embodiment of the present disclosure, processing the black level in the synthetic noise image can reduce the impact of inaccurate black level on the training of the machine learning model, thereby improving the denoising performance of the trained machine learning model trained based on the processed synthetic noise image.

[0099] According to another embodiment of this disclosure, an information processing apparatus 800 is also provided.

[0100] Figure 8 A functional block diagram of an information processing apparatus 800 according to an embodiment of the present disclosure is shown, such as... Figure 8As shown, the information processing apparatus 800 includes a third processing unit 802, which can be configured to input an image to be processed into a trained machine learning model obtained by the information processing device 300 or the information processing method S700, in order to perform denoising processing on the image to be processed.

[0101] Since a trained machine learning model with improved denoising performance is obtained through the information processing device 300 or the information processing method S700, the information processing apparatus 800 according to embodiments of the present disclosure is able to remove noise from the image to be processed more accurately.

[0102] As an example, the image to be processed is a short-exposure image taken in low-light conditions. The information processing device 800 can more accurately remove noise from short-exposure images taken in low-light conditions.

[0103] Corresponding to the above-described information processing apparatus embodiments, this disclosure also provides embodiments of information processing methods.

[0104] Figure 9 This is a flowchart illustrating an example of the information processing method S900 according to an embodiment of the present disclosure.

[0105] The information processing method S900 according to an embodiment of this disclosure begins at S902.

[0106] In S904, the image to be processed is input into a trained machine learning model obtained by the information processing device 300 or the information processing method S700 to perform denoising processing on the image to be processed.

[0107] Information processing method S900 ends in S906.

[0108] Since a trained machine learning model with improved denoising performance is obtained through the information processing device 300 or the information processing method S700, the information processing method S900 according to the embodiments of the present disclosure is able to remove noise from the image to be processed more accurately.

[0109] The basic principles of the present invention have been described above in conjunction with specific embodiments. However, it should be noted that those skilled in the art will understand that all or any step or component of the method and apparatus of the present invention can be implemented in any computing device (including processors, storage media, etc.) or network of computing devices, in the form of hardware, firmware, software or a combination thereof. This can be achieved by those skilled in the art using their basic circuit design knowledge or basic programming skills after reading the description of the present invention.

[0110] Furthermore, this invention also proposes a program product storing machine-readable instruction code. When the instruction code is read and executed by a machine, the method described above according to embodiments of the present invention can be performed.

[0111] Accordingly, the storage medium used to carry the program product storing the machine-readable instruction code is also included in the disclosure of this invention. Storage media include, but are not limited to, floppy disks, optical disks, magneto-optical disks, memory cards, memory sticks, etc.

[0112] When the present invention is implemented via software or firmware, the transmission from a storage medium or network to a computer with a dedicated hardware architecture (e.g., Figure 10 The general-purpose computer 1000 shown is equipped with the programs that constitute the software, and when various programs are installed, the computer is able to perform various functions, etc.

[0113] exist Figure 10 In this system, the Central Processing Unit (CPU) 1001 performs various processes based on programs stored in the Read-Only Memory (ROM) 1002 or programs loaded into the Random Access Memory (RAM) 1003 from the Storage Section 1008. The RAM 1003 also stores data required as needed when the CPU 1001 performs various processes, etc. The CPU 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An Input / Output Interface 1005 is also connected to the bus 1004.

[0114] The following components are connected to the input / output interface 1005: input section 1006 (including keyboard, mouse, etc.), output section 1007 (including display, such as cathode ray tube (CRT), liquid crystal display (LCD), etc., and speakers, etc.), storage section 1008 (including hard disk, etc.), and communication section 1009 (including network interface card, such as LAN card, modem, etc.). The communication section 1009 performs communication processing via a network, such as the Internet. If necessary, a drive 1010 may also be connected to the input / output interface 1005. Removable media 1011, such as disk, optical disk, magneto-optical disk, semiconductor memory, etc., are installed on the drive 1010 as needed, so that computer programs read from them can be installed into the storage section 1008 as needed.

[0115] When the above series of processes are implemented through software, the program constituting the software is installed from a network such as the Internet or a storage medium such as removable media 1011.

[0116] Those skilled in the art will understand that such storage media are not limited to Figure 10The illustration shows a removable medium 1011 containing programs, distributed separately from the device to provide programs to users. Examples of removable media 1011 include disks (including floppy disks (registered trademark)), optical disks (including optical disc read-only memory (CD-ROM) and digital versatile disks (DVD)), magneto-optical disks (including mini-discs (MD) (registered trademark)), and semiconductor memory. Alternatively, the storage medium may be ROM 1002, a hard disk included in storage section 1008, etc., containing programs and distributed to users along with the device containing them.

[0117] It should also be noted that in the apparatus, method, and system of the present invention, the components or steps can be decomposed and / or recombined. These decompositions and / or recombinations should be considered equivalent solutions of the present invention. Furthermore, the steps performing the above series of processes can naturally be executed in the order described, but are not necessarily required to be executed in chronological order. Some steps can be performed in parallel or independently of each other.

[0118] Finally, it should be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Furthermore, unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0119] While embodiments of the present invention have been described in detail above with reference to the accompanying drawings, it should be understood that the embodiments described above are merely illustrative and do not constitute a limitation thereof. Those skilled in the art can make various modifications and alterations to the above embodiments without departing from the spirit and scope of the present invention. Therefore, the scope of the present invention is defined only by the appended claims and their equivalents.

[0120] This technology can also be implemented as follows.

[0121] Appendix 1. An information processing device, comprising:

[0122] The processing circuit is configured as follows:

[0123] Generate a synthetic noisy image based on the input original image; and

[0124] The black level in the synthetic noise image is processed, and a machine learning model for image denoising is trained based on the processed synthetic noise image to obtain the trained machine learning model.

[0125] Appendix 2. The information processing device according to Appendix 1, wherein the processing circuit is configured to acquire predetermined image acquisition parameters from the original image, obtain a corresponding bias image based on the correspondence between the predetermined image acquisition parameters and a bias image for processing black levels, and process the black levels in the synthetic noise image based on the obtained bias image.

[0126] Appendix 3. The information processing device according to Appendix 2, wherein,

[0127] The correspondence is generated in the following way:

[0128] In low-light environments, a bias image corresponding to each predetermined image acquisition parameter is obtained based on the black image captured by the image acquisition device for each predetermined image acquisition parameter.

[0129] Appendix 4. The information processing device according to Appendix 3, wherein,

[0130] The image acquisition device is a digital photography device, and the predetermined image acquisition parameters include the exposure time and ISO of the digital photography device; and

[0131] The correspondence is generated in the following way:

[0132] The cover of the digital photography device is closed, a black image is captured for each parameter pair of the digital photography device, including exposure time and sensitivity, and the black image is preprocessed to obtain an offset image corresponding to each parameter pair.

[0133] Note 5. According to the information processing device described in Note 4, the preprocessing of the black image includes: linearizing the black image based on the black level and white level of the digital photography device.

[0134] Note 6. The information processing apparatus according to any one of Notes 2 to 5, wherein the processing circuit is configured to add the obtained bias image to the synthetic noise image.

[0135] Note 7. The information processing apparatus according to any one of Notes 2 to 5, wherein the processing circuit is configured to select an image block from a predetermined position in the obtained bias image and add a bias value calculated based on the selected image block to the synthetic noise image.

[0136] Note 8. The information processing apparatus according to any one of Notes 2 to 5, wherein the processing circuit is configured to select a pixel from a predetermined position in the obtained bias image and add the value of the selected pixel to the synthetic noise image.

[0137] Appendix 9. The information processing device according to Appendix 1, wherein the processing circuit is configured as follows:

[0138] Bias values ​​are drawn from a range of bias values ​​that follow a predetermined distribution, and

[0139] The black level of the image patch in the synthetic noise image is processed based on the extracted bias value, and the machine learning model is trained based on the processed image patch.

[0140] Note 10. The information processing device according to Note 9, wherein the predetermined distribution is a Gaussian distribution or a uniform distribution.

[0141] Note 11. The information processing apparatus according to Note 9 or 10, wherein the processing circuit is configured to randomly sample the bias value from the numerical range.

[0142] Note 12. The information processing apparatus according to any one of Notes 1 to 11, wherein the processing circuit is configured to noise-encode the original image based on noise parameters obtained from the original image, thereby generating the synthetic noise image.

[0143] Note 13. The information processing apparatus according to Note 12, wherein the processing circuit is configured as follows:

[0144] Based on the black and white levels obtained from the original image, the original image is linearized to obtain a linearized image; and

[0145] The linearized image is subjected to the noise reduction process to generate the synthetic noise image.

[0146] Note 14. The information processing apparatus according to any one of Notes 1 to 13, wherein the original image is a long-exposure image captured by the image acquisition device in a long-exposure state, and the synthesized noise image is used to simulate a short-exposure image captured by the image acquisition device in a low-light environment.

[0147] Note 15. The information processing apparatus according to any one of Notes 1 to 14, wherein,

[0148] The original image includes RGB data in RAW format.

[0149] Note 16. The information processing apparatus according to any one of Notes 1 to 15, wherein,

[0150] The machine learning model is a neural network.

[0151] Appendix 17. An information processing apparatus, comprising:

[0152] The processing circuit is configured as follows:

[0153] The image to be processed is input into a trained machine learning model obtained by the information processing device according to any one of Appendices 1 to 16 to denoise the image to be processed.

[0154] Note 18. The information processing apparatus according to Note 17, wherein the image to be processed is a short-exposure image obtained in a low-light environment.

[0155] Appendix 19. An information processing method, comprising:

[0156] Generate a synthetic noisy image based on the input original image; and

[0157] The black level in the synthetic noise image is processed, and a machine learning model for image denoising is trained based on the processed synthetic noise image to obtain the trained machine learning model.

[0158] Appendix 20. An information processing method, comprising:

[0159] The image to be processed is input into a trained machine learning model obtained by the information processing device according to any one of Appendices 1 to 16 to denoise the image to be processed.

[0160] Appendix 21. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed, perform the information processing method according to Appendix 19 or 20.

Claims

1. An information processing device, comprising: The processing circuit is configured as follows: Generate a synthetic noisy image based on the input original image; as well as The black levels in the synthesized noisy image are processed, and a machine learning model for image denoising is trained based on the processed synthesized noisy image to obtain the trained machine learning model. The processing circuit is configured to obtain predetermined image acquisition parameters from the original image, obtain a corresponding bias image based on the correspondence between the predetermined image acquisition parameters and a bias image used for processing black levels, and process the black levels in the synthetic noise image based on the obtained bias image.

2. The information processing device according to claim 1, wherein, The correspondence is generated in the following way: In low-light environments, a bias image corresponding to each predetermined image acquisition parameter is obtained based on the black image captured by the image acquisition device for each predetermined image acquisition parameter.

3. The information processing device according to claim 2, wherein, The image acquisition device is a digital photography device, and the predetermined image acquisition parameters include the exposure time and ISO of the digital photography device; and The correspondence is generated in the following way: The cover of the digital photography device is closed, a black image is captured for each parameter pair of the digital photography device, including exposure time and sensitivity, and the black image is preprocessed to obtain an offset image corresponding to each parameter pair.

4. The information processing device according to claim 3, wherein, Preprocessing the black image includes linearizing the black image based on the black and white levels of the digital photography device.

5. The information processing device according to any one of claims 1 to 4, wherein, The processing circuit is configured to add the obtained bias image to the synthesized noise image.

6. The information processing apparatus according to any one of claims 1 to 4, wherein, The processing circuit is configured to select an image block from a predetermined position in the obtained bias image and add a bias value calculated based on the selected image block to the synthesized noise image.

7. The information processing apparatus according to any one of claims 1 to 4, wherein, The processing circuit is configured to select pixels from predetermined positions in the obtained bias image and add the values ​​of the selected pixels to the synthesized noise image.

8. The information processing device according to claim 1, wherein, The processing circuit is configured as follows: Bias values ​​are drawn from a range of bias values ​​that follow a predetermined distribution, and The black level of the image patch in the synthetic noise image is processed based on the extracted bias value, and the machine learning model is trained based on the processed image patch.

9. The information processing device according to claim 8, wherein, The predetermined distribution is either a Gaussian distribution or a uniform distribution.

10. The information processing device according to claim 8 or 9, wherein, The processing circuit is configured to randomly sample the bias value from the numerical range.

11. The information processing apparatus according to any one of claims 1 to 4, wherein, The processing circuit is configured to noise-encode the original image based on noise parameters obtained from the original image, thereby generating the synthetic noise image.

12. The information processing device according to claim 11, wherein, The processing circuit is configured as follows: Based on the black and white levels obtained from the original image, the original image is linearized to obtain a linearized image of the original image; as well as The linearized image is subjected to the noise reduction process to generate the synthetic noise image.

13. The information processing apparatus according to any one of claims 1 to 4, wherein, The original image is a long-exposure image captured by the image acquisition device under long-exposure conditions, and the synthesized noise image is used to simulate a short-exposure image captured by the image acquisition device under low-light conditions.

14. The information processing apparatus according to any one of claims 1 to 4, wherein, in, The original image includes RGB data in RAW format.

15. The information processing apparatus according to any one of claims 1 to 4, wherein, The machine learning model is a neural network.

16. An information processing apparatus, comprising: The processing circuit is configured as follows: The image to be processed is input into a trained machine learning model obtained by the information processing device according to any one of claims 1 to 15 to perform denoising processing on the image to be processed.

17. The information processing apparatus according to claim 16, wherein, The image to be processed is a short-exposure image taken in low-light conditions.

18. An information processing method, comprising: Generate a synthetic noisy image based on the input original image; as well as The black levels in the synthesized noisy image are processed, and a machine learning model for image denoising is trained based on the processed synthesized noisy image to obtain the trained machine learning model. Specifically, predetermined image acquisition parameters are obtained from the original image; a corresponding bias image is obtained based on the correspondence between the predetermined image acquisition parameters and a bias image used for processing black levels; and the black levels in the synthesized noise image are processed based on the obtained bias image.

19. An information processing method, comprising: The image to be processed is input into a trained machine learning model obtained by the information processing device according to any one of claims 1 to 15 to perform denoising processing on the image to be processed.

20. A computer-readable storage medium having stored thereon computer-executable instructions that, when executed, perform the information processing method according to claim 18 or 19.

Images