Image processing device

Abstract

To provide a learning model that can reduce noise in a simpler manner with higher precision than ever before.SOLUTION: An image processing device generates a learning model that reduces noise included in an image obtained by a prescribed imaging device. Extraction means extracts, as a noise image, an image of an optical black region which is included in each of images captured by the prescribed imaging device and to which light having passed through an optical system of the prescribed imaging device is not applied. Combining means combines the noise images to create a student image to a teacher image. Learning means gives the student image to the learning model as input to perform learning.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing device. [Background technology]

[0002] Images captured by digital cameras and the like may contain noise. Conventionally, such noise has been reduced by digital filters and the like. In recent years, it has been proposed to reduce noise in an image by training a learning model to learn noise and using the trained learning model (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-086284 A Summary of the Invention [Problem to be solved by the invention]

[0004] According to Patent Document 1, it is proposed to calculate the noise to be added to the teacher image based on the ISO sensitivity, and to generate a training image (student image) by adding the calculated noise to the teacher image. This has the advantage that a large number of student images can be obtained. On the other hand, noise (thermal noise) that occurs depending on the temperature of the image sensor and bright spot noise that occurs due to the incidence of radiation such as cosmic rays can depend on the individual product differences of the image sensor. It is extremely difficult to prepare a noise calculation formula for each individual difference. Therefore, the present invention aims to provide a learning model that can reduce noise more simply and accurately than before. [Means for solving the problem]

[0005] The present invention relates to, for example, An image processing device that generates a learning model for reducing noise contained in an image acquired by a predetermined imaging device, an extraction means for extracting, as a noise image, an image of an optical black region included in each of the plurality of images captured by the predetermined imaging device, the image being not irradiated with light that has passed through an optical system of the predetermined imaging device; a synthesis means for synthesizing the noise image with a teacher image to generate a student image; a learning means for inputting the student image to the learning model and learning the learning model so that a difference between an output image output from the learning model and the teacher image becomes small; The present invention provides an image processing device having the following features. Effect of the Invention

[0006] According to the present invention, a learning model capable of reducing noise more simply and accurately than ever before is provided. [Brief description of the drawings]

[0007] [Figure 1] Diagram explaining an image processing system [Diagram 2] FIG. 1 is a diagram illustrating an imaging device. [Diagram 3] FIG. 1 is a diagram illustrating an information processing device. [Figure 4] A diagram explaining the developing section [Diagram 5] FIG. 1 is a diagram illustrating a teacher image generating device. [Figure 6] FIG. 1 is a diagram illustrating a learning processing device. [Figure 7] A diagram illustrating a student image generating device. [Figure 8] Diagram explaining demosaic processing [Figure 9] A diagram explaining how to extract noise regions [Figure 10] A diagram explaining how to extract noise regions [Figure 11] Diagram explaining how to add noise [Figure 12] Diagram explaining how to add noise [Figure 13] Diagram explaining how to add noise [Figure 14] Diagram explaining the learning method [Figure 15] A diagram explaining how student images are generated. [Figure 16] A diagram illustrating a student image generating device. [Figure 17] Diagram explaining the effect DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims, and not all combinations of features described in the embodiments are essential to the invention. Two or more features among the multiple features described in the embodiments may be arbitrarily combined. In addition, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0009] (1) Image processing system FIG. 1 shows an image processing system. The imaging device 100 is a digital still camera, a digital video camera, or a surveillance camera that captures still images or moving images. The imaging device 100 may be mounted on an artificial satellite or a spacecraft. The noise image group 101 includes a plurality of noise images captured by the imaging device 100. When a plurality of imaging devices 100 are present, the noise image group 101 is generated for each of the plurality of imaging devices 100 individually. The original image set 102 is a data set consisting of a plurality of color images (e.g., sRGB images or device RGB format images). This data set may be, for example, free material provided on the Internet. The device RGB format image refers to an sRGB format image adjusted to the RGB color gamut displayed on a monitor or the like.

[0010] The information processing device 110 is a computer such as a personal computer (PC). In Fig. 1, one information processing device 110 is collectively shown, but in reality, the information processing device 110 may be formed by multiple computers. The information processing device 110 has multiple functions (a student image generating device 111, a teacher image generating device 112, a learning processing device 115, and an image processing device 117). These multiple functions may be realized by multiple computers.

[0011] The student image generation device 111 generates a student image group 113 from the original image set 102 and the noise image group 101. The student image group 113 includes a plurality of color images acquired by the imaging device 100 to which noise has been added.

[0012] The teacher image generation device 112 generates a teacher image group 114 from the original image set 102. Note that if the original image set 102 can be used as the teacher image group 114 as it is, such as when the original image set 102 is in demosaic RGB format, the teacher image generation device 112 is omitted.

[0013] The learning processing device 115 inputs the student image group 113 as a student image and the teacher image group 114 as a teacher image to generate a learning model 116. For example, a first student image included in the student image group 113 and a first teacher image included in the teacher image group 114 are each generated from a common original image (one original image included in the original image set 102). The learning processing device 115 generates a first output image by inputting the first student image to the learning model 116, obtains an error between the first output image and the first teacher image using an error function, and updates coefficients in the learning model 116 so as to minimize the error. This update process is performed for N student images included in the student image group 113 and N teacher images included in the teacher image group 114, each of which corresponds to one of the N student images. As a result, a learned learning model 116 is generated.

[0014] The image processing device 117 generates an output image by reducing noise contained in an input image using the trained learning model 116. That is, the image processing device 117 has a noise reduction function. The image processing device 117 may be the imaging device 100. That is, the trained learning model 116 may be written to the imaging device 100, and the noise reduction process may be performed in the imaging device 100.

[0015] When the original image set 102 is image data in device RGB format, the student image group 113 is image data in RAW format, and the teacher image group 114 is image data in device RGB format or demosaic RGB format, the learning model 116 is a model capable of simultaneously executing demosaic (development) processing and noise reduction processing. It is sufficient for the learning model 116 to be able to realize noise reduction processing, but in the following, for convenience of explanation, the learning model 116 will execute the demosaic (development) processing and noise reduction processing collectively.

[0016] (2) Structure of the imaging device FIG. 2 shows the structure of the imaging device 100. The CPU 201 is a processor that controls the imaging device 100 according to a control program stored in the memory 202. The image processing function of the CPU 201 may be realized by a hardware circuit such as an application specific integrated circuit (ASIC). The lens unit 205 has, for example, an optical lens, a focus adjustment function, and a zoom function. The focus adjustment function and the zoom function may be omitted. The light blocking unit 206 is also optional and has a light blocking mechanism such as an aperture or a mechanical shutter. The amount of light blocking may be adjusted by moving the light blocking plate or the aperture blades by a motor or the like. The imaging element 207 is a semiconductor element that converts an optical signal into an electrical signal, such as a CMOS image sensor. The temperature sensor 204 detects the temperature of the imaging device 100, particularly the temperature of the imaging element 207. The communication circuit 203 is a communication circuit for communicating with the information processing device 110. The communication circuit 203 and the information processing device 110 may communicate directly with each other, or indirectly with each other via an access point, a base station, or the like. The control circuit 208 controls the focus or focal length of the lens unit 205, the light blocking amount of the light blocking unit 206, and the exposure time (shutter speed) and sensitivity of the image sensor 207 according to instructions from the CPU 201. The CPU 201 generates RAW data 221 based on an image signal output from the image sensor 207 and stores the RAW data 221 in the memory 202. The memory 202 may include a ROM, a RAM, a memory card, and the like. The CPU 201 may transmit the RAW data 221 to the information processing device 110 via the communication circuit 203. When the image processing device 117 is mounted on the image capturing device 100, the image processing device 117 develops a mosaic image (e.g., the RAW data 221) and generates a demosaic color image (device RGB data 222). The device RGB data 222 may be image data in a demosaic RGB format. Furthermore, the CPU 201 stores in the memory 202 the imaging conditions (e.g., temperature, sensitivity, exposure time, and focal length of the imaging device 100) applied when acquiring the RAW data 221 in association with the RAW data 221. In the following, the file format of the RAW data 221 includes both image data and imaging conditions.

[0017] (3) Structure of information processing device 3 shows the structure of the information processing device 110. The CPU 301 controls the information processing device 110 in accordance with a program 306 stored in the storage device 302. The CPU 301 may realize the student image generating device 111, the teacher image generating device 112, the learning processing device 115, and the image processing device 117 by executing the program 306. The communication circuit 303 is a circuit for communicating with the imaging device 100. The input device 304 is a pointing device, a keyboard, or the like, and receives instructions from a user. The display device 305 is a display that displays information to the user.

[0018] The storage device 302 is, for example, a storage device that stores a ROM, a RAM, a memory card, a solid-state drive (SSD), and a hard disk drive (HDD). The storage device 302 may further store the noise image set 101, the original image set 102, the student image set 113, the teacher image set 114, and the learning model 116.

[0019] (4) Functions of image processing device FIG. 4 shows functions of the image processing device 117. The demosaic unit 411 demosaics the RAW data 221, which is a Bayer-type mosaic image, to generate a demosaic image. As is well known, the demosaic process includes a process of interpolating the pixel value of a pixel of interest from the pixel values ​​of surrounding pixels of the same color. The noise reduction unit 412 reduces noise contained in the demosaic image. This noise refers to noise generated in an electric circuit provided between the image sensor 207 and the CPU 201. For example, various noises may be included, such as noise caused by the heat of the image sensor 207, noise caused by radiation such as cosmic rays being irradiated onto the image sensor 207, noise caused by variations in analog gain, and noise caused by differences in sensitivity of cells. In addition, if the noise reduction unit 412 reduces noise, false signals caused by noise during processing by the CPU 201 or CPU 301, such as false colors that may be generated by the demosaic process, may also be reduced. In this embodiment, the learning model 116 operates at least as the noise reduction unit 412, but may operate as both the demosaicing unit 411 and the noise reduction unit 412. The noise reduction unit 412 reduces noise in the demosaiced image to generate a noise-reduced image.

[0020] The image processing unit 400 executes image processing for generating a color image (device RGB data 222) from the output image output from the noise reduction unit 412. For example, the image processing unit 400 has a white balance unit 413, a color conversion unit 414, a gamma correction unit 415, a tone mapping unit 416, and the like. However, this is only an example, and the image processing unit 400 may have at least one of these, or may have an image processing function different from these. Alternatively, the image processing unit 400 may be omitted. Note that the case where the image processing unit 400 has at least one of these includes, for example, a case where the image processing unit 400 has only the white balance unit 413, or a case where the image processing unit 400 has the remaining three functions excluding the tone mapping unit 416.

[0021] In the following, as an example, the image processing unit 400 will be described as having a white balance unit 413, a color conversion unit 414, a gamma correction unit 415, and a tone mapping unit 416.

[0022] The white balance unit 413 adjusts the white balance of the noise-reduced color image (e.g., image data in demosaic RGB format) output from the noise reduction unit 412. Note that the demosaic RGB format image data refers to the format of the image data (image data before being processed by the white balance unit 413, color conversion unit 414, gamma correction unit 415, and tone mapping unit 416) output from the noise reduction unit 412, including the linear RGB format and the sRGB format. The color conversion unit 414 converts the color of the color image whose white balance has been adjusted. For example, the color conversion unit 414 corrects (converts) the color of the input color image using a color correction matrix. The gamma correction unit 415 corrects the gradation characteristics of the color image output from the color conversion unit 414. In order to accurately reproduce the color tones on the output device, the tone mapping unit 416 determines the range of gradations included in the color image output from the gamma correction unit 415, remaps the gradations of the color image to a narrow color gamut range according to the output device, and generates the device RGB data 222.

[0023] (5) Teacher image generation device FIG. 5 shows the function of the teacher image generating device 112. The inverse image processing unit 500 executes image processing for generating the teacher image group 114 from the original image set 102. The inverse image processing unit 500 is configured to execute the inverse conversion process of the image processing executed in the image processing unit 400. For example, the inverse image processing unit 500 has an inverse white balance unit 513, an inverse color conversion unit 514, an inverse gamma correction unit 515, an inverse tone mapping unit 516, and the like. However, this is only an example, and the inverse image processing unit 500 may have at least one of these, or may have an image processing function different from these. Alternatively, the inverse image processing unit 500 may be omitted. In any case, the inverse image processing unit 500 is paired with the image processing unit 400, and may be configured to execute the opposite image processing (which may be called inverse image processing or inverse conversion) to the image processing in the image processing unit 400. In an extreme case, the image processing unit 400 is omitted, and the inverse image processing unit 500 is accordingly omitted.

[0024] Here, as an example, the reverse image processing (which may be called reverse image processing or reverse conversion) of the teacher image generating device 112 basically executes image processing reverse to the image processing executed by the white balance unit 413, the color conversion unit 414, the gamma correction unit 415, and the tone mapping unit 416 in the image processing device 117. The output image output by the learning model 116 is a color image (e.g., image data in demosaic RGB format) processed by the demosaic unit 411 and the noise reduction unit 412. Therefore, as the teacher image, a color image (e.g., image data in demosaic RGB format) before the white balance unit 413, the color conversion unit 414, the gamma correction unit 415, and the tone mapping unit 416 act is required.

[0025] The inverse tone mapping unit 516 performs inverse tone mapping on each color image included in the input original image set 102. The inverse gamma correction unit 515 performs inverse gamma correction on the color image output from the inverse tone mapping unit 516. The inverse color conversion unit 514 performs inverse color conversion on the color image output from the inverse gamma correction unit 515. The inverse white balance unit 513 performs inverse white balance processing on the color image output from the inverse color conversion unit 514. This generates a teacher image group 114 (e.g., image data in demosaic RGB format) that can be compared with the output image output by the learning model 116.

[0026] (6) Learning processing device FIG. 6 shows the function of the learning processing device 115. The learning model 116 is based on a neural network, and is a model that outputs an output image group 616 to an output layer based on a student image group 113 given to an input layer. An intermediate layer (hidden layer) is provided between the input layer and the output layer, and multiple nodes exist in the intermediate layer. When passing data from one node to the next node, a coefficient (weight) is multiplied. Therefore, the learning model 116 may be understood as a collection of multiple coefficients (weights applied between multiple nodes in a neural network). The model execution unit 601 inputs the student image group 113 to the learning model 116 and outputs an output image group 616. Multiple pixels (pixel values) from the output layer form one output image.

[0027] The error calculation unit 602 calculates an error from the output image group 616 and the teacher image group 114, and passes the error to the update unit 603. Each error is calculated from one output image included in the output image group 616 and one teacher image in the teacher image group 114. Here, the original image that is the source of the input image (student image) that is the source of one output image is the same as the original image that is the source of one teacher image. In other words, the student image and the teacher image may be managed in association with identification information of a common original image.

[0028] The update unit 603 updates the coefficients in the learning model 116 so as to reduce the error output from the error calculation unit 602. The error is calculated pixel by pixel. As the learning depth increases, the error becomes smaller.

[0029] (7) Student image generation device 7 shows the structure of the student image generating device. An extracting unit 700 extracts a noise region to be added to each original image included in the original image set 102 from the noise image group 101. The extracting unit 700 is composed of, for example, an image capturing condition acquiring unit 701, an image selecting unit 702, a region size acquiring unit 703, and a noise region cutting unit 704.

[0030] Each of the multiple noise images included in the noise image group 101 is an image acquired from an element in the imaging element 207 through which only dark current flows. The amount of dark current may vary depending on the individual difference of the imaging device 100 and the imaging conditions. The imaging conditions when each of the multiple noise images included in the noise image group 101 was acquired by the imaging device 100 may be the same, similar, or completely different. Therefore, multiple noise images acquired under closer imaging conditions are required. For example, multiple noise images acquired under imaging conditions closer to the imaging conditions under which the user actually wants to reduce noise may be required. The imaging condition acquisition unit 701 acquires imaging conditions 711 of each noise image included in the noise image group 101 from the storage device 302. When the file format of the noise image is a file format that can include the imaging conditions, the imaging condition acquisition unit 701 acquires the imaging conditions 711 together with the noise image from the file format. The image selection unit 702 compares the imaging conditions 711 of the noise images included in the noise image group 101, and selects a plurality of noise images (e.g., a predetermined number) having the same or similar imaging conditions 711. The region size acquisition unit 703 acquires the size of the noise region cut out from the noise image. The size of the noise region may be a fixed value, or may be dynamically determined according to the size of the teacher image.

[0031] The noise region cutout unit 704 cuts out a noise region from each selected noise image. That is, the size of the noise region is smaller than the size of the noise image.

[0032] The noise adding unit 705 generates a student image by adding the noise area extracted by the extraction unit 700 to the original image. In this way, a student image group 113 consisting of a plurality of student images to which noise specific to the imaging device 100 has been added is generated from a plurality of original images included in the original image set 102.

[0033] In reality, since the image output from the noise adding unit 705 is a color image (device RGB), inverse image processing (linearization processing) and mosaic processing are required. Therefore, in addition to the above-mentioned inverse tone mapping unit 516, inverse gamma correction unit 515, inverse color conversion unit 514, and inverse white balance unit 513, a mosaic unit 706 is required downstream of the noise adding unit 705. The inverse tone mapping unit 516 performs inverse tone mapping on the color image to which noise is added and input from the noise adding unit 705. The inverse gamma correction unit 515 performs inverse gamma correction on the color image output from the inverse tone mapping unit 516. The inverse color conversion unit 514 performs inverse color conversion on the color image output from the inverse gamma correction unit 515. The inverse white balance unit 513 performs inverse white balance processing on the color image output from the inverse color conversion unit 514. The mosaic unit 706 generates a student image by converting the color image into a mosaic image (Bayer image). By applying the inverse image processing to the multiple original images included in the original image set 102, the student image group 113 is generated. However, this is just one example, and the original image set 102 may include image information in RAW format or may be in demosaic RGB format. In these cases, the inverse image processing unit may have at least one of these, or may have an image processing function different from these. Alternatively, the inverse image processing unit may be omitted. In any case, the inverse image processing unit is paired with the image processing unit, and may be configured to perform image processing (which may be called inverse image processing or inverse conversion) that is opposite to the image processing in the image processing unit. In an extreme case, the image processing unit may be omitted, and the inverse image processing unit may also be omitted accordingly.

[0034] (8) Demosaic processing 8 shows the demosaic process executed by the demosaic unit 411. The RAW data 221 generated by the image sensor 207 having a Bayer array type color filter becomes a Bayer image (mosaic image). In a Bayer image, there are missing R pixels, G pixels, and B pixels. Therefore, in the development process (demosaic process), the pixel values ​​of the missing pixels are interpolated using the pixel values ​​of the surrounding pixels. This results in a color image (e.g., device RGB data 222). If the accuracy of the interpolation calculation is low, false colors may occur.

[0035] (9) Acquiring a noise image FIG. 9 shows a method of acquiring a noise image. [0] In the following, a "noise image" is an image containing noise. In contrast, a "noise region" is a pixel region containing noise that is cut out from a noise image and is synthesized with an original image when generating a student image from a set of original images. Since the noise region is a part of the noise image, the noise region may also be called a noise image. The pixel region 900 is a region in which a plurality of photoelectric conversion elements are arranged in the image sensor 207. The image circle 901 is a region in which light that has passed through the lens unit 205 is imaged. The image circle 901 changes according to the focal length of the lens unit 205. In a single-focus type lens unit 205, the size of the image circle 901 is almost constant. In the example shown in FIG. 9, there is a pixel region outside the image circle 901 that is not irradiated with light. Such a pixel region may be called an optical black region 902. Since light is not irradiated in the optical black region 902, only dark current flows through the photoelectric conversion elements present therein. That is, an image consisting only of noise components is generated in the optical black region 902. In this example, a plurality of noise regions 903 are cut out from the optical black region 902. In Fig. 9, the optical black region 902 exists parallel to the short side of the pixel region 900, but the optical black region 902 may exist parallel to the long side of the pixel region 900.

[0036] Fig. 10 shows that it is possible to make the entire pixel region 900 an optical black region 902 by intentionally blocking light with the light blocking unit 206 described in Fig. 1. In the example of Fig. 10, the light blocking unit 206 is required, but more samples (noise region 903) can be extracted by one imaging.

[0037] (10) How to add noise 11 shows the generation of a student image 1120 to which noise has been added from a noise image 1100 and an original image 1110. Here, it is assumed that the original image 1110 is logically divided into a plurality of pixel regions 1111, and the size of each pixel region 1111 is equal to the size of the noise region 903. The CPU 301 determines the cutout size of the noise region 903 based on the size of the pixel region 1111.

[0038] In this example, a plurality of noise regions 903 are extracted from one noise image 1100. Each of the plurality of noise regions 903 is synthesized with one of a plurality of pixel regions 1111 constituting an original image 1110. The relationship between the plurality of noise regions 903 and the plurality of pixel regions 1111 may be determined randomly or based on some rule. By employing randomization, it may be possible to suppress overlearning for a specific noise.

[0039] In this example, a plurality of noise regions 903 are extracted from one noise image 1100, but this is merely one example.

[0040] Fig. 12 shows that only one noise region 903 is extracted from one noise image 1100. According to Fig. 12, one noise region 903a is extracted from noise image 1100a. One noise region 903b is extracted from noise image 1100b. One noise region 903c is extracted from noise image 1100c.

[0041] In this case, the positions of the noise regions 903a, 903b, and 903c extracted from the noise images 1100a, 1100b, and 1100c may be the same or different. In the latter case, the positions of the noise regions 903a, 903b, and 903c may be determined randomly so as not to overlap. This will also help to suppress overlearning.

[0042] As illustrated in Fig. 13, the number of noise regions 903 extracted from multiple noise images 1100a, 1100b, 1100c, ... may be different. In this example, two noise regions 903a, 903b are extracted from the noise image 1100a. Two noise regions 903c, 903d are extracted from the noise image 1100b. One noise region 903e is extracted from the noise image 1100c. The positions of the noise regions 903 extracted from each noise image 1100 may be fixed or may be different. In the latter case, the extraction positions may be determined randomly.

[0043] In this way, one student image 1120 is completed by synthesizing the noise region 903 over the entire original image 1110. However, in reality, the student image 1120 is completed by subsequently applying inverse image processing and mosaic processing.

[0044] (11) Flowchart FIG. 14 is a flowchart showing the learning process of the learning model 116 executed by the CPU 301 of the information processing device 110.

[0045] In S1401 , the CPU 301 (student image generating device 111 , teacher image generating device 112 ) acquires the teacher original image set 102 from the storage device 302 .

[0046] In S1402 , the CPU 301 (student image generating device 111 ) acquires the noise image group 101 from the imaging device 100 .

[0047] In S1403, the CPU 301 (student image generating device 111) generates a student image group 113 based on the original image set 102 and the noise image group 101. A detailed example of S1403 will be described later with reference to FIG.

[0048] In S1404, the CPU 301 (teacher image generating device 112) generates a group of teacher images 114 for comparison from the original image set 102. Note that S1404 may be executed between S1401 and S1405. S1404 may be executed before S1402 and S1403, may be executed after S1402 and S1403, or may be executed in parallel with S1402 and S1403.

[0049] In S1405, the CPU 301 (learning processing device 115) executes learning for the learning model 116 using the student image group 113 and the teacher image group 114 for comparison.

[0050] In S1406, the CPU 301 stores the trained learning model 116 in the storage device 302. When the image processing device 117 is mounted on the imaging device 100, the trained learning model 116 is transmitted to the imaging device 100 via the communication circuit 303. The imaging device 100 receives the trained learning model 116 via the communication circuit 203 and stores it in the memory 202.

[0051] FIG. 15 is a flowchart showing details of the student image generation process.

[0052] In S1501 , the CPU 301 (imaging condition acquisition unit 701 ) acquires the imaging conditions 711 for each of the multiple noise images 1100 included in the noise image group 101 .

[0053] In S1502, the CPU 301 (image selection unit 702) compares the imaging conditions 711 of the multiple noise images 1100, and selects multiple noise images having similar imaging conditions 711. Here, the number of noise images to be selected may be determined in advance. Alternatively, a range in which the imaging conditions can be determined to be similar to each other may be determined in advance, and multiple imaging conditions 711 that fall within that range may be selected. For example, when the imaging condition 711 is the temperature (imaging temperature) at the time when the noise image 1100 was acquired, multiple noise images 1100 associated with an imaging temperature that is equal to or higher than the lower limit temperature and lower than the upper limit temperature are selected.

[0054] In S1503, the CPU 301 (noise region cutout unit 704) cuts out the noise region 903 from the selected multiple noise images 1100.

[0055] In S1504, the CPU 301 (noise adding unit 705) composites the noise region 903 with the original image set 102. That is, the noise region 903 is composited with each of the multiple original images 1110 included in the original image set 102. As a result, a color image with noise added is formed.

[0056] In S1505, the CPU 301 (the inverse tone mapping unit 516, the inverse gamma correction unit 515, the inverse color conversion unit 514, and the inverse white balance unit 513) executes inverse image processing on the color image to which noise has been added.

[0057] In S1506, the CPU 301 (mosaic unit 706) performs mosaic generation on the color image to which the inverse image processing has been applied, thereby generating a student image 1120.

[0058] (12) Modification of the position of the noise adding part For example, in FIG. 7, the noise adding unit 705 is arranged before the inverse tone mapping unit 516, but this is only one example. FIG. 16 shows an example in which the noise adding unit 705 is arranged after the mosaic unit 706. In this case, the inverse tone mapping unit 516 performs inverse tone mapping on the original image 1110 included in the original image set 102. The inverse gamma correction unit 515 processes the output image from the inverse tone mapping unit 516. The inverse color conversion unit 514 processes the output image from the inverse gamma correction unit 515. The inverse white balance unit 513 processes the output image from the inverse color conversion unit 514. The mosaic unit 706 processes the output image from the inverse white balance unit 513. The noise adding unit 705 synthesizes a noise region 903 with the output image from the mosaic unit 706. As a result, a student image group 113 to which noise has been added may be generated.

[0059] From the viewpoint of accurately reproducing secondary spurious signals that occur when noise components undergo image processing at a later stage, it may be preferable to sample noise at the stage where image data is in device RGB format. From this viewpoint, the noise adding unit 705 may be disposed anywhere between the stage before the inverse tone mapping unit 516 and the stage before the mosaic unit 706. For example, in FIG. 16, the noise adding unit 705 may be disposed between the inverse tone mapping unit 516 and the inverse gamma correction unit 515. In FIG. 16, the noise adding unit 705 may be disposed between the inverse gamma correction unit 515 and the inverse color conversion unit 514. In FIG. 16, the noise adding unit 705 may be disposed between the inverse color conversion unit 514 and the inverse white balance unit 513. In FIG. 16, the noise adding unit 705 may be disposed between the inverse white balance unit 513 and the mosaic unit 706. In other words, the noise image may be added to the input image of the inverse tone mapping unit 516, the input image of the inverse gamma correction unit 515, the input image of the inverse color conversion unit 514, the input image of the inverse white balance unit 513, the input image of the mosaic unit 706, or the output image of the mosaic unit 706.

[0060] The sampling points of the noise image 1100 correspond to the arrangement of the noise adding unit 705 .

[0061] (13) Comparison of effects FIG. 17 is a diagram for explaining the effect of this embodiment. An image 1601 shows an image output by inputting an image captured by the imaging device 100 mounted on a satellite to the learning model 116 trained by a student image without noise (i.e., a learning model having only a demosaic function). An image 1602 shows an image output by inputting an image captured by the imaging device 100 mounted on a satellite to the learning model 116 trained by a student image with noise (i.e., a learning model having a demosaic function and a noise reduction function). It can be seen that noise remains in the image 1601. It can be seen that noise has been reduced by the learning model 116 in the image 1602. In this way, noise is accurately reduced by having the learning model 116 learn a student image generated using a noise image acquired by the imaging device 100.

[0062] (14) Technical ideas derived from the embodiments [Point 1] The information processing device 110 is an example of an image processing device that generates a learning model 116 that reduces noise contained in an image acquired by a predetermined imaging device 100. The CPU 301 and the extraction unit 700 operate as an extraction means that extracts an image of an optical black area (e.g., optical black area 902, light-shielded pixel area 900) that is included in each of a plurality of images captured by the predetermined imaging device 100 and is not irradiated with light that has passed through the optical system (e.g., lens unit 205) of the predetermined imaging device 100 as a noise image (noise image 1100, noise area 903). The CPU 301 and the noise adding unit 705 operate as a synthesis means that synthesizes a noise image with a teacher image (e.g., original image set 102) to generate a student image (e.g., student image group 113). The CPU 301 and the learning processing device 115 operate as a learning means that provides a student image as an input to the learning model 116 to learn the learning model.

[0063] According to the first aspect, a learning model 116 capable of reducing noise more simply and accurately than before is provided. In particular, a student image is generated using an actual noise image acquired by an imaging device 100 that generates an image to be subjected to noise reduction. In other words, since the source of the noise image and the source of the image to be subjected to noise reduction match, a high noise reduction effect is expected. Note that, as a learning method, a method of updating the learning model so that an output image output from the learning model 116 approaches a teacher image may be adopted. For example, a method of updating the learning model (weighting coefficient) may be adopted so that a difference (error) between an output image output from the learning model 116 and a teacher image becomes small.

[0064] [Point 2] As shown in Fig. 11 to Fig. 13, a teacher image (original image 1110) may be composed of a plurality of partial images (pixel regions 1111) smaller than the size of the teacher image. As shown in Fig. 12, the synthesis means (noise adding unit 705) may generate a student image by synthesizing each of a plurality of partial images constituting the teacher image with one of a plurality of noise images (noise regions 903) obtained one by one from a plurality of images captured by a predetermined imaging device 100. This makes it difficult for overlearning to occur for a specific noise image 1100, and a high noise reduction effect can be obtained.

[0065] [Point 3] 11 to 13, the synthesis means (noise adding unit 705) may randomly select a noise image to be applied to each of the partial images from among the multiple noise images. This makes it difficult for overlearning to occur for a specific noise image 1100, and thus makes it possible to obtain a higher noise reduction effect.

[0066] [Point 4] The noise images applied to each of the multiple partial images may be selected so as not to overlap with each other. This makes it difficult for overfitting to occur for a specific noise image 1100, and may result in a higher noise reduction effect.

[0067] [Point 5] The multiple images (e.g., noise images 1100a, 1100b, ...) from which multiple noise images used to generate one student image are extracted are each captured by a specific imaging device 100 under the same imaging conditions. As a result, if the imaging conditions are the same, the tendency for noise to occur will be similar. Therefore, by selecting multiple noise images captured under the same imaging conditions, a learning model 116 with a higher noise reduction effect will be obtained.

[0068] [Point 6] The imaging conditions may include at least one of the temperature, sensitivity, and exposure time of a given imaging device 100. These are parameters that contribute to the occurrence of noise, and therefore may be suitable as criteria for selecting a noise image.

[0069] [Point 7] The image input to the learning model 116 may be a mosaic image. The output image output from the learning model 116 may be a demosaic image corresponding to the mosaic image. In this case, the learning model 116 is a learning model that can simultaneously realize a demosaic function in addition to a noise reduction function. However, the above-mentioned learning model 116 may be a model that has a noise reduction function that inputs a color image (e.g., device RGB) and outputs a color image (e.g., device RGB) and does not include a demosaic function.

[0070] [Point 8] The mosaic image may be a Bayer image, or conversely, a mosaic image other than a Bayer array may be applied.

[0071] [Point 9] The Bayer image may be a RAW image. Conversely, a Bayer image other than a RAW image may be used.

[0072] [Point 10] The synthesis means (e.g., student image generation device 111) may synthesize a noise image with a teacher image to generate a first image (e.g., noise addition unit 705), apply inverse tone mapping processing to the first image to generate a second image (e.g., inverse tone mapping unit 516), apply inverse gamma correction to the second image to generate a third image (e.g., inverse gamma correction unit 515), apply inverse color conversion to the third image to generate a fourth image (e.g., inverse color conversion unit 514), apply inverse white balance processing to the fourth image to generate a fifth image (e.g., inverse white balance unit 513), and apply mosaic processing to the fifth image to generate a student image that is a mosaic image (e.g., mosaic unit 706).

[0073] [Point 11] The teacher image generating device 112 operates as a generating means for generating a comparison image from the teacher image to be compared with the output image in the learning means. The generating means (e.g., teacher image generating device 112) may apply an inverse tone mapping process to the teacher image to generate a sixth image (e.g., an inverse tone mapping unit 516), apply an inverse gamma correction to the sixth image to generate a seventh image (e.g., an inverse gamma correction unit 515), apply an inverse color conversion to the seventh image to generate an eighth image (e.g., an inverse color conversion unit 514), and apply an inverse white balance process to the eighth image to generate a comparison image (e.g., an inverse white balance unit 513).

[0074] [Point 12] As illustrated in FIG. 16, the synthesis means (student image generation device 111) may apply an inverse tone mapping process to the teacher image to generate a first image (e.g., inverse tone mapping unit 516), apply an inverse gamma correction to the first image to generate a second image (e.g., inverse gamma correction unit 515), apply an inverse color conversion to the second image to generate a third image (e.g., inverse color conversion unit 514), apply an inverse white balance process to the third image to generate a fourth image (e.g., inverse white balance unit 513), apply mosaic processing to the fourth image to generate a fifth image (e.g., inverse color conversion unit 514), and synthesize a noise image with the fifth image to generate a student image that is a mosaic image (e.g., noise addition unit 705).

[0075] [Point 13] As illustrated in FIG. 7 and FIG. 16, the synthesis means (student image generating device 111) applies an inverse tone mapping process to the teacher image to generate a first image, applies an inverse gamma correction to the first image to generate a second image, applies an inverse color conversion to the second image to generate a third image, applies an inverse white balance process to the third image to generate a fourth image, and applies mosaic processing to the fourth image to generate a student image that is a mosaic image. Here, the noise image is synthesized with any of the first image, the second image, the third image, and the fourth image. For example, in FIG. 7, the noise adding unit 705 is disposed before the inverse tone mapping unit 516, and in FIG. 16, it is disposed after the mosaic unit 706, but these are merely examples. As described with reference to FIG. 16, the noise adding unit 705 may be disposed anywhere from before the inverse tone mapping unit 516 to after the mosaic unit 706. In other words, the noise image may be added to the input image of the inverse tone mapping unit 516, the input image of the inverse gamma correction unit 515, the input image of the inverse color conversion unit 514, the input image of the inverse white balance unit 513, the input image of the mosaic unit 706, or the output image of the mosaic unit 706.

[0076] [Point 14] 9, an image captured by a given imaging device 100 is a rectangular image. A noise image (e.g., a noise region 903) may be extracted from a rectangular optical black region 902 that exists parallel to a short side or a long side of the rectangular image.

[0077] [Point 15] 9, the area of ​​the rectangular optical black region 902 is larger than the area of ​​the noise image (e.g., noise region 903). The extraction means (e.g., extraction unit 700) may randomly determine the position of the noise image to be extracted from the rectangular optical black region 902, and extract the noise image from the determined position in the rectangular optical black region 902. Since the optical black region 902 is a region that is not irradiated with light, it is considered to include noise caused by dark current, etc.

[0078] [Point 16] The predetermined imaging device 100 may be a camera mounted on an artificial satellite. In this case, the noise includes bright spot noise caused by cosmic rays being incident on the predetermined imaging device 100. This makes it possible to accurately reduce bright spot noise caused by cosmic rays.

[0079] [Point 17] The image processing device 117 functions as a noise reduction device that reduces noise contained in an input image. The communication circuits 203 and 303 function as input means that input an image captured by a predetermined imaging device 100 as an input image to the trained learning model 116 generated by the image processing device described in any one of the aspects 1 to 16. The CPUs 201 and 301 function as acquisition means that acquire an output image corresponding to the input image input from the input means from the learning model 116.

[0080] [Point 18] The lens unit 205 is an example of an optical system. The imaging element 207 is an example of an imaging element that converts light incident through the optical system into an image signal. The CPU 201 and the image processing device 117 are an example of a noise reduction device according to the seventeenth aspect that reduces noise from an image corresponding to an image signal acquired by the imaging element. As illustrated in FIG. 9, the long side of the imaging element 207 is larger than the diameter of an image circle 901 of the optical system. This makes it possible to acquire a noise area 903 from an optical black area 902 even without a light shielding unit 206.

[0081] [Point 19] A learning method executed by an image processing device to generate a learning model for reducing noise included in an image acquired by a predetermined imaging device, comprising: An extraction step (e.g., S1501 to S1503) of extracting, as a noise image, an image of an optical black region that is included in each of a plurality of images captured by a predetermined imaging device and is not irradiated with light that has passed through an optical system of the predetermined imaging device; A synthesis process (e.g., S1504 to S1506) of synthesizing a noise image with a teacher image to generate a student image; A learning process (e.g., S1405) of providing a student image as an input to the learning model to learn the learning model; A learning method having:

[0082] [Point 20] The program 306 is an example of a program that causes a computer to function as the image processing device according to any one of the first to sixteenth aspects.

[0083] The invention is not limited to the above-described embodiment, and various modifications and variations are possible within the scope of the gist of the invention. [Explanation of symbols]

[0084] 100...Imaging device 110...Information processing device 116...Learning Model 115...Learning processing device 700...Extraction part 705...Noise adding section

Claims

1. An image processing device that generates a learning model for reducing noise contained in an image acquired by a predetermined imaging device, an extraction means for extracting, as a noise image, an image of an optical black region included in each of the plurality of images captured by the predetermined imaging device, the image being not irradiated with light that has passed through an optical system of the predetermined imaging device; a synthesis means for synthesizing the noise image with a teacher image to generate a student image; a learning means for providing the student image as an input to the learning model to learn the learning model; The image processing device includes:

2. the teacher image is composed of a plurality of partial images smaller than the size of the teacher image, The image processing device described in claim 1, wherein the synthesis means generates the student image by synthesizing each of the plurality of partial images constituting the teacher image with one of a plurality of noise images obtained one by one from the plurality of images captured by the specified imaging device.

3. The image processing apparatus according to claim 2 , wherein the synthesizing means randomly selects a noise image to be applied to each of the plurality of partial images from the plurality of noise images.

4. The image processing apparatus according to claim 3 , wherein the noise images applied to each of the plurality of partial images are selected so as not to overlap.

5. The image processing apparatus according to claim 1 , wherein the images from which the noise images used to generate one student image are extracted are each acquired by the predetermined imaging device under the same imaging conditions.

6. The image processing device according to claim 5 , wherein the image capturing conditions include at least one of a temperature, a sensitivity, and an exposure time of the predetermined image capturing device.

7. The image input to the learning model is a mosaic image, The image processing device according to claim 1 , wherein an output image output from the learning model is a demosaic image corresponding to the mosaic image.

8. The image processing apparatus according to claim 7 , wherein the mosaic image is a Bayer image.

9. The image processing device according to claim 8 , wherein the Bayer image is a RAW image.

10. The synthesis means is generating a first image by synthesizing the noise image with the teacher image; applying an inverse tone mapping process to the first image to generate a second image; applying inverse gamma correction to the second image to generate a third image; applying an inverse color transform to the third image to generate a fourth image; applying an inverse white balance process to the fourth image to generate a fifth image; applying mosaic to the fifth image to generate the mosaic image of the student image; The image processing device according to claim 7.

11. The learning means further includes a generating means for generating a comparison image from the teacher image to be compared with the output image, The generating means includes: applying an inverse tone mapping process to the teacher image to generate a sixth image; applying inverse gamma correction to the sixth image to generate a seventh image; applying an inverse color transform to the seventh image to generate an eighth image; applying a reverse white balance process to the eighth image to generate the comparison image; The image processing device according to claim 10.

12. The synthesis means is applying an inverse tone mapping process to the teacher image to generate a first image; applying inverse gamma correction to the first image to generate a second image; applying an inverse color transform to the second image to generate a third image; applying an inverse white balance process to the third image to generate a fourth image; applying mosaicking to the fourth image to generate a fifth image; synthesizing the noise image with the fifth image to generate the student image, which is the mosaic image; The image processing device according to claim 7.

13. The synthesis means is applying an inverse tone mapping process to the teacher image to generate a first image; applying inverse gamma correction to the first image to generate a second image; applying an inverse color transform to the second image to generate a third image; applying an inverse white balance process to the third image to generate a fourth image; applying a mosaic to the fourth image to generate the mosaic image of the student image; The noise image is synthesized with any one of the first image, the second image, the third image, or the fourth image. The image processing device according to claim 7.

14. The image captured by the predetermined imaging device is a rectangular image, The image processing apparatus according to claim 1 , wherein the noise image is extracted from a rectangular optical black region that exists parallel to a short side or a long side of the rectangular image.

15. an area of ​​the rectangular optical black region is greater than an area of ​​the noise image; The extraction means includes: Randomly determining a position of the noise image to be extracted from the rectangular optical black region; The image processing apparatus of claim 14 , further comprising: extracting the noise image from the determined location in the rectangular optical black region.

16. the predetermined imaging device is a camera mounted on an artificial satellite, The image processing device according to claim 1 , wherein the noise includes bright spot noise caused by cosmic rays being incident on the predetermined imaging device.

17. A noise reduction device for reducing noise contained in an input image, comprising: an input unit that inputs an image acquired by the predetermined imaging device as an input image to the learned learning model generated by the image processing device according to any one of claims 1 to 16; an acquisition means for acquiring an output image corresponding to the input image input from the input means from the learning model; A noise reduction device having the following:

18. An optical system; an image sensor that converts the light incident through the optical system into an image signal; The noise reduction device according to claim 17, which reduces noise from an image corresponding to an image signal acquired by the imaging element; having An imaging device, wherein a long side of the imaging element is larger than a diameter of an image circle of the optical system.

19. A learning method executed by an image processing device to generate a learning model for reducing noise included in an image acquired by a predetermined imaging device, comprising: an extraction step of extracting, as a noise image, an image of an optical black region included in each of the plurality of images captured by the predetermined imaging device, the image being not irradiated with light that has passed through an optical system of the predetermined imaging device; a synthesis step of synthesizing the noise image with a teacher image to generate a student image; a learning step of providing the student image as an input to the learning model to learn the learning model; A learning method having:

20. A program that causes a computer to function as the image processing device according to any one of claims 1 to 16.

Images