Image processing device and image processing system

Abstract

To obtain training data reduced in image degradation caused by demosaicing even when training images are demosaiced.SOLUTION: An image processing device 100 provided herein comprises a first processing unit 11 for applying mosaic processing to a color image, and a second processing unit 12 for applying first demosaic processing to the image processed by the first processing unit 11, and is configured to generate training data to be used for training of a model using the image processed by the second processing unit 12.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to image processing. [Background technology]

[0002] One method used by imaging devices such as digital cameras to acquire color information from a subject is a single-chip method in which a color filter is applied to the image sensor, resulting in different colors being acquired depending on the pixel position. In single-chip imaging, the process of converting a RAW image from the image sensor into a color image consisting of three RGB channels is called demosaicing. Challenges with demosaicing include image degradation such as false colors, moiré, and zipper noise. Non-Patent Document 1 describes a technology that uses deep learning to suppress image degradation during demosaicing. Furthermore, many high-performance image processing technologies utilizing deep learning have been proposed in recent years. Patent Document 1 describes a technology for learning noise reduction using training data consisting of a set of training images and images to which artificial noise has been added and then digital gain has been applied. Such deep learning-based technologies generally require a large number of training images. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-41375 [Non-patent literature]

[0004] [Non-Patent Document 1] M. Gharbi et al, “Deep joint demosaicking and denoising”, [online], [Retrieved March 6, 2020],<https: / / groups.csail.mit.edu / graphics / demosaicnet / data / demosaic.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] Images collected as training images using imaging devices or via the Internet contain image quality degradation due to demosaicing. If these images are used as training data for model training, the output images from the model will contain image quality degradation due to demosaicing. However, technologies for suppressing image quality degradation during demosaicing, including those described in Non-Patent Document 1, target images before demosaicing (RAW images). On the other hand, images collected as training images are usually images after demosaicing, so even if technologies for suppressing image quality degradation during demosaicing, including those described in Non-Patent Document 1, are applied to training images, training data in which image quality degradation due to demosaicing is suppressed cannot be obtained.

[0006] Therefore, an object of the present invention is to obtain training data in which image quality degradation due to demosaicing is suppressed even when the learning images are demosaiced. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized by having a first processing means that performs mosaic processing on a color image, a second processing means that performs a first demosaic processing on the image processed by the first processing means, and a first generation means that generates training data to be used for model training based on the image processed by the second processing means. [Effects of the Invention]

[0008] According to the present invention, even if the learning images are demosaiced, training data can be obtained in which image quality degradation due to demosaicing is suppressed. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of the internal configuration of an image processing device. [Figure 2] FIG. 10 is a diagram illustrating an example of a color array. [Figure 3]10 is a flowchart illustrating a process executed by the image processing apparatus. [Figure 4] FIG. 1 is a diagram illustrating an example of a device configuration of an image processing system. [Figure 5] FIG. 2 is a diagram illustrating an example of the functional configuration of each device in the system. [Figure 6A] 4 is a flowchart showing a process executed by the image processing system. [Figure 6B] 4 is a flowchart showing a process executed by the image processing system. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[0011] [Embodiment 1] In this embodiment, a method for generating training data from collected training images will be described. Training images are images that have been demosaiced. The training images contain image quality degradation caused by demosaicing. Hereinafter, training images will be referred to as color images. In this embodiment, training data is generated by performing high-quality demosaicing on a mosaic image generated by performing mosaic processing on a color image.

[0012] The configuration of the image processing device according to this embodiment will be described with reference to FIG. FIG. 1(a) shows an example of the hardware configuration of an image processing device 100. The image processing device 100 includes a CPU 101, a GPU 102, a RAM 103, a ROM 104, and an HDD 105. These components are interconnected via a bus 106. The CPU (Central Processing Unit) 101 controls the entire image processing device 100. The CPU 101 may execute various processes using a GPU (Graphics Processing Unit) 102 together with the CPU 101. The CPU 101 reads a program stored in the ROM 104 or the like, loads it into the RAM 103, and executes it to realize the processing of the flowcharts described below. The RAM 103 is a volatile memory that temporarily stores images and execution results of various processes. The ROM 104 is a non-volatile memory that stores programs and various data. The HDD (Hard Disk Drive) 105 is a rewritable secondary storage device that stores programs, model parameters, collected learning images, generated training data, etc.

[0013] 1(b) shows an example of the functional configuration of the image processing device 100. The image processing device 100 functions as a first processing unit 11, a second processing unit 12, and a learning unit 13 by the CPU 101 executing a program stored in the ROM 104 or the like. The storage unit 10 stores a large number of collected learning images. The learning images are color images consisting of three channels of RGB. The color images may be collected by any method, such as by using an imaging device equipped with a general CMOS sensor, or via the Internet. The storage unit 10 may be implemented using the HDD 105 or an external storage device.

[0014] The first processing unit 11 acquires a color image from the storage unit 10 and generates a mosaic image by performing mosaic processing on the acquired color image. In this embodiment, the mosaic processing is a thinning process. Specifically, the first processing unit 11 thins out predetermined pixels from the color image to generate a one-channel image consisting of a predetermined color array that imitates a RAW image. FIG. 2 shows an example of a color array used in the thinning process. As shown in FIG. 2, an example of the color array is a Bayer array, whose basic unit is a 2×2 pattern in which R, G, G, and B are arranged from the upper left to the lower right. The first processing unit 11 outputs the mosaic image to the second processing unit 12.

[0015] The second processing unit 12 performs high-quality demosaic processing on the mosaic image output from the first processing unit 11. Specifically, the second processing unit 12 performs high-quality demosaic processing on the mosaic image to suppress image quality degradation caused by demosaic, such as false colors, moiré, and zipper noise. A deep learning model is used for the high-quality demosaic processing. For example, the technology described in Non-Patent Document 1 is used. The second processing unit 12 also performs predetermined image processing on the image after the high-quality demosaic processing (hereinafter referred to as the demosaic image). The second processing unit 12 outputs the image obtained by performing the predetermined image processing on the demosaic image to the learning unit 13 as training data. The learning unit 13 uses the training data output from the second processing unit 12 to learn a model that outputs an image.

[0016] The first processing unit 11 performs image processing on the color image before mosaic processing. This image processing is processing (hereinafter referred to as inverse image processing) that converts an image that has undergone predetermined image processing on the demosaic image by the second processing unit 12 back to the state before the predetermined image processing. Here, the predetermined image processing performed on the demosaic image by the second processing unit 12 is an example of the first image processing. The inverse image processing performed on the mosaic image by the first processing unit 11 is an example of the second image processing.

[0017] 3 is a flowchart showing the processing executed by the image processing device 100 according to this embodiment. Hereinafter, each step (process) will be represented by adding an S before the reference numeral. In S301, the CPU 101 acquires a color image from the storage unit 10 and performs inverse image processing on the acquired color image. In S302, the CPU 101 performs thinning processing on the color image that has been subjected to the inverse image processing in S301 to generate a mosaic image. In S303, the CPU 101 performs the high-quality demosaic processing described above on the mosaic image generated in S302 to generate a demosaic image.

[0018] In S304, CPU 101 performs image processing such as gamma correction, color conversion, brightness correction, and contrast correction on the demosaic image generated in S303. Here, the inverse image processing performed in S301 is set to have a strict or approximately inverse mapping relationship with the image processing applied in this step. In this step, CPU 101 performs brightness correction that doubles pixel values, and then performs gamma correction that raises pixel values ​​to the 1 / 2.2 power. In this case, in S301, CPU 101 first performs de-gamma correction that raises pixel values ​​to the 2.2 power, and then performs brightness correction that raises pixel values ​​to the 0.5 power. In this manner, the CPU 101 generates training data from the color images stored in the storage unit 10. The CPU 101 executes the processes of S301 to S304 on a large number of color images stored in the storage unit 10, thereby generating a large amount of training data.

[0019] In S305, CPU 101 uses the generated large amount of training data to perform training on a model that outputs an image. The training may be aimed at reducing image quality degradation due to demosaicing, or may be aimed at suppressing image quality degradation due to demosaicing in conjunction with another primary objective. In other words, the input image paired with the training data as a correct answer image may be the original color image from which the training data was derived, or may be an image obtained by processing that color image. For example, the training may be image conversion training, the primary objective of which is to reduce noise in the input image, or image generation training, the primary objective of which is to generate a realistic image. The processing of this flowchart then ends.

[0020] According to the present embodiment, even if the collected training images are demosaiced, by performing mosaic processing and high-quality demosaicing on the training images, training data can be generated in which image quality degradation due to demosaicing is suppressed. By using the training data generated in this manner to train a model, image degradation due to demosaicing in the output image can be suppressed.

[0021] [Embodiment 2] In this embodiment, a method is described in which a model is trained using training images to output an image in which both image quality degradation due to demosaicing and noise in the input image are reduced, and the trained model is used to perform inference on a captured image. Note that descriptions that overlap with those in the first embodiment will be omitted.

[0022] FIG. 4 shows an example of the device configuration of an image processing system according to this embodiment. The image processing system according to this embodiment includes an image processing device 100, an imaging device 200, and an inference device 300. The hardware configuration of the image processing device 100 of embodiment 2 is similar to that of the image processing device 100 of embodiment 1, and therefore a description thereof will be omitted. Note that in this embodiment, the image processing device 100, the imaging device 200, and the inference device 300 are configured as separate devices, but some or all of the devices may be configured integrally.

[0023] The imaging device 200 includes a CPU 201, a communication I / F unit 202, a RAM 203, a ROM 204, an imaging element 206, and an optical system 207. These units are interconnected via a bus 205. The CPU 201 controls the entire imaging device 200. The CPU 201 reads out a program stored in the ROM 204 or the like, expands it in the RAM 203, and executes it to realize processing of a flowchart by the imaging device 200, which will be described later. The RAM 203 temporarily stores images and execution results of various processes. The ROM 204 stores programs and various data. The communication I / F unit 202 is an interface for communicating with an external device. The imaging device 200 transmits a captured image to the inference device 300 via the communication I / F unit 202. The imaging element 206 converts light that has passed through the optical system 207 into an image signal. The CPU 201 generates a captured image from the image signal converted by the imaging element 206.

[0024] The inference device 300 includes a CPU 301, a GPU 302, a RAM 303, a ROM 304, a HDD 305, and a communication I / F unit 306. These units are interconnected via a bus 307. The CPU 301 controls the entire inference device 300. The CPU 301 may execute various processes using the GPU 302 together with the CPU 301. The CPU 301 reads out a program stored in the ROM 304 or the like, expands it in the RAM 303, and executes it, thereby realizing the processing of a flowchart by the inference device 300, which will be described later. The RAM 303 temporarily stores images and the execution results of various processes. The ROM 304 stores programs and various data. The HDD 305 stores programs, model parameters of models trained by the image processing device 100, captured images received from the imaging device 200, and the like. The communication I / F unit 306 is an interface for communicating with external devices. The CPU 301 receives a captured image from the imaging device 200 via the communication I / F unit 306 .

[0025] Fig. 5 shows an example of the functional configuration of each device in the image processing system. The image processing device 100 functions as each functional unit shown in Fig. 5 when the CPU 101 executes a program stored in the ROM 104 or the like. The imaging device 200 functions as each functional unit shown in Fig. 5 when the CPU 201 executes a program stored in the ROM 204 or the like. The inference device 300 functions as each functional unit shown in Fig. 5 when the CPU 301 executes a program stored in the ROM 304 or the like. Of the functional units of the image processing device 100 shown in Fig. 5, those that are similar to those described in the first embodiment are given the same reference numerals and will not be described again.

[0026] The image processing device 100 includes a storage unit 10, a first processing unit 11, a second processing unit 12, a learning unit 13, a third processing unit 15, and a first model storage unit 16. The storage unit 10 and the first model storage unit 16 are stored in an HDD 105. The third processing unit 15 adds artificial noise, which imitates noise generated in the imaging device 200, to the mosaic image generated by the first processing unit 11. The artificial noise may be generated according to a method described in Patent Document 1, for example.

[0027] The third processing unit 15 also performs lightweight demosaic processing on the mosaic image to which artificial noise has been added. Lightweight demosaic processing is characterized by being more susceptible to image quality degradation due to demosaic, such as false colors, moire, and zipper noise, compared to the high-quality demosaic processing performed by the second processing unit 12. The lightweight demosaic processing may be, for example, rule-based processing. A look-up table (LUT) or the like is used as the rule-based processing. In this case, for example, the relationship between input data and output data may be created in advance as an LUT and stored in the memory of the device. The third processing unit 15 processes the mosaic image to which artificial noise has been added, with reference to the LUT, and outputs the result as an image after lightweight demosaic processing. The high-quality demosaic processing is an example of a first demosaic processing. The lightweight demosaic processing is an example of a second demosaic processing. Furthermore, the third processing unit 15 performs predetermined image processing such as gamma correction, color conversion, brightness correction, contrast correction, etc. on the image after the light demosaicing process. The predetermined image processing here is the same as the predetermined image processing (first image processing) performed on the demosaiced image by the second processing unit 12. The first model storage unit 16 stores the model parameters of the trained model trained by the training unit 13.

[0028] The imaging device 200 includes an imaging unit 20, a fourth processing unit 21, and a transmission unit 22. The imaging unit 20 converts light that has passed through an optical system 207 into a signal using an imaging element 206 such as a CMOS, and outputs the signal as an image. The image output from the imaging unit 20 is a RAW image and a mosaic image. The fourth processing unit 21 performs a lightweight demosaic process on the image output from the imaging unit 20. The lightweight demosaic process performed by the fourth processing unit 21 is the same as the lightweight demosaic process performed by the third processing unit 15. Note that "same" here includes "approximately the same." The fourth processing unit 21 also performs predetermined image processing such as gamma correction, color conversion, brightness correction, and contrast correction on the image after the lightweight demosaic process. The predetermined image processing here is the same as the predetermined image processing (first image processing) performed on the demosaic image by the second processing unit 12. Note that "same" here includes "approximately the same." The fourth processing unit 21 outputs the image obtained by performing the predetermined image processing on the image after the lightweight demosaic process to the transmission unit 22 as a captured image. The transmission unit 22 uses the communication I / F unit 202 to transmit the captured image output from the fourth processing unit 21 to the inference device 300. Note that the transmission unit 22 may transmit via an SDI cable or the like, or may transmit via wireless communication or internet communication.

[0029] The inference device 300 includes a receiving unit 31, an inference unit 32, and a second model storage unit 33. The receiving unit 31 receives the captured image transmitted from the imaging device 200 using the communication I / F unit 306. Note that the receiving unit 31 may receive the captured image via an SDI cable or the like, or via wireless communication or internet communication. The inference unit 32 inputs the captured image received by the receiving unit 31 into a trained model based on the model parameters stored in the second model storage unit 33, thereby executing an inference process. The second model storage unit 33 stores model parameters of the trained model trained by the training unit 13 of the image processing device 100.

[0030] 6A and 6B are flowcharts showing the processing executed by the image processing system according to this embodiment. First, the processing executed by the image processing device 100 will be described using the flowchart in FIG. The processes in S601 to S603 are the same as those in S301 to S303 in FIG. In S604, the CPU 101 adds artificial noise to the mosaic image generated in S602. In S605, the CPU 101 performs the aforementioned lightweight demosaic processing on the image to which the artificial noise has been added in S603. The processes of S604 and S605 are executed in parallel with S603.

[0031] The process of S606 is similar to the process of S304, but differs in that the processing targets include both images to which the process of S603 has been applied and images to which the processes of S604 and S605 have been applied. The processing of S607 is similar to the processing of S305 in that the image that has been through the processing of S603 is used as training data, but the image that has been through the processing of S604 to S605 is used as an input image, and conversion from the input image to training data is learned. By the processing of S607, the trained model trained in S607 is stored in HDD 105. In S608, the trained model stored in HDD 105 is copied to HDD 305 of inference device 300. The copying may be performed, for example, via an external storage device or via a network. This completes the processing of the flowchart in FIG. 6A.

[0032] 6B, a description will be given of the processing executed by the imaging device 200 and the inference device 300. Steps S611 to S614 are processing executed by the imaging device 200. In S611, the CPU 201 captures an image using the image sensor 206 and the optical system 207. In this step, a mosaic image is output. In S612, the CPU 201 performs a lightweight demosaic process on the mosaic image output in S611. The lightweight demosaic process here is the same as the process in S605. In S613, the CPU 201 performs image processing such as gamma correction, color conversion, brightness correction, and contrast correction on the image that has been subjected to the light mosaic processing in S612. The image processing here is the same as the processing in S606. In S614, the CPU 201 transmits the image that has been subjected to image processing in S613 to the inference device 300 as a captured image. This completes the processing of the imaging device 200 in the flowchart of FIG. 6B.

[0033] Steps S615 and S616 are processes performed by the inference device 300. In S615 , the CPU 301 receives the captured image from the imaging device 200 . In S616, the CPU 301 reads out the trained model copied in S608 from the HDD 305, and inputs the captured image received in S615 into the trained model thus read out, thereby executing the inference process. This completes the processing of inference device 300 in the flowchart of FIG. 6B.

[0034] According to the present embodiment as described above, demosaiced training images that contain image quality degradation due to demosaicing are used to perform training to generate images in which both the image quality degradation and noise in the input image are reduced, and the trained model can be used to infer captured images.

[0035] The present invention can also be realized by providing a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The above-described embodiments are merely illustrative examples of how the present invention can be implemented, and the technical scope of the present invention should not be interpreted as being limited by them. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0036] The disclosure of each of the above-described embodiments includes the following configurations, methods, and programs. (Configuration 1) a first processing means for performing mosaic processing on a color image; a second processing means for performing a first demosaic process on the image processed by the first processing means; a first generation means for generating training data to be used for model learning based on the image processed by the second processing means; 1. An image processing device comprising: (Configuration 2) 2. The image processing device according to configuration 1, wherein the mosaic processing is a process for generating a one-channel image having a predetermined color arrangement by thinning out predetermined pixels. (Configuration 3) the first generating means performs first image processing on the image processed by the second processing means; 3. The image processing device according to configuration 1 or 2, wherein the first processing means performs a second image processing on the color image before performing a mosaic process, converting the image that has been subjected to the first image processing into a state before the first image processing. (Configuration 4) 4. The image processing device according to any one of configurations 1 to 3, wherein the color image is an image that has been subjected to demosaic processing. (Configuration 5) 5. The image processing device according to any one of configurations 1 to 4, wherein the first demosaic processing is processing for suppressing image degradation caused by demosaic processing. (Configuration 6) 6. The image processing device according to configuration 5, wherein the image degradation includes at least one of false color, moire, and zipper noise. (Configuration 7) 7. The image processing device according to any one of configurations 1 to 6, further comprising a learning means for learning the model using the training data. (Configuration 8) a third processing means for performing a second demosaic processing different from the first demosaic processing on the image processed by the first processing means; a second generation means for generating an input image to be used for training the model based on the image processed by the third processing means; and 8. The image processing device according to configuration 7, wherein the learning means performs learning of the model using the input image and the training data. (Configuration 9) The first demosaic processing is a process using a deep learning model, 9. The image processing device according to configuration 8, wherein the second demosaic processing is a rule-based processing. (Configuration 10) the first generating means performs first image processing on the image processed by the second processing means; the second generating means performs the first image processing on the image processed by the third processing means; 10. The image processing device according to configuration 8 or 9, wherein the first processing means performs second image processing on the color image before performing mosaic processing, converting the image that has been subjected to the first image processing into a state before the first image processing. (Configuration 11) The image processing device according to any one of configurations 8 to 10, wherein the third processing means adds noise to the image processed by the first processing means before performing the second demosaic processing. (Configuration 12) An image processing system including the image processing device according to any one of configurations 8 to 11, an imaging means for capturing an image using an imaging element to generate a mosaic image; a fourth processing means for performing the same processing as the second demosaic processing on the mosaic image; inference means for performing inference using the image processed by the fourth processing means and the model trained by the training means; An image processing system comprising: (Configuration 13) The first demosaic processing is a process using a deep learning model, 13. The image processing system according to configuration 12, wherein the second demosaic processing is a rule-based processing. (Configuration 14) the first generating means performs first image processing on the image processed by the second processing means; the second generating means performs the first image processing on the image processed by the third processing means; the fourth processing means performs the same processing as the first image processing on the image that has been subjected to the same processing as the second demosaic processing, The image processing system according to configuration 12 or 13, wherein the first processing means performs second image processing on the color image before performing mosaic processing, converting the image that has been subjected to the first image processing back to its state before the first image processing. (method) a first processing step of performing mosaic processing on the color image; a second processing step of performing a first demosaic process on the image processed in the first processing step; a first generation step of generating training data to be used for model learning based on the image processed in the second processing step; An image processing method comprising: (program) 12. A program for causing a computer to function as the image processing device according to any one of the first to eleventh aspects. [Explanation of symbols]

[0037] 100: Image processing device, 200: Imaging device, 300: Inference device

Claims

1. a first processing means for performing mosaic processing on a color image; a second processing means for performing a first demosaic process on the image processed by the first processing means; a first generation means for generating training data to be used for model learning based on the image processed by the second processing means; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the mosaic processing is a process for generating a one-channel image having a predetermined color arrangement by thinning out predetermined pixels.

3. the first generating means performs first image processing on the image processed by the second processing means; 2. The image processing device according to claim 1, wherein the first processing means performs a second image processing on the color image before performing mosaic processing, converting the image that has been subjected to the first image processing into a state before the first image processing.

4. 2. The image processing apparatus according to claim 1, wherein the color image is a demosaic processed image.

5. 2. The image processing device according to claim 1, wherein the first demosaic processing is processing for suppressing image degradation caused by demosaic processing.

6. 6. The image processing device according to claim 5, wherein the image degradation includes at least one of false color, moire, and zipper noise.

7. 2. The image processing apparatus according to claim 1, further comprising: a learning unit that uses the training data to learn the model.

8. a third processing means for performing a second demosaic processing different from the first demosaic processing on the image processed by the first processing means; a second generation means for generating an input image to be used for training the model based on the image processed by the third processing means; and 8. The image processing apparatus according to claim 7, wherein the learning means performs learning of the model using the input image and the training data.

9. the first demosaic processing is processing using a deep learning model, 9. The image processing apparatus according to claim 8, wherein the second demosaicing process is a rule-based process.

10. the first generating means performs first image processing on the image processed by the second processing means; the second generating means performs the first image processing on the image processed by the third processing means; 9. The image processing device according to claim 8, wherein the first processing means performs a second image processing on the color image before performing mosaic processing, converting the image that has been subjected to the first image processing into a state before the first image processing.

11. 9. The image processing device according to claim 8, wherein the third processing means adds noise to the image processed by the first processing means before performing the second demosaicing process.

12. An image processing system including the image processing device according to claim 8, an imaging means for capturing an image using an imaging element to generate a mosaic image; a fourth processing means for performing the same processing as the second demosaic processing on the mosaic image; an inference means for performing inference using the image processed by the fourth processing means and the model trained by the training means; An image processing system comprising:

13. the first demosaic processing is processing using a deep learning model, 13. The image processing system according to claim 12, wherein the second demosaicing process is a rule-based process.

14. the first generating means performs first image processing on the image processed by the second processing means; the second generating means performs the first image processing on the image processed by the third processing means; the fourth processing means performs the same processing as the first image processing on the image that has been subjected to the same processing as the second demosaic processing, 13. The image processing system according to claim 12, wherein the first processing means performs a second image processing on the color image before performing mosaic processing, converting the image that has been subjected to the first image processing into a state before the first image processing.

15. a first processing step of performing mosaic processing on the color image; a second processing step of performing a first demosaic processing on the image processed in the first processing step; a first generation step of generating training data to be used for model learning based on the image processed in the second processing step; An image processing method comprising:

16. A program for causing a computer to function as the image processing device according to claim 1.

Images