Image processing device, its control method, and program

Abstract

To provide an image processing apparatus, a control method thereof, and a program capable of performing processing that suppresses deterioration in gradation of an image with N-bit depth exceeding M-bit while using a system that performs a convolutional neural network (CNN) with M-bit precision.SOLUTION: An image processing apparatus comprises: first generation means configured to generate a plurality of images having an M-bit depth per pixel by inputting an input image having a bit depth of N bits, and clipping M-bit segments from mutually different positions within the N bits; second generation means including a neural network for an M-bit-depth image and configured to generate, by inputting the plurality of images generated by the first generation means into the neural network, a plurality of output images corresponding to each of the plurality of images being input; and third generation means configured to generate an image having the N-bit depth from the plurality of output images generated by the second generation means.SELECTED DRAWING: None

Description

[Technical Field]

[0001] The present invention relates to an image processing apparatus, a control method thereof, and a program. [Background technology]

[0002] In recent years, many image processing methods using deep learning have been proposed. Convolutional neural networks (CNNs), which are generally used for deep learning inference, take a long time to calculate. For this reason, a method has been proposed in which the calculation precision of CNNs is increased to INT8 (8-bit precision).

[0003] Images captured by cameras can contain noise depending on the camera settings and the circumstances at the time. The deep learning technology mentioned above has been proposed to be used to remove such noise.

[0004] Patent Document 1 discloses a method for inferring images with improved resolution and contrast when noise is removed using a neural network. In Patent Document 1, when a RAW image is input into a neural network for learning, gamma correction is performed according to the output format that the user will view, thereby improving the accuracy of inference.

[0005] In Patent Document 2, before removing noise by deep learning, an image is passed through deep learning for region estimation to separate it into regions by noise level. Then, pixel values ​​for each region are converted into values ​​that take noise into account, and then passed through deep learning to remove noise, thereby improving the accuracy of noise removal. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] Japanese Patent Application Publication No. 2019-121252 [Patent Document 2] Japanese Patent Application Publication No. 2018-077786 Summary of the Invention [Problem to be solved by the invention]

[0007] However, the above conventional example does not take into consideration the fact that the calculation precision of CNN is INT8. The sensors used in cameras have a bit precision greater than 8 bits, and when CNN with INT8 is used, there is a problem that the gradation is lost due to the reduction of bits.

[0008] The present invention aims to provide a technology that performs processing that suppresses the reduction in gradation of images with an N-bit depth exceeding M bits, even when using a system that uses an M-bit CNN. [Means for solving the problem]

[0009] In order to solve this problem, for example, an image processing device of the present invention has the following arrangement: a first generating means for receiving an input image having a bit depth of N bits and extracting M bits at different positions from the N bits to generate a plurality of images having a bit depth of M bits per pixel; a second generating means having a neural network for images with an M-bit depth, which receives the plurality of images generated by the first generating means and generates a plurality of output images corresponding to the plurality of images; and third generating means for generating the image of N bit depth from the plurality of output images generated by the second generating means. [Effects of the Invention]

[0010] According to the present invention, it is possible to perform processing that suppresses a decrease in the gradation of an image with an N-bit depth exceeding M bits, while using a system that performs CNN with M-bit precision. [Brief explanation of the drawings]

[0011] [Figure 1]FIG. 1 is a block diagram of a basic system for performing inference, which is common to all embodiments. [Figure 2] FIG. 1 is a block diagram of a basic system for performing learning, which is common to all embodiments. [Figure 3A] 10 is a flowchart of an inference process according to the first embodiment. [Figure 3B] 10 is a flowchart of an inference process according to the first embodiment. [Figure 4] FIG. 1 is a conceptual diagram of 8-bit extraction common to all embodiments. [Figure 5] FIG. 1 is a conceptual diagram of a bit shift common to all embodiments. [Figure 6] 10 is a flowchart of a learning process according to the first embodiment. [Figure 7] FIG. 1 is a conceptual diagram of a neural network common to all embodiments. [Figure 8A] 10 is a flowchart of an inference process according to the second embodiment. [Figure 8B] 10 is a flowchart of an inference process according to the second embodiment. [Figure 9] FIG. 10 is a conceptual diagram of region division according to the second embodiment. [Figure 10A] 10 is a flowchart of an inference process according to the third embodiment. [Figure 10B] 10 is a flowchart of an inference process according to the third embodiment. [Figure 11] 10 is a flowchart of an inference process according to the fourth embodiment. [Figure 12] FIG. 10 is a conceptual diagram of cutting out upper bits common to the fourth to sixth embodiments. [Figure 13] FIG. 10 is a conceptual diagram of lower-order bit clipping common to the fourth to sixth embodiments. [Figure 14] FIG. 10 is a conceptual diagram of bit splicing common to the fourth to sixth embodiments. [Figure 15] 10 is a flowchart of a learning process according to the fourth embodiment. [Figure 16] 13 is a flowchart of an inference process according to the fifth embodiment. [Figure 17] 13 is a flowchart of a learning process according to the fifth embodiment. [Figure 18] 13 is a flowchart of an inference process according to the sixth embodiment. [Figure 19] 13 is a flowchart of a learning process according to the sixth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0012] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claimed invention. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0013] First, an image processing device that performs inference according to this embodiment will be described. Fig. 1 is a configuration diagram of an image processing device 100. In Fig. 1, a CPU 106, a ROM 105, a RAM 107, an image processing unit 104, an optical lens 101, an image sensor 102, a frame memory 103, a video output driving unit 108, a display driving unit 110, and a metadata extraction unit 112 are connected to an internal bus 113. These units are capable of exchanging data with each other via the internal bus 113.

[0014] The optical lens 101 is an optical element consisting of a lens and a motor for driving the lens. The optical lens 101 operates based on a control signal and can optically enlarge or reduce an image and adjust the focal length, etc. Furthermore, if the amount of incident light needs to be adjusted, the amount of light can be adjusted to a desired brightness by controlling the aperture area (F-number) of the iris. Light passing through the lens is imaged by the image sensor 102. The image sensor 102 uses a CCD sensor or CMOS sensor, and serves to convert the optical image formed on its detection surface into an electrical signal. The image sensor 102 is driven based on a control signal to reset the charge in the pixels and control the readout timing. The image sensor 102 also has the function of performing gain processing on pixel signals read out as electrical analog signals (voltage values) and converting analog signals into digital signals.

[0015] The image processing unit 104 performs various image processing on the image input from the image sensor 102. For example, this includes correction of the amount of light in the peripheral parts of the image caused by the characteristics of the optical lens 101, correction of sensitivity variations between pixels of the image sensor 102, color correction, flicker correction, etc. The image processing unit 104 also has a function of performing noise reduction processing using neural network parameters, the details of which will be described later.

[0016] The frame memory 103, commonly called RAM (Random Access Memory), is a device that temporarily stores video signals and can read them out when needed. Because video signals contain a huge amount of data, high-speed, high-capacity memory is required. In recent years, DDR4-SDRAM (Dual Data Rate 4 - Synchronous Dynamic RAM) and other memory types have become popular. This frame memory 103 enables a variety of processing. For example, it is an essential device for image processing, such as synthesizing images that differ over time or extracting only the required area.

[0017] The image processing device 100 has a central processing unit (CPU) 106 as a control unit for controlling each function. The CPU 106 functions as the brain of the device. A read-only memory (ROM) and a random access memory (RAM) 107 are connected to the CPU 106. The ROM 105 is a non-volatile storage element that stores programs for operating the CPU 106, various adjustment parameters, and the like. Programs read from the ROM 105 are expanded in the volatile RAM 107 and executed by the CPU 106. Generally, the RAM 107 is slower and has a smaller capacity than the frame memory 103.

[0018] A metadata extraction unit 112 extracts metadata information such as lens driving conditions and sensor driving conditions. The image generated by the image processing unit 104 is output to the outside of the image processing device 100 via a video output driving unit 108 and a video terminal 109. Typical interfaces include SDI (Serial Digital Interface), HDMI (registered trademark) (High Definition Multimedia Interface), and various other interfaces such as DisplayPort (registered trademark), which make it possible to display real-time images on an external monitor or the like.

[0019] The image generated by the image processing unit 104 is displayed on the display unit 111 via the display drive unit 110. The display unit 111 is a display device that can be viewed by the user and can display, for example, images processed by the image processing unit 104 and setting menus, allowing the user to check the operating status of the image processing device 100. The display unit 111 may be a small, low-power device such as an LCD (Liquid Crystal Display) or an organic EL (Electro Luminescence) display. It may also be equipped with a resistive or capacitive thin-film element known as a touch panel. The CPU 106 generates text to inform the user of the setting status of the image processing device 100 and a menu for setting the image processing device 100, and displays these on the display unit 111 by superimposing them on the image processed by the image processing unit 104. In addition to text information, it is also possible to superimpose shooting assist displays such as a histogram, vectorscope, waveform monitor, zebra, peaking, and false color.

[0020] 2 shows a system for performing learning in this embodiment. The system is composed of an image capturing device 200, an image processing system 210, a display device 220, and a storage device 230.

[0021] The camera control unit 201 controls the driving of the optical lens 101 and the image sensor 102 based on an instruction signal (command) received from the camera communication connection unit 212. The image signal receiving unit 211 receives an image signal (image data) from the imaging device 200.

[0022] The GPU 213 performs various calculations under the control of the CPU 106. The GPU 213 in this embodiment is used at least when performing learning by deep learning. Since a huge amount of calculation is required particularly during learning, a GPU with higher processing power for image processing than a CPU is used. The GPU 213 may also be used to generate images to be displayed on the display device 220. In this case, the images generated under the control of the GPU 213 are displayed on the display device 220 via a display drive unit 216 and a display device connection unit 217.

[0023] The storage device 230 is used to store a huge amount of image data as learning images and to store network parameters generated during learning. The image processing system 210 exchanges data with the storage device 230 via a storage drive unit 214 and a storage connection unit 215 provided in the system.

[0024] In this embodiment, the learning process is performed using the system shown in FIG. 2 , and the inference process is performed using the image processing device 100 shown in FIG. 1 . However, this is not a limitation. For example, the inference process can also be performed using the system shown in FIG. 2 . Furthermore, in this embodiment, the training images are assumed to be images in a Bayer array. However, images captured using a three-chip image sensor or a vertical color separation image sensor such as a FOVEON sensor may also be used. The same applies to other arrays (e.g., honeycomb structures, filter arrays of X-Trans CMOS sensors) in addition to the Bayer array. In the case of images in a Bayer array, they may be left as a single Bayer array image, or may be separated into individual color channels and used as training images. Furthermore, in this embodiment, the training image input to the neural network and the image output from the neural network are assumed to be one image, but the neural network may also input and output multiple images.

[0025] [First embodiment] A first embodiment will be described below. In the following, an example will be described in which an image with a depth of M bits per pixel is input and noise processing of the image is performed using a neural network for images with a depth of N bits, which is less than M bits. In this embodiment, to provide a specific example, a case will be described in which M is 14 and N is 8.

[0026] 3A, 3B, 4 and 5, the inference process performed by the image processing device 100 will be described. The image sensor 102 of this embodiment outputs 14-bit image data per pixel.

[0027] First, in S301 of the flowchart in FIG. 3A, the CPU 106 sets the neural network parameters recorded in the ROM 105 in the neural network in the image processing unit 104. Next, in S302, the CPU 107 acquires a first image from the image sensor 102 and supplies the acquired first image (an image having a 14-bit depth) to the image processing unit 104. Then, in S303, the CPU 106 controls the image processing unit 104 to perform correction processing. The correction processing here is correction processing for eliminating variations in the optical lens 101 and the image sensor 102, such as correction of peripheral light intensity or correction of variations in sensitivity for each pixel. However, if correction processing is not necessary, this step may not be performed.

[0028] Next, in S304, the CPU 106 controls the image processing unit 104 to apply a digital gain. In the digital gain processing, if the pixel value has an offset, the offset is subtracted, then the gain is applied, and then the offset is added.

[0029] Next, in S305, the CPU 106 controls the image processing unit 104 to generate a second image by subtracting an offset from each pixel value of the image to which the digital gain was applied in S304. The offset here refers to the black level added by the image sensor 102. In this embodiment, an example in which an offset is subtracted is shown, but if the process of subtracting the offset does not need to be performed, this step does not need to be performed.

[0030] Next, in S306, the CPU 106 controls the image processing unit 104 to generate a third image by extracting 8 bits at a first position of each pixel value of the second image. The 8-bit extraction positions here are 8 bits from the most significant bit position (bit 13) to the least significant bit, as shown in FIG. 4(a). Note that the numbers in each box in the illustration indicate bit positions. A 14-bit pixel value is represented by bit 0 (least significant bit) to bit 13 (most significant bit).

[0031] Next, in S307, the CPU 106 controls the image processing unit 104 to input the third image to the first neural network. The first neural network here is a neural network that is optimally trained to perform noise removal on the image from which the upper 8 bits have been cut out.

[0032] Next, in S308, CPU 106 controls image processing unit 104 to generate a fourth image in which each pixel value of the second image is limited to 4095 (12 bits). In other words, if a pixel value in the second image is 4095 or less, the pixel value constituting the fourth image is expressed as is in 12 bits, and if a pixel value in the second image is greater than 4095, the pixel value is changed to the maximum value "4095" that can be expressed in 12 bits. The "limit" used below should be understood to have the same meaning.

[0033] Next, in S309, CPU 106 controls image processing unit 104 to generate a fifth image by extracting 8 bits at a second position of each pixel value of the fourth image. The 8-bit extraction position here is 8 bits extracted from a position shifted 2 bits to the right from the most significant bit, as shown in FIG. 4(b). Next, in S310, CPU 106 controls image processing unit 104 to input the fifth image to a second neural network. Here, the second neural network is a neural network that is optimally trained to perform noise reduction on an image shifted 2 bits to the right from the most significant bit.

[0034] Next, in S311, CPU 106 controls image processing unit 104 to generate a sixth image in which each pixel value of the second image is limited to 1023. Then, in S312, CPU 106 controls image processing unit 104 to generate a seventh image in which 8 bits at a third position of each pixel value of the sixth image are cut out. The 8-bit cut-out position here is 8 bits cut out from a position shifted 4 bits to the right from the most significant bit, as shown in FIG. 4(c). Next, in S313, CPU 106 controls image processing unit 104 to input the seventh image to a third neural network. Here, the third neural network is a neural network that is optimally trained to perform noise reduction on an image shifted 4 bits to the right from the most significant bit.

[0035] Next, in S314, the CPU 106 controls the image processing unit 104 to generate a ninth image by shifting each pixel value of the eighth image output from the first neural network to the left by six bits. Here, each pixel value of the ninth image has data in the upper eight bits and zeros in the lower six bits 8 to 13, as shown in FIG. 5(a).

[0036] Next, in S315, the CPU 106 controls the image processing unit 104 to generate an 11th image by shifting each pixel value of the 10th image output from the second neural network to the left by 4 bits. Here, as shown in Fig. 5(b), data is entered into 8 bits of each pixel value of the 11th image, starting from the least significant bit and shifted 4 bits to the left, and the remaining bits 22 to 27 are filled with 0.

[0037] Next, in S316, the CPU 106 controls the image processing unit 104 to generate a 13th image by shifting each pixel value of the 12th image output from the third neural network to the left by two bits. Here, as shown in Fig. 5(c), data is entered into 8 bits from the least significant bit to the left by two bits, and the remaining bits 36 to 41 are filled with 0.

[0038] Next, in S317 (FIG. 3B), CPU 106 initializes variable I, which indicates each pixel, to 0. Then, in S318, CPU 106 determines whether image processing unit 104 has performed, for all pixels, the selection process for determining which pixel value to output from the image generated in S314 to S316. If CPU 106 determines that the selection process for all pixels has been completed, CPU 106 proceeds to S325. On the other hand, if CPU 106 determines that an unselected pixel exists, CPU 106 proceeds to S319.

[0039] In S319, CPU 106 determines whether the value of the I-th pixel of the ninth image is greater than or equal to 4096. If it is determined that the value of the I-th pixel is greater than or equal to 4096, CPU 106 advances the process to S320. Next, in S320, CPU 106 selects and outputs a pixel of the ninth image as the I-th pixel of the image data to be generated.

[0040] On the other hand, if CPU 106 determines in S319 that the I-th pixel value of the ninth image is smaller than 4096, the process proceeds to S321. In S321, CPU 106 determines whether the I-th pixel value of the ninth image is greater than or equal to 1024. If CPU 106 determines that the I-th pixel value is greater than or equal to 1024, the process proceeds to S322. In S322, CPU 106 selects and outputs a pixel of the eleventh image as the I-th pixel of the image data to be generated.

[0041] On the other hand, if the CPU 106 determines in S321 that the value of the I-th pixel of the ninth image is smaller than 1024, the process proceeds to S323. In S323, the CPU 106 selects and outputs the pixel of the thirteenth image as the I-th pixel of the image data to be generated.

[0042] Next, in S324, the CPU 106 increments the variable I, returns the process to S318, and repeats S318 to S324 until the selection process for all pixels is completed.

[0043] In the final step, S325, the CPU 106 controls the image processing unit 104 to add an offset to the image data obtained above to generate a fourteenth image. This step is not performed if S305 was not performed. In this embodiment, the limit values ​​are 4095 and 1023 as specific examples, and the threshold values ​​for determination are 4096 and 1024 as specific examples. However, these are not limited to these examples; any limit value or threshold value may be used. In this embodiment, pixel values ​​of each neural network are selected to generate the final image. However, this is not limiting; other methods may be used, such as summing, averaging, or weighted averaging. In this embodiment, digital gain is applied before input to the neural network. However, digital gain may also be applied after passing through the neural network. In this case, each neural network is prepared by optimally training the image before digital gain is applied.

[0044] 6 and 7, a learning process performed by the imaging device 200, image processing system 210, display device 220, and storage device 230 will be described. Fig. 6 is a flowchart of the learning process.

[0045] First, in S601, the CPU 106 acquires training images (noise images) and correct images (teacher images) from the storage device 230. Here, a training image is an image containing noise. A correct image is an image that contains the same object as the training image but is noise-free (or has very little noise). Training images can be generated by adding noise through simulation to a correct image that is less affected by noise. Alternatively, an image captured of the same object as the correct image under conditions where noise may actually occur (e.g., high sensitivity setting) may be used. In this case, for example, the training images are images captured with low sensitivity, and the correct image is an image captured with high sensitivity or an image captured under low illumination that has been corrected with sensitivity correction to have a brightness similar to that of the correct image. Noise patterns and object structures (such as edges) that are not included in the learning process cannot be accurately inferred in the subsequent inference process. Therefore, in S601, the CPU 106 acquires multiple training images and their corresponding correct images so that various noise patterns and object structures are included. The amount of noise may be a single amount, or a mixture of multiple amounts of noise.

[0046] Next, in S602, the CPU 106 performs a process of extracting 8 bits from each pixel value of the training image and the ground truth image acquired in S601. The extraction of 8 bits is as shown in FIGS. 4(a) to 4(c), and is performed appropriately depending on the pattern to be learned. It is preferable to normalize the training image and the ground truth image by dividing them by the upper limit value of the signal (saturation brightness value). Here, taking into account that pixel values ​​are input in 8 bits during inference, the pixel values ​​are reduced to INT8 precision for learning, but learning can also be performed with FP32 precision.

[0047] Next, in S603, the CPU 106 selects at least one of the multiple training images generated in S602, and inputs the selected training image to the neural network of the image processing unit 104 to generate an output image. At this time, the amount of noise in the training image used in the learning process may be the same as or different from that of the other training images.

[0048] Here, the processing performed by the neural network 700 will be described with reference to Fig. 7. Fig. 7 is a diagram showing a neural network, and takes a convolutional neural network (CNN) as an example, but the present embodiment is not limited to this, and a GAN (Generative Adversarial Network) or the like may be used, a network may have a skip connection, or a recurrent network such as an RNN (Recurrent Neural Network) may be used.

[0049] In FIG. 7 , input image 701 represents an image to be input to the neural network or a feature map (described later). Convolution symbol 702 represents a convolution operation. Convolution matrix 703 is a filter that performs a convolution operation on input image 701. Bias 704 is added to the result output by the convolution operation between input image 701 and convolution matrix 703. Feature map 705 is the result of the convolution operation to which bias 704 has been added. Note that for simplicity, FIG. 7 depicts a small number of neurons, hidden layers, and channels. However, the number of neurons, layers, and the number and weights of connections between neurons are not limited to these. Furthermore, when implementing the system on an FPGA or the like, the number of connections and weights between neurons may be reduced. In this embodiment, the learning and inference processes are performed collectively for multiple color channels. However, the learning and inference processes may also be performed individually for each color.

[0050] In CNN, a feature map of an input image is obtained by performing a convolution operation on the input image with a certain filter. The size of the filter does not matter. In the next layer, a different feature map is obtained by performing a convolution operation on the feature map of the previous layer with a different filter. In each layer, an input signal is multiplied by a filter weight that represents the strength of the connection, and the result is summed with a bias. An activation function is applied to this result to obtain an output signal for each neuron. The weights and biases in each layer are called network parameters, and their values ​​are updated through a learning process. Examples of commonly used activation functions include the sigmoid function and the ReLU function. In this embodiment, the following formula (1) Leaky ReLU function is used, but this is not limited to this. f(x)=max(x,x×0.2) …(1) In formula (1), max(...) represents a function that outputs the maximum value among multiple arguments in the parentheses.

[0051] Next, in S604, the CPU 106 performs image processing on the neural network output image and the ground truth image, respectively. By matching the conditions of the image processing performed in the inference process with those of the image processing performed in the learning process, the inference accuracy of the noise reduction process during inference can be improved. Note that image processing may be performed at any time before S604 and S605. For example, it may be performed on the input side of the neural network. By applying multiple patterns of noise levels for training images used in the learning process, it is possible to effectively remove noise even if a captured image with a noise level outside the learning level is input during inference. If the number of training images is insufficient, padding processes such as cropping, rotation, and flipping may be performed. In this case, the ground truth image must also be subjected to the same processing.

[0052] Next, in S605, the CPU 106 calculates the error between the output image processed in S604 and the correct image. The correct image also has the same color component arrangement as the output image. The error is generally calculated using the mean square error of each pixel or the sum of the absolute values ​​of the differences between each pixel, but other indices may also be used.

[0053] In the next step S606, the CPU 106 updates each parameter using the backpropagation algorithm so as to reduce the error calculated in step S605. However, this embodiment is not limited to this. The amount of update for each parameter may be fixed or may be variable.

[0054] In S607, CPU 106 determines whether a predetermined termination condition has been met. If the condition has not been met, the process returns to S601 and learning is resumed. On the other hand, if CPU 106 determines that the predetermined termination condition has been met, the process proceeds to S608. The predetermined termination condition here may be that the number of learning attempts has reached a specified value, or that the error has fallen below a certain predetermined value. Alternatively, the process may be terminated when the error has almost completely stopped decreasing, or at the user's discretion.

[0055] Next, in S608, CPU 106 stores information related to the network parameters updated by learning, the structure of the neural network, and the like in storage device 230. Storage device 230 may be used to save the output network parameters. In this embodiment, the description is given on the assumption that the information is stored in a storage device, but other storage media may also be used.

[0056] In the next step S609, the CPU 106 determines whether network parameters have been acquired under all conditions. As mentioned above, by matching the conditions for the inference process and the learning process, the inference accuracy of the noise reduction process during inference can be improved. If image processing is performed under multiple conditions during inference (or the conditions are switched), it is desirable to have network parameters for each condition.

[0057] If CPU 106 determines in S609 that network parameters for all conditions have been acquired, it advances the process to S611. If CPU 106 determines that network parameters for all conditions have not been acquired, it advances the process to S610, changes the conditions, and then performs the flow from S601 again. Network parameters are stored in a parameter storage area for each condition. The parameter storage area may be ROM 105 or RAM 107. If necessary, the network parameters stored in the parameter storage area may be stored in storage device 230.

[0058] In S611, the CPU 106 quantizes the parameters of the neural network trained in FP32 to INT8. The bit width and data type are not limited to this, and FP16 parameters may be used, or quantization may be performed to INT4.

[0059] Finally, in S612, the CPU 106 stores the quantized network parameters in a parameter storage area.

[0060] As described above, by acquiring and storing optimal network parameters for each condition, it is possible to obtain a neural network whose inference accuracy is less affected by image processing that changes depending on the conditions.

[0061] For processes other than noise reduction, the learning process can be performed by similarly preparing pairs of training and reference images through simulation. For super-resolution, training images can be prepared by downsampling the reference image. The reference and training images may or may not be resized. For deblurring or deblurring, training images can be prepared by applying a blur function to the reference image. For white balance correction, training images can be prepared by comparing a reference image with a properly white-balanced image with an image that is not properly adjusted or corrected. The same applies to color correction such as color matrix correction. For missing image interpolation, training images can be obtained by removing missing elements from the reference image. For demosaicing, training images can be prepared by preparing a reference image using a three-chip image sensor and then resampling the reference image using a Bayer array or similar. For color component estimation, training images can be prepared by reducing the color components of the reference image. For dehazing, training images can be prepared by adding scattered light, simulated from a physical phenomenon, to a reference image without haze. In addition, when multiple frames are consecutive, such as in a moving image, noise can be removed more effectively by inputting a desired number of frames in the depth direction into the neural network.

[0062] [Second embodiment] Next, the inference processing performed by the image processing device 100 in the second embodiment according to the present invention will be described with reference to FIGS. 8A, 8B, and 9. FIG.

[0063] In the flowcharts of FIGS. 8A and 8B, the same processing steps as those in the first embodiment are assigned the same step numbers as those in FIG. 3, and the description thereof will be omitted.

[0064] S301 to S304 are the same as those in the first embodiment (FIG. 3). Next, in S805, the CPU 106 controls the image processing unit 104 to obtain an average value of pixel values ​​for each predetermined area. Here, the predetermined area means, for example, dividing the image into nine 3×3 areas as shown in FIG. 9, and obtaining an average value of pixel values ​​for each area. The area division is not limited to this, and the image may be divided into smaller or larger areas, and may be divided into shapes other than rectangular areas. Furthermore, when dividing the area, the areas may be divided so that they overlap. Furthermore, the value obtained for each area does not have to be an average value.

[0065] Next, in S806 (FIG. 8B), CPU 106 initializes a variable I indicating each pixel to 0. Next, in S807, CPU 106 determines whether all of the regions divided in S805 have been selected. If it is determined that selection of all regions has been completed, CPU 107 advances the process to S325. On the other hand, if it is determined that there is an unselected region, CPU 107 advances the process to S808.

[0066] In S808, CPU 106 controls image processing unit 104 to calculate the average value of the I-th region and determines whether the average value is 4096 or greater. If the average value of the I-th region is 4096 or greater, CPU 106 proceeds to S809. In S809, CPU 106 sets the parameters of the first neural network recorded in ROM 105 to the neural network in image processing unit 104. Next, in S810, CPU 106 controls image processing unit 104 to generate a fifteenth image by extracting 8 bits at the first position of each pixel value of the I-th region. Next, in S811, CPU 106 controls image processing unit 104 to input the fifteenth image to the neural network. Next, in S812, CPU 106 controls image processing unit 104 to generate a seventeenth image by shifting each pixel value of the sixteenth image output from the neural network to the left by 6 bits.

[0067] On the other hand, if the CPU 106 determines in S808 that the average value of the I-th region is smaller than 4096, the process proceeds to S813. In S813, the CPU 106 controls the image processing unit 104 to calculate the average value of the I-th region and determines whether the calculated value is 1024 or greater. If the CPU 106 determines that the average value of the I-th region is 1024 or greater, the process proceeds to S814. In S814, the CPU 106 sets the parameters of the second neural network recorded in the ROM 105 in the neural network in the image processing unit 104. Next, in S815, the CPU 106 controls the image processing unit 104 to generate an 18th image in which each pixel value of the I-th region is limited to 4095. Next, in S816, the CPU 106 controls the image processing unit 104 to generate a 19th image in which 8 bits at the second position of each pixel value of the I-th region are cut out. Next, in S817, CPU 106 controls image processing unit 104 to input the 19th image to the neural network. Then, in S818, CPU 106 controls image processing unit 104 to generate a 21st image by shifting each pixel value of the 20th image output from the neural network to the left by 4 bits.

[0068] On the other hand, if it is determined in S813 that the average value of the I-th region is smaller than 1024, the CPU 106 proceeds to S819. In S819, the CPU 106 sets the parameters of the third neural network recorded in ROM 105 to the neural network in the image processing unit 104. Next, in S820, the CPU 106 controls the image processing unit 104 to generate a 22nd image by limiting each pixel value of the I-th region to 1023. Then, in S821, the CPU 106 controls the image processing unit 104 to generate a 23rd image by cutting out 8 bits at the third position of each pixel value of the I-th region. Next, in S822, the CPU 106 controls the image processing unit 104 to input the 11th image to the neural network. Then, in S823, the CPU 106 controls the image processing unit 104 to generate a 25th image by shifting each pixel value of the 24th image output from the neural network left by 2 bits. The bit extraction and bit shifting described here are the same as the methods described with reference to FIGS.

[0069] In S824, the CPU 106 increments the variable I, returns the process to S807, and repeats the above process for all regions.

[0070] In the final step S825, the CPU 106 controls the image processing unit 104 to generate a 26th image by adding an offset to the image generated from the neural network output image for each region. In this embodiment, the limit values ​​are 4096 and 1024 as specific examples of thresholds for determining whether the limit values ​​are 4095 or 1023, but the limit values ​​are not limited to these. Any value or threshold may be used for the limit. Furthermore, in this embodiment, a digital gain is applied before input to the neural network, but a method of applying a digital gain after passing through the neural network is also possible. In this case, each neural network is prepared that has been optimally trained using images before digital gain is applied.

[0071] [Third embodiment] Next, the inference processing performed by the image processing device 100 in the third embodiment will be described with reference to the flowcharts of Figures 10A and 10B. In the flowcharts of Figures 10A and 10B, the same processing steps as in the first embodiment are assigned the same step numbers as in Figure 3, and their description will be omitted.

[0072] Steps S302 to S304 are the same as those in the first embodiment (FIG. 3). Next, in step S1004, the CPU 106 controls the image processing unit 104 to detect an area where a desired object (subject) is located. Object detection may be performed using deep learning, or a method using machine learning or a support vector machine. Next, in step S1005, the CPU 106 controls the image processing unit 104 to generate a 27th image by subtracting an offset from each pixel value of the area detected in step S1004. The offset here refers to the black level added by the image sensor 102. Next, in step S1006, the CPU 106 calculates the average value of each pixel value of the 27th image.

[0073] In S1007 (FIG. 10B), CPU 106 determines whether the average value of the 27th image is 4096 or greater. If it is determined that the average value of the 27th image is 4096 or greater, CPU 106 proceeds to S1008. In S1008, CPU 106 sets the parameters of the first neural network recorded in ROM 105 to the neural network in image processing unit 104. Then, in S1009, CPU 106 controls image processing unit 104 to generate a 28th image by extracting 8 bits at the first position of each pixel value of the 27th image. Next, in S1010, CPU 106 controls image processing unit 104 to input the 28th image to the neural network. Next, in S1011, CPU 106 controls image processing unit 104 to generate a 30th image by shifting each pixel value of the 29th image output from the neural network to the left by 6 bits.

[0074] On the other hand, if the CPU 106 determines in S1007 that the average value of the 27th image is smaller than 4096, the process proceeds to S1012. In S1012, the CPU 106 controls the image processing unit 104 to calculate the average value of the 27th image and determines whether the calculated value is 1024 or greater. If the CPU 106 determines that the average value of the 27th image is 1024 or greater, the process proceeds to S1013. In S1013, the CPU 106 sets the parameters of the second neural network recorded in ROM 105 in the neural network in the image processing unit 104. Then, in S1014, the CPU 106 controls the image processing unit 104 to generate a 31st image by limiting each pixel value of the 27th image to 4095. Next, in S1015, the CPU 106 controls the image processing unit 104 to generate a 32nd image by clipping 8 bits at a second position of each pixel value of the second image. Then, in S1016, CPU 106 controls image processing unit 104 to input the seventh image to the neural network. Then, in S1017, CPU 106 controls image processing unit 104 to generate a 34th image by shifting each pixel value of the 33rd image output from the neural network to the left by 4 bits.

[0075] On the other hand, if it is determined in S1012 that the average value of the 27th image is smaller than 1024, CPU 106 proceeds to S1018. In S1018, CPU 106 sets the parameters of the third neural network recorded in ROM 105 to the neural network in image processing unit 104. Then, in S1019, CPU 106 controls image processing unit 104 to generate a 35th image in which each pixel value of the 27th image is limited to 1023. Then, in S1020, CPU 106 controls image processing unit 104 to generate a 36th image in which 8 bits at the third position of each pixel value of the 35th image are cut out. Next, in S1021, CPU 106 controls image processing unit 104 to input the 36th image to the neural network. Then, in S1022, CPU 106 controls image processing unit 104 to generate a 38th image by shifting each pixel value of the 37th image output from the neural network to the left by 2 bits. The bit extraction and bit shifting described here are the same as the methods described using Figures 4 and 5.

[0076] Finally, in S1023, the CPU 106 controls the image processing unit 104 to generate a 39th image with the offset added. In this embodiment, the limit values ​​are 4096 and 1024 as specific examples of thresholds for determining 4095 and 1023, but the limit values ​​are not limited to these and any value or threshold may be used for the limit. Also, in this embodiment, digital gain is applied before input to the neural network, but a method of applying digital gain after passing through the neural network is also possible. In this case, each neural network is prepared that has been optimally trained on images before digital gain is applied.

[0077] [Fourth embodiment] The inference processing performed by the image processing device 100 in the fourth embodiment will be described with reference to Figures 11 to 14(c). In the flowchart in Figure 11, the same processing steps as in the first embodiment are assigned the same step numbers as in Figure 3, and their description will be omitted.

[0078] Steps S301 to S304 are the same as those in the first embodiment (FIG. 3). Next, in step S1105, the CPU 106 controls the image processing unit 104 to generate a 40th image by extracting the most significant bits from each pixel value of the first image to which the digital gain was applied in step S304. For example, if the bit depth of each pixel value is 16 bits, an image is generated by extracting the most significant 8 bits as shown in FIG. 12(a). If the bit depth of each pixel value is 14 bits, an image is generated by extracting the most significant 8 bits as shown in FIG. 12(b). Alternatively, the most significant 7 bits may be extracted as shown in FIG. 12(c), or the most significant 6 bits may be extracted as shown in FIG. 12(d). When the most significant 7 bits are extracted, the most significant bit is shifted one bit to the right so that the bit depth becomes 8 bits. Alternatively, the least significant bit is shifted one bit to the left. When the most significant 6 bits are extracted, the most significant bit is shifted two bits to the right so that the bit depth becomes 8 bits. Or, shift the least significant bit two bits to the left.

[0079] In S1106, the CPU 106 controls the image processing unit 104 to generate a 41st image by extracting the least significant bits from each pixel value of the image to which the digital gain was applied in S304. For example, if the bit depth of each pixel value is 16 bits, an image is generated by extracting the least significant 8 bits as shown in FIG. 13(a). If the bit depth of each pixel value is 14 bits, an image is generated by extracting the least significant 8 bits as shown in FIG. 13(b). Alternatively, the least significant 7 bits may be extracted as shown in FIG. 13(c), or the least significant 6 bits may be extracted as shown in FIG. 13(d). If the least significant 7 bits are extracted, the most significant bit is shifted one bit to the right so that the bit depth becomes 8 bits. Alternatively, the least significant bit is shifted one bit to the left. If the least significant 6 bits are extracted, the most significant bit is shifted two bits to the right so that the bit depth becomes 8 bits. Alternatively, the least significant bit is shifted two bits to the left.

[0080] In S1107, the CPU 106 controls the image processing unit 104 to input the 40th image and the 41st image to the neural network. The neural network here is a neural network that performs optimally trained noise removal by evaluating noisy training images divided into images extracted from the most significant bits and images extracted from the least significant bits with noiseless target images divided into images extracted from the most significant bits and images extracted from the least significant bits. In this embodiment, the 40th image and the 41st image are combined in the channel direction to form a multi-channel image and input to the neural network, but this is not limiting and the images may also be combined in the vertical and horizontal directions and input.

[0081] In S1108, the CPU 106 controls the image processing unit 104 to generate a 44th image by bit-combining the corresponding pixel values ​​of the 42nd image and the 43rd image output from the neural network. The bit-combining here refers to the process of combining the corresponding pixel values ​​of the 42nd image and the 43rd image output from the neural network, with the 43rd image being the most significant bits and the 43rd image being the least significant bits. The relationship between the images output from the neural network and the bit positions will be described later in the learning process. For example, as shown in FIG. 14(a), the pixel values ​​of the 42nd image are combined into 16 bits with the most significant 8 bits and the pixel values ​​of the 43rd image being the least significant 8 bits. To combine the pixel values ​​into 14 bits, the most significant or least significant 7 bits of the 8 bits of the pixel values ​​of the 42nd image may be combined into 14 bits with the most significant or least significant 7 bits of the 8 bits of the pixel values ​​of the 43rd image being the least significant bits, as shown in FIG. 14(b). Alternatively, the 8 bits of the pixel value of the 42nd image and the 6 most significant or least significant bits of the 8 bits of the pixel value of the 43rd image may be joined together, or the 6 most significant or least significant bits of the 8 bits of the pixel value of the 42nd image and the 8 bits of the pixel value of the 43rd image may be joined together. Furthermore, as shown in FIG. 14(c), the 8 bits of the pixel value of the 43rd image and the 8 bits of the pixel value of the 43rd image may be joined together to form 14 bits. In this case, the lowest 2 bits of the pixel value of the 42nd image and the highest 2 bits of the pixel value of the 43rd image are overlapped to form new 2 bits. This overlapping process may be performed by calculating an average value, or one of the bits may be selected.

[0082] In this embodiment, a digital gain is applied before inputting the image to the neural network, but a method of applying a digital gain after passing the image through the neural network is also possible. In this case, a neural network that has been optimally trained using images before applying the digital gain is prepared.

[0083] In this embodiment, the pixel values ​​of the image are treated as 16 bits or 14 bits, but this embodiment can be applied to pixel values ​​of 9 bits or more. In addition, the bit extraction shown in Figures 12(a) to 13(d) and the bit joining shown in Figures 14(a) to 14(c) can be changed depending on the number of bits of the pixel value, and any combination is acceptable.

[0084] Next, with reference to FIG. 15, a learning process performed by the imaging device 200, image processing system 210, display device 220, and storage device 230 in this embodiment will be described.

[0085] In the flowchart of FIG. 15, steps in which the same processing as in the first embodiment is performed are assigned the same step numbers as in FIG. 6, and the description thereof will be omitted.

[0086] S601 in Fig. 15 is the same as that in Fig. 6. In the next step S1502, the CPU 106 performs processing to extract the most significant bits from each pixel value of the training image acquired in S601. The most significant bits are extracted as shown in Fig. 12(a) to Fig. 12(d), and are selected appropriately according to the pattern to be learned.

[0087] In S1503, the CPU 106 performs a process of extracting lower bits from each pixel value of the training image acquired in S601. The extraction of lower bits is as shown in Figures 13(a) to 13(d), and is appropriately selected according to the pattern to be learned.

[0088] Next, in S1504, the CPU 106 performs a process of extracting the most significant bits from each pixel value of the correct image acquired in S601. The most significant bits are extracted as shown in Figures 12(a) to 12(d), and are selected appropriately according to the pattern to be learned.

[0089] In S1505, the CPU 106 performs a process of extracting lower bits from each pixel value of the correct image acquired in S601. The extraction of lower bits is as shown in Figures 13(a) to 13(d), and is selected appropriately according to the pattern to be learned.

[0090] It is preferable to normalize the training images and the ground truth images by dividing them by the upper limit of the signal (saturation brightness value). Here, considering that pixel values ​​are input in 8 bits during inference, the pixel values ​​are reduced to INT8 precision for learning, but learning can also be performed with FP32 precision.

[0091] In S1506, the CPU 106 combines the training image generated in S1502 and the training image generated in S1503 in the channel direction to create a multi-channel image, and inputs the resulting image to the neural network of the image processing unit 104 to generate an output image. At this time, the amount of noise in the training image used in the learning process may be the same as that in the other training images, or may be different. The output image here uses the image generated in S1504 with the most significant bits extracted and the image generated in S1505 with the least significant bits extracted as the correct answer image, so two images are output: one extracted as the most significant bits and the other extracted as the least significant bits. The subsequent steps S604 to S612 are the same as those in FIG. 6.

[0092] As described above, by acquiring and storing optimal network parameters for each condition, it is possible to obtain a neural network whose inference accuracy is less affected by image processing that changes depending on the conditions.

[0093] For processes other than noise reduction, the learning process can be performed by similarly preparing pairs of training and reference images through simulation. For super-resolution, training images can be prepared by downsampling the reference image. The reference and training images may or may not be resized. For deblurring or deblurring, training images can be prepared by applying a blur function to the reference image. For white balance correction, training images can be prepared by comparing a reference image with a properly white-balanced image with an image that is not properly adjusted or corrected. The same applies to color correction such as color matrix correction. For missing image interpolation, training images can be obtained by removing missing elements from the reference image. For demosaicing, training images can be prepared by preparing a reference image using a three-chip image sensor and then resampling the reference image using a Bayer array or similar. For color component estimation, training images can be prepared by reducing the color components of the reference image. For dehazing, training images can be prepared by adding scattered light, simulated from a physical phenomenon, to a reference image without haze. In addition, when multiple frames are consecutive, such as in a moving image, noise can be removed more effectively by inputting a desired number of frames in the depth direction into the neural network.

[0094] [Fifth embodiment] Next, the inference processing performed by the image processing device 100 of the fifth embodiment will be described with reference to the flowchart of Fig. 16. In the flowchart of Fig. 16, steps that perform the same processing as in Fig. 3 are given the same step numbers, and their descriptions will be omitted. Also, in Fig. 16, steps that perform the same processing as in Fig. 11 are given the same step numbers, and their descriptions will be omitted. S301 to S304 in Fig. 16 are the same as those in Fig. 3. Also, S1105 to S1106 in Fig. 16 are the same as those in Fig. 11.

[0095] In S1607, the CPU 106 controls the image processing unit 104 to input the 40th image and the 41st image to the neural network. The neural network here is a neural network that optimally learns to perform noise reduction by evaluating a noisy training image, which is divided into an image extracted from the most significant bits and an image extracted from the least significant bits, with a noiseless target image that has been subjected to gradation compression processing after subtracting an offset. The offset here refers to the black level added by the image sensor 102. The gradation compression processing here can be performed using various methods, such as gamma correction.

[0096] In the next step S1608, the CPU 106 controls the image processing unit 104 to generate a 46th image by performing gradation expansion processing on each pixel value of the 45th image output from the neural network. The gradation expansion processing here can be performed using various methods, such as de-gamma correction.

[0097] In S1609, the CPU 106 controls the image processing unit 104 to generate a 47th image by adding an offset to each pixel value of the fifth image.

[0098] In this embodiment, a digital gain is applied before inputting the image to the neural network, but a method of applying a digital gain after passing the image through the neural network is also possible. In this case, a neural network that has been optimally trained using images before applying the digital gain is prepared.

[0099] In this embodiment, the pixel values ​​of the image are treated as 16 bits or 14 bits, but this embodiment can be applied to pixel values ​​of 9 bits or more. In addition, the bit extraction shown in Figures 12(a) to 13(d) and the bit joining shown in Figures 14(a) to 14(c) can be changed depending on the number of bits of the pixel value, and any combination is acceptable. Next, the learning process performed by the imaging device 200, image processing system 210, display device 220, and storage device 230 in this embodiment will be described with reference to the flowchart in FIG.

[0100] In the flowchart of Fig. 17, steps that perform the same processing as in the flowchart of Fig. 6 are assigned the same step numbers as in Fig. 6, and their explanations will be omitted. Also, in the flowchart of Fig. 17, steps that perform the same processing as in the flowchart of Fig. 15 are assigned the same step numbers as in Fig. 15, and their explanations will be omitted.

[0101] S601 has the same step number as in Fig. 6. S1502 and S1503 have the same step numbers as in Fig. 15.

[0102] In S1704, the CPU 106 performs a process of subtracting an offset from each pixel value of the correct answer image acquired in S601. In S1705, the CPU 106 performs a gradation compression process on each pixel value of the correct answer image generated in S1704.

[0103] It is preferable to normalize the training images and the ground truth images by dividing them by the upper limit of the signal (saturation brightness value). Here, considering that pixel values ​​are input in 8 bits during inference, the pixel values ​​are reduced to INT8 precision for learning, but learning can also be performed with FP32 precision.

[0104] In S1706, the CPU 106 combines the training image generated in S1502 and the training image generated in S1503 in the channel direction to create a multi-channel image, and inputs the resulting image to the neural network of the image processing unit 104 to generate an output image. At this time, the amount of noise in the training image used in the learning process may be the same as that of the other training images, or may be different. The output image here uses the image that has been subjected to gradation compression processing generated in S1705 as the correct image, and therefore an image in which gradation compression processing has been applied to each pixel value is output. The following steps S604 to S612 are the same as the processing of the same step numbers in Figure 6.

[0105] As described above, by acquiring and storing optimal network parameters for each condition, it is possible to obtain a neural network whose inference accuracy is less affected by image processing that changes depending on the conditions.

[0106] For processes other than noise reduction, the learning process can be performed by similarly preparing pairs of training and reference images through simulation. For super-resolution, training images can be prepared by downsampling the reference image. The reference and training images may or may not be resized. For deblurring or deblurring, training images can be prepared by applying a blur function to the reference image. For white balance correction, training images can be prepared by comparing a reference image with a properly white-balanced image with an image that is not properly adjusted or corrected. The same applies to color correction such as color matrix correction. For missing image interpolation, training images can be obtained by removing missing elements from the reference image. For demosaicing, training images can be prepared by preparing a reference image using a three-chip image sensor and then resampling the reference image using a Bayer array or similar. For color component estimation, training images can be prepared by reducing the color components of the reference image. For dehazing, training images can be prepared by adding scattered light, simulated from a physical phenomenon, to a reference image without haze. In addition, when multiple frames are consecutive, such as in a moving image, noise can be removed more effectively by inputting a desired number of frames in the depth direction into the neural network.

[0107] [Sixth embodiment] The inference processing performed by the image processing device 100 in the sixth embodiment will be described with reference to the flowchart in FIG.

[0108] In the flowchart of FIG. 18, steps that perform the same processing as in the flowchart of FIG. 3 are assigned the same step numbers as in FIG. 3, and their explanations will be omitted.

[0109] Steps S301 to S304 are the same as those in Fig. 3. In step S1805, the CPU 106 controls the image processing unit 104 to generate a 48th image by subtracting an offset from each pixel value of the image to which the digital gain was applied in step S304. The offset here refers to the black level added by the image sensor 102.

[0110] In S1806, CPU 106 controls image processing unit 104 to generate a 49th image by performing gradation compression processing on each pixel value of the 48th image. The gradation compression processing here can be performed using various methods, such as gamma correction.

[0111] In the next step S1807, the CPU 106 controls the image processing unit 104 to input the 49th image into the neural network. The neural network here is a neural network that performs optimally trained noise removal by evaluating a noisy training image that has been offset and gamma-processed with a noise-free target image that has been divided into an image extracted from the upper bits and an image extracted from the lower bits.

[0112] Then, in S1808, the CPU 106 controls the image processing unit 104 to generate a 52nd image by bit-combining corresponding pixel values ​​of the 50th image and the 51st image output from the neural network. The bit-combining here is the same as the method described with reference to FIG. 14.

[0113] In this embodiment, a digital gain is applied before inputting the image to the neural network, but a method of applying a digital gain after passing the image through the neural network is also possible. In this case, a neural network that has been optimally trained using images before applying the digital gain is prepared.

[0114] In this embodiment, the pixel values ​​of the image are treated as 16 bits or 14 bits, but this embodiment can be applied to pixel values ​​of 9 bits or more. In addition, the bit extraction shown in Figures 12(a) to 13(d) and the bit joining shown in Figures 14(a) to 14(c) can be changed depending on the number of bits of the pixel value, and any combination is acceptable.

[0115] Next, the learning process performed by the imaging device 200, image processing system 210, display device 220, and storage device 230 in this embodiment will be described with reference to the flowchart in FIG.

[0116] In the flowchart of Fig. 19, steps that perform the same processing as in the flowchart of Fig. 6 are assigned the same step numbers as in Fig. 6, and descriptions thereof will be omitted. Also, in the flowchart of Fig. 19, steps that perform the same processing as in the flowchart of Fig. 15 are assigned the same step numbers as in Fig. 15, and descriptions thereof will be omitted.

[0117] S601 in Fig. 19 is the same as that in Fig. 6. In the following S1902, the CPU 106 performs a process of subtracting an offset from each pixel value of the training image acquired in S601. Then, in S1903, the CPU 106 performs a gradation expansion process on each pixel value of the training image generated in S1902.

[0118] It is preferable to normalize the training images and the ground truth images by dividing them by the upper limit of the signal (saturation brightness value). Here, considering that pixel values ​​are input in 8 bits during inference, the pixel values ​​are reduced to INT8 precision for learning, but learning can also be performed with FP32 precision.

[0119] Steps S1504 to S1505 are the same as the processes of the same step numbers in FIG. 15. At S1906, the CPU 106 inputs the training image generated at S1903 to the neural network of the image processing unit 104 to generate an output image. At this time, the amount of noise of the training image used in the learning process may be the same as that of the other training images, or may be different. The output image here uses the image generated at S1504 from which the upper bits are extracted and the image generated at S1505 from which the lower bits are extracted as the correct answer image, so two images are output: one extracted as the upper bits and the other extracted as the lower bits. Steps S604 to S612 are the same as the processes of the same step numbers in FIG. 6.

[0120] As described above, by acquiring and storing optimal network parameters for each condition, it is possible to obtain a neural network whose inference accuracy is less affected by image processing that changes depending on the conditions.

[0121] For processes other than noise reduction, the learning process can be performed by similarly preparing pairs of training and reference images through simulation. For super-resolution, training images can be prepared by downsampling the reference image. The reference and training images may or may not be resized. For deblurring or deblurring, training images can be prepared by applying a blur function to the reference image. For white balance correction, training images can be prepared by comparing a reference image with a properly white-balanced image with an image that is not properly adjusted or corrected. The same applies to color correction such as color matrix correction. For missing image interpolation, training images can be obtained by removing missing elements from the reference image. For demosaicing, training images can be prepared by preparing a reference image using a three-chip image sensor and then resampling the reference image using a Bayer array or similar. For color component estimation, training images can be prepared by reducing the color components of the reference image. For dehazing, training images can be prepared by adding scattered light, simulated from a physical phenomenon, to a reference image without haze. In addition, when multiple frames are consecutive, such as in a moving image, noise can be removed more effectively by inputting a desired number of frames in the depth direction into the neural network.

[0122] Although the present invention has been described in detail above based on preferred embodiments thereof, the present invention is not limited to these specific embodiments, and various forms within the scope of the gist of the present invention are also included in the present invention. Parts of the above-described embodiments may be combined as appropriate.

[0123] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0124] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0125] 100...image processing device, 200...imaging device, 210...image processing system, 220...display device, 230...storage device, 101...optical lens, 102...imaging element, 103...frame memory, 104...image processing unit, 106...CPU, 108...video output driving unit, 110...display driving unit

Claims

1. a first generating means for receiving an input image having a bit depth of N bits and extracting M bits at different positions from the N bits to generate a plurality of images having a bit depth of M bits per pixel; a second generating means having a neural network for images of M bit depth, which receives the plurality of images generated by the first generating means and generates a plurality of output images corresponding to the plurality of images; a third generating means for generating an image of the N-bit depth from the plurality of output images generated by the second generating means; 1. An image processing device comprising:

2. The neural network includes a plurality of neural networks optimally trained for each bit extraction position.

2. The image processing device according to claim 1, wherein:

3. The third generating means generates the image by selecting, for each pixel, which neural network output image to use, using an image output from the neural network through an image cut out from the most significant bit position.

3. The image processing device according to claim 1, wherein the image processing device is a computer.

4. means for dividing the input image into a plurality of regions and calculating the brightness of each region; a means for selecting, for each region, a bit position at which the image is to be extracted in accordance with the brightness calculated for each region; and means for selecting, for each region, which neural network to use in accordance with the brightness calculated for each region.

4. The image processing device according to claim 1, wherein the image processing device is a computer.

5. means for detecting a subject from the input image; means for calculating the brightness of an area including the subject; a means for selecting, for each region, at which bit position to extract the image in accordance with the calculated brightness; The method further comprises means for selecting which neural network to use in accordance with the calculated brightness.

5. The image processing device according to claim 1, wherein the image processing device is a computer.

6. The first generating means generates an M-bit depth image by limiting the N-bit depth image with a maximum value represented by the most significant bit position and the least significant bit 0 when extracting the M bits from the N-bit depth image, and then extracting the M bits.

6. The image processing device according to claim 1, wherein the image processing device is a computer.

7. The neural network is optimally trained by inputting a plurality of M-bit depth images generated by the first generation means in a manner that expands them in one dimension, and outputs the expanded images in one dimension.

7. The image processing device according to claim 1, wherein the image processing device is a computer.

8. The third generating means expands and combines a plurality of images generated from the neural network in one dimension.

8. The image processing device according to claim 1, wherein the image processing device is a computer.

9. a first generation step of inputting an input image having a bit depth of N bits and extracting M bits from different positions in the N bits to generate a plurality of images having a bit depth of M bits per pixel; a second generation step having a neural network for images with an M-bit depth, inputting the plurality of images generated in the first generation step and generating a plurality of output images corresponding to the plurality of images; a third generation step of generating the N-bit depth image from the plurality of output images generated in the second generation step; An image processing method comprising:

10. A program that, when read and executed by a computer, causes the computer to execute each step of the method of claim 9.

Images