Information processing device, information processing method, and program

Abstract

This technology provides the ability to estimate images using a correction map generated by a low-bit-depth neural network, even if the data structures of the input and output images are different. [Solution] The information processing device includes: an image acquisition means for acquiring a first image with a first bit depth; an image conversion means for performing image processing on the first image to convert it into a second image; an image quantization means for converting the first image or the second image into a third image with a second bit depth lower than the first bit depth; a correction map estimation means for estimating a correction map of correction bit depths to correct the second image based on the third image; and an image correction means for correcting the second image based on the correction map to generate a corrected image.

Description

[Technical Field]

[0001] This invention relates to image processing technology. [Background technology]

[0002] In recent years, various methods using neural networks (NN) have been developed for image quality enhancement processing. Here, image quality enhancement processing refers to image processing such as noise reduction, aberration correction, demosaicing, and super-resolution processing.

[0003] Modern neural networks (NNs) are becoming increasingly large-scale, not just for high-resolution image processing, and higher performance tends to result in greater computational requirements. To enable these high-performance NNs to run on devices with limited computing resources, such as embedded systems, or to speed up processing on general-purpose computers, there is a great deal of research into lightweighting techniques that reduce the size and computational load of NNs while maintaining performance as much as possible.

[0004] One known method for reducing the size of neural networks (NNs) is to quantize their weights and features to a low bit depth. By quantizing an NN, the computational complexity and size can be reduced while maintaining its structure, making it possible to run it on equipment with limited computing resources. Furthermore, even on general-purpose computers, quantization can enable the use of high-throughput arithmetic instructions, potentially leading to faster processing speeds.

[0005] However, when attempting to estimate a high-quality image using a neural network (NN) with a lower bit depth than the desired output image, the output image will have fewer gradations than originally intended, resulting in lower image quality performance compared to an NN with a bit depth greater than or equal to the desired output image.

[0006] Patent Document 1 proposes a method for denoising 14-bit images by inferring the noise component of the image using an 8-bit neural network (NN), and then obtaining a denoised image by subtracting the noise component, which is the difference image between the input image and the denoised image, from the 14-bit input image. Although the image inferred by the NN is 8 bits, the final image maintains 14 bits of grayscale by subtracting the noise component from the 14-bit input image. On the other hand, because the NN is 8 bits, it is possible to process it at high speed.

[0007] Non-patent document 1 discloses fake quantization learning. Non-patent document 2 discloses a non-uniform quantization method. [Prior art documents] [Patent Documents]

[0008] [Patent Document 1] Japanese Patent Publication No. 2024-65787 [Non-patent literature]

[0009] [Non-Patent Document 1] Jacob et al., "Quantization and Training of Neural Networks for Efficient Integer-Arithmetic-Only Inference", CVPR2018 [Non-Patent Document 2] Yamamoto et al., "Learnable Companding Quantization for Accurate Low-bit Neural Network", CVPR2021 [Overview of the project] [Problems that the invention aims to solve]

[0010] However, when the data structures of the input image and the output image are different, such as in the case of demosaicking processing that converts a Bayer image (1 channel) into an RGB image (3 channels), it is impossible to generate a correction map with a low-bit-depth NN and estimate the image.

[0011] Therefore, the present invention provides a technique that can estimate an image using a correction map generated by a low-bit-depth NN even when the data structures of the input image and the output image are different.

Means for Solving the Problems

[0012] To solve this problem, for example, the information processing apparatus of the present invention has the following configuration. That is, image acquisition means for acquiring a first image with a first bit depth, image conversion means for performing image processing on the first image and converting it into a second image, image quantization means for converting the first image or the second image into a third image with a second bit depth lower than the first bit depth, correction map estimation means for estimating a correction map with a correction bit depth for correcting the second image based on the third image, image correction means for correcting the second image based on the correction map and generating a corrected image, and is provided with.

Effects of the Invention

[0013] According to the present invention, it is possible to provide a technique for estimating a high-quality image with a NN having a low bit depth.

Brief Description of the Drawings

[0014] [Figure 1] A diagram showing the hardware configuration of the information processing apparatus in the first embodiment. [Figure 2] A block diagram showing the functional configuration of the information processing apparatus during inference and learning in the first embodiment. [Figure 3] A diagram for explaining the differential estimation NN in the first embodiment. [Figure 4]A diagram illustrating the nonlinear transformation in the bit depth transformation process in the first embodiment. [Figure 5] Flowchart of the inference process in the first embodiment. [Figure 6] Flowchart of the learning process of the neural network in the first embodiment. [Figure 7] A diagram illustrating the piecewise linear function used in the bit depth conversion process in Modification Example 1. [Figure 8] Flowchart of the image quantization process in modified example 3. [Figure 9] A diagram showing the functional configuration of the information processing device during inference and learning in modified example 4. [Figure 10] A flowchart of the inference process performed by the information processing device in the modified example 4. [Figure 11] Flowchart of the inference process in modified example 5. [Figure 12] Flowchart of the inference process in modified example 6. [Modes for carrying out the invention]

[0015] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0016] (First Embodiment) As a first embodiment, an information processing device that performs image quality enhancement processing using a neural network (NN) will be described below as an example.

[0017] <Overview> This embodiment relates to a process for estimating high-resolution images using machine learning. In the first embodiment, demosaicing is used as a specific example.

[0018] This embodiment describes the inference process of a neural network (NN) for demosaicing images and the method for training the NN. The bit depth of the image to be processed is 14 bits, and the bit depth of the NN weights and the intermediate features (hereinafter referred to as "NN bit depth") is 8 bits. However, the bit depth of the image to be processed only needs to be higher than the bit depth of the NN, and the bit depth is not limited to these values.

[0019] The reason for the effectiveness of this embodiment will be briefly explained using an example of the demosaicing process performed in this embodiment. For example, if we want to convert a 14-bit Bayer image to an RGB image, if we estimate the RGB image itself with a neural network (NN) that has 8-bit weights and intermediate feature bit depths, the NN will output an 8-bit RGB image, making accurate estimation difficult. Therefore, we consider an NN that estimates a correction map to correct an RGB image that has undergone lightweight demosaicing. The lightweight demosaicing process may be a rule-based process or a process performed by machine learning. From now on, we will refer to the RGB image that has undergone lightweight demosaicing as a simplified RGB image. The simplified RGB image, although simplified, outputs an image that is close to the result of the high-performance demosaicing process that we want to achieve with the NN. The NN infers a map to correct the simplified RGB image, and the correction of the simplified RGB image can obtain a sufficiently high-quality image with 8 bits of information. Also, the larger the ratio between the pixel values ​​of the simplified RGB image and the corresponding pixel values ​​of the correction map, the greater the improvement in image quality. Furthermore, since the image quality of the simplified RGB image is already at a certain level, there is basically no significant deviation from the simplified RGB image. Therefore, the ratio between the pixel values ​​of the simplified RGB image and the corresponding pixel values ​​of the correction map will inevitably be larger. By converting to 8 bits with finer gradations, a higher quality RGB image can be obtained.

[0020] In the following sections, we will first describe the hardware configuration of the information processing device. Then, we will describe the functional configuration and operation of the inference and learning processes, respectively.

[0021] <Hardware Configuration> Figure 1 shows the hardware configuration of the information processing device 100 in the first embodiment. Note that the inference processing and learning processing may be performed on the same information processing device or on different information processing devices. The information processing device 100 includes a CPU 101, ROM 102, RAM 103, storage unit 104, input unit 105, display unit 106, communication unit 107, and bus 108. The CPU 101, ROM 102, RAM 103, storage unit 104, input unit 105, display unit 106, and communication unit 107 are connected to each other via the bus 108 so that they can send and receive data.

[0022] CPU101 stands for Central Processing Unit and is a processor. The CPU101 reads computer programs (hereinafter also referred to as programs), such as control programs, stored in either ROM102 or memory unit104, loads them into RAM103, and executes them, thereby controlling the entire information processing device 100, realizing all or part of the functions described later, and executing all or part of the processing.

[0023] The information processing device 100 may have other processors such as an MPU (Micro Processing Unit), GPU (Graphics Processing Unit), NPU (Neural Processing Unit), and QPU (Quantum Processing Unit) in place of or in addition to the CPU 101. Furthermore, the information processing device 100 may have multiple processors of the same type, each performing a different function.

[0024] Furthermore, some or all of the functions of the information processing device 100 may be implemented by one or more circuits, such as an ASIC (Application Specific Integrated Circuit) and a PLD (Programmable Logic Device) including an FPGA (Field Programmable Gate Array).

[0025] ROM102 stands for Read Only Memory and is a non-volatile type of memory. ROM102 stores programs such as the BIOS (Basic Input Output System).

[0026] RAM103 stands for Random Access Memory and is a memory that allows for high-speed data reading and writing. RAM103 temporarily stores various data from its components. Furthermore, RAM103 also functions as a work area when the CPU101 executes a program. In this case, programs such as control programs are loaded into RAM103 in a state that can be executed by the CPU101.

[0027] The memory unit 104 stores the program executed by the CPU 101, various data that the program in this embodiment is to process, and data necessary when the program is executed. For example, the memory unit 104 stores images that are the target of inference processing (demosaiking processing), images used in learning processing, and various parameters. The storage unit 104 can use non-volatile storage devices such as HDDs (Hard Disk Drives), SSDs (Solid State Drives), flash memory, and various optical media as its medium.

[0028] The input unit 105 receives input such as instructions and information from the user and outputs it to the CPU 101. The input unit 105 may receive input from the user via input devices such as a keyboard and a mouse.

[0029] The display unit 106 receives display screen information from the CPU 101 and displays the display screen on the display device. The display unit 106 may display the display screen on a display device such as a liquid crystal display or an organic EL (Electro-Luminescence) display.

[0030] The communication unit 107 may be an interface for sending and receiving data with an external device. The communication unit 107 is connected to an external device via a network such as a LAN (Local Area Network) or a WAN (Wide Area Network).

[0031] <Functional configuration during inference processing> Figure 2 is a block diagram showing the functional configuration of the information processing device 100 during inference and learning. Figure 2(a) is a block diagram showing the functional configuration of the information processing device 100 during inference. The information processing device 100 includes a storage unit 201, an image acquisition unit 202, an image conversion unit 203, an image quantization unit 204, a correction map estimation unit 205, and an image correction unit 206. Each functional component will be briefly described below. The CPU 101 may implement all or part of the image acquisition unit 202, image conversion unit 203, image quantization unit 204, correction map estimation unit 205, and image correction unit 206 by reading and executing an inference program.

[0032] The image acquisition unit 202 acquires a high-bit (14-bit in this case) Bayer image to be demosaicing from the storage unit 201 as an input image. In this embodiment, the Bayer image has an RGGB Bayer array, but the array and other elements are not limited to this.

[0033] The image conversion unit 203 obtains a simplified RGB image with the data structure of the output image from a high-bit (14-bit in this case) Bayer image by performing demosaicing. The demosaicing performed here may be a lightweight rule-based process. For example, the image conversion unit 203 may perform linear interpolation on each of the R, G, and B pixels in the demosaicing process. The demosaicing process does not need to be rule-based as long as it is lightweight; it may be a process acquired through learning, such as in machine learning. The simplified RGB image does not reach the image quality that the neural network would ideally want to infer, but it is close to it.

[0034] The image quantization unit 204 performs quantization on the high-bit Bayer image with a bit depth of 14 bits obtained from the image acquisition unit 202, converting it into a low-bit Bayer image of an unsigned 8-bit integer. The image quantization unit 204 converts to a low-bit Bayer image using the same uniform quantization method as the bit depth conversion layer 306 described later. Here, the bit depth of the NN and the bit depth of the low-bit depth image are matched to 8 bits, but they may be different. The bit depth of the NN should be lower than the bit depth of the input image and greater than or equal to the bit depth of the low-bit Bayer image.

[0035] The correction map estimation unit 205 inputs the 8-bit Bayer image 301 obtained from the image quantization unit 204 to an 8-bit difference estimation NN and estimates a correction map with 8-bit grayscale and a value range of signed 15-bit integers (correction bit depth). The correction map in this embodiment may be a map corresponding to the difference between an RGB image of the image quality to be inferred by the difference estimation NN and a simplified processed RGB image. Figure 3 is a diagram illustrating the difference estimation NN. Details of the processing of the difference estimation NN will be described later using Figure 3.

[0036] The image correction unit 206 derives a higher-quality 14-bit RGB image by subtracting (for example, subtracting) a correction map estimated by the correction map estimation unit 205, which has 8-bit grayscale and a value range of signed 15-bit integers, from the simplified RGB image obtained from the image conversion unit 203. The high-quality 14-bit RGB image is an example of a corrected image. The image correction unit 206 may also add the correction map to the simplified RGB image. In this case, the correction map estimation unit 205 generates a correction map corresponding to the addition.

[0037] Figure 3(a) illustrates the structure of the 8-bit difference estimation NN in the correction map estimation unit 205. The difference estimation NN is also called the correction map estimation NN. In the following description, the difference estimation NN may be referred to simply as NN. The difference estimation NN has multiple intermediate layers 302 and a final layer 303.

[0038] The hidden layer 302 exists, for example, from the first hidden layer 302-1 to the nth hidden layer 302-n. When there is no need to distinguish between multiple hidden layers, they are simply referred to as hidden layer 302. Each hidden layer is a neural network (NN) that holds signed 8-bit integer weights and outputs unsigned 8-bit integers.

[0039] The final layer 303 is located after the last nth intermediate layer 302-n. The final layer 303 is a neural network (NN) that has signed 8-bit integer weights and outputs signed 15-bit integers with an 8-bit grayscale range.

[0040] In a difference estimation neural network having an intermediate layer 302 and a final layer 303, the first intermediate layer 302-1 acquires an unsigned 8-bit Bayer image 301, and the final layer 303 outputs a correction map estimate 309 with 8-bit grayscale and a signed 15-bit integer range. Here, the number of intermediate layers can be any number and is not particularly limited.

[0041] The internal structure of the first intermediate layer 302-1 through the nth intermediate layer 302-n is common, and the internal structure of each intermediate layer will be explained using the first intermediate layer 302-1 as a representative example.

[0042] The intermediate layer 302-1 includes a convolutional layer 304-1, a ReLU layer 305-1, and a bit depth conversion layer 306-1.

[0043] The convolutional layer 304-1 performs a convolution operation with signed 8-bit integer weights. In the convolutional operation, the convolutional layer 304-1 multiplies the signed 8-bit integer weights (including biases) with the unsigned 8-bit integer Bayer image 301 and outputs a signed 16-bit integer result.

[0044] The ReLU layer 305-1 performs a nonlinear transformation called ReLU (Rectified Linear Unit). ReLU outputs values ​​less than or equal to 0 as 0. Therefore, the ReLU layer 305-1 converts the intermediate features of the input signed 16-bit integer into an unsigned 15-bit integer.

[0045] The bit depth conversion layer 306-1 converts the unsigned 15-bit integer data converted by the ReLU layer 305-1 into an unsigned 8-bit integer. In this embodiment, the bit depth conversion is performed by uniformly quantizing 15 bits to 8 bits, but a non-uniform quantization method, such as that described in Non-Patent Document 2, may also be used. Details of the processing in the bit depth conversion layer 306-1 will be described later with reference to Figure 3(b).

[0046] Next, the configuration of the final layer 303 will be described. The final layer 303 includes a convolutional layer 307 and a final bit depth conversion layer 308.

[0047] The convolutional layer 307, like the convolutional layer 304, performs a convolution operation with signed 8-bit integer weights and outputs a corrected map of signed 16-bit integers.

[0048] The final bit depth conversion layer 308 converts a correction map of signed 16-bit integers into a correction map with 8-bit grayscale and a value range of signed 15-bit integers. The final bit depth conversion layer 308 may use a non-uniform quantization method, such as that described in Non-Patent Literature 2, to convert from 16-bit grayscale to 8-bit grayscale. Here, non-uniform quantization is a method that reduces quantization errors by devising the grayscale representation during decimation and finely representing the range of input that is effective for accuracy, and is a method that can be expected to improve the accuracy of the quantization neural network. The final bit depth conversion layer 308 devises a non-uniform quantization method to accurately quantize the correction map that is effective for improving image quality. Details of the processing in the final bit depth conversion layer 308 will be described later with reference to Figure 3(c).

[0049] The NN structure is not limited to that shown in Figure 3(a), and other structures such as U-Net may be used. The convolutional layers 304, 307, and ReLU layer 305 are also not limited to these, and other linear and nonlinear transformations may be used. Furthermore, the type and number of layers in the hidden layer 302 are not limited and do not have to be the same as the final layer 303. In addition, the bit depth of the Bayer image 301 may be greater than 8 bits.

[0050] <Operation during inference processing> Figure 5 is a flowchart of the inference process performed by the information processing device 100. However, the information processing device 100 does not necessarily have to perform all the steps described in this flowchart. When the inference process is started, the storage unit 201 stores the image to be processed, for example, a high-bit Bayer image.

[0051] In S501, the image acquisition unit 202 acquires a high-bit Bayer image from the storage unit 201 as the input image to be subjected to demosaicing. Here, the high-bit Bayer image may be an unsigned 14-bit integer Bayer image.

[0052] In S502, the image conversion unit 203 performs demosaicing as image processing to convert the unsigned 14-bit integer Bayer image acquired in S501 and obtain a high-bit simplified RGB image. The high-bit simplified RGB image may be an image of an unsigned 14-bit integer.

[0053] In S503, the image quantization unit 204 converts the unsigned 14-bit integer Bayer image acquired in S501 into an unsigned 8-bit integer Bayer image 301 through quantization processing.

[0054] In S504, the correction map estimation unit 205 estimates a correction map by calculating an estimated value in the form of a signed 15-bit integer with an 8-bit grayscale range from the unsigned 8-bit integer Bayer image 301 obtained in S503.

[0055] More specifically, the correction map estimation unit 205 inputs the unsigned 8-bit integer Bayer image 301 obtained in S502 to the difference estimation NN in Figure 3(a), and sequentially processes each intermediate layer 302 and the final layer 303. As a result, the correction map estimation unit 205 outputs a correction map estimate 309 with an 8-bit grayscale and a value range of signed 15-bit integers. Here, we will explain the case where the biases of the convolutional layers 304 and 307 are "0" and the weights are represented as "signed 8-bit integers".

[0056] First, the convolutional layer 304 performs a convolution operation using signed 8-bit integer weights and an unsigned 8-bit integer Bayer image 301, resulting in the output of a signed 16-bit integer intermediate feature. The ReLU layer 305 converts negative values ​​to "0" and outputs positive values ​​as they are, resulting in an output represented as an unsigned 15-bit integer. The bit depth conversion layer 306 converts the unsigned 15-bit integer obtained by the ReLU layer 305 into an unsigned 8-bit integer.

[0057] The convolutional layer 307 performs a convolution operation using the weights of a signed 8-bit integer and the intermediate features of an unsigned 8-bit integer, resulting in a corrected map of a signed 16-bit integer as its output. The final bit depth conversion layer 308 converts the signed 16-bit integer obtained in the convolutional layer 307 into a corrected map of a signed 15-bit integer with an 8-bit grayscale range.

[0058] Figure 3(b) is a flowchart illustrating the processing in the bit depth conversion layer 306. This process converts an unsigned 15-bit integer input to an unsigned 8-bit integer.

[0059] In S311, the bit depth conversion layer 306 normalizes the unsigned 15-bit integer output by the ReLU layer 305. Specifically, the bit depth conversion layer 306 performs the process shown in equation (1) on the intermediate feature x output by the ReLU layer 305.

number

[0060] Here, β is 2 15 Set to -1. The output of this process will be a 15-bit grayscale real value with a value range of [0,1]. Also, in this embodiment, β is 2 15 The process was designed to normalize by -1, but x can be set to any minimum and maximum value. inter Alternatively, you could clip the data and normalize it using the difference between the minimum and maximum values ​​mentioned above to obtain real values ​​with 15 bits or less in the range [0,1].

[0061] In S312, the bit depth conversion layer 306 converts the normalized intermediate features obtained in S311 into unsigned 8-bit integer intermediate features by rounding. The map of unsigned 8-bit integer intermediate features is an example of an intermediate correction map. Specifically, the bit depth conversion layer 306 applies a process like that shown in equation (2) to the output of S311.

number

[0062] Here, s inter is 2 8 -1, and the parentheses on the right side indicate the operation of rounding to the nearest decimal place. The bit depth conversion layer 306 scales real values ​​to [0,2 8 The bit depth conversion layer 306 obtains an unsigned 8-bit integer value by first setting it to the range of -1 and then rounding the decimal part. The bit depth conversion layer 306 converts the unsigned 15-bit integer value output by the ReLU layer 305 into an unsigned 8-bit integer through this process. In this embodiment, the bit depth conversion layer 306 processes using a uniform quantization method that does not perform nonlinear processing during quantization, but it may also process using a non-uniform quantization method as described in Non-Patent Literature 2.

[0063] FIG. 3(c) is a flowchart for explaining the processing in the final bit depth conversion layer 308. This processing is for converting an input represented by a certain bit depth into a different bit depth. Also, at this time, tone conversion is performed so that non-uniform tone expression (tones in a certain range are expressed finely and tones in other ranges are expressed coarsely) is performed. This tone conversion corresponds to a non-uniform quantization method as performed in Non-Patent Document 2.

[0064] In S321, the final bit depth conversion layer 308 normalizes the intermediate features obtained in the convolutional layer 307. Here, let the intermediate features be x, and assume that x is a map with a width W, a height H, and 3 channels. Normalization is, for example, a process of taking the absolute value of the intermediate features as in Equation (3), clipping so that the value is not more than α, and then normalizing to the range [0, 1].

Equation

[0065] Here, the parameter α of the clipping range is 2 14 -1. The parameter α may be optimized from a plurality of candidates by Bayesian optimization or the like so that the image quality of the prepared evaluation image improves. At that time, a general quantification index such as PSNR (Peak signal-to-noise ratio) may be used as an index of the image quality to be optimized, but it is not limited to this.

[0066] In S322, the final bit depth conversion layer 308 applies a function indicated by a non-linear transformation f Θ to the normalized intermediate features obtained in S321 as shown in the following Equation (4).

Equation

[0067] FIG. 4 is a diagram for explaining the non-linear transformation process in the bit depth conversion process. FIG. 4(a) is the non-linear transformation f ΘThis is a diagram of a tone curve representing the transformation. In this embodiment, we will explain the case where the nonlinear transformation can be represented by a tone curve as shown in Figure 4(a). First, the normalized map x' obtained in S321 is input to the tone curve to obtain a nonlinearly transformed map. The tone curve is transformed so that the gradation becomes finer at lower values ​​and coarser as the value increases. The range of the nonlinearly transformed map is [0,1] and takes the form of a 14-bit real value.

[0068] In S323, the final bit depth conversion layer 308 converts the output of S322 into an unsigned 7-bit integer by rounding. Specifically, the final bit depth conversion layer 308 uses the following formula (5).

number

[0069] Here, s1=2 7 -1, and the parentheses on the right side indicate the operation of rounding to the nearest decimal place. The final bit depth conversion layer 308 scales the 7-bit real value to [0,2 7 By setting the value to the range of -1 and then rounding the decimal part, an unsigned 7-bit integer value is obtained. Note that the final bit depth conversion layer 308 takes the absolute value of x in S321, so a 7-bit integer value is obtained here, not an 8-bit integer.

[0070] In S324, the final bit depth conversion layer 308 normalizes the 7-bit integer value obtained in S323 again. The final bit depth conversion layer 308 uses the same normalization coefficient as s1 in S323 to make the range of the normalized map a 7-bit real value in the range [0,1]. Specifically, the final bit depth conversion layer 308 uses the following formula (6).

number

[0071] In S325, the final bit depth conversion layer 308 performs an inverse nonlinear transformation f, which is the inverse function of the nonlinear transformation used in S322, on the output obtained in S324.Θ -1 The inverse nonlinear transformation process is applied by the final bit depth transformation layer 308. Θ -1 The value that was nonlinearized in S322 is converted back to linear by applying this. The range of the converted map is [0,1] and takes the value of a 7-bit real value. Specifically, the final bit depth conversion layer 308 uses the following formula (7).

number

[0072] In S326, the final bit depth conversion layer 308 converts the 7-bit real value output in S325 into an 8-bit signed 15-bit integer by rounding. Specifically, the final bit depth conversion layer 308 uses the following formula (8).

number

[0073] Here, s² = 2 14 -1, and the parentheses on the right side indicate the operation of rounding to the nearest decimal place. The final bit depth conversion layer 308 scales the real value to [0,2 14 By first setting the range to -1 and then rounding the decimal part, the value range becomes [0,2 14 This obtains a 7-bit integer value of -1. Since sign(x) outputs the sign of x, the final value obtained is [-2 14 ,2 14 It is an 8-bit integer value with a range of -1.

[0074] Since the Bayer image 301 input to the difference estimation NN and the weights and features of the intermediate layers of the difference estimation NN are represented in 8 bits, it is difficult to accurately infer 9 bits or more of gradation as the final output of the difference estimation NN with a high-speed model. Therefore, the final bit depth conversion layer 308 of this embodiment applies nonlinear processing to convert to a low bit depth, as in the processing of S321 to S326, and then applies inverse nonlinear processing to return the value range to the original bit depth. Through this processing, the final bit depth conversion layer 308 can express the correction portion with a small absolute value that greatly contributes to image quality with finer gradation while making the gradation of the correction map low bit depth, thereby suppressing the degradation of image quality that occurs with low bit gradation. Note that although the final bit depth conversion layer 308 converts the gradation to 8 bits in the processing of S321 to S323, it is not limited to 8 bits, and any bit depth less than or equal to the parameter α clipped in S321 is acceptable.

[0075] Furthermore, the final bit depth conversion layer 308 may implement the combined processing of S321 to S326 by performing arithmetic operations, or it may implement it using a lookup table (LUT) as shown in Figure 4(b). Figure 4(b) is a diagram of a lookup table showing the final bit depth conversion. The LUT in Figure 4(b) may correspond to the tone curve in Figure 4(a). By using a LUT, the final bit depth conversion layer 308 can speed up these nonlinear conversion processes and inverse nonlinear conversion processes. In this LUT, regions with small absolute values ​​of the correction map are converted with fine gradations, and as the absolute value of the correction map increases, the conversion is performed with coarser gradations. When using a LUT, the final bit depth conversion layer 308 may clip the input x with respect to the sign of parameter α before performing the conversion with the LUT. By using the LUT shown in Figure 4(b), the final bit depth conversion layer 308 can represent the range of relatively large image quality values ​​with relatively fine gradations.

[0076] In S505, the image correction unit 206 corrects the simplified RGB image obtained in S502 by subtracting the estimated value of the correction map obtained in S504, which has an 8-bit grayscale and a signed 15-bit integer range. This allows the image correction unit 206 to derive an estimated value of a higher-quality 14-bit RGB image, which is an image obtained by demosaicing the Bayer image.

[0077] <Functional configuration during learning process> In this embodiment, it is assumed that the model is trained within the framework of pseudo-quantization learning, as described in Non-Patent Document 1. In pseudo-quantization learning, the model's weights and intermediate features are not represented as integers but as floating-point numbers, unlike during inference, and are pseudo-quantized to 8-bit levels before use. When calculating the loss during forward propagation, the 8-bit quantized values ​​are used, and during backpropagation, the pre-quantized values ​​such as 32 bits are used, enabling minute parameter updates and further reducing errors during inference. After training the model within the framework of pseudo-quantization learning, the parameters are converted to integers using the parameter integerization unit 210 described later, and these integers are used during inference.

[0078] Figure 2(b) shows the functional configuration of the information processing device during learning. The information processing device 100 includes a storage unit 201, a learning data acquisition unit 207, an image quantization unit 204, a correction map estimation unit 205, an error calculation unit 208, a parameter update unit 209, and a parameter integerization unit 210. The CPU 101 may implement all or part of the learning data acquisition unit 207, image quantization unit 204, correction map estimation unit 205, error calculation unit 208, parameter update unit 209, and parameter integerization unit 210 by reading and executing a learning program. The storage unit 201 and the image quantization unit 204 are the same as in the inference configuration (Figure 2(a)), so their explanation is simplified or omitted.

[0079] The learning data acquisition unit 207 acquires the input Bayer image and GT (Ground Truth) image to be used for learning from the storage unit 201. The input Bayer image is generated by extracting pixels corresponding to the RGGB array of the Bayer image from the R, G, and B components of the RGB image, respectively. The GT image may be the difference between a simplified RGB image obtained by processing the generated Bayer image by the image conversion unit 203 and the ideal RGB image. The GT image obtained by this processing is a ground truth map that serves as the correct data, and can be said to be an ideal correction map that corrects the simplified RGB image to the ideal RGB. In this embodiment, the input image and GT image are generated in advance and stored in the storage unit 201, but the RGB image may be stored in the storage unit 201 and the image conversion unit 203 may generate them each time using the RGB image. The input image and GT image have a 14-bit depth.

[0080] The correction map estimation unit 205 obtains the model of the difference estimation NN from the memory unit 201. Then, the correction map estimation unit 205 inputs the 8-bit depth Bayer image obtained from the image quantization unit 204 into the 8-bit depth difference estimation NN and estimates a correction map with 8-bit grayscale and a value range of signed 15-bit integers.

[0081] The correction map estimation unit 205 uses the weights and intermediate features of the difference estimation NN model, which are represented as floating-point numbers rather than integers, as in the inference process, and then quantizes them to a pseudo-8-bit grayscale.

[0082] The error calculation unit 208 calculates the loss to the estimation result of the correction map as an error. Specifically, the error calculation unit 208 calculates the error between the estimated value of the correction map, which is an 8-bit grayscale correction map with a signed 15-bit integer range estimated by the correction map estimation unit 205, and the GT image obtained from the training data acquisition unit 207. The specific method for calculating the error will be described later.

[0083] The parameter update unit 209 updates the parameters of the difference estimation neural network based on the error obtained by the error calculation unit 208 and stores them in the storage unit 201.

[0084] The parameter integerization unit 210 quantizes the weights and output of the pseudo-quantized learning difference estimation NN and converts them to integers. The quantization method for converting to integers can be any known NN quantization method, and therefore no explanation is provided. As a result, the parameter integerization unit 210 obtains the same output before and after the integer conversion.

[0085] <Operation during the learning process> Figure 6 is a flowchart of the NN learning process performed by the information processing device 100. However, the information processing device 100 does not necessarily have to perform all the steps described in the flowchart in Figure 6.

[0086] In S601, the learning data acquisition unit 207 acquires a correction map, which will become the GT image, and an input Bayer image from the storage unit 201 as learning data. The bit depth of the correction map and the Bayer image is 14 bits.

[0087] In S602, the image quantization unit 204 performs quantization processing to convert the 14-bit depth Bayer image acquired in S601 into an 8-bit depth Bayer image and output it.

[0088] In S603, the correction map estimation unit 205 estimates a correction map with 8-bit grayscale and a value range of signed 15-bit integers using the same procedure as in S504, and obtains an estimated value of the correction map. That is, the correction map estimation unit 205 estimates a correction map with 8-bit grayscale and a value range of signed 15-bit integers from the 8-bit depth Bayer image obtained in S602.

[0089] In S604, the error calculation unit 208 calculates the loss Loss1 as the error for the estimation result of the correction map. The purpose of calculating the error is to learn to correctly estimate the RGB image, which is the difference between the simplified RGB image and the correction map, by correctly estimating the correction map from the Bayer image. As shown in equation (9), the error calculation unit 208 calculates the error for each element of the correction map estimation result C obtained from S603. inf And, correction map C, which is GT obtained with S601.gt The sum of the absolute values ​​of the differences between (also called the L1 distance) is calculated as the loss, Loss1. However, the type of loss is not limited to this.

number

[0090] In S605, the parameter update unit 209 updates the NN parameters using backpropagation based on the loss 1 calculated in S604. The parameters updated here refer to the weights of the convolutional layer 304 and convolutional layer 307 that constitute the difference estimation NN shown in Figure 3(a).

[0091] In S606, the parameter update unit 209 stores the updated parameters of the difference estimation NN in the storage unit 201. Then, the parameter update unit 209 loads the weights into the difference estimation NN. Steps S601 to S606 constitute one iteration of training.

[0092] In S607, the parameter update unit 209 determines whether to terminate the learning process. The parameter update unit 209 may determine to terminate the learning process if the value of the loss Loss1 obtained by formula (9) falls below a predetermined threshold. Alternatively, the parameter update unit 209 may determine to terminate the learning process if a predetermined number of learning sessions have been performed. The parameter update unit 209 returns to S601 and repeats the process until it determines to terminate the learning process. On the other hand, if the parameter update unit 209 determines to terminate the learning process, it proceeds to S608.

[0093] In S608, the parameter integerization unit 210 integerizes the parameters of the difference estimation NN.

[0094] As explained above, in the first embodiment, during the inference process, a correction map is estimated by a neural network with a bit depth lower than the bit depth of the image to be processed. Then, by subtracting the estimated correction map from the simplified RGB image, the first embodiment can estimate and derive an RGB image corresponding to the data structure of the output image, even if the data structures of the input image and the output image are different.

[0095] Furthermore, the first embodiment can accurately represent the correction map by applying a non-uniform quantization method (nonlinear transformation processing) in the final layer of a low-bit-depth neural network. As a result, the first embodiment can estimate and output high-quality images while reducing the processing load.

[0096] (Variation 1) Modification 1 describes a form in which a piecewise linear function is used in the final bit depth conversion layer 308 of the final layer 303. That is, Modification 1 is a nonlinear conversion f Θ A piecewise linear function is used. Modification 1 allows for more flexible setting of which range of the input to refine the gradation.

[0097] Furthermore, the piecewise linear function may be a function that defines the slope of each equally spaced interval, as in Non-Patent Document 2. In this case, the piecewise linear function will represent intervals with larger slopes with finer gradations.

[0098] Figure 7 illustrates the piecewise linear function used in the bit depth conversion process of the final bit depth conversion layer 308. This piecewise linear function has five equally spaced intervals in the input domain [0,1], and the slope γ of each interval i Of the ranges (i=1~5), the slope γ2 (=(β3-β2) / 0.2) in the second interval (x=0.2~0.4) is the largest. Using this piecewise linear function, the correction map output by the final bit depth conversion layer 308 will be a map that most finely represents the gradation within the range of the second interval.

[0099] In addition, if a function obtained by piecewise linear approximation of the tone curve of the first embodiment is used, the output finally obtained from the final bit depth conversion layer 308 is converted so that the gradation becomes finer for small inputs and coarser for large inputs. The final bit depth conversion layer 308 may determine the slope of each interval of the piecewise linear function using Bayesian optimization or other methods, and may select multiple candidates and optimize to improve the image quality of a pre-prepared evaluation image. In this case, the final bit depth conversion layer 308 may use general quantification metrics such as PSNR as the image quality index that is the target of optimization.

[0100] Alternatively, the final bit depth conversion layer 308 may learn the parameters of the piecewise linear function by backpropagation, as described in Non-Patent Document 2.

[0101] As explained above, in Modification 1, the final bit depth conversion layer 308 performs a nonlinear conversion f Θ By using a piecewise linear function, the degree of freedom in shape can be increased, and the degree of freedom in gradation expression can be increased compared to the first embodiment. As a result, Modification 1 can effectively suppress image quality degradation due to quantization. Furthermore, Modification 1 can learn parameters such as the slope of the piecewise linear function together with the weights of the neural network by backpropagation, and the optimal gradation expression for improving image quality can be efficiently determined.

[0102] (Variations 1-2) In the above modified example 1, when learning the piecewise linear function and the weights of the NN, in S604, the error calculation unit 208 may calculate the loss Loss1, which is the error with respect to the estimation result of the correction map, as follows. Specifically, the C obtained in S603 inf , C obtained with S601 gt , and C gt A weighted map w has the same width and height as the simplified RGB image used to generate it, and each pixel has a different weight value. i This is prepared in advance. And C inf and C gtTo reduce the loss that brings the two points closer together, a weighting is applied to each pixel. The error calculation unit 208 may calculate the loss Loss1 based on formula (10) when using the L1 distance.

number

[0103] Here, a weighted map w i This may be determined according to the relationship between the image quality index and the pixel value I. For example, if the image quality index is expressed as a function g(I) of the pixel value I, the pixel values ​​of an ideal RGB image may be input into the function g(I) to obtain a map with the same width and height. Alternatively, the values ​​of the obtained map may be further divided by the maximum value of the map to normalize the map, and this normalized map may be used as the weighted map.

[0104] For example, if a graph with pixel values ​​on the horizontal axis and image quality index g(I) on the vertical axis does not increase monotonically but has a local maximum, then pixels with pixel values ​​close to the local maximum of the graph will receive a greater weight in the loss calculation in equation (10) w. i This increases the size of the pixels. As a result, learning of these pixels is prioritized. This promotes learning of NN weights and nonlinear transformation parameters that improves the image quality of regions that affect image quality.

[0105] As explained above, in Modification 1-2, loss weighting is performed so that the accuracy of correction map estimation increases for pixels with pixel values ​​that contribute significantly to image quality. This makes it possible to focus on improving the accuracy of demosaicing processing in areas where the image quality improvement effect is high.

[0106] (Modification 2) Modification 2 is configured such that, during the learning process, the S322 operation performed by the final bit depth conversion layer 308 constituting the final layer 303 is replaced with an identity map, and the nonlinear transformation is implicitly performed within the NN. In other words, unlike the first embodiment, Modification 2 does not explicitly perform the nonlinear transformation in S322. As a result, Modification 2 can accurately represent the correction map even with a small number of grayscale levels while avoiding the increased processing load due to the nonlinear transformation during the inference process, thereby improving the accuracy of the demosaicing process. In the following description of Modification 2, the focus will be on the parts that differ from the processing of the first embodiment, and similar processing will be simplified or omitted.

[0107] <Operation during the learning process> In S601, the learning data acquisition unit 207 acquires the input image and the GT image from the storage unit 201. The input image is a Bayer image, and the GT image is a correction map. The correction map has been pre-converted using a nonlinear transformation and converted to a signed 8-bit integer. Specifically, the nonlinear transformation and conversion to a signed 8-bit integer of the correction map are performed in the same manner as in the processing of S321 to S323. The correction map after these transformations is used as the GT image. The type of nonlinear transformation may be a tone curve, as used in the first embodiment, but is not limited to this. Here, the bit depth of the correction map and the Bayer image is set to 14 bits.

[0108] In S602, the signed 14-bit integer Bayer image acquired as the input image in S601 is converted to a signed 8-bit integer Bayer image using the same quantization process as in S503.

[0109] In S603, the correction map estimation unit 205 obtains an estimated value of the correction map, which is an 8-bit grayscale integer with a signed 15-bit range, using the same procedure as in S504. However, in this embodiment, when the correction map estimation unit 205 processes the final bit depth conversion layer 308 of the difference estimation NN in S503, it replaces the nonlinear transformation applied in the nonlinear transformation processing of S322 with an identity map. Also, the processing in S324 to S326 is performed only during the inference process and not during the learning process.

[0110] In S604, the error calculation unit 208 calculates a loss Loss1 for the estimation result of the correction map. The error calculation unit 208 defines the loss Loss1 such that it decreases as the estimated value of the correction map obtained in S603 approaches the GT image, which is the correction map. For example, the error calculation unit 208 may calculate the L1 distance, which is the sum of the differences in the absolute values ​​of each element, as in the first embodiment, but the type of loss is not limited to this.

[0111] <Operation during inference processing> In S503, the correction map estimation unit 205 modifies the processing in the final bit depth conversion layer 308 of the difference estimation NN. Specifically, the final bit depth conversion layer 308 does not perform the processing in S322 that was performed in the first embodiment. This is because, by performing the learning process described above in this modified example, the NN is trained so that the result of the nonlinear transformation is directly output at the start of Figure 3(c).

[0112] Furthermore, as mentioned above, the final bit depth conversion layer 308 performs the processing S324 to S326, which was not performed during the learning process, during the inference process.

[0113] As explained above, in Modification 2, during the learning process, the final bit depth conversion layer 308 is configured to implicitly perform a nonlinear transformation within the neural network. This allows Modification 2 to accurately represent the correction map even with a small number of grayscale levels while avoiding an increase in processing load due to the nonlinear transformation during the inference process, and is expected to improve the accuracy of the demosaicing process.

[0114] (Variation 3) In Modification 3, a method is described in which a nonlinear processing is applied to an unsigned 14-bit integer Bayer image in the image quantization unit 204 to obtain an unsigned 8-bit integer Bayer image using a non-uniform quantization method. Figure 8 is a diagram illustrating Modification 3 of the first embodiment.

[0115] The correction map estimation unit 205 aims to represent images that significantly contribute to image quality with finer gradations. For this purpose, it is desirable that the input data be converted to 8 bits in an appropriate state. More specifically, it is desirable to represent the low-luminance region, which contributes to image quality, with finer gradations.

[0116] Figure 8(a) is a flowchart of the image quantization process performed by the image quantization unit 204 in the modified example 3.

[0117] In the S801, the image quantization unit 204 normalizes the 14-bit integer Bayer image. Specifically, it performs the process shown in equation (11) on the 14-bit integer Bayer image.

number

[0118] Here, γ is 2 14 Set to -1. This process results in a 14-bit grayscale real value with a range of [0,1] as the output.

[0119] In S802, the image quantization unit 204 performs a nonlinear transformation f on the normalized Bayer image acquired by S801. Φ Apply this.

number

[0120] Figure 8(b) shows the nonlinear transformation f in this embodiment. Φ This is a diagram. The nonlinear transformation is performed so that the gradation becomes finer above the black level (OB level). The black level is a 14-bit numerical value that represents the level of the pixel value that serves as the reference for black. Pixel values ​​below the black level are ultimately determined to be black. Since values ​​below the black level are uniformly determined to be black, values ​​greater than the black level contain information as an image. Therefore, it is important to convert the pixel values ​​above the black level into finer gradations. In this embodiment, the black level is set to 2048.

[0121] The range of values ​​for a nonlinearly transformed Bayer image is real values ​​in the range [0,1].

[0122] In S803, the image quantization unit 204 converts the nonlinearly transformed Bayer image acquired by S802 into an unsigned 8-bit integer value. Specifically, the image quantization unit 204 applies the process shown in equation (13) to the output of S902.

number

[0123] Here, s input =2 8 -1, and the parentheses on the right side indicate rounding to the nearest integer. The scale of a 14-bit real number is [0,2 8 By reducing the value to the range of -1 and then rounding the decimal part, an unsigned 8-bit integer value can be obtained.

[0124] As explained above, in Modification 3, the image quantization unit 204 applies a nonlinear process to the 14-bit Bayer image to obtain an unsigned 8-bit image using a non-uniform quantization method. As a result, Modification 3 can accurately represent the correction map, and an improvement in the accuracy of the demosaicing process can be expected.

[0125] (Modification 4) In Modification 4, the correction map estimation unit 205 converts the simplified RGB image output by the image conversion unit 203 into a correction map to obtain an RGB image. Figure 9 is a block diagram showing the functional configuration of the information processing device during inference and learning in Modification 4.

[0126] Figure 9(a) shows the functional configuration of the information processing device during inference in this modified example. The explanation will focus on the configuration modified from the original first embodiment, and configurations similar to the first embodiment will be simplified or omitted.

[0127] The image quantization unit 204 applies quantization processing to the 14-bit simplified RGB image (high-bit simplified RGB image) obtained from the image conversion unit 203, converting it into an unsigned 8-bit integer simplified RGB image (low-bit simplified RGB image). In this modified example, the image quantization unit 204 uses the same uniform quantization method as the bit depth conversion layer 306 described later, but a non-uniform quantization method may also be used. Here, the bit depth of the NN and the bit depth of the low-bit depth image are matched (8 bits), but the bit depths may be different. The bit depth of the NN should be lower than the bit depth of the input image and greater than or equal to the bit depth of the low-bit Bayer image.

[0128] The correction map estimation unit 205 inputs the simplified RGB image, which is an unsigned 8-bit integer obtained from the image quantization unit 204, into an 8-bit neural network (NN) and estimates a correction map with 8-bit grayscale and a value range of signed 15-bit integers. In this modified example, the correction map is the difference map between the RGB image of the quality that the NN originally wanted to infer and the simplified RGB image.

[0129] The image correction unit 206 derives a higher-quality 14-bit RGB image by subtracting a correction map, estimated by the correction map estimation unit 205, which has 8-bit grayscale and a value range of signed 15-bit integers, from the simplified RGB image obtained from the image conversion unit 203.

[0130] Figure 10 is a flowchart of the inference process performed by the information processing device in Modification 4. Steps S501 and S502 are the same as those in Figure 5, so their explanation is omitted.

[0131] In S1003, the image quantization unit 204 performs quantization processing to convert the simplified RGB image of an unsigned 14-bit integer acquired in S502 into a simplified RGB image of an unsigned 8-bit integer.

[0132] In S1004, the correction map estimation unit 205 obtains an estimated value of a correction map with 8-bit grayscale and a signed 15-bit integer range from the simplified RGB image of unsigned 8-bit integers obtained in S1003.

[0133] In S1005, the correction map estimation unit 205 subtracts the estimated value of the correction map obtained in S1004 from the simplified RGB image of an unsigned 14-bit integer obtained in S502. This allows the correction map estimation unit 205 to derive an estimated value of the RGB image, which is the image obtained by demosaicing the Bayer image.

[0134] Figure 9(b) shows the functional configuration of the information processing device during inference in the modified example 4. Similar to the explanation in Figure 9(a), the explanation will focus on the configuration modified from the original first embodiment, and the explanation of the configuration that is the same as in the first embodiment will be simplified or omitted. In addition, the image quantization unit 204 and the correction map estimation unit 205 have the same functional configuration during inference, so their explanations will be simplified or omitted.

[0135] The training data acquisition unit 207 acquires the input image and GT (Ground Truth) image to be used for training from the storage unit 201. The high-bit simplified RGB image used as the input image may be generated by the image conversion unit 203 converting a Bayer image, which is generated by extracting pixels corresponding to the RGGB array of the Bayer image from the R, G, and B components of the RGB image, respectively. The GT image is the difference between the high-bit simplified RGB image obtained by processing the generated Bayer image by the image conversion unit 203 and the ideal RGB image. In this modified example, the input image and GT image are generated in advance and stored in the storage unit 201, but the RGB image may be stored in the storage unit 201 and the training data acquisition unit 207 may generate the GT image each time using the RGB image and the image conversion unit 203. The input image and GT image have a 14-bit depth.

[0136] As explained above, according to Modification 4, the correction map estimation unit 205 can obtain an RGB image by converting the data obtained by quantizing the high-bit simplified RGB image output by the image conversion unit 203 into 8-bit grayscale data into a correction map.

[0137] Alternatively, the correction map estimation unit 205 may not input the simplified RGB image to the neural network (NN), but may instead input a Bayer image generated by extracting pixels corresponding to the RGGB array of the Bayer image from the simplified RGB image, based on the R, G, and B components, respectively.

[0138] (Variation 5) Modification 5 describes a method in which the image correction unit 206 obtains an RGB image by multiplying the simplified RGB image by the correction map estimation unit 205.

[0139] Although the functional configuration of the information processing device remains unchanged, the processing of the correction map estimation unit 205, which estimates the correction map, and the image correction unit 206 differs from previous examples, so we will explain the differences in detail.

[0140] The correction map estimation unit 205 outputs a correction map that corresponds to the ratio between the simplified RGB image and the higher-quality RGB image corresponding to the correction image, using a neural network (also called a ratio estimation neural network), rather than a correction map that corresponds to the difference between the simplified RGB image and the higher-quality RGB image corresponding to the correction image. The correction map estimation unit 205 may output a correction map with 8-bit grayscale and a value range of signed 15-bit integers.

[0141] The image correction unit 206 derives a higher-quality 14-bit RGB image by multiplying the 14-bit simplified RGB image obtained from the image conversion unit 203 with a correction map estimated by the correction map estimation unit 205, which has an 8-bit grayscale range and a value range of signed 15-bit integers. The image correction unit 206 may also divide the simplified RGB image by the correction map instead of multiplying. In this case, the correction map estimation unit 205 generates a correction map corresponding to the division.

[0142] Figure 11 is a flowchart of the inference process in Modification Example 5. Steps S501 to S503 are the same as those in the flowchart in Figure 5, so their explanation is omitted.

[0143] In the S504-1100, the correction map estimation unit 205 generates and outputs a correction map that shows the ratio between the simplified RGB image and the higher-quality RGB image.

[0144] In steps S505-1101, the image correction unit 206 multiplies the 14-bit Bayer image obtained in S501 by the estimated value of the correction map obtained in S504. This allows the image correction unit 206 to derive an estimated value for the RGB image, which is the image obtained by demosaicing the Bayer image.

[0145] As explained above, according to Modification 5, the correction map estimation unit 205 generates a correction map based on the ratio of the simplified RGB image to the higher-quality RGB image. As a result, the image correction unit 206 can obtain an RGB image by multiplying the simplified RGB image by the correction map output by the correction map estimation unit 205.

[0146] (Experimental variation 6) In Modification 6, the image correction unit 206 explains how to obtain an RGB image by multiplying the simplified RGB image by the correction map output by the correction map estimation unit 205 and taking the difference between that map and the simplified RGB image.

[0147] Although the functional configuration of the information processing device remains unchanged, the correction map estimation unit 205, which estimates the correction map, and the processing within the image correction unit 206 differ from that of Modification 5. Therefore, the explanation of Modification 6 will focus on the differences.

[0148] The correction map estimation unit 205 generates a correction map using a neural network (also called a difference ratio estimation neural network) that compares the difference map between the simplified RGB image and a higher-quality RGB image corresponding to the correction image, as well as a map corresponding to the ratio with the simplified RGB image. The correction map estimation unit 205 generates a correction map with 8-bit grayscale and a value range of signed 15-bit integers.

[0149] The image correction unit 206 generates a difference map by multiplying the simplified RGB image obtained from the image conversion unit 203 with the estimated correction map. The image correction unit 206 may also divide the simplified RGB image by the correction map instead of multiplying. In this case, the correction map estimation unit 205 generates a correction map corresponding to the division.

[0150] The image correction unit 206 derives a higher-quality 14-bit RGB image by subtracting a difference map from the simplified RGB image obtained from the image conversion unit 203. Alternatively, the image correction unit 206 may add the correction map to the simplified RGB image instead of subtracting it. In this case, the correction map estimation unit 205 generates a correction map corresponding to the addition.

[0151] Figure 12 is a flowchart of the inference process in modified example 6. Since steps S501 to S504-1200 are the same as the process in step S504 of the flowcharts in Figures 5 and 11, the explanation is omitted.

[0152] In steps S505-1201, the image correction unit 206 multiplies the simplified RGB image obtained in S502 by the estimated value of the correction map obtained in S504. This allows the image correction unit 206 to derive a difference map between the simplified RGB image and a higher-quality RGB image.

[0153] In S505-1202, the image correction unit 206 subtracts the difference map obtained in S505-1201 from the simplified RGB image obtained in S502. This allows the image correction unit 206 to derive an estimated value for the RGB image, which is the image obtained by demosaicing the Bayer image.

[0154] As explained above, according to Modification 6, the image correction unit 206 can obtain an RGB image by taking the difference between the simplified RGB image and the simplified RGB image, which is obtained by multiplying the simplified RGB image by the correction map output by the correction map estimation unit 205 and the simplified RGB image.

[0155] It goes without saying that the processing of these embodiments, including the modified versions, is not limited to demosaicing processing, and can be applied to other image processing, such as noise reduction, aberration correction, and high-resolution processing to super-resolution (or high resolution), with a similar configuration. The information processing device may enable image processing such as demosaicing processing, noise reduction, aberration correction, and high-resolution processing, and may perform image processing according to the user's selection. In addition, the nonlinear processing performed by the image quantization unit 204 and the correction map estimation unit 205 may be improved by ensembling multiple nonlinear processing operations.

[0156] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the present invention can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0157] The disclosures herein include the following information processing devices, information processing methods, and programs. (Item 1) Image acquisition means for acquiring a first image with a first bit depth, Image conversion means that performs image processing on the first image to convert it into a second image, Image quantization means for converting the first image or the second image into a third image with a second bit depth lower than the first bit depth, A correction map estimation means for estimating a correction map of correction bit depth for correcting the second image based on the third image, Image correction means for correcting the second image based on the correction map to generate a corrected image, An information processing device characterized by comprising: (Item 2) The correction map estimation means estimates a correction map corresponding to the difference between the second image and the corrected image based on the third image, The image correction means corrects the second image by adding or subtracting the correction map to the second image. The information processing device described in item 1, characterized by the features described herein. (Item 3) The correction map estimation means estimates a correction map corresponding to the ratio between the second image and the corrected image based on the third image, The image correction means corrects the second image by multiplying or dividing the second image by the correction map. The information processing device described in item 1, characterized by the features described herein. (Item 4) The correction map estimation means estimates a difference map corresponding to the difference between the second image and the corrected image, and the correction map corresponding to the ratio with the second image. The image correction means corrects the second image by adding or subtracting a difference map obtained by multiplying the second image by the correction map. The information processing device described in item 1, characterized by the features described herein. (Item 5) The correction map estimation means includes a neural network with a third bit depth that is lower than the first bit depth and greater than or equal to the second bit depth. The neural network estimates an intermediate correction map of the third bit depth from the third image, and converts the bit depth of the intermediate correction map of the third bit depth to the correction bit depth to estimate the correction map. An information processing device according to any one of items 1 to 3, characterized by the features described in item 1 to 3. (Item 6) The neural network includes a bit depth conversion layer that generates the correction map, The bit depth conversion layer is After applying a nonlinear transformation process to the intermediate correction map, the bit depth is transformed to generate the third bit depth intermediate correction map. After applying an inverse nonlinear transformation process using the inverse function of the nonlinear transformation process to the intermediate correction map of the third bit depth, the bit depth is converted to the correction bit depth to generate the correction map of the third bit depth with respect to the gradation. The information processing device described in item 5. (Item 7) The bit depth conversion layer converts the bit depth using a lookup table or arithmetic operations including operations using piecewise linear functions. The information processing device described in item 6, characterized by the features described herein. (Item 8) The third bit depth is equal to the second bit depth. An information processing device according to any one of items 5 to 7, characterized in that it is an information processing device. (Item 9) The image quantization means generates the third image from the first image by a process including a nonlinear transformation process. An information processing device according to any one of items 1 to 8, characterized by the above. (Item 10) The image quantization means generates the third image by a nonlinear transformation process that converts pixel values ​​greater than the black level into finer gradations than the black level pixel values. The information processing device according to item 9, characterized in that it is a processing device. (Item 11) The image acquisition means acquires an image of a Bayer array including the first bit depth as the first image, The image correction means outputs a 3-channel RGB image with the first bit depth as the corrected image. An information processing device according to any one of items 1 to 10, characterized by the features described in item 1 to 10. (Item 12) The image correction means outputs the image obtained by removing noise from the first image as the corrected image. An information processing device according to any one of items 1 to 11, characterized by the features described in item 1 to 11. (Item 13) The image correction means outputs an image in which the aberrations of the first image have been corrected as the corrected image. An information processing device according to any one of items 1 to 12, characterized in that it is the same as described in item 1 to 12. (Item 14) The image correction means outputs a high-resolution image obtained by increasing the resolution of the first image as the corrected image. An information processing device according to any one of items 1 to 13, characterized by the features described in item 1 to 13. (Item 15) The image conversion means converts the first image into the second image with the first bit depth, The image quantization means converts the second image into the third image. An information processing device as described in any one of items 1 through 14. (Item 16) The image quantization means converts the first image into the third image. An information processing device as described in any one of items 1 through 15. (Item 17) An information processing device for learning a neural network, A learning data acquisition means for acquiring a first image of a first bit depth or a second image obtained by image processing of the first image, and a ground truth map of a first bit depth which will be the ground truth data for the correction map, Image quantization means for converting the acquired first or second image into a third image with a second bit depth lower than the first bit depth, Based on the third image, the neural network estimates the correction map, which is used to estimate the correction map of the correction bit depth for correcting the second image. An update means for updating the parameters of the neural network based on the error between the correct map and the correction map, An information processing device characterized by comprising: (Item 18) The aforementioned neural network is After applying a nonlinear transformation process to the third image, the bit depth is transformed to generate an intermediate correction map of a third bit depth that is lower than the first bit depth and greater than or equal to the second bit depth. After applying an inverse nonlinear transformation process using the inverse function of the nonlinear transformation process to the intermediate correction map of the third bit depth, the bit depth is converted to the correction bit depth to generate the correction map of the third bit depth with respect to the gradation. The information processing device described in item 17, characterized by the features described herein. (Item 19) An image acquisition process to acquire a first image with a first bit depth, An image conversion step involves applying image processing to the first image to convert it into a second image, Image quantization steps include converting the first image or the second image into a third image with a second bit depth lower than the first bit depth, A correction map estimation step is performed to estimate a correction map of correction bit depth for correcting the second image based on the third image, Image correction step to correct the second image based on the correction map to generate a corrected image, An information processing method comprising the following: (Item 20) A program to cause a computer to function as one of the information processing devices described in any one of items 1 through 18.

[0158] The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of symbols]

[0159] 100... Information processing unit, 202... Image acquisition unit, 203... Image conversion unit, 204... Image quantization unit, 205... Correction map estimation unit, 206... Image correction unit, 306... Bit depth conversion layer, 308... Final bit depth conversion layer, 207... Training data acquisition unit, 209... Parameter update unit.

Claims

1. Image acquisition means for acquiring a first image with a first bit depth, Image conversion means that performs image processing on the first image to convert it into a second image, Image quantization means for converting the first image or the second image into a third image with a second bit depth lower than the first bit depth, A correction map estimation means for estimating a correction map of correction bit depth for correcting the second image based on the third image, Image correction means for correcting the second image based on the correction map to generate a corrected image, An information processing device characterized by comprising:

2. The correction map estimation means estimates a correction map corresponding to the difference between the second image and the corrected image based on the third image, The image correction means corrects the second image by adding or subtracting the correction map to the second image. The information processing apparatus according to feature 1.

3. The correction map estimation means estimates a correction map corresponding to the ratio between the second image and the correction image based on the third image, The image correction means corrects the second image by multiplying or dividing the second image by the correction map. The information processing apparatus according to feature 1.

4. The correction map estimation means estimates a difference map corresponding to the difference between the second image and the corrected image, and the correction map corresponding to the ratio with the second image. The image correction means corrects the second image by adding or subtracting a difference map obtained by multiplying the second image by the correction map. The information processing apparatus according to feature 1.

5. The correction map estimation means includes a neural network with a third bit depth that is lower than the first bit depth and greater than or equal to the second bit depth. The neural network estimates an intermediate correction map of the third bit depth from the third image, and converts the bit depth of the intermediate correction map of the third bit depth to the correction bit depth to estimate the correction map. The information processing apparatus according to feature 1.

6. The neural network includes a bit depth conversion layer that generates the correction map, The bit depth conversion layer is After applying a nonlinear transformation process to the intermediate correction map, the bit depth is transformed to generate the third bit depth intermediate correction map. After applying an inverse nonlinear transformation process using the inverse function of the nonlinear transformation process to the intermediate correction map of the third bit depth, the bit depth is converted to the correction bit depth to generate the correction map of the third bit depth with respect to the gradation. The information processing apparatus according to claim 5.

7. The bit depth conversion layer converts the bit depth using a lookup table or arithmetic operations including operations using piecewise linear functions. The information processing apparatus according to feature 6.

8. The third bit depth is equal to the second bit depth. The information processing apparatus according to feature 5.

9. The image quantization means generates the third image from the first image by a process including a nonlinear transformation process. The information processing apparatus according to feature 1.

10. The image quantization means generates the third image by a nonlinear transformation process that converts pixel values ​​greater than the black level into finer gradations than the black level pixel values. The information processing apparatus according to feature 9.

11. The image acquisition means acquires an image of a Bayer array including the first bit depth as the first image. The image correction means outputs a three-channel RGB image of the first bit depth as the corrected image. The information processing apparatus according to feature 1.

12. The image correction means outputs the image obtained by removing noise from the first image as the corrected image. The information processing apparatus according to feature 1.

13. The image correction means outputs an image in which the aberrations of the first image have been corrected as the corrected image. The information processing apparatus according to feature 1.

14. The image correction means outputs a high-resolution image obtained by increasing the resolution of the first image as the corrected image. The information processing apparatus according to feature 1.

15. The image conversion means converts the first image into the second image with the first bit depth, The image quantization means converts the second image into the third image. The information processing apparatus according to claim 1.

16. The image quantization means converts the first image into the third image. The information processing apparatus according to claim 1.

17. An information processing device for learning a neural network, A learning data acquisition means for acquiring a first image of a first bit depth or a second image obtained by image processing of the first image, and a ground truth map of a first bit depth which will be the ground truth data for the correction map, Image quantization means for converting the acquired first or second image into a third image with a second bit depth lower than the first bit depth, Based on the third image, the neural network estimates the correction map for correction of the bit depth used to correct the second image; An update means for updating the parameters of the neural network based on the error between the correct map and the correction map, An information processing device characterized by comprising:

18. The aforementioned neural network is After applying a nonlinear transformation process to the third image, the bit depth is transformed to generate an intermediate correction map of a third bit depth that is lower than the first bit depth and greater than or equal to the second bit depth. After applying an inverse nonlinear transformation process using the inverse function of the nonlinear transformation process to the intermediate correction map of the third bit depth, the bit depth is converted to the correction bit depth to generate the correction map of the third bit depth with respect to the gradation. The information processing apparatus according to feature 17.

19. An image acquisition step to acquire a first image with a first bit depth, An image conversion step involves applying image processing to the first image to convert it into a second image, Image quantization steps include converting the first image or the second image into a third image with a second bit depth lower than the first bit depth, A correction map estimation step is performed to estimate a correction map of correction bit depth for correcting the second image based on the third image, Image correction step to correct the second image based on the correction map to generate a corrected image, An information processing method comprising the following:

20. A program for causing a computer to function as one of the means of an information processing device according to any one of claims 1 to 18.

Images