Image processing apparatus, computer-readable storage medium, and image processing method

Abstract

An image processing apparatus, a computer-readable storage medium, and an image processing method that combine the advantages of image processing of RAW image data and the advantages of image processing of full-color image data to improve image quality. Includes: a first processing section that detects corresponding pixels of reference image data selected from a plurality of RAW image data and each of a plurality of reference image data included in the plurality of RAW image data, and generates synthetic RAW image data by synthesizing the reference image data and the plurality of reference image data based on the corresponding pixels; a mapping generation section that generates mapping information that is obtained by associating a pixel position of the synthetic RAW image data with information derived from at least one RAW image data from the plurality of RAW image data; and a second processing section that performs image processing different from the first processing section on the synthetic RAW image data after demosaicing based on the mapping information.

Description

Technical Field

[0001] This disclosure relates to an image processing apparatus, a computer-readable storage medium, and an image processing method. Background Technology

[0002] Patent Document 1 discloses a noise reduction apparatus. The apparatus aligns one reference RAW image data from N+1 raw (RAW) image data generated by an image sensor with the N RAW image data, and mixes the pixel values ​​for each pixel, thereby generating a noise-reduced image. For moving subject regions in the RAW image data, the mixing ratio is set small, prioritizing information from the reference RAW image data. A low-pass filter is applied to the synthesized image data obtained after mixing to smooth the pixel values ​​at the pixel positions corresponding to the moving subject regions.

[0003] The technique of using multiple image data to reduce noise is called multi-frame noise reduction (MFNR), while the technique of generating a noise-reduced image from a single image is called single-frame noise reduction (SFNR). The device described in Patent Document 1 applies MFNR to the region of a stationary subject and SFNR to the region of a moving subject. This effectively reduces the overall noise of the image.

[0004] The process involves converting N+1 RAW image data into full-color image data through color interpolation (de-mosaicing), white balance, applying a linear matrix to improve color reproducibility, and contour enhancement to improve visual clarity. This data is then encoded using a compression codec such as that of the Joint Photographic Experts Group (JPEG) and stored in the recording and playback unit. Furthermore, the N+1 full-color image data stored in the recording and playback unit undergo the same processing as described above for generating a composite image from N+1 RAW image data, thereby generating a composite image. The apparatus described in Patent Document 1 achieves circuit generalization by making the processing of RAW image data the same as the processing of full-color image data.

[0005] Existing technical documents

[0006] Patent documents

[0007] Patent Document 1: Japanese Patent Application Publication No. 2012-186593 Summary of the Invention

[0008] The problem that the invention aims to solve

[0009] Furthermore, RAW image data before depixelation is not in a visually perceptible state, but rather contains information close to the sensor data itself, maintaining information that conforms to natural laws. Therefore, it is appropriate to implement processing methods such as predicting values ​​using natural laws. Conversely, even if image processing can be performed effectively on RAW image data, there is a concern that the image processing effect may be weakened due to subsequent depixelation or other processing.

[0010] Since de-mosaiced full-color image data (Red Green Blue (RGB) image data or Luminance Chrominance Chroma (YUV) image data) is in a visually perceptible state, it is suitable for processing that directly affects the final output image, such as careful design or final adjustments to visualability. However, image processing performed on full-color image data involves intentionally manipulating pixel values ​​through filtering or correction, and therefore sometimes fails to maintain information that conforms to natural laws. Therefore, there are concerns that even when performing processing on full-color image data to predict values ​​using natural laws, accurate information may not be obtained.

[0011] The apparatus described in Patent Document 1 performs the same processing on image data before and after demosaicing, without considering the advantages of image processing on RAW image data and the advantages of image processing on full-color image data. This disclosure provides a technique that combines the advantages of image processing on RAW image data and the advantages of image processing on full-color image data to improve image quality.

[0012] Technical means to solve the problem

[0013] An image processing apparatus according to an embodiment of this disclosure includes a first processing unit, a mapping generation unit, and a second processing unit. The first processing unit detects corresponding pixels of a reference image data selected from multiple RAW image data sets and multiple reference image data sets included in the multiple RAW image data sets. Furthermore, the first processing unit synthesizes the reference image data and the multiple reference image data sets based on the corresponding pixels to generate synthesized RAW image data. The mapping generation unit generates mapping information, which is obtained by associating the pixel positions of the synthesized RAW image data with information derived from at least one of the multiple RAW image data sets. The second processing unit performs image processing on the de-mosaicized synthesized RAW image data that differs from that of the first processing unit.

[0014] The effects of the invention

[0015] This disclosure allows for the combination of the advantages of image processing for RAW image data and the advantages of image processing for full-color image data to improve image quality. Attached Figure Description

[0016] Figure 1 This is a block diagram illustrating the hardware structure of the image processing apparatus in an embodiment.

[0017] Figure 2 This is a block diagram illustrating the image processing performed by the image processing device.

[0018] Figure 3 This is a block diagram illustrating the process of bypassing a portion of the preprocessing.

[0019] Figure 4 This is a summary diagram illustrating the first extended process.

[0020] Figure 5 This is a flowchart of an image processing method.

[0021] Explanation of symbols

[0022] 1: Image processing device

[0023] 10: Image Sensor

[0024] 11: Processor

[0025] 12: Memory

[0026] 121: Pipeline Processing Module

[0027] 123: Switch Module

[0028] 124: First extended processing module (an example of the first processing unit)

[0029] 125: Mapping Generation Module (An Example of a Mapping Generation Unit)

[0030] 126: Second Extended Processing Module (An Example of a Second Processing Unit)

[0031] D1: RAW image data

[0032] D3: Intermediate image data Detailed Implementation

[0033] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. Furthermore, in the following description, the same or equivalent components will be labeled with the same symbols, and repeated descriptions will be omitted.

[0034] (Structure of an image processing device)

[0035] Figure 1This is a block diagram illustrating the hardware structure of the image processing apparatus in an embodiment. Figure 1 The image processing apparatus 1 shown is, for example, a device with a camera function such as a smartphone. The image processing apparatus 1 sequentially applies a series of image processing steps to the image data. Hereinafter, the process of the series of image processing steps will also be referred to as an image processing pipeline.

[0036] like Figure 1 As shown, the image processing apparatus 1 includes an image sensor 10, a processor 11, a memory 12 (an example of a storage unit), a storage unit 13, an input unit 14, and an output unit 15. The processor 11 is connected to the image sensor 10, the memory 12, the storage unit 13, the input unit 14, and the output unit 15 in a manner that enables communication between them.

[0037] Image sensor 10 is a solid-state imaging element that outputs RAW image data. RAW image data is color image data recorded using a mosaic array. An example of a mosaic array is a Bayer array. Image sensor 10 may include a continuous shooting function. In this case, image sensor 10 generates multiple consecutive RAW image data. Processor 11 is a computing device that executes an image processing pipeline, and for example, it is an image signal processor (ISP) optimized for image signal processing. Processor 11 can be not only an ISP, but also a graphics processing unit (GPU) or a central processing unit (CPU). Depending on the type of image processing in the image processing pipeline, the ISP can be combined with a GPU or CPU to execute each image processing task. Processor 11 executes the image processing pipeline for each RAW image data output from image sensor 10.

[0038] Memory 12 and memory bank 13 are storage media. Figure 1 In the example shown, memory 12 stores modules of the program executed by processor 11, definition data, RAW image data, and intermediate image data and mapping information, as described later. Memory 12 is, for example, Synchronous Dynamic Random Access Memory (SDRAM). Storage unit 13 stores image data processed by the image processing pipeline. Image data processed by the image processing pipeline is, for example, RGB image data or YUV image data. Storage unit 13 is, for example, a Hard Disk Drive (HDD). Furthermore, there is no particular limitation on memory 12 and storage unit 13 as long as they are storage media. Memory 12 and storage unit 13 may comprise a single piece of hardware.

[0039] The memory 12 includes a pipeline processing module 121 for executing an image processing pipeline. The processor 11 executes the image processing pipeline with reference to the memory 12. Furthermore, the memory 12 stores definition data 122 and a switching module 123 for implementing the switching of the image processing pipeline (described later). Moreover, the memory 12 stores a first extended processing module 124 (an example of a first processing unit) and a mapping generation module 125 (an example of a mapping generation unit) executed during the switching of the image processing pipeline (described later), and a second extended processing module 126 (an example of a second processing unit) executed after the image processing pipeline is executed.

[0040] Input unit 14 is a user interface that accepts user operations, such as operation buttons. Output unit 15 is a machine that displays image data, such as a display device. Input unit 14 and output unit 15 may include a single piece of hardware such as a touch screen.

[0041] (Overview of an image processing pipeline)

[0042] Figure 2 This is a block diagram illustrating image processing performed by an image processing device. The processor 11 executes the image processing pipeline P2 based on the pipeline processing module 121. First, the processor 11 loads RAW image data stored in the memory 12 (load processing P1). Then, the processor 11 sequentially performs multiple image processing operations on the RAW image data (image processing pipeline P2). Finally, the processor 11 obtains YUV image data as the processing result of the image processing pipeline P2. The pipeline processing module 121 is a program that enables the processor 11 to function in performing the aforementioned actions. The processor 11 can output the processing result to the output unit 15 or store it in the memory 13 (display / storage processing P3). Figure 2 In the example shown, processor 11 is also able to perform image processing before display / storage processing P3, as will be described later.

[0043] The pipeline processing module 121 causes the processor 11 to sequentially execute preprocessing P20, white balance processing P21, demosaic processing P22, color correction processing P23 and postprocessing P24 as an image processing pipeline P2.

[0044] In preprocessing P20, image processing is performed on the Bayer format image data, which is RAW image data. Details of preprocessing P20 will be described later. Next, in white balance processing P21, the intensity of each color component of the RGB in the image data for which preprocessing P20 has been performed is corrected. Next, in desacrifice processing P22, for the image data for which white balance processing P21 has been performed, pixels lacking color information in the Bayer format are interpolated to generate RGB image data. Next, in color correction processing P23, the RGB image data is color corrected. Finally, in post-processing P24, color space conversion from RGB format to YUV format and image processing on the YUV format are performed. Furthermore, Figure 2 The image processing pipeline P2 shown is an example, and various forms can be adopted. The image processing pipeline P2 is not limited to... Figure 2 The example shown can perform Figure 2 The diagram shows changes to the order of image processing, deletion of image processing, and addition of new image processing.

[0045] (Extension of the image processing pipeline)

[0046] Image processing apparatus 1 extends the image processing pipeline P2. Specifically, image processing apparatus 1 performs image processing (first extended processing P4) in a stage preceding the de-mosaic processing P22, replacing a portion of the image processing in image processing pipeline P2. Furthermore, image processing apparatus 1 performs image processing (second extended processing P6) on the image data processed by image processing pipeline P2. Image processing apparatus 1 outputs the processing result of second extended processing P6 to output unit 15 or stores it in storage unit 13.

[0047] The image processing apparatus 1 may selectively execute extended functions of the image processing pipeline, or it may be configured to always execute them. Hereinafter, an example of selectively executing extended functions of the image processing pipeline is disclosed, but the image processing apparatus 1 may also omit a structure for selectively executing extended functions.

[0048] Image processing apparatus 1 has the function of selecting, as needed, the processing to be bypassed for each image processing in image processing pipeline P2. After bypassing the target image processing, image processing apparatus 1 executes a first extended processing P4 to replace the target image processing. The target image processing is the image processing to be bypassed among the image processing included in image processing pipeline P2. The first extended processing P4 is a processing different from each image processing in image processing pipeline P2. Thus, image processing pipeline P2 is given the option of a new image processing. Image processing pipeline P2 and the first extended processing P4 are executed by processor 11.

[0049] The specific image processing to be bypassed depends on the content of the first extended processing P4. The user creates definition data 122 to bypass one or more image processing processes included in the image processing pipeline P2 according to the content of the first extended processing P4. Definition data 122 is stored in memory 12. Definition data 122 includes a definition indicating which image processing is the target image processing, and a definition indicating that the image processing executed after bypass is the first extended processing P4. By setting the method to bypass image processing processes that degrade the performance of the first extended processing P4, the user can transfer image data that fully utilizes the capabilities of the first extended processing P4 to subsequent processing.

[0050] The following is an example of the first extended processing P4, which is a noise reduction process based on multiple image data. Noise reduction based on multiple image data is called MFNR, which reduces the noise contained in the image data by calculating the average value of the pixels in the continuously captured image data.

[0051] In MFNR (Multiple Motion Not Recognition) implementation, moving objects are detected based on the differences in pixel values ​​between images to prevent multiple blurring caused by synthesis. For proper moving object detection, it is necessary to determine whether the differences in pixel values ​​between images are due to object movement or noise present in the image. Therefore, it is important to appropriately estimate the intensity of noise contained in the image. Typically, noise intensity can be estimated with good accuracy using factory-calibrated data and based on the ISO (International Organization for Standardization) ISO sensitivity at the time of shooting. This is because these intensities vary according to natural laws.

[0052] Furthermore, the image processing pipeline P2 sometimes includes processes that alter the noise intensity of an image. Such processes include, for example, SFNR (Single-Frequency Noise Reduction) or peripheral illumination reduction correction. SFNR uses low-frequency filters to reduce noise from an image. Peripheral illumination reduction correction increases the brightness of the image's peripheral areas to improve the situation where the amount of light at the image's periphery is reduced due to lens characteristics. The noise intensity estimation method using natural laws does not rely on image data that has undergone noise intensity alteration processing; therefore, when using such image data, there are concerns that sufficient estimation accuracy cannot be achieved.

[0053] Therefore, when performing the first extended processing P4, definition data 122 is generated in a manner that bypasses the processing that changes the noise intensity of the image. Thus, more appropriate image processing can be achieved by inputting image data with noise characteristics as a prerequisite to the first extended processing P4.

[0054] Processor 11 bypasses object image processing in image processing pipeline P2 based on switching module 123. For example, processor 11 determines whether predetermined switching conditions are met. These predetermined switching conditions are conditions used to determine whether bypassing is necessary and are predetermined. The predetermined switching conditions may be, for example, receiving a user operation that enables the first extended processing P4, or the timing or environment meeting predetermined conditions. Furthermore, if the predetermined switching conditions are met, processor 11 bypasses object image processing in image processing pipeline P2 according to defined data 122. Switching module 123 is a program that functions to cause processor 11 to perform a series of actions to bypass the object image processing. Figure 2 In the example shown, the process switches from preprocessing P20 to the first extended processing P4 midway through, and executes the first extended processing P4. The processing result of the first extended processing P4 is returned to the image processing pipeline P2, where white balance processing P21, de-mosaic processing P22, color correction processing P23, and post-processing P24 are executed sequentially.

[0055] (Details of the switching process)

[0056] Figure 3 This is a block diagram illustrating a process that bypasses a portion of the preprocessing. For example... Figure 3 As shown, preprocessing P20 constitutes a series of image processing steps, including basic processing P201, adaptive bilateral filter (ABF) application processing P203, darkness correction processing P204, and brightness adjustment processing P205. In basic processing P201, basic correction and linearization are performed on the input Bayer format image data as RAW image data D1. In ABF application processing P203, the ABF (Adaptive Bilateral Filter) is applied to the image data resulting from basic processing P201 to adjust edge intensity. In darkness correction processing P204, a pre-acquired black level is subtracted from the image data resulting from ABF application processing P203 to remove noise. In brightness adjustment processing P205, the image data resulting from darkness correction processing P204 is corrected by increasing the brightness of the surrounding areas of the image. With the illumination adjustment process P205 completed, a single processed RAW image data D2 is generated as the result of the preprocessing P20. A processed RAW image data D2 is generated for a single RAW image data D1.

[0057] Since the first extended processing P4 is noise reduction processing, therefore in Figure 3In the example shown, a switching process P202 is incorporated between the basic processing P201 and the ABF application processing P203 based on definition data 122. In switching process P202, if the specified switching conditions are met, the processor 11 switches the image processing pipeline P2 based on definition data 122. This generates intermediate image data D3 instead of a single processed RAW image data D2. Intermediate image data D3 is the result of image processing immediately preceding the object image processing, and one is generated for each RAW image data D1. Intermediate image data D3 becomes the processing object of the first extended processing P4. For example, the processor 11 sequentially reads N consecutive shots of RAW image data D1, performs preprocessing P20 on each, and generates N corresponding intermediate image data D3s. The processor 11 can also save the intermediate image data D3 in memory 12 each time it is created.

[0058] Processor 11 executes first extended processing P4 based on first extended processing module 124. First extended processing P4 is a different processing method from the individual image processing methods in image processing pipeline P2. This provides image processing options to image processing pipeline P2. Processor 11, referring to memory 12, generates a single composite RAW image data based on N intermediate image data.

[0059] Define data 122 as the output destination of a single composite RAW image data generated by the first extended processing P4. Figure 2 In the example shown, definition data 122 stores white balance processing P21 as the output destination. The first extended processing module 124 causes the processor 11 to run, outputting the generated single composite RAW image data to the output destination defined by definition data 122. Thus, the processing result of the first extended processing P4 is transferred to white balance processing P21 to execute image processing pipeline P2. In this way, a single composite image data in YUV format (composite YUV image data) can be obtained from the single composite RAW image data.

[0060] Processor 11 outputs the composite image data processed by image processing pipeline P2 to output unit 15, or stores it in storage unit 13 (display / storage processing P3). Here, output switching processing P25 can be executed after image processing pipeline P2. In output switching processing P25, processor 11 determines either display / storage processing P3 or image processing pipeline P2 as the output destination of image processing pipeline P2. Processor 11 determines the output destination based on definition data 122. Definition data 122 contains the output destination of the image data generated by image processing pipeline P2. Figure 2In the example shown, definition data 122 stores the second extended processing P6 and the display / storage processing P3 as output destinations. For example, the processor 11 determines whether a predetermined output switching condition is met. The predetermined output switching condition is a condition used to determine the output destination of the image processing pipeline P2 as the second extended processing P6, and is predetermined. The predetermined output switching condition may be, for example, receiving a user operation that enables the first extended processing P4 or the second extended processing P6, or it may be that the time or environment meets the predetermined condition. Furthermore, if the predetermined output switching condition is met, the processor 11 determines the output destination of the image processing pipeline P2 as the second extended processing P6 according to the definition data 122. If the predetermined output switching condition is not met, the processor 11 determines the output destination of the image processing pipeline P2 as the display / storage processing P3 according to the definition data 122. The switching module 123 is a program that enables the processor 11 to function in a manner that performs the aforementioned actions.

[0061] When the output destination of image processing pipeline P2 is determined to be the second extended processing P6, processor 11 executes the second extended processing P6 based on the second extended processing module 126. Processor 11 performs image processing on the synthesized YUV image data output from image processing pipeline P2. Thus, processing P7, which includes image processing pipeline P2, first extended processing P4, mapping generation processing P5, output switching processing P25, and second extended processing P6, is executed by processor 11.

[0062] (Details of the first extended processing)

[0063] The operation of the processor 11, as defined by the first extended processing module 124, is as follows.

[0064] (1) Obtain N intermediate image data D3

[0065] As described above, an intermediate image data D3 is generated for a single RAW image data D1. Therefore, processor 11 repeats the following process N times (N is an integer greater than or equal to 3): inputting the RAW image data D1 output from image sensor 10 into image processing pipeline P2, generating intermediate image data D3, and storing it in memory 12. Thus, N intermediate image data D3s are stored in memory 12. Processor 11 retrieves N intermediate image data D3s from memory 12.

[0066] (2) Pretreatment for synthesis

[0067] Processor 11 selects reference image data from multiple RAW image data sets. Reference image data is RAW image data that serves as a reference for processing such as position alignment or color correction. For example, the reference image data is the image data initially read from the multiple RAW image data sets in the first extended process P4. By initially reading the image data initially stored in memory 12 from the multiple RAW image data sets, processor 11 can use the earliest RAW image data in the time series as the reference image data. In the first extended process P4, N-1 RAW image data sets read after the reference image data become multiple reference image data sets. These multiple reference image data sets are then combined with the reference image data sets respectively.

[0068] Processor 11 detects corresponding pixels between reference image data and multiple reference image data. A corresponding pixel is a pixel (a pixel depicting the same subject) that corresponds between the reference image data and a reference image data. Processor 11 calculates a Global Motion Vector (GMV) representing the overall motion of the image between the reference image data and the reference image data. Next, processor 11 aligns the reference image data and the reference image data based on the GMV. Then, processor 11 calculates the difference between the pixel values ​​of the reference image data and the reference image data at each pixel location. Processor 11 designates pixels with a difference of 0 or less as corresponding pixels and stores them in memory 12 as ghost map information corresponding to the pixel locations. That is, the ghost map information is information obtained by associating the presence or absence of corresponding pixels with the pixel locations. Processor 11 performs the above processing for each reference image data to generate ghost map information for each reference image data.

[0069] Figure 4 (A) Figure 4 (J) is a schematic diagram illustrating the first extended process. Figure 4 (A) Figure 4 (D) is an example of RAW image data captured in a time series, with the subject being a moving vehicle. Figure 4 (A) is an example of baseline image data. Figure 4 (B) Figure 4 (D) is an example of reference image data. Figure 4 (E) is with Figure 4 (B) shows an example of ghost mapping information corresponding to the reference image data. The areas (pixels) indicated in black in the figure are those determined to have a corresponding pixel in the reference image data. The areas (pixels) indicated in white in the figure are those determined not to have a corresponding pixel in the reference image data. Similarly, Figure 4 (F) is with Figure 4 (C) shows an example of ghost mapping information corresponding to the reference image data. Figure 4 (G) is with Figure 4 The image shown in (D) is an example of ghost mapping information corresponding to reference image data. Thus, ghost mapping information is generated for each reference image data.

[0070] (3) Synthesis treatment

[0071] Processor 11 synthesizes reference image data with each reference image data. Processor 11 synthesizes each pixel of the reference image data with each pixel of the reference image data. Processor 11 determines the weight for each pixel position during synthesis by referring to ghost mapping information generated between the reference image data and the reference image data. When the pixel position of the synthesized object is associated with information indicating the absence of a corresponding pixel, processor 11 reduces the weight during synthesis compared to when the pixel position of the synthesized object is associated with information indicating the presence of a corresponding pixel. For example, processor 11 may set the synthesis weight to 1 when the pixel position of the synthesized object is associated with information indicating the presence of a corresponding pixel, and set the synthesis weight to 0 when the pixel position of the synthesized object is associated with information indicating the absence of a corresponding pixel. Thus, in the synthesis of each pixel of the reference image data and each pixel of the reference image data, the pixel value of the corresponding pixel is reflected. For example, Figure 4 (B) shows the pixel values ​​of the reference image data and Figure 4 The weighted sum of the ghost mapping information shown in (E) Figure 4 The pixel values ​​of each pixel in the reference image data shown in (A) are synthesized. The processor 11 synthesizes the reference image data with each reference image data by performing the aforementioned processing on each reference image data. As a result, the pixel values ​​of corresponding pixels in the reference image data are averaged with the pixel values ​​of corresponding pixels in each of the multiple reference image data, thereby generating synthesized RAW image data with reduced noise. Figure 4 (J) is an example of synthesized RAW image data, and is Figure 4 (B) Figure 4 The pixel values ​​of the reference image data shown in (D) are obtained using the method described above. Figure 4 The result is the composite of the pixel values ​​of each pixel in the reference image data shown in (A).

[0072] Furthermore, the method for determining the synthesis weights is not limited to the method described above. The processor 11 can also estimate the noise level at each pixel location based on the pixel values ​​of the reference image data to determine the synthesis weights. As mentioned above, the noise level is estimated using factory-shipped correction data and based on the ISO sensitivity at the time of shooting. This generates noise level mapping information that establishes a correlation between pixel locations and accurate noise levels. The processor 11 can also adjust the synthesis weights by comparing the difference between the pixel values ​​of the reference image data and a pixel value of the reference image data with the noise level at the pixel location. For example, if the square of the pixel value difference and the difference in noise level are within a reference value, the processor 11 does not change the synthesis weights; if the square of the pixel value difference and the difference in noise level are not within a reference value, the processor 11 changes the synthesis weights to 0. Additionally, the processor 11 can also estimate the presence or absence of texture in the image and change the synthesis method accordingly.

[0073] (Mapping generation process)

[0074] Processor 11, based on mapping generation module 125, accumulates ghost mapping information for each reference image data to generate accumulated mapping information (an example of mapping information) (mapping generation process P5). Accumulated mapping information is obtained by associating the pixel positions of the synthesized RAW image data with information derived from at least one RAW image data from multiple RAW image data sets. An example of the derived information is information related to the presence or absence of corresponding pixels. Figure 4 H is an example of accumulated mapping information. Figure 4 (H) is through the Figure 4 (E)~ Figure 4 The ghost mapping information shown in (G) is accumulated to obtain the information. The generated accumulated mapping information is stored in memory 12. The accumulated mapping information is used in the second extended process P6, which will be described later.

[0075] (Details of the second extended processing)

[0076] The second extended process P6 is an image processing process different from the image processing included in the first extended process P4 and the image processing pipeline P2. An example of the second extended process P6 is a noise reduction process different from the first extended process P4. The second extended process P6 takes the processing result of the image processing pipeline P2 as its processing object. That is, the second extended process P6 is a noise reduction process (SFNR) that takes the synthetic image data in single YUV format as its processing object. The processor 11 uses smoothing filters such as low-frequency filters to reduce noise in the synthetic image data. Here, the processor 11 retrieves the accumulated mapping information generated in the mapping generation process P5 from the memory 12 and determines the pixel positions where the smoothing filters are applied.

[0077] The processor 11 applies a smoothing filter to the pixel values ​​of the synthesized image data locations that are associated with information indicating the absence of corresponding pixels, based on accumulated mapping information. As a result, noise is removed from pixels with reduced weights synthesized in the first extended processing P4 (i.e., pixels that were not adequately denoised in the first extended processing P4). Thus, during the denoising processing of YUV image data, accurate information about corresponding pixels generated from multiple RAW image data can be utilized, allowing for accurate determination of the pixel locations where the smoothing filter is applied.

[0078] (Image processing methods)

[0079] Figure 5 This is a flowchart of an image processing method. Figure 5 The flowchart shown begins, for example, when a RAW image data D1 is read from memory 12. Figure 5 As shown, as a first extended process (step S10: an example of a first processing step), processor 11 performs MFNR. In the first extended process (step S10), noise-reduced synthetic RAW image data is generated based on multiple RAW image data. Then, as a mapping generation process (step S12: an example of a mapping generation step), processor 11 generates cumulative mapping information. Then, as a de-mosaic process (step S14: an example of a de-mosaic step), processor 11 converts the image format of the synthetic RAW image data generated in the first extended process (step S10). Processor 11 converts the synthetic RAW image data into RGB image data or YUV image data. As a second extended process (step S16: an example of a second processing step), processor 11 performs SFNR on the RGB image data or YUV image data obtained in the de-mosaic process (step S14). At this time, processor 11 applies a smoothing filter to pixels at pixel locations where MFNR has not been sufficiently performed, referring to the cumulative mapping information generated in the mapping generation process (step S12). Through the above steps, Figure 5 The image processing method shown has ended.

[0080] (Summary of Implementation Methods)

[0081] Image processing on RGB or YUV image data involves intentionally manipulating pixel values ​​through filtering or correction, which sometimes fails to maintain information that conforms to natural laws. Therefore, when performing MFNR on multiple RGB or YUV image datasets, it's sometimes impossible to accurately estimate the amount of noise, leading to overestimation or underestimation of ghosting information. For example, if the noise level is higher than expected, pixels that actually exist may be judged as nonexistent, significantly reducing the effectiveness of noise reduction. In contrast, when performing MFNR on multiple RAW image datasets, statistical measures of the sensor data, which can be explained by natural laws, can be utilized. Since the noise level follows the pixel values, ghosting information can be estimated with good accuracy. Therefore, in MFNR, using multiple RAW image datasets as the object yields higher performance than using multiple RGB or YUV image datasets.

[0082] Furthermore, when performing SFNR on RAW image data, even if noise is appropriately reduced in the RAW image data state, numerous subsequent processing steps can lead to uneven noise levels in the output. This can result in concerns about reduced output image quality. In contrast, when performing SFNR on RGB or YUV image data, this issue is less likely to occur because the SFNR process is closer to the output processing. Therefore, in SFNR, using RGB or YUV image data as the object generally yields higher performance compared to using RAW image data.

[0083] In the image processing apparatus 1, MFNR is performed using multiple RAW image data before performing the demosaicing process P22. The RAW image data prior to P22 maintains information conforming to natural laws. The image processing apparatus 1 predicts the amount of noise using natural laws and adjusts the composition weights of corresponding pixels based on the estimated noise amount, thereby appropriately removing noise. Pixels at positions where no corresponding pixel exists are not reduced in noise and are passed to the next processing step. Additionally, accumulated mapping information is generated by accumulating the ghost mapping information of each of the multiple RAW image data. Since the accumulated mapping information is generated based on the RAW image data, i.e., pixel values ​​without artificial processing, it more accurately represents the information of the corresponding pixels compared to the case where accumulated mapping is generated based on RGB or YUV image data.

[0084] Furthermore, after performing the demosaicing process P22, the image processing apparatus 1 determines the pixel locations where no corresponding pixel exists based on the accumulated mapping information, and performs SFNR at the determined pixel locations. As a result, the noise of pixels whose noise level was not reduced by MFNR is removed. Thus, the image processing apparatus 1 processes RGB or YUV image data using information derived from RAW image data, thereby combining the advantages of image processing on RAW image data with the advantages of image processing on full-color image data to improve image quality.

[0085] (Modified Example)

[0086] The embodiments of this disclosure have been described above, but this disclosure is not limited to the described embodiments. For example, in the described embodiments, the first extended processing P4 is shown to be a noise reduction process, but the first extended processing P4 can be set to any processing. For example, the first extended processing P4 can also be image compositing that is not aimed at noise removal (HDR compositing: High-Dynamic-Range rendering), etc. Furthermore, Figure 5 The mapping generation process shown can be performed before the second extension process, or it can be performed after the de-mosaic process.

[0087] The operation of the image processing apparatus 1 in the described embodiment can also be achieved by an image processing program that enables a computer to perform its functions. Furthermore, in this embodiment, the image processing apparatus 1 may not include the image sensor 10, the storage unit 13, the input unit 14, and the output unit 15.

[0088] In the described embodiment, an example of cumulative mapping information is shown, but the mapping information is not limited to cumulative mapping information. For example, the mapping information could be noise amount mapping information generated to determine the composition weights in the first extended processing P4, which establishes a correlation between pixel position and noise amount. Alternatively, it could be noise amount mapping information generated to determine the composition weights in image composition processing not aimed at noise removal, which establishes a correlation between pixel position and noise amount. Regarding the noise amount, it is not limited to methods that estimate it using factory-shipped correction data or based on the ISO sensitivity at the time of shooting; it can also be estimated based on a statistic that represents the deviation of pixel values ​​between the selected pixel of interest from the image data and the surrounding pixels located around the pixel of interest. For example, variance is known as a statistic representing this deviation of pixel values.

[0089] When the mapping information is noise level mapping information, the following SFNR can be performed in the second extended processing P6: increase the strength of the smoothing filter at pixel positions where the estimated noise level is above a first threshold, and decrease the strength of the smoothing filter at pixel positions where the estimated noise level is below a second threshold. The second threshold is a value below the first threshold. Thus, the processor 11 can perform image processing by applying the smoothing filter to the pixel values ​​of the pixel positions in the synthesized RAW image data with a smoothing strength corresponding to the noise level. Even in this variant, the image processing apparatus 1 uses information derived from RAW image data to process RGB or YUV image data, thereby combining the advantages of image processing on RAW image data with the advantages of image processing on full-color image data to improve image quality.

Claims

1. An image processing apparatus, comprising: The first processing unit detects the corresponding pixels of the reference image data selected from multiple RAW image data and the multiple reference image data contained in the multiple RAW image data, and synthesizes the reference image data and the multiple reference image data based on the corresponding pixels to generate synthetic RAW image data. A mapping generation unit generates mapping information, which is obtained by associating the pixel positions of the synthesized RAW image data with information derived from at least one of the plurality of RAW image data. as well as The second processing unit performs image processing on the de-mosaiced composite RAW image data based on the mapping information generated by the mapping generation unit, which is different from that performed by the first processing unit.

2. The image processing apparatus according to claim 1, wherein, The information derived from the at least one RAW image data is information related to the presence or absence of the corresponding pixel.

3. The image processing apparatus according to claim 2, wherein, The first processing unit performs image processing that averages the corresponding pixels of the reference image data with the corresponding pixels of the plurality of reference image data. The second processing unit performs image processing by applying a smoothing filter to the pixel values ​​of the pixel positions in the synthesized RAW image data that are associated with information indicating the absence of the corresponding pixel.

4. The image processing apparatus according to claim 1, wherein, The information derived from the at least one RAW image data is a noise level estimated based on a statistic that represents the deviation of pixel values ​​between a pixel of interest and its surrounding pixels.

5. The image processing apparatus according to claim 4, wherein, The second processing unit performs image processing by applying a smoothing filter to the pixel values ​​at the pixel locations of the synthesized RAW image data with a smoothing intensity corresponding to the noise level.

6. A computer-readable storage medium storing an image processing program thereon, enabling a computer to function as: a first processing unit that detects corresponding pixels of a reference image data selected from a plurality of RAW image data and a plurality of reference image data contained in the plurality of RAW image data, and synthesizes the reference image data and the plurality of reference image data based on the corresponding pixels to generate composite RAW image data; A mapping generation unit generates mapping information, which is obtained by associating the pixel positions of the synthesized RAW image data with information derived from at least one of the plurality of RAW image data. as well as The second processing unit performs image processing on the de-mosaiced composite RAW image data based on the mapping information generated by the mapping generation unit, which is different from that performed by the first processing unit.

7. An image processing method, comprising: The first processing step involves detecting the corresponding pixels of the reference image data selected from multiple RAW image data and the multiple reference image data contained in the multiple RAW image data, and synthesizing the reference image data and the multiple reference image data based on the corresponding pixels to generate synthetic RAW image data. The mapping generation step generates mapping information, which is obtained by associating the pixel positions of the synthesized RAW image data with information derived from at least one RAW image data among the plurality of RAW image data. as well as The second processing step involves performing image processing on the de-mosaiced synthetic RAW image data, based on the mapping information generated by the mapping generation step, that is different from the image processing in the first processing step.

Images