Image processing apparatus, image pickup apparatus, image processing method, and storage medium

Abstract

The present disclosure provides an image processing apparatus, an image pickup apparatus, an image processing method, and a storage medium. The image processing apparatus obtains image data of a plurality of frames, aligns the image data of the plurality of frames, and synthesizes the image data of the plurality of frames that have been aligned. The image data includes first data and second data, the first data being data of pixels of an effective pixel region of an image sensor and being constituted by signals corresponding to a predetermined color component arrangement, and the second data being data of pixels outside the effective pixel region of the image sensor. The image processing apparatus is capable of aligning the first data independently of the second data.

Description

Technical Field

[0001] This invention relates to image processing apparatus, camera equipment, image processing methods, and computer-readable storage media. In particular, this invention relates to image compositing technology. Background Technology

[0002] A technique for reducing image jitter is known to be achieved by combining N images taken consecutively with an exposure time of 1 / N of the desired exposure time (Japanese Patent Application Laid-Open No. 2007-243775).

[0003] When combining multiple images, alignment between them is necessary. However, if images are simply aligned for pixels within the valid pixel area, pixels outside the valid pixel area will be combined in an misaligned state. Therefore, problems arise when using values ​​from pixels outside the valid pixel area. Japanese Patent Application Publication No. 2007-243775 does not consider this issue. Summary of the Invention

[0004] The present invention was developed in view of the problems of such conventional techniques. One aspect of the present invention provides an image processing apparatus and an image processing method that enable the appropriate use of pixel values ​​outside the effective pixel region for image synthesis.

[0005] According to one aspect of the present invention, an image processing apparatus is provided, comprising: an acquiring component for acquiring image data of a plurality of frames; an alignment component for aligning the image data of the plurality of frames; and a synthesis component for synthesizing the aligned image data of the plurality of frames, wherein the image data includes first data and second data, the first data being data of pixels in an effective pixel region of an image sensor and consisting of signals corresponding to a predetermined color component arrangement, and the second data being data of pixels outside the effective pixel region of the image sensor, and the alignment component being capable of aligning the first data independently of the second data.

[0006] According to another aspect of the present invention, a camera device is provided, comprising: an image sensor; and an image processing device according to the present invention, wherein the image data of the plurality of frames are image data captured over time using the image sensor.

[0007] According to another aspect of the present invention, an image processing method is provided, comprising: obtaining image data of a plurality of frames; aligning the image data of the plurality of frames; and synthesizing the aligned image data of the plurality of frames, wherein the image data includes first data and second data, the first data being data of pixels in an effective pixel region of an image sensor and consisting of signals corresponding to a predetermined color component arrangement, and the second data being data of pixels outside the effective pixel region of the image sensor, the alignment being capable of aligning the first data independently of the second data.

[0008] According to another aspect of the invention, a computer-readable storage medium is provided storing a program that enables a computer to be used as an image processing device according to the invention.

[0009] Other features of the invention will become apparent from the following description of typical embodiments (with reference to the accompanying drawings). Attached Figure Description

[0010] Figure 1 This is a block diagram illustrating an example of the functional configuration of a camera device used as an image processing device according to an embodiment.

[0011] Figure 2 This is a schematic diagram showing the pixel area of ​​an image sensor.

[0012] Figure 3 This is a schematic diagram illustrating the RAW data synthesis process according to the first embodiment.

[0013] Figure 4 This is a flowchart relating to the operation performed in the composite image capturing mode according to the first embodiment.

[0014] Figure 5 This is a diagram illustrating the background of the second embodiment.

[0015] Figure 6 This is a diagram illustrating an overview of the second embodiment.

[0016] Figure 7 This is a flowchart relating to the operation performed in the composite image capturing mode according to the second embodiment. Detailed Implementation

[0017] In the following, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments are not intended to limit the scope of the claimed invention. Several features are described in the embodiments, but the invention is not limited to requiring all of these features, and multiple such features can be suitably combined. Furthermore, in the drawings, the same reference numerals denote the same or similar configurations, and redundant descriptions thereof are omitted.

[0018] Note that embodiments of the invention applied to imaging devices such as digital cameras will be described below. However, imaging functionality is not essential to the invention, and the invention can be implemented in any electronic device capable of processing image data. Examples of such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), smartphones, game consoles, robots, drones, and dashboard cameras. However, these are merely examples, and the invention can also be applied to other electronic devices.

[0019] Figure 1 This is a block diagram illustrating an example of the functional configuration of a camera device 10 used as an image processing device according to an embodiment of the present invention.

[0020] The control unit 113 includes one or more processors, RAM, and ROM capable of executing programs. The control unit 113 can load programs stored in the ROM into the RAM and execute the programs using the processor. The control unit 113 controls [various functions] by executing programs. Figure 1 The illustrated functional blocks describe the operation of the components of the camera device 10 and / or communicatively connected external devices. The functions of the camera device 10 can be implemented by executing a program through the control unit 113. Note that the functions that can be implemented by executing a program through the control unit 113 can be implemented using hardware (e.g., ASIC, FPGA, etc.). The ROM is, for example, electrically rewritable and stores programs executable by the processor of the control unit 113, as well as settings, GUI data, etc., of the camera device 10. For example, the control unit 113 can be in the form of a system-on-a-chip (SoC), a system-in-package (SiP), etc.

[0021] The imaging lens 100 has multiple lenses and aperture stops, and generates an optical image of the subject. The image sensor 102 is, for example, a CMOS image sensor, and has multiple photoelectric conversion units (pixels) arranged in a two-dimensional pattern.

[0022] Figure 2 This diagram schematically illustrates the pixel arrangement in an image sensor 102. The image sensor 102 has an effective pixel region 200 and an optical black (OB) region 201 outside the effective pixel region as areas for arranging pixels. The effective pixel region 200 is the area used to capture an image of the subject (the exposed area). On the other hand, the OB region 201 is the area not used to capture an image of the subject (the unexposed area). The pixels in the OB region 201 may be provided with a light-shielding film to, for example, prevent light from entering. An additional OB region 202 may be provided above or below the effective pixel region 200. The OB region 202 can be used to calculate black levels and can also be used for line offset correction on a vertical pixel line basis.

[0023] Apart from the presence of a light-shielding film, the pixels arranged in OB region 201 have the same structure as the pixels arranged in effective pixel region 200. If a color filter is set for the pixels in effective pixel region 200, a color filter is also set for the pixels in OB region 201 in the same manner as for the pixels in effective pixel region 200.

[0024] The effective pixel region 200 and the OB region 201 are regions of the image sensor 102, but for convenience, this specification will use the same terminology to refer to the image obtained by the image sensor 102. For example, "image" or "effective pixel region" refers to the image region obtained by the pixels of the effective pixel region 200 of the image sensor 102 in an image of a single frame obtained by capturing an image.

[0025] A color filter consists of an array of unit filters of different colors, with one unit filter set for each pixel. For example, a primary color Bayer array color filter consists of an array of four types of unit filters (R (red), G1 (green), G2 (green), and B (blue)) with a repeating unit of 2×2 pixels. In the following text, a pixel with unit filter G1 will be referred to as the "G1 pixel." The same applies to pixels with other types of unit filters.

[0026] For example, pixels arranged in OB region 201 are used to detect noise components in the photoelectric conversion elements (e.g., photodiodes) included in the pixels. For example, by subtracting the signal values ​​of pixels in OB region 201 from the signal values ​​of pixels in effective pixel region 200, offsets caused by noise components can be removed, and the black level of the image can be corrected.

[0027] The pixels included in the image sensor 102 generate pixel signals having values ​​corresponding to the amount of charge generated during the charge accumulation period. In shooting with the mechanical shutter 101 open and closed, the charge accumulation period corresponds to the exposure period. In shooting with the mechanical shutter 101 kept open, the charge accumulation period corresponds to the period from the time the accumulated charge is reset until the exposure period has elapsed. Typically, the former corresponds to still image shooting, and the latter to moving image shooting; however, the latter can also correspond to still image shooting.

[0028] When a charge accumulation period ends, a frame of pixel signal (analog image signal) is read from the image sensor 102. A frame of analog image signal includes pixel signals from the effective pixel region 200 and the OB region 201. The analog image signal is converted into a digital image signal (a set of pixel signals in digital format) by the A / D conversion unit 103. If the image sensor 102 can output a set of pixel signals in digital format, then the A / D conversion unit 103 is not required.

[0029] At this stage, each pixel signal has only one color component, depending on the color of the unit filter possessed by the corresponding pixel. In this specification, such a digital format image signal consisting of pixel signals (pixel signals having only one color component corresponding to the color of the unit filter) corresponding to an array of predetermined color components is referred to as "RAW (raw) data".

[0030] RAW data output from A / D conversion unit 103 or image sensor 102 is first stored in memory 114. Memory 114 is used to temporarily store RAW data, image data processed by signal processing unit 111, etc.

[0031] For each pixel type, the OB integration unit 104 calculates the average pixel value of the pixels in the OB region 201 of the RAW data. Here, the image sensor 102 has a primary color Bayer array color filter, so the OB integration unit 104 calculates the average pixel value of the R pixels, G1 pixels, G2 pixels, and B pixels in the OB region 201. This average pixel value is used as the black level.

[0032] The OB clamping unit 105 applies OB clamping processing (correcting black levels based on pixel values ​​in OB region 201) to pixel values ​​in the effective pixel region 200. Specifically, the OB clamping processing can be a process for subtracting the average pixel value calculated by the OB integration unit 104. The OB clamping processing allows for the suppression of bright black, color shifts, etc., in the image obtained from the pixel signals in the effective pixel region 200. The OB clamping processing uses black levels based on pixel type. For example, for the pixel value of R pixels in the effective pixel region 200, the black level obtained from R pixels in the OB region is subtracted. The OB clamping unit 105 applies the OB clamping processing to each composite source image in the composite source image and to the composite image generated by the composite unit 108.

[0033] The shadow correction unit 106 applies shadow correction to pixel values ​​in the effective pixel area 200. Shadow correction corrects for brightness reduction based on pixel position caused by optical characteristics such as the imaging lens 100 and microlenses included in the pixels. Therefore, in shadow correction, a gain based on pixel position is applied.

[0034] The white balance (WB) processing unit 107 applies white balance adjustment processing to the shadow-corrected image. The white balance adjustment processing applies gain according to pixel type (R, G1, G2, B) to the pixel values ​​that have already undergone shadow correction.

[0035] The misalignment detection unit 109 detects the misalignment amount in each frame of a plurality of frames of the synthesized source image. The misalignment amount can be an absolute misalignment amount with the misalignment amount in the first frame set to 0, or it can be a relative misalignment amount relative to the immediately preceding frame. When detecting an absolute misalignment amount, the first frame image is used as a reference image.

[0036] The misalignment detection unit 109 can detect the amount of misalignment in the motion vector of the image, which is the effective pixel region 200 between frames. The misalignment detection unit 109 can use any known technique (e.g., a method using template matching between frames or a method using the output of a gyroscope sensor, etc., provided in the camera device 10) to detect the misalignment. The misalignment detection unit 109 outputs the detected misalignment to the misalignment correction unit 110 or stores the misalignment in the memory 114.

[0037] The misalignment correction unit 110 aligns the image of the frame to be synthesized based on the misalignment amount detected by the misalignment detection unit 109. The misalignment correction unit 110 can perform alignment, for example, by changing the coordinate values ​​of each pixel in the frame to be synthesized according to the misalignment amount.

[0038] Note that the misalignment correction unit 110 has two modes: a mode that aligns only with the image from the effective pixel region 200, and a mode that aligns with the image including both the effective pixel region 200 and the OB region 201 (e.g., the entire frame). In this way, the misalignment correction unit 110 can align with the data (first data) in the effective pixel region 200 independently of the data (second data) in the OB region 201. For example, the mode of the misalignment correction unit 110 can be set by the control unit 113.

[0039] Even if the mode aligns the entire frame is set, if the condition that the alignment causes a decrease in the synthesis accuracy of the OB region 201 is met, the control unit 113 can switch to a mode that aligns only the image of the effective pixel region 200. For example, if the number of frames overlapping in the OB region 201 is less than a predetermined percentage of the total number of frames N, the control unit 113 can switch to a mode that aligns only the image in the effective pixel region 200. The number of frames overlapping in the OB region 201 can be determined based on the amount of misalignment detected by the misalignment detection unit 109 for each frame.

[0040] The compositing unit 108 composites multiple images using one of a plurality of selectable compositing modes and generates a composite image. It is sufficient to apply the compositing process to the effective pixel region and OB region of the reference image, rather than applying it to the entire frame.

[0041] The compositing unit 108 stores the generated composite image in the memory 114. In this embodiment, it is assumed that the camera device 10 can select an additive mode, an average additive mode, or a comparison brightness mode as the compositing mode. These compositing modes are examples, and other compositing modes may be selectable, or only one compositing mode may exist.

[0042] The image compositing methods in each compositing mode will be described below. Here, it is assumed that N frames of images (where N is an integer greater than or equal to 2) are being composited. The pixels constituting the images in each frame have coordinates (x, y) in an xy orthogonal coordinate system, and the brightness value of the pixel at coordinate (x, y) is represented by I_i(x, y) (where i = 1 to N), and the brightness value of the pixel at coordinate (x, y) in the composite image is represented by I(x, y). The compositing unit 108 calculates the brightness value I(x, y) of each pixel in the composite image according to the compositing mode as follows.

[0043] • Addition mode

[0044] I(x,y)=I_1(x,y)+I_2(x,y)+...+I_N(x,y)

[0045] In the additive mode, the compositing unit 108 adds the brightness values ​​of pixels at the same coordinates in each frame to generate a composite image. For example, the additive mode is used when compositing N frames of images taken with an exposure of 1 / N of the appropriate exposure to produce an image with proper exposure.

[0046] • Sum-and-Average Mode

[0047] I(x,y)=(I_1(x,y)+I_2(x,y)+...+I_N(x,y)) / N

[0048] In the additive averaging mode, the synthesis unit 108 generates a composite image whose brightness value for each pixel is the average of N frames by dividing the brightness value obtained as in the additive mode by the number of frames N. For example, the additive averaging mode is used to reduce noise in images captured at high sensitivity.

[0049] • Brighter mode

[0050] I(x,y)=max(I_1(x,y),I_2(x,y),...,I_N(x,y))

[0051] Here, `max()` is a function that extracts the maximum value of the elements in the parentheses. It obtains a composite image composed of the highest brightness values ​​among N pixels with the same coordinates in each frame. For example, comparing brightness patterns is effective when compositing images of fireworks or starry skies.

[0052] For coordinates where there is no pixel value to be synthesized due to alignment-based coordinate changes, the synthesizing unit 108 may instead use pixel values at those coordinates in another predetermined frame (for example, the first frame) in the addition mode and the addition averaging mode. The synthesizing unit 108 may perform synthesis using only existing pixel values in the addition averaging mode or the comparatively bright mode (in the addition averaging mode, the divisor is changed to the number of frames used for addition (<N)). Note that these are merely several conceivable examples, and other methods may be used to perform the synthesis.

[0053] In addition, the synthesizing unit 108 ensures that an image outside the effective pixel region 200 is not synthesized with the effective pixel region in the reference image. The synthesizing unit 108 may process pixel values outside the effective pixel region 200 whose coordinates have been changed by alignment to positions overlapping with the effective pixel region in the reference image in the same manner as when no pixel values are present.

[0054] The signal processing unit 111 applies development processing to (unsynthesized) RAW data, data of a synthesized image generated by the synthesizing unit 108 (synthesized RAW data), and the like. Note that the development processing is applied to an image region corresponding to the effective pixel region 200. "Development processing" is a general term for multiple types of image processing such as color interpolation processing, tone correction processing (gamma processing), and the like. The color interpolation processing is processing for interpolating values of color components that cannot be obtained at the time of shooting, and is also referred to as "demosaicing". As a result of applying the color interpolation processing, each pixel will have a plurality of color components (for example, RGB or YCbCr) required for a color image, and the data will no longer be RAW data.

[0055] The signal processing unit 111 may apply, to the developed image data, detection processing such as detection of characteristic regions (for example, a face region and a human body region) and motion thereof and person recognition processing, as well as synthesis processing, scaling processing, encoding and decoding processing. In addition, the signal processing unit 111 may apply various types of image processing, for example, data processing including header information generation processing, evaluation value calculation processing such as generating signals and evaluation values for autofocus detection (AF), calculating evaluation values for automatic exposure control (AE), and the like. Note that these are examples of image processing that can be applied by the signal processing unit 111, and are not intended to limit the image processing applied by the signal processing unit 111.

[0056] The recording unit 112 records RAW data, synthesized RAW data, developed image data, audio data accompanying these data, and the like in a recording medium such as a memory card or the like, in accordance with a shooting mode and recording settings.

[0057] "Operation unit 115" is a general term for input devices (switches, buttons, dials, touch panels, etc.) that enable users to give various instructions to the camera device 10.

[0058] The display unit 116 is, for example, a touch screen, and is used to display live view images, playback images, GUI, and settings and information of the camera device 10.

[0059] Figure 4 The flowchart illustrates the operation of the camera device 10 in the composite image capturing mode. The composite image capturing mode is a mode for recording composite images by combining images of multiple frames captured over time. In the composite image capturing mode, one of the aforementioned composite modes is set. Although this section describes still image capture, this process can also be performed as a process for generating a single frame of a captured motion image (in which the individual frames are used as a composite image).

[0060] In step S101, the control unit 113 receives the setting of the number of frames N to be synthesized from the user via the operation unit 115. The number of frames N can be preset in the same way as the setting of the synthesis mode.

[0061] In step S102, the control unit 113 detects that the user has input a shooting instruction through the operation unit 115. This shooting instruction can be made, for example, by fully pressing the shutter button included in the operation unit 115. Upon detecting the input of the shooting instruction, the control unit 113 executes step S103.

[0062] In step S103, the control unit 113 performs processing for capturing one frame. For example, the exposure conditions (shutter speed, aperture value, and ISO) at the time of shooting can be made to the appropriate exposure obtained by performing AE processing using the live view image before step S102 (e.g., when the shutter button is detected to be half-pressed). Note that when the composite mode is additive mode, the control unit 113 sets the exposure conditions to 1 / N of the appropriate exposure based on the number of frames N set in step S101.

[0063] The control unit 113 saves the RAW data acquired from the shooting to the memory 114. When shooting the first frame, the control unit 113 skips step S104 and executes step S105, and executes step S104 when shooting the second frame and subsequent frames.

[0064] In step S104, the misalignment detection unit 109 detects the amount of misalignment in the image (RAW data) obtained from the most recently captured instance. Here, it is assumed that the misalignment is detected in the image (reference image) in the first frame for all images in the second and subsequent frames. Template matching is used to detect the misalignment, where a portion of the reference image is used as a template.

[0065] Here, to improve the accuracy of misalignment detection, the misalignment detection unit 109 generates multiple templates from the reference image and detects the misalignment of each individual template. Then, the misalignment detection unit 109 determines the final misalignment based on the detected misalignments. The misalignment detection unit 109 may, for example, determine the final misalignment using the most frequent misalignment or the average of the misalignments, but it can also be determined using other methods.

[0066] In step S105, the OB integration unit 104 uses the pixel values ​​of the OB region 201 in the image (RAW data) obtained in the most recently captured instance to calculate the black level of each pixel type.

[0067] In step S106, the OB clamping unit 105 applies OB clamping processing to the pixels in the valid pixel region of the image (RAW data) obtained from the most recently captured instance using the black level calculated in step S105.

[0068] In step S107, the misalignment correction unit 110 uses the misalignment amount detected in step S104 to align the image (RAW data) obtained from the most recently captured instance with the reference image. The reference image is then used to... Figure 3 Describe the alignment process.

[0069] Figure 3 The compositing process is illustrated schematically, in which the image (RAW data) 302 of the second frame is aligned and composited with the image (RAW data) 301 of the first frame, which serves as the reference image. Image 301 has an effective pixel region 301a and an OB region 301b. Similarly, image 302 has an effective pixel region 302a and an OB region 302b.

[0070] Here, it is assumed that the misalignment correction unit 110 is set to align only the image within the valid pixel region. Therefore, in the aligned image 302', only the valid pixel region 302a' has moved, while the OB region 302b has not moved. As a result, a region 302c without pixels appears in the aligned image 302'. When the misalignment correction unit 110 is set to align the entire frame, the coordinate information is changed internally, but there is no significant change between the images 302 before and after alignment.

[0071] In step S108, the synthesis unit 108 synthesizes the image aligned in step S107 ( Figure 3 The image 302' in the composite is combined with the reference image or the composite image up to the previous frame (composite RAW data). The compositing unit 108 performs the compositing process as described above according to the pre-set mode.

[0072] Figure 3 An example of compositing processing is shown in an additive averaging mode or a comparative brightness mode, excluding the region 302c of non-existent pixels due to alignment issues from the composite. As a result, the effective pixel region 310a of the composite image 310 includes the region 310c of the second frame image that was not composited. For the OB region 310b of the composite image 310, the OB region 301b of the first frame and the OB region 302b of the second frame are composited using a mode-based method.

[0073] Note that if the pixel signal is affected by noise in the negative direction, the pixel value may become less than 0 when the black level is reduced through the OB clamping process in step S106. To prevent this, a predetermined positive offset can be added to the composite image before saving the composite image in memory 114. If an offset is added to the composite image, the image to be composited is given the same offset as the offset added to the composite image before applying the OB clamping process in step S106. After applying the OB clamping process and before applying the composite process in step S108, the offset is removed from both images.

[0074] although Figure 4 The flowchart illustrates sequential compositing relative to the reference image, starting from the second frame. However, it's possible to generate composite images from the second to the Nth frame, and then composite these images with the reference image at the end. In this case, compositing from the second to the Nth frame can be performed only in the overlapping areas across all frames. For the valid pixel areas of the reference image, specifically the portions from the second to the Nth frame where no composite images are generated, the method according to the compositing mode described above can be applied, assuming no pixel values ​​are available for compositing.

[0075] In step S109, the control unit 113 determines whether the shooting of the set number of frames N has been completed. If it is determined that the shooting is complete, the control unit 113 executes step S110; otherwise, it executes step S103 and shoots the next frame. For example, assuming that the control unit 113 continuously shoots images as long as the shooting instruction is continuously input. For example, if the input of the shooting instruction is stopped before the shooting of the set number of frames N is completed, the control unit 113 can discard the results of the processing up to this point and return to the shooting standby state.

[0076] In step S110, the OB integration unit 104 calculates the black level based on the pixel values ​​of the OB region of the composite image stored in the memory 114. Except that the pixel values ​​in the OB region are the composite pixel values, the calculation of the black level is the same as that in step S105.

[0077] In step S111, the OB integration unit 104 calculates a value (e.g., variance) indicating the variation of pixel values ​​in the OB region of the composite image. The variance value can be calculated for each pixel type (color), or a single variance value can be calculated for all pixels. Steps S110 and S111 can be performed in parallel by the OB integration unit 104.

[0078] In step S112, the OB clamping unit 105 applies OB clamping processing to the pixel values ​​in the effective pixel region of the composite image using the black level calculated in step S110 based on the composite image. Although OB clamping is applied to the image before compositing in step S106, the reason for applying OB clamping processing after compositing is to reduce black float caused by the compositing process.

[0079] For example, when combining multiple frames of images taken at high ISO in a brighter mode, each frame contains a significant amount of noise. The brighter mode composition selects the highest brightness value at each coordinate. Therefore, it is more likely that a brightness value higher than the original brightness value will be selected for each coordinate due to noise. Therefore, by applying OB clamping processing to the composite image in the same manner as to the effective pixel area 200, using the black level of the pixels in the OB area 201 (the composite OB area) already composed in the brighter mode, black float caused by composition can be suppressed. Although the preceding description has discussed composition in the brighter mode as a typical example of black float caused by composition, black float caused by composition also occurs in other composition modes; therefore, OB clamping processing using the black level of the pixels in the composite OB area is applied to the composite image.

[0080] In step S113, the WB processing unit 107 applies white balance adjustment processing to the effective pixel area of ​​the composite image (RAW data) after OB clamping processing.

[0081] In step S114, the signal processing unit 111 applies image processing to the RAW data that has already undergone white balance adjustment. The signal processing unit 111 changes the parameters of the noise reduction processing applied in the image processing based on the variance value calculated in step S111. Specifically, the signal processing unit 111 changes the noise reduction processing parameters such that if the variance value is greater than a predetermined threshold, a stronger noise reduction processing is applied compared to when the variance value is not greater than the predetermined threshold. This is because the noise level is higher when the variance value is greater than the threshold. If there are more than three different intensities of noise reduction processing, two or more thresholds can be used to adjust the intensity of the noise reduction processing. The signal processing unit 111 generates an image data file containing the image data from the image data after image processing.

[0082] In step S115, the recording unit 112 records the image data file generated in step S114 to a recording medium such as a memory card, an external device, or the like. Instead of the developed image data, or in addition to the developed image data, it can also record composite RAW data of the effective pixel region before the OB clamping process is applied in step S112. Information required for the external device to perform the processing in steps S112 to S114 (the black level calculated in step S110 and the variance calculated in step S111) can be added to the composite RAW data and recorded in this state. N frames of RAW data from the composite source can be recorded, allowing the external device to perform the processing in steps S104 to S114.

[0083] When recording by recording unit 112 ends, control unit 113 terminates. Figure 4 The operation shown is executed, and for example, the user returns to the shooting standby state.

[0084] As described above, according to this embodiment, when combining RAW data from multiple frames, the alignment of the RAW data can be applied separately to the data in the effective pixel area and the data in the OB area. By independently aligning the data in the effective pixel area, it is possible to prevent the data in the OB area from being combined with the data in the effective pixel area, thus suppressing the degradation of the quality of the combined image. Furthermore, since there is no misalignment in the OB area between the frames to be combined, the black level can be accurately calculated from the combined OB area, which effectively suppresses black floats in the image caused by the combination.

[0085] Second Embodiment

[0086] The second embodiment of the present invention will now be described. This embodiment relates to a case where line offset correction is performed in addition to OB clamping processing. The characteristics of the multiple circuit elements formed in an image sensor vary due to manufacturing errors, etc. For example, the difference in pixel signal offset caused by variations in the characteristics of amplifiers provided in each pixel line can be seen as a stripe pattern in the image.

[0087] Line offset correction is a process used to correct for differences in pixel signal offsets that may occur in units of horizontal pixel lines, vertical pixel lines, etc. The difference in offset of each pixel line can be detected from the pixels in the OB region. Therefore, line offset correction can be applied to the pixel signal read from the effective pixel region based on the difference in offset of each pixel line detected in the OB region.

[0088] However, if the alignment between the effective pixel region and the OB region is different, the position of the pixel line requiring line offset correction will be inconsistent between the OB region and the aligned effective pixel region. Specifically, when compositing images from multiple frames, the alignment between the OB region and the effective pixel region can be different for each frame. Therefore, the offset of each pixel line detected in the composited OB region will be inconsistent with the offset of the pixel signal in the composited effective pixel region, thus preventing proper line offset correction.

[0089] Reference Figure 5 This describes an example where proper line offset correction cannot be performed. Although line offset correction for horizontal pixel lines will be described here, it also applies to line offset correction for vertical pixel lines. When performing line offset correction for vertical pixel lines, use... Figure 2 The OB region 202 is shown.

[0090] Figure 5 The compositing process is schematically illustrated, in which the second frame image (RAW data) 302 is aligned and composited with the first frame image (RAW data) 301, which serves as the reference image, using alignment amounts applied respectively to the OB region and the effective pixel region. Image 301 has an effective pixel region 301a and an OB region 301b. Similarly, image 302 has an effective pixel region 302a and an OB region 302b.

[0091] It is also assumed here that the misalignment correction unit 110 is set to align only the image in the valid pixel region. Therefore, in the aligned image 302', only the valid pixel region 302a' has moved, and the OB region 302b has not moved. As a result, a region 302c with no pixels appears in the aligned image 302'.

[0092] In the first frame image 301 (reference image), two horizontal pixel lines with a signal offset that is larger (or smaller) by a set amount than the other horizontal pixel lines form a bright (or dark) stripe pattern 501. Similarly, in the second frame image 302, two horizontal pixel lines form a bright (or dark) stripe pattern 502.

[0093] Because the positions of the horizontal pixel lines in the image sensor are fixed, the stripe pattern always forms in the same position in the captured image. Therefore, before aligning the effective pixel area 302a of the image in the second frame with the effective pixel area 301a of the image in the first frame, the stripe pattern forms in the same position in the images in the first and second frames.

[0094] However, as a result of aligning the effective pixel region 302a with the effective pixel region 301a of the image in the first frame, the number and position of the stripe pattern in the image of the second frame change in the effective pixel region 302a' of the aligned image.

[0095] As a result, the effective pixel region 310a of the composite image 310 of the first frame and the second frame includes not only the stripe pattern 501 present in the effective pixel region 301, but also the stripe pattern 502' present in the effective pixel region 302a'.

[0096] On the other hand, focusing on the OB region, OB regions 301b in the first frame and 302b in the second frame are composited without alignment. Therefore, the number and position of the stripe patterns in OB region 310b in the composite image 310 are the same as before composite.

[0097] Line offset correction will be described here. Line offset correction includes detecting the presence and position of the stripe pattern and determining the amount of offset correction. Line offset correction can be performed, for example, by signal processing unit 111.

[0098] For example, the detection of the presence and location of a stripe pattern can be performed on a vertically extending sequence of pixel signals formed by averaging pixel values ​​in region OB 310b in the horizontal direction, and on the detection of pixels and their locations that have a signal level whose absolute value of the difference from a representative signal level is greater than or equal to a threshold. The representative signal level can be the average or median of the signal levels. The threshold can be determined experimentally beforehand.

[0099] If a pixel exists with a signal level whose absolute difference from the representative signal level is greater than or equal to a threshold, then the position of that pixel corresponds to the position of the pixel line to be corrected, and the difference between the signal level of that pixel and the representative signal level can be used as the offset correction amount. Note that the pixel line to be corrected and its position can also be detected. Different methods than those described herein can be used to determine the offset correction amount, etc.

[0100] For the composite image 310, it is conceivable to apply line offset correction to the effective pixel region 310a based on the stripe pattern detected in the OB region 310b. In this case, the stripe pattern 502' of the image (here, the image of the second frame) in which the vertical alignment of the effective pixel region in the image to be composited is retained in the effective pixel region 511a of the corrected image 511.

[0101] This embodiment addresses the aforementioned problem by generating a composite image for line offset correction. Specifically, a composite image for line offset correction is generated by aligning the OB regions of each frame image to be synthesized with the OB region of the reference image based on the alignment amount of the effective pixel regions.

[0102] Therefore, in this embodiment, the following content is generated.

[0103] • A composite image (first composite image) obtained by aligning and combining effective pixel areas.

[0104] • A composite image (second composite image) obtained by aligning and synthesizing the extracted OB regions for line offset correction.

[0105] • A composite image (third composite image) obtained by compositing the OB region without aligning the OB region, used for OB clamping (black level adjustment).

[0106] Figure 6 Showing in with Figure 5 The example shown uses a composite image for line offset correction in the same compositing process. Figure 6 As shown, the compositing unit 108 extracts the OB region 302b of the second frame image 302, which is composed of the first frame image 301 (reference image), and applies the vertical component of the alignment amount of the effective pixel region 302a to generate an aligned image 601. Then, the compositing unit 108 combines the aligned OB region image 601 with the OB region 301b extracted from the reference image to generate a composite image 602 for line offset correction.

[0107] Note that when compositing three or more frames, the compositing unit 108 extracts the OB region for each frame image other than the reference image, applies the vertical component of the alignment amount of the effective pixel region and aligns it, and then sequentially composites the image with the OB region extracted from the reference image. In this way, the compositing unit 108 can generate a composite image for line offset correction.

[0108] Note that when aligning the OB region, only the vertical component of the alignment amount for the effective pixel region is applied, because the line offset correction targets horizontal pixel lines. When the line offset correction targets vertical pixel lines, only the horizontal component of the alignment amount for the effective pixel region is applied to the OB region 202 of the frame image to align the OB region.

[0109] It should also be noted that because OB regions extracted from each frame are used, the composite image used for line offset correction does not affect the synthesis of OB regions in each frame. The synthesis unit 108 performs the synthesis for line offset correction separately from the synthesis of OB regions for OB clamping, the synthesis of effective pixel regions, etc.

[0110] In the composite image 602 used for line offset correction, a stripe pattern exists at the same location as the aligned and composite effective pixel region 310a. Therefore, the stripe pattern can be appropriately corrected by applying line offset correction to the composite effective pixel region 310a using the composite image 602 used for line offset correction. Figure 6 As shown, no stripe pattern exists in the effective pixel region 511a of the image 511' after online offset correction.

[0111] When the application uses line offset correction for the composite image 602, the signal processing unit 111 can use the region of the composite image 602 that is common to all OB regions used to generate the composite image 602. In this case, during line offset correction, the signal processing unit 111 does not use regions of OB regions that were only composited from a portion of the frame (such as...). Figure 6 (The area indicated by 602a, etc.).

[0112] Figure 7 This is a flowchart illustrating the operation of adding the line offset correction process described in this embodiment to the composite image capturing mode of the camera device 10 described in the first embodiment. Steps S701 to S708 are... Figure 4 Steps S101 to S108 are the same, and steps S711 to S714 are the same. Figure 4 Steps S109 to S112 are the same, and steps S716 to S718 are the same. Figure 4 Steps S113 to S115 are the same, so they will not be described.

[0113] In step S709, the control unit 113 instructs the signal processing unit 111 to extract the OB region from the most recently captured frame image and supply the OB region to the misalignment correction unit 110. When step S709 is executed first, the control unit 113 instructs the signal processing unit 111 to also extract the OB region of the reference image and supply the OB region to the synthesis unit 108. The signal processing unit 111 extracts the OB region from the frame image according to the instruction from the control unit 113 and supplies the OB region to the misalignment correction unit 110.

[0114] Control unit 113 further instructs misalignment correction unit 110 to apply the alignment processing to the OB region using the alignment amount applied to the effective pixel region in the alignment processing in step S707, and supplies the result to synthesis unit 108. Misalignment correction unit 110 applies the alignment processing to the OB region supplied by signal processing unit 111 and supplies the result to synthesis unit 108. As described above, in the alignment processing, misalignment correction unit 110 applies only the vertical component of the alignment amount when extracting OB region 201, and only the horizontal component of the alignment amount when extracting OB region 202. Misalignment correction unit 110 supplies the OB region to synthesis unit 108 with the alignment processing applied.

[0115] In step S710, the control unit 113 instructs the synthesis unit 108 to generate a composite image for line offset correction. The synthesis unit 108 also synthesizes the aligned OB region supplied by the unaligned correction unit 110 with the recently generated composite image stored in the memory 114. When step S710 is executed first, the synthesis unit 108 synthesizes the aligned OB region supplied by the unaligned correction unit 110 with the OB region of the reference image supplied from the signal processing unit 111. The synthesis unit 108 stores the composite image of the OB region in the memory 114.

[0116] The scenario described here involves performing alignment and compositing processes to generate a composite image for offset correction after the compositing process of the OB region 301b and the effective pixel region 302a of the frame image for recording is completed. However, the alignment and compositing processes for generating the composite image of the OB region for offset correction can also be performed in steps S707 and S708.

[0117] After the OB clamping process in step S714 is completed, in step S715, the control unit 113 instructs the signal processing unit 111 to perform line offset correction. The signal processing unit 111 reads the composite image for offset correction stored in the memory 114 and detects the presence and position of the stripe pattern. If no stripe pattern is detected, the signal processing unit 111 notifies the control unit 113 of the detection result (no pattern detected) and does not apply line offset correction. In this case, the control unit 113 executes step S716 and instructs the WB processing unit 107 to perform white balance adjustment processing.

[0118] On the other hand, if a stripe pattern is detected, the signal processing unit 111 detects the position of the stripe pattern and determines the offset correction value for each detected stripe pattern. Then, the signal processing unit 111 applies line offset correction to the composite image stored in the memory 114 that has undergone OB clamping. Subsequent processing is the same as in the first embodiment, and therefore will not be described here.

[0119] In this embodiment, an image with aligned and composited OB regions is generated for line offset correction. Then, using the composite image of the OB regions generated for line offset correction, the line offset correction is applied to the composite image of the effective pixel regions. This allows for appropriate correction of stripe patterns caused by variations in the circuitry characteristics of the image sensor, in addition to the effects of the first embodiment.

[0120] Other embodiments

[0121] The foregoing embodiments describe the generation of composite images during shooting in a camera device. However, as mentioned above, the shooting function is not essential in this invention. Therefore, the present invention can also be implemented in an image processing device that acquires a frame of RAW data captured and recorded over time and performs step S104 (S704) and subsequent processing, instead of capturing an image in step S103 (S703). In this case, the determination process in step S109 (S711) determines whether the set number of frames has been acquired.

[0122] Furthermore, the line offset correction described in the second embodiment does not need to be performed during shooting. For example, by recording a composite image of the OB region used for line offset correction in association with a composite image of the effective pixel region, line offset correction can be performed in a device different from the device used to record the image and at any desired timing.

[0123] Alternatively, by recording the OB region as RAW data, the generation of the composite image and line offset correction can be performed on a different device than the device used to record the image and at any desired timing.

[0124] One or more functional blocks described as separate from the image sensor 102 in the foregoing embodiments can be implemented in the image sensor 102. By making the image sensor 102 a stacked image sensor, the A / D conversion unit 103, memory 114, misalignment detection unit 109, misalignment correction unit 110, and synthesis unit 108 can be implemented in the image sensor 102.

[0125] In this case, a composite image of the effective pixel area can be generated by image sensor 102. A composite image of the OB area used for line offset correction can also be generated in image sensor 102. Image sensor 102 can output the data of the composite image of the effective pixel area and the data of the composite image of the OB area used for line offset correction as separate image data, or it can combine those image data into a single image data for output. If line offset correction can also be performed within the image sensor, then the image sensor does not need to output the data of the composite image of the OB area used for line offset correction.

[0126] Although the second embodiment describes always generating a composite image of the OB region for line offset correction, this composite image can be generated only when it is determined that line offset correction is needed. For example, whether line offset correction is needed can be determined by comparing shooting conditions (such as ISO sensitivity and shutter speed at the time of shooting) and environmental conditions (such as sensor temperature and ambient temperature) with predetermined thresholds.

[0127] Other embodiments

[0128] The embodiments of the present invention can also be implemented by providing software (programs) that perform the functions of the above embodiments to a system or device via a network or various storage media, and the computer or central processing unit (CPU) or microprocessor unit (MPU) of the system or device reads out and executes the program.

[0129] While the invention has been described with reference to exemplary embodiments, it should be understood that the invention is not limited to the disclosed exemplary embodiments. The scope of the appended claims should be accorded the broadest interpretation to cover all such modifications and equivalent structures and functions.

Claims

1. An image processing apparatus, comprising: The acquisition component is used to acquire image data from multiple frames. An alignment component is used to align the image data of the plurality of frames; as well as A compositing component is used to synthesize image data from the already aligned plurality of frames. The image data includes first data and second data. The first data consists of pixel data from the effective pixel area of ​​the image sensor and is composed of signals corresponding to a predetermined color component arrangement. The second data consists of pixel data from outside the effective pixel area of ​​the image sensor. The alignment component is capable of aligning the first data independently of the second data, and The synthesis component processes pixels of the second data whose coordinates have been changed to the position in the region overlapping with the first data through the alignment as having no pixel value.

2. The image processing apparatus according to claim 1, in, The alignment component aligns the first data but not the second data.

3. The image processing apparatus according to claim 1, in, The synthesis component synthesizes the first data of the plurality of frames and the second data of the plurality of frames, and the method for synthesizing the first data is the same as the method for synthesizing the second data.

4. The image processing apparatus according to claim 1, in, The second data is the data of the unexposed pixels.

5. The image processing apparatus according to claim 4, in, The second data is used to adjust the black level of the first data.

6. The image processing apparatus according to claim 5, in, The adjustment of the black level is performed on the individual image data of the multiple frames and the composite image data synthesized by the compositing component.

7. The image processing apparatus according to any one of claims 1 to 6, further comprising: A calculation unit is used to calculate values ​​representing changes in the second data, which represent the composite image data synthesized by the compositing unit. The intensity of the noise reduction processing applied in the imaging processing of the synthetic image data is determined based on the value representing the change.

8. The image processing apparatus according to claim 1, in, The alignment component aligns the second data included in the image data of the plurality of frames based on an alignment amount used to align the first data included in the image data of the plurality of frames. The compositing component generates data for a first composite image obtained by compositing the aligned first data, and generates data for a second composite image obtained by compositing the aligned second data. The image processing device further includes: A correction component is used to correct the data of the first synthetic image based on the data of the second synthetic image.

9. The image processing apparatus according to claim 8, in, The correction component corrects the stripe pattern that appears in the first composite image in units of pixel lines due to the image sensor used to generate the image data of the plurality of frames.

10. The image processing apparatus according to claim 9, in, When the correction component corrects the stripe pattern appearing in the horizontal pixel line, the alignment component uses the horizontal component and the vertical component of the alignment amount used to align the first data included in the image data of the plurality of frames to align the second data.

11. The image processing apparatus according to claim 9, in, When the correction component corrects the stripe pattern appearing in the vertical pixel line, the alignment component uses the horizontal component of the alignment amount used to align the first data included in the image data of the plurality of frames and the horizontal component of the vertical component to align the second data.

12. The image processing apparatus according to any one of claims 8 to 11, further comprising: A recording component for recording data of the first synthesized image in association with data of the second synthesized image.

13. The image processing apparatus according to any one of claims 8 to 11, in, The compositing component also generates data for a third composite image obtained by compositing the second data that has not yet been aligned.

14. The image processing apparatus according to claim 13, in, The data from the third composite image is used to adjust the black level of the data from the first composite image.

15. The image processing apparatus according to any one of claims 8 to 11, in, The second data is the data of the unexposed pixels.

16. A camera device, comprising: Image sensor; as well as The image processing apparatus according to any one of claims 1 to 15, The image data of the multiple frames are image data captured over time using the image sensor.

17. An image processing method, comprising: Obtain image data from multiple frames; Align the image data of the multiple frames; as well as The image data of the multiple aligned frames are then synthesized. The image data includes first data and second data. The first data consists of pixel data from the effective pixel area of ​​the image sensor and is composed of signals corresponding to a predetermined color component arrangement. The second data consists of pixel data from outside the effective pixel area of ​​the image sensor. Wherein, the alignment can be performed independently of the second data on the first data, and In this process, the pixels of the second data whose coordinates have been changed in the alignment to the region overlapping with the first data are treated as having no pixel value.

18. The image processing method according to claim 17, in, The alignment includes aligning the second data included in the image data of the plurality of frames based on an alignment amount used to align the first data included in the image data of the plurality of frames. The compositing includes generating data for a first composite image obtained by compositing the aligned first data, and generating data for a second composite image obtained by compositing the aligned second data. The image processing method further includes correcting the data of the first synthetic image based on the data of the second synthetic image.

19. A computer-readable storage medium storing a program that, when executed by a computer, implements the steps of the image processing method according to claim 17 or 18.

20. A computer program product comprising a program that, when executed by a computer, implements the steps of the image processing method according to claim 17 or 18.

Images