Image processing apparatus, imaging apparatus, and image processing method

Abstract

To provide an image processing device, an imaging device and an image processing method, capable of suppressing deterioration in quality of a composite image due to a difference in a state of environmental light when photographing an image to be synthesized.SOLUTION: An image processing device detects a motion vector between a reference frame and a current frame, and generates a composite image by aligning and combining the reference frame and the current frame based on the motion vector. When the color of the environmental light differs between when shooting the reference frame and when shooting the current frame, the image processing device corrects information used for alignment according to the color of the environmental light.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an imaging apparatus, and an image processing method, and particularly to an image synthesis technique.

Background Art

[0002] There is known a technique for generating an image with appropriate exposure by synthesizing a plurality of frames of images obtained by photographing the same scene under conditions of insufficient exposure (Patent Document 1).

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] When photographing a scene where the color of ambient light can change greatly in a short time (for example, scenes such as events, stages, attractions, etc.), the color tone of the image can differ depending on the timing of shooting. The difference in color tone between frames can reduce the detection accuracy of movement between frames, that is, the alignment accuracy between frames, and thus becomes a factor reducing the quality of the synthesized image.

[0005] In one aspect of the present invention, there is provided an image processing apparatus, an imaging apparatus, and an image processing method capable of suppressing a decrease in the quality of a synthesized image due to a difference in the state of ambient light at the time of shooting the images to be synthesized.

Means for Solving the Problems

[0006] The above-mentioned objective is to provide a detection means for detecting motion vectors between a reference frame and the current frame, an alignment means for aligning the reference frame and the current frame based on the motion vectors, a generation means for generating a composite image by combining the frames aligned by the alignment means, and a correction means for correcting the information used for alignment according to the color of the ambient light when the ambient light color differs between the time the reference frame was captured and the time the current frame was captured. The system includes a control means that supplies a coefficient to a correction means corresponding to the type of light source that was lit during the capture of the reference frame and the current frame from among multiple light sources with different spectral distributions, and the correction means uses the coefficient to correct the image data of the reference frame and / or the current frame, which is information used for alignment, and supplies it to a detection means. This is achieved by an image processing device characterized by the above. [Effects of the Invention]

[0007] According to the present invention, it is possible to provide an image processing apparatus, an imaging apparatus, and an image processing method that can suppress the deterioration of the quality of the composite image due to differences in ambient light conditions when capturing the images to be synthesized. [Brief explanation of the drawing]

[0008] [Figure 1] Schematic diagram of the entire imaging system according to the embodiment [Figure 2] Block diagram showing an example of the functional configuration of the imaging device according to the embodiment. [Figure 3] Block diagram showing an example of the functional configuration of a computer according to the embodiment. [Figure 4] Flowchart relating to the operation of the imaging system in the first embodiment [Figure 5] Diagram relating to the correction process in the first embodiment [Figure 6] Figure relating to shading correction in the first embodiment [Figure 7] Flowchart showing the details of processes S404-S405 in Figure 4 [Figure 8] Figure relating to motion detection in the first embodiment [Figure 9] Block diagram showing an example of the functional configuration of a computer according to the second embodiment. [Figure 10] Flowchart relating to the operation of the imaging system in the second embodiment [Modes for carrying out the invention]

[0009] The present invention will be described in detail below with reference to the attached drawings, based on exemplary embodiments thereof. Note that the following embodiments do not limit the invention to the claims. Furthermore, while multiple features are described in the embodiments, not all of them are essential to the invention, and the multiple features may be combined arbitrarily. In addition, in the attached drawings, the same or similar configurations are given the same reference numeral, and redundant descriptions are omitted.

[0010] In the following embodiments, the present invention will be described in relation to its implementation using computer equipment (personal computers, tablet computers, media players, PDAs, etc.). However, the present invention can be implemented using any electronic device capable of handling image data. Such electronic devices include imaging devices (digital cameras), smartphones, game consoles, robots, drones, and dashcams. These are examples, and the present invention can be implemented using other electronic devices as well.

[0011] ●(First Embodiment) Figure 1 is a schematic diagram showing an example of an imaging system 1 using a computer (hereinafter referred to as PC) 301 as an example of an image processing device according to this embodiment. The imaging system 1 includes an imaging device 201, an illumination device 101, a proximity sensor 102, and a PC 301. The imaging system 1 of this embodiment photographs a subject 100 that is located within a specific range, and the illumination range of the illumination device 101 and the shooting range of the imaging device 201 are basically fixed.

[0012] The lighting device 101 has a plurality (here, three) of light sources with different emission colors (spectral distributions). The type of the light source is arbitrary, but here it is assumed to be a light source capable of short-time emission like a flash. The lighting device 101 is arranged to illuminate a predetermined range. Although the lighting and extinguishing of each light source of the lighting device 101 are controlled by the PC 301, the PC 301 only controls the timing of lighting, and the combination of the light sources to be lit (lighting pattern) may be controlled by the lighting device 101. In any case, the PC 301 can grasp which light source the lighting device 101 lights at what timing.

[0013] The imaging device 201 is communicably connected to the PC 301. The PC 301 can remotely control the operation of the imaging device 201. Also, the PC 301 can acquire the image data acquired by the imaging device 201 through imaging from the imaging device 201.

[0014] The proximity sensor 102 separately detects that the subject 100 has approached and that the subject 100 exists at a predetermined imaging position, and notifies the detection result to the PC 301. The proximity sensor 102 may be an infrared sensor, a weighing scale installed on the floor, a proximity wireless communicator, or the like. The proximity sensor 102 uses, for example, the distance of an object detected by the infrared sensor, the weight measured by the weighing scale installed on the floor, or the fact that the proximity wireless communicator has communicated with an external device to detect that the subject 100 has approached or that the subject 100 exists at a specific position. The communication by the proximity wireless communicator assumes communication with a device (such as a smartphone) possessed by the subject 100.

[0015] In the example shown in FIG. 1, the PC 301 controls the operations of the lighting device 101 and the imaging device 201. However, the operations of the lighting device 101 and the imaging device 201 may be controlled by each device itself, or may be controlled by an external device different from the PC 301. Also, the proximity sensor 102 may be connected to the imaging device 201.

[0016] FIG. 2 is a block diagram showing an example of the functional configuration of the imaging device 201. The imaging optical system 219 is composed of a plurality of lenses including a movable lens. In FIG. 2, the focus lens 202 is shown as the movable lens, but movable lenses for changing the angle of view or correcting image blur may also be included. The focus lens 202 is driven by a motor 212 controlled by a lens movement circuit 211. Although not shown in FIG. 2, the imaging optical system 219 may have an aperture and an actuator for adjusting the aperture degree of the aperture. When having an aperture, the aperture may be used as a mechanical shutter.

[0017] The imaging optical system 219 forms an optical image on the imaging surface of the imaging device 203. The imaging device 203 may be a known CCD or CMOS color image sensor having a color filter of a primary color Bayer array, for example. The imaging device 203 has a pixel array in which a plurality of pixels are two-dimensionally arranged. Each pixel accumulates charges corresponding to the incident light amount by photoelectric conversion. The imaging circuit 204 reads out signals having voltages corresponding to the amounts of charges accumulated during the exposure period from each pixel, thereby obtaining a group of pixel signals (analog image signals) representing the subject image formed on the imaging surface.

[0018] The imaging device 203 is configured to be movable in each of the horizontal, vertical, and rotational directions by a sensor movement motor 214, and realizes a sensor shift type blur correction function. The operation of the sensor movement motor 214 is controlled by a sensor movement circuit 213.

[0019] The imaging circuit 204 performs noise reduction processing, gain adjustment processing, etc. on the analog image signal read from the imaging device 203, and then outputs it to an A / D conversion circuit 205. The A / D conversion circuit 205 converts the analog image signal input from the imaging circuit 204 into a digital image signal. The A / D conversion circuit 205 outputs the digital image signal to a RAM 206, an AE processing circuit 207, an AF processing circuit 208, and a communication driver 220.

[0020] The control unit 215 has rewritable non-volatile memory (ROM), volatile memory (RAM), and a program-executable processor (CPU). The control unit 215 controls the operation of each block of the imaging device 201 and realizes the functions of the imaging device 201 by loading programs stored in ROM into RAM and executing them with the CPU. In addition to programs that the CPU can execute, the ROM stores setting values ​​for the imaging device 201, graphical user interface (GUI) data such as menu screens, etc. The RAM is used to load programs that the CPU will execute and to store values ​​necessary during program execution.

[0021] The timing generator (TG) 216 supplies timing signals to the imaging circuit 204 and the sensor driver 217, in accordance with the control of the control unit 215. The sensor driver 217 controls operations such as charge accumulation and readout of the image sensor 203. The imaging circuit 24 and the sensor driver 217 perform predetermined operations based on the timing signals supplied from the TG 216.

[0022] The AE processing circuit 207 generates evaluation values ​​used for automatic exposure control (AE) based on digital image data and supplies them to the control unit 215. The AF processing circuit 208 generates evaluation values ​​used for autofocus detection (AF) based on digital image data and supplies them to the control unit 215. The control unit 215 can set how the AE processing circuit 207 and the AF processing circuit 208 use the digital image data to generate evaluation values, depending on the settings of the imaging device 201, etc. The control unit 215 can also apply subject detection processing, etc., to the digital image signal stored in the RAM 206 as needed. The control unit 215 may also use the results of the subject detection processing to generate evaluation values ​​in the AE processing circuit 207 and the AF processing circuit 208.

[0023] The control unit 215 determines the exposure conditions based on the AE evaluation value, for example, by referring to a program diagram pre-stored in ROM. If the imaging optical system 219 does not have an aperture, the control unit 215 can fix the aperture value to wide open and determine the exposure time and sensitivity according to the AE evaluation value.

[0024] The AF processing circuit 208 generates evaluation values ​​for contrast AF or phase-detection AF. When generating evaluation values ​​(defocus amount) for phase-detection AF, the image sensor 204 must be capable of generating a signal for phase-detection. Alternatively, a configuration using a signal obtained from a dedicated AF sensor may be used. In any case, the AF processing circuit 208 can generate AF evaluation values ​​based on known technology.

[0025] Assuming that the AF processing circuit 208 generates evaluation values ​​for phase-detection AF, the control unit 215 determines the drive direction and drive amount of the focus lens based on the defocus amount. Note that when the shooting distance of the subject 100 is constant and there is no need to adjust the focus, the generation of AF evaluation values ​​and the driving of the focus lens are unnecessary.

[0026] The motion sensor 210 is a gyroscope or accelerometer, and outputs signals corresponding to the translational and rotational movements of the imaging device 201 around its axis. The blur detection circuit 209 processes the output of the motion sensor 210 to calculate the magnitude of the translational and axial movements of the imaging device 201 and outputs it to the control unit 215.

[0027] The control unit 215 drives the image sensor 203 through the sensor movement circuit 213 to counteract the movement of the imaging device 201 input from the blur detection circuit 209.

[0028] In this embodiment, the imaging device 201 can perform AE and AF processing (shooting preparation operation) in response to instructions from the PC 301 or detection of the subject 100 by the proximity sensor 102, and then execute shooting processing including image stabilization.

[0029] The communication driver 220 is the communication interface between the imaging device 201 and the PC 301. The communication driver 220 may be a wired and / or wireless interface that the PC 301 generally has, such as a USB interface or a wireless LAN interface. In this embodiment, digital image signals, evaluation values, and the movement of the imaging device 201 are supplied from the imaging device 201 to the PC 301 through the communication driver 220.

[0030] Next, the image processing device PC301 will be described using Figure 3. Here, the PC301 is assumed to consist of a main unit 302 and external devices 330 to 332. However, the external devices 330 to 332 may be built into the main unit 302.

[0031] The external devices connected to the main unit 302 are a display device 330, an operating member 331, and a storage device 332. The display device 330 is, for example, a monitor using a liquid crystal display or an organic EL display, and may have a touch panel function. The operating member 331 is, for example, a keyboard and a pointing device (mouse, trackpad, etc.). The storage device 332 may be, for example, a hard disk drive or an SSD. The storage device 332 stores programs that the CPU 311 can execute, such as the OS and applications, as well as various settings and data files.

[0032] The communication driver 320 is a communication interface with the imaging device 201. The communication driver 320 may be a communication interface of the same standard as the communication driver 220 of the imaging device 201. The communication driver 320 is also a communication interface with the illumination device 101 and the proximity sensor 102.

[0033] The CPU 311 controls the operation of the PC 301's functional blocks and connected external devices by loading programs stored in, for example, the ROM 318 or storage device 332 into the system memory 319 and executing them, thereby realizing the operation of the PC 301 described later. In this embodiment, the CPU 311 executes an image processing application on a predetermined basic software (OS) to realize the operation described later.

[0034] ROM318 is, for example, a rewritable non-volatile memory that stores programs that the CPU311 can execute, as well as settings for the PC301 and the user. System memory 319 is used to load programs that the CPU will execute and to store values ​​that are needed during program execution.

[0035] RAM312 temporarily stores digital image signals (image data) received from the imaging device 201. A portion of RAM312 is also used as video memory for the display device 330.

[0036] The correction circuit 313 corrects the image data stored in the RAM 312 based on the light source information provided by the CPU 311. The correction circuit 313 outputs the corrected image data to the motion vector detection circuit 314 and the subject detection circuit 315.

[0037] The motion vector detection circuit 314 uses image data corrected by the correction circuit 313 for multiple frames to detect motion vectors between frames. In this embodiment, the motion vector detection circuit 314 divides a reference frame (for example, the most recently captured frame) into multiple blocks and detects the motion vector between the current frame (the frame to be detected) and each block. The motion vector can be detected by a known method, for example, template matching using the blocks of the reference frame as a template to search for the region with the highest correlation within the current frame. The motion vector detection circuit 314 outputs the detected motion vector information to the subject detection circuit 315.

[0038] The subject detection circuit 315 applies subject detection processing to the current frame. The subject detection circuit 315 detects a predetermined type of subject and outputs the position and size of the subject area, detection confidence level, etc., to the CPU 311 as detection results.

[0039] The subject detection circuit 315 can be fitted with any known subject detection process. However, if a main subject is set in the imaging device 201 and information regarding the main subject area can be obtained as supplementary information to the image data received by the communication driver 320, then it is not necessary to apply the subject detection process.

[0040] If information about the main subject cannot be obtained from the imaging device 201, the subject detection circuit 315 applies subject detection processing to the image data stored in the RAM 312 or the image data output by the correction circuit 313. For example, the subject detection circuit 315 can detect the area of ​​the subject by using methods such as detecting characteristic areas unique to the subject from the color of pixels or the contour of the area, or by detecting characteristic parts of the subject using a pre-prepared template.

[0041] If the subject to be detected is a face, the eyebrows, eyes, mouth, etc., are detected as feature regions, and the face region can be detected based on the positional relationship of the feature regions and the color of the region containing the feature regions. Furthermore, if a face region is detected, the torso and limb regions can be detected based on the shape of the regions connected to the face region, thereby detecting the human body region.

[0042] If multiple subject regions are detected in a single frame, the subject detection circuit 315 may determine the primary subject region by a known method. For example, the subject detection circuit 315 can determine the primary subject region as the subject region that satisfies one or more conditions, such as being the largest in size, being closest to the center of the screen, and having the highest detection confidence. It may also assign a ranking of likelihood of each subject region being the primary subject region. The subject detection circuit 315 outputs this determined information as part of the detection results to the CPU 311.

[0043] The image deformation and cropping circuit 316 applies rotation, scaling, geometric deformation, trimming, etc., to the supplied image data according to the instructions of the CPU 311.

[0044] The light source characteristic detection circuit 317 detects the characteristics of the ambient light (illumination light from the lighting device 101) during shooting (spectral distribution, white balance coefficient value, color temperature, etc.). Details of ambient light characteristic detection will be described later.

[0045] Next, the operation of the imaging system 1 will be explained using the flowchart shown in Figure 4. Here, the imaging device 201 is set to an operating mode that performs operations in cooperation with the PC 301, and the image processing application is running on the PC 301, making it possible to operate in cooperation with the illumination device 101 and the imaging device 201.

[0046] In S401, the control unit 215 of the imaging device 201 and the CPU 311 of the PC 301 perform initialization operations as needed. For example, the imaging device 201 moves the focus lens 202 and the image sensor 203 to their initial positions. Information may also be exchanged between the imaging device 201 and the PC 301, or between the illumination device 101 and the PC 301. Once the initialization operation is complete, the imaging device 201 waits for instructions from the PC 301.

[0047] In S402, the CPU 311 waits for input from the proximity sensor 102 indicating that a subject has been detected in close proximity. Upon detecting input from the proximity sensor 102, the CPU 311 instructs the imaging device 201 to perform pre-shooting via the communication driver 320. Alternatively, instead of using the proximity sensor 102, the imaging device 201 may output a live view image to the PC 301, and the subject detection circuit 315 may apply subject detection processing to the live view image to detect that the subject has entered (approached) the shooting range.

[0048] In S403, the control unit 215 performs pre-shooting in response to instructions from PC301. Pre-shooting is the process of acquiring still images or video data for multiple frames so that PC301 can determine the shooting conditions. Therefore, it is not necessary to shoot at the maximum possible resolution; for example, a video for live view may be sent to PC301.

[0049] Furthermore, since blur or out-of-focus images are acceptable as long as they do not hinder the determination of shooting conditions, priority can be given to executing AE and AF processing at high speed. Also, when performing blur correction, it is not necessary to initialize the position of the image sensor 205 for each frame. The control unit 215 sequentially transmits the image data for the specified number of frames from the PC 301 to the PC 301 via the communication driver 220.

[0050] In S404, the CPU 311 begins receiving image data from the imaging device 201 via the communication driver 320 and sequentially stores it in the RAM 312. The CPU 311 then uses the subject detection circuit 315 and the motion vector detection circuit 314 to predict the movement of the subject. Note that the image data obtained in pre-shooting does not necessarily need to be corrected by the correction circuit 313 (but it may be).

[0051] The subject detection circuit 315 sequentially applies subject detection processing to the image data stored in the RAM 312 and outputs the results of the subject detection processing to the CPU 311 for each frame. Based on the results of the subject detection processing, the CPU 313 instructs the motion vector detection circuit 314 to detect motion vectors for both the subject area and the background area.

[0052] The motion vector detection circuit 314 detects motion vectors for individual subject areas and the background between the current frame and a reference frame (for example, the frame immediately preceding the current frame), and outputs them to the CPU 311.

[0053] The CPU 311 predicts the movement of the subject based on the motion vector detected by the motion vector detection circuit 314.

[0054] In S405, the CPU 311 determines the shooting conditions (exposure time and sensitivity) for the actual shooting and transmits them to the imaging device 201 via the communication driver 320, along with a predetermined number of composite frames. Details of the operation of S405 will be described later.

[0055] In S406, the CPU 311 determines whether or not it has received input from the proximity sensor 102 indicating that the subject 100 has reached the shooting position. If it determines that it has, it proceeds to S407; otherwise, it continues processing the image data received from the imaging device 201 from S404 onwards.

[0056] In S407, the CPU 311 instructs the imaging device 201 to take a picture. The control unit 215 of the imaging device 201 takes a specified number of pictures according to the shooting conditions instructed by the PC 301. In S407, for this shooting, the control unit 215 prioritizes image quality, generally performing continuous shooting of still images. AF processing and image stabilization also prioritize image quality over processing speed. In image stabilization, continuous shooting is treated as a single shot, and the position of the image sensor 203 is not initialized between frames. The control unit 215 transmits image data to the PC 301 in parallel with the shooting.

[0057] In S408, the CPU 311 controls the correction circuit 313 to apply a correction to the image data received from the imaging device 201 and stored in the RAM 312, according to the ambient light characteristics detected by the light source characteristic detection circuit 317. Details of the ambient light characteristic detection process and correction process will be described later.

[0058] In S409, the CPU 311, similar to S404, uses the subject detection circuit 315 and the motion vector detection circuit 314 to acquire the motion vector between the subject area and the background for each frame from the second frame onward.

[0059] In S410, the CPU 311 uses the image deformation and extraction circuit 316 to perform inter-frame alignment based on the motion vector acquired in S409. The image deformation and extraction circuit 316 sequentially aligns the images from the second frame onward with the image of the first frame (reference image). The image deformation and extraction circuit 316 may also cut out or deform parts of the image as needed. The aligned image data is sequentially stored in the RAM 312.

[0060] In S411, CPU311 determines whether (number of composite frames - 1) alignments have been completed. If it determines that the alignments have been completed, it executes S412; otherwise, it repeats the process from S407. Since no alignment is performed on the reference image, the number of alignments is 1 less than the number of composite frames.

[0061] In step S412, the CPU 311 sequentially supplies the reference image data and the aligned image data from the RAM 312 to the image deformation and extraction circuit 316 to generate a composite image. The image deformation and extraction circuit 316 generates the composite image by averaging the image data pixel by pixel using the number of composite elements. The CPU 311 records the generated composite image data in the storage device 332.

[0062] (Correction process) Using Figures 5 and 6, the details of the image data correction process performed in S408 (and possibly S404) will be explained. In this embodiment, the correction process is performed based on the characteristics of the color filter of the image sensor 203. Here, as an example, the case in which the color filter is a primary color Bayer array will be explained. In the primary color Bayer array, rows in which R (red) and G1 (green) are arranged alternately and rows in which G2 (green) and B (blue) are arranged alternately are arranged alternately, with R, G1, G2, and B corresponding to 2 pixels horizontally and 2 pixels vertically as repeating units.

[0063] Figure 5 shows an example of the response characteristics of four pixels corresponding to the repeating units R·G1·G2·B of a primary color Bayer array color filter. Figure 5(A) shows the response characteristics of the four pixels (relative output level relationship) when receiving reference white light through the imaging optical system 219. The gray area indicates the magnitude of the output level.

[0064] Therefore, the response characteristics are measured in advance for each lighting pattern of ambient light (mainly the light that the lighting device 101 uses to illuminate the subject), and coefficients are calculated to convert them to a reference response characteristic. Then, by applying the coefficient corresponding to the lighting pattern of the lighting device 101 when the image was captured to the pixel values ​​according to the filter color, differences in ambient light color can be corrected. The response characteristics can be obtained, for example, from image data obtained by photographing a white screen illuminated by the lighting device 101.

[0065] For example, suppose that the response characteristics shown in Figures 5(B) to 5(D) are obtained for the first to third light sources of the lighting device 101. In this case, a coefficient to be applied to the pixel value is calculated for each type of light source so that the response characteristics of the pixels equipped with the R·G1·G2·B color filters are the same as in Figure 5(A). For example, R501 / R502 is obtained as a conversion coefficient for the R pixel value when the first light source is lit. Even when multiple light sources are lit simultaneously, a conversion coefficient can be determined for each combination of light sources. Alternatively, a conversion coefficient obtained by averaging the conversion coefficients for each individual light source may be used.

[0066] Although the reference ambient light color is set to white, it may also be the ambient light color corresponding to one of the lighting patterns of the lighting device 101. Basically, it is sufficient to correct the image data so that it is image data taken under ambient light of the same color. Therefore, the image data of the reference frame may be corrected to the ambient light color at the time the current frame for motion detection was taken, or conversely, the image data of the current frame may be corrected.

[0067] Furthermore, shading (vignetting) of the imaging optical system 219 and the image sensor 203 may be reflected in the conversion coefficient. As shown in Figure 6, shading tends to decrease as the distance from the optical axis or the intersection of the optical axis and the screen increases in both the X (horizontal) and Y (vertical) directions. The shading characteristics differ depending on the combination of the imaging optical system and the image sensor. Therefore, similar to the response characteristics due to the light source, the shading characteristics can be measured in advance, and a correction coefficient to remove the effect of shading can be determined.

[0068] The conversion coefficients for each lighting pattern of the lighting device 101 are assumed to be pre-stored in the light source characteristic detection circuit 317. When shading is considered, the conversion coefficients corresponding to the increase in pixel height are stored for each lighting pattern. Alternatively, the conversion coefficients and the coefficients for correcting shading may be stored separately in the light source characteristic detection circuit 317, and the correction circuit 313 may apply the conversion coefficients and the shading correction coefficients corresponding to the increase in pixel height.

[0069] In this embodiment, the CPU 311 controls the light emission of the illumination device 101. By synchronizing the shooting timing of the imaging device 201 with the light emission timing of the illumination device 101, the lighting pattern of the illumination device during the shooting of each frame can be identified. Therefore, the CPU 311 can obtain a coefficient corresponding to the lighting pattern during the shooting of the frame to be corrected from the light source characteristic detection circuit 317 and supply it to the correction circuit 313.

[0070] For example, the correction circuit 313 can apply correction by multiplying the product of two conversion coefficients by the individual pixel values ​​of the image data for one frame stored in the RAM 312. The corrected output signal is then recorded in a predetermined area in the RAM 312.

[0071] As characteristics of ambient light, the ratio of pixel output levels according to the type of color filter was used, but spectral characteristics, white balance coefficient values, color temperature, and the central wavelength and full width at half maximum of the illumination may also be used.

[0072] When using spectral characteristics, the conversion coefficient for each pixel can be determined from the specific energy value for each wavelength of light emitted from the emitted light source and the response characteristics of each pixel to the wavelength of light. Since the white balance coefficient is a coefficient used to determine the color under a white light source, the white balance coefficient corresponding to the color of the color filter can be used as the conversion coefficient. When using color temperature, similar to the white balance coefficient value, the conversion coefficient for each pixel can be determined from the value that represents the original color under a white light source. When using the center wavelength and full width at half maximum (FMAX) of the illumination, the response ratio at the center wavelength is set to 1, and the response ratio at the FMAX wavelength is set to 0.5. A coefficient is stored that is a simplified calculation of the response characteristics obtained by interpolating between these two values. By comparing the wavelength that is the center of the response characteristics of each pixel with the stored coefficient, the conversion coefficient for each pixel can be determined.

[0073] (Determining shooting conditions) This section describes the process for predicting the subject's movement and determining the shooting conditions (exposure time, sensitivity) in S404 and S405.

[0074] Figure 7 is a flowchart detailing the operation of S404 and S405. In S701, the CPU 311 instructs the subject detection circuit 315 to apply subject detection processing to the frame to be processed. As described above, the subject detection circuit 315 outputs the detection result to the CPU 311.

[0075] In S702, CPU311 determines whether a subject area that meets the predetermined confidence criteria in S701 has been detected. If it is determined that a subject area has been detected, it executes S703; otherwise, it executes S711. If multiple subject areas are present, the determination is made for the primary subject area. Below, for the sake of clarity and ease of understanding, we will explain the case where only one subject area is detected.

[0076] In S703, the CPU 311 sets the subject area as the area to be detected for motion detection. For example, as shown in Figure 8(A), suppose a subject area (in this case, the area of ​​a person) that meets a predetermined confidence level is detected near the center of the screen. In this case, the smallest rectangular area (shown in gray) that encloses the area of ​​the person is divided into multiple blocks, and the motion vector detection circuit 314 detects the motion vector for each block.

[0077] Figure 8(A) shows arrows indicating the motion detected for each block within the subject area. The size of the arrow indicates the magnitude of the motion, and its direction indicates the direction of the motion in the X and Y directions of the screen. The CPU 311 performs a known clustering process based on the detected amount of motion and selects blocks with similar motion. Through the clustering process, the second-to-last block from the right edge, which has a different direction of motion from the other blocks, is excluded from the blocks in which motion was detected.

[0078] The CPU 311 averages the motion detected for the selected block to determine the motion of the subject area. The size of the block is predetermined, taking into account the motion detection accuracy of the motion vector detection circuit 314.

[0079] Meanwhile, in S711, the CPU 311 divides the entire screen into multiple blocks, and the motion vector detection circuit 314 detects the motion vector for each block. Figure 8(B) shows an example where the entire screen is divided into 9 sections in the X direction and 7 sections in the Y direction.

[0080] In S712, CPU311 performs clustering based on motion vectors to separate the moving subject area from the background area. In Figure 8(B), the gray blocks without arrows indicate areas where motion cannot be detected (or where the reliability of detection does not meet the criteria). Through clustering, CPU311 considers the shaded areas as the subject area and calculates the motion of the subject area by averaging the motion of each block.

[0081] In S704, CPU311 analyzes the movement of the subject area detected in S703 or S712. From the time proximity is detected in S402 until arrival at the shooting position is detected in S406, steps S403 to S405 are repeatedly executed, allowing for continuous detection of the subject area's movement.

[0082] The CPU311 performs frequency analysis on the continuously detected movement of the subject area to understand the trend of the subject's movement. Specifically, the CPU311 extracts the amount of movement at the central frequency and the surrounding frequencies of the subject area's movement. The central frequency of movement is determined by the frequency analysis to be the frequency with the highest frequency. The mean and standard deviation of the frequencies that appeared during the period are then calculated, and the range of the central frequency ± standard deviation is defined as the surrounding frequency band. The frequency can be expressed in units of, for example, 0.1 Hz.

[0083] In S705, the CPU 311 calculates the difference in image magnification (Δβ) based on the distance between the point where proximity was detected and the shooting position. If the shooting position is closer to the imaging device 201 than the point where proximity was detected, the image magnification at the shooting position will be greater. Since the magnitude of motion on the screen is affected by the image magnification, the motion predicted in S403 to S405 will be Δβ times at the shooting position.

[0084] Next, in S706, the predicted amount of movement at the shooting point is calculated. The average value of (frequency of occurrence of that frequency × amount of movement at that frequency × frequency) is found for the center frequency and surrounding frequency bands of the subject's movement, and this value is multiplied by an appropriate difference Δβ between the magnification and image magnification, taking into account unexpected subject movement, to obtain the predicted amount of movement at the shooting point.

[0085] If the difference in image magnification due to the distance between the point where proximity is detected and the shooting position is negligible, steps S705 to S706 may be skipped or Δβ may be set to 1.

[0086] In S707, CPU311 checks the lighting status of the illumination device for the shooting in S403 and S407, and if there is a difference, it performs preprocessing to determine the shooting conditions, taking the influence of the difference into consideration.

[0087] In S403, the AE processing circuit 207 determines the shooting conditions to ensure proper exposure for each shot. However, since S407 is designed for image compositing, determining the shooting conditions to ensure proper exposure for each shot may result in an incorrect exposure for the composite image. The CPU 311 stores the light emission pattern during S407 shooting in, for example, the RAM 312, to establish a relationship between the images to be composited.

[0088] Even if the number of light sources illuminated differs for each shot, the exposure levels of the resulting image data need to be roughly the same. Therefore, the CPU 311 lists the ratios of illumination brightness for each lighting pattern during each shot, with the case where all three light sources of the lighting device 101 are illuminated set to 1, and can use this to equalize the exposure levels of each individual shot.

[0089] In S708, CPU311 determines the shooting conditions. The exposure time is determined based on the amount of subject movement predicted in S706. Exposure time = Allowable blur amount ÷ Predicted motion amount This is the decision. The acceptable blur can be set to a value of, for example, 50 μm on the image plane, but other values ​​may be used depending on the requirements for the composite image.

[0090] Next, the CPU 311 determines the shooting sensitivity. First, the CPU 311 determines the sensitivity that will yield proper exposure when all three light sources of the lighting device 101 emit light simultaneously for the determined exposure time. Using this sensitivity and the list created in S707, it determines the sensitivity according to the emission pattern for each shot. The shooting timing in S407 and the emission timing of the lighting device 101 are synchronized. Therefore, the continuous shooting interval is determined by the emission interval from the lighting device, and the number of continuous shots is determined by the number of times the lighting device emits light.

[0091] In this embodiment, an example of performing blur correction by driving the image sensor 203 was described, but it is also applicable to inventions that perform blur correction using a blur correction lens provided in the imaging optical system 219. Although the alignment blending was described using additive averaging blending as an example, it can also be applied to additive blending or comparative brightness blending.

[0092] In this embodiment, the PC 301 receives the image data to be synthesized from the imaging device 201, but it is also possible to synthesize pre-recorded image data. In this case, the lighting status of the illumination device 101 at the time of shooting may be recorded as metadata in the image data, or it may be obtained by detecting the characteristics of ambient light from the image data.

[0093] As described above, according to this embodiment, when combining image data from multiple frames taken under different ambient light conditions, the differences in ambient light conditions at the time of shooting are corrected before detecting movement between frames. Specifically, as an example of correcting the information used for alignment, the optical characteristics of the light source were used to correct the image data. As a result, alignment between frames can be performed with high accuracy, and a high-quality composite image can be generated.

[0094] ●(Second Embodiment) Next, a second embodiment of the present invention will be described. Figure 9 is a block diagram showing an example of the functional configuration of PC301' according to the second embodiment. The difference from the first embodiment is that instead of detecting motion between frames using image data corrected according to the characteristics of the light source, the detected motion vector is corrected according to the characteristics of the light source.

[0095] In Figure 9, components identical to those in the first embodiment are denoted by the same reference numerals as in Figure 3, and their descriptions are omitted. In this embodiment, a motion vector correction circuit 901 is provided instead of the correction circuit 313.

[0096] Next, the operation of the imaging system 1 in this embodiment will be explained using the flowchart shown in Figure 10. In Figure 10, steps that perform the same operations as in the first embodiment are given the same reference numerals as in Figure 4, and their explanations are omitted. In this embodiment, S408 is omitted, and motion vector correction processing S1001 is added. Below, only the operations that differ from the first embodiment will be explained.

[0097] In the motion vector detection process at S409, no correction is performed by the correction circuit; the image data stored in RAM312 is used as is.

[0098] In S1001, the CPU 311 corrects the motion vector detected in S409 using the motion vector correction circuit 901. First, the CPU 311 determines whether the lighting pattern of the lighting device 101 during shooting is different between the frames used to detect the motion vector. If it is determined that the patterns are different, it performs the correction; otherwise, it does not perform the correction.

[0099] If it is determined that motion vector correction is necessary, the CPU 311 determines which of the R, G1, G2, or B pixel output values ​​is the maximum for the reference frame (e.g., the most recent frame) at the time of motion vector detection. Here, the region for determining the output value can be the template region used for motion vector detection within the reference frame. The average value obtained for each type of filter for pixels within the template region may be used as the output value, or the pixel values ​​contained in one repeating unit in the central part of the template region may be used directly as the output value.

[0100] Next, CPU311 similarly determines which filter has the highest output value for the current frame when motion vectors are detected. For the current frame, it calculates the output value in the region determined to have the highest correlation with the template, in the same way as for the reference frame.

[0101] In motion vector detection using template matching, a portion of a reference image is used as a template, and the region with the highest correlation value in the reference image is used as the destination for motion vector detection. However, template matching tends to calculate high correlation values ​​for regions where the texture is strongly expressed. Therefore, if the ambient light conditions (especially color) change between frames, the output value will change depending on the type of pixel (filter) even with the same texture, and the position of the color with the highest output value under the changed ambient light will be determined to be the position with the highest correlation value.

[0102] Therefore, in order to obtain the correct motion vector, it is necessary to correct for the shift in the position of the maximum correlation value due to changes in ambient light. Note that the necessity of correction may be determined before performing the correction. For example, the CPU 311 can compare the output value of the pixel with the maximum output value with the output values ​​of neighboring pixels, and if the difference with the output values ​​of neighboring pixels is greater than a certain value, it can determine that correction of the motion vector is necessary.

[0103] This section explains an example of a motion vector correction method. When ambient light conditions change between frames, template matching is likely to result in the highest correlation value being associated with the color with the highest output value in the current frame. Therefore, the correct destination position is likely to be either a pixel of the color with the highest output value in the reference frame, or a pixel of the color with the highest output value in the reference frame, located near the pixel of the color with the highest output value in the current frame.

[0104] The motion vector correction circuit 901 stores the color with the highest output value (e.g., R) at the starting coordinate position of the motion vector in the reference frame, and compares it with the output values ​​of R that exist around the coordinate position determined to be the destination in the current frame. At this time, the characteristic information of the light source at the time of shooting of each frame obtained from the light source characteristic detection circuit 317 is used to convert it into an output value equivalent to that which would occur if it were shot under the same lighting conditions, thereby correcting for the effects of changes in ambient light.

[0105] The motion vector correction circuit 901 then determines that the coordinate position of the R pixel near the destination in the current frame, which has the closest value to the output value at the starting coordinate position held for the reference frame, is the correct destination, and corrects the motion vector. This makes it possible to obtain the correct motion vector even when the lighting conditions change between frames.

[0106] For a simpler approach, the endpoint of the motion vector may be corrected to the position of the pixel with the highest output value in the reference frame within the repeating unit (R·G1·G2·B) of the color filter that includes the pixel of the position detected as the destination in the current frame.

[0107] In this embodiment, instead of correcting the image data using the optical properties of the light source as an example of correcting the information used for alignment, the motion vector detected between frames is corrected using the optical properties of the light source. This eliminates the need to correct the entire frame, thus enabling faster processing.

[0108] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0109] Furthermore, some of the operations described above as being performed by the control unit 215 or CPU 311 may be performed using dedicated circuits. Also, the computer may have an imaging function, or the imaging device may be capable of performing the computer's image processing as described in the above embodiment. That is, the imaging device may perform the image processing as described in the above embodiment when aligning and combining image data acquired by itself.

[0110] The present invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0111] 101...Lighting device, 102...Proximity sensor, 201...Imaging device, 215...Control unit, 301...PC, 313...Correction circuit, 314...Motion vector detection circuit, 315...Subject detection circuit, 317...Light source characteristic detection circuit, 311...CPU

Claims

1. A detection means for detecting the motion vector between a reference frame and the current frame, Alignment means for aligning the reference frame and the current frame based on the motion vector, A generation means that generates a composite image by combining frames aligned by the aforementioned alignment means, If the ambient light color differs between the time the reference frame was photographed and the time the current frame was photographed, a correction means is provided to correct the information used for alignment according to the ambient light color. The system includes a control means that supplies a coefficient to the correction means corresponding to the type of light source that was lit during the capture of the reference frame and the current frame, among a plurality of light sources with different spectral distributions. The correction means corrects the image data of the reference frame and / or the current frame, which is information used for alignment, using the coefficient and supplies it to the detection means. An image processing apparatus characterized by the following:

2. The image processing apparatus according to claim 1, wherein the coefficient corrects the image data to a value obtained when the image data is captured under ambient light of a specific color.

3. The image processing apparatus according to claim 1, characterized in that the correction means corrects the position of the endpoint of the motion vector detected by the detection means, which is information used for alignment.

4. The image processing apparatus according to claim 3, wherein the correction means corrects the position of the endpoint of the motion vector detected by the detection means based on the output values ​​corresponding to the same color among the output values ​​of pixels provided with the same color color filter in the image sensor used for shooting, for the reference frame and the current frame.

5. The image processing apparatus according to any one of claims 1 to 4, characterized in that the ambient light includes illumination by an illumination device having a plurality of light sources with different spectral distributions.

6. The image processing apparatus according to any one of claims 1 to 5, characterized in that the detection means detects the motion vector of the subject area and the motion vector of the background area.

7. The image processing apparatus according to claim 6, characterized in that the subject area is the area of ​​a person.

8. The image processing apparatus according to any one of claims 1 to 7, further comprising determination means for determining shooting conditions when photographing the reference frame and the current frame based on image data acquired from an imaging device.

9. Imaging means, A detection means for detecting the motion vector between a reference frame captured by the aforementioned imaging means and the current frame, Alignment means for aligning the reference frame and the current frame based on the motion vector, A generation means that generates a composite image by combining frames aligned by the aforementioned alignment means, If the ambient light color differs between the time the reference frame was photographed and the time the current frame was photographed, a correction means is provided to correct the information used for alignment according to the ambient light color. The system includes a control means that supplies a coefficient to the correction means corresponding to the type of light source that was lit during the capture of the reference frame and the current frame, among a plurality of light sources with different spectral distributions. The correction means corrects the image data of the reference frame and / or the current frame, which is information used for alignment, using the coefficient and supplies it to the detection means. An imaging device characterized by the following features.

10. Detecting the motion vector between the reference frame and the current frame, Aligning the reference frame and the current frame based on the motion vector, The process involves combining the aforementioned aligned frames to generate a composite image, If the ambient light color differs between the time the reference frame was captured and the time the current frame was captured, the information used for alignment shall be corrected according to the ambient light color. The process involves determining a coefficient corresponding to the type of light source that was lit during the capture of the reference frame and the current frame, among a plurality of light sources with different spectral distributions. The correction involves correcting the image data of the reference frame and / or the current frame, which is information used for alignment, using the coefficient. The above detection involves detecting the motion vector using the corrected reference frame and / or current frame data. An image processing method characterized by the following:

11. A program for causing a computer to function as each of the means of the image processing apparatus described in any one of claims 1 to 8.

Images