Image processing device, imaging device, image processing method, image processing program, and recording medium

Abstract

To provide an image processing device, an imaging device, an image processing method, and an image processing program that perform different color correction between frames of a moving image according to a change in a light source.SOLUTION: An image processing device includes a moving image acquisition unit 302 that acquires moving image data, a frame identification unit 310 that identifies a reference frame and a correction frame from a plurality of frames that constitute the moving image data, a first image region determination unit 320 that determines an image area used to calculate a first correction coefficient in the frame, determines a first reference image region from a reference frame, and determines a first corrected image region corresponding to the first reference image region from the correction frame, a first color signal acquisition unit 330 that acquires a first reference color signal from the color image data of the first reference image region and acquiring a first corrected color signal from the color image data of the first corrected image region, and a first correction coefficient calculator 340 that calculates a first correction coefficient for matching the first corrected color signal to the first reference color signal.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, an image processing program, and a recording medium, and more particularly to a technique for performing different color corrections between frames of a moving image in response to changes in a light source. [Background technology]

[0002] The image processing device described in Patent Document 1 calculates an integral value of the signal amount for each row of an image to be corrected for flicker removal, and applies this integral value to detect flicker components contained in each row of the image frame. The detected flicker components are data corresponding to the flicker waveform of actual lighting. Flicker removal is performed by executing a correction process that applies a flicker correction coefficient consisting of an inverse phase pattern of the flicker components.

[0003] Furthermore, the image correction device described in Patent Document 2 is an imaging device that uses a CMOS image sensor and captures images using a rolling shutter, and is equipped with a flicker correction circuit that corrects flicker in an original image represented by the output of the CMOS image sensor. The area correction coefficient calculation circuit of the flicker correction circuit divides the original image into M vertical and N horizontal divisions, calculates an area average value by averaging pixel signals for each divided region, and calculates an area reference value that does not include flicker components by averaging the area average values ​​for multiple frames for each divided region. The flicker correction circuit corrects the original image using an area correction coefficient calculated from the ratio between the area reference value and the area average value of the original image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2011-160090 [Patent Document 2] Japanese Patent Application Laid-Open No. 2008-109370 Summary of the Invention

[0005] One embodiment of the technique of the present disclosure provides an image processing device, an imaging device, an image processing method, an image processing program, and a recording medium that perform different color corrections between frames of a moving image in accordance with changes in a light source. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present invention includes one or more processors, and the one or more processors are configured to execute a moving image acquisition process for acquiring moving image data captured by an imaging element; a frame identification process for identifying a reference frame and a correction frame for the reference frame from a plurality of frames constituting the moving image data acquired in the moving image acquisition process; a first image area determination process for determining an image area to be used in calculating a first correction coefficient within the frame, determining a first reference image area from the reference frame, and determining a first correction image area corresponding to the first reference image area from the correction frame; a first color signal acquisition process for acquiring a first reference color signal from color image data of the first reference image area and acquiring a first correction color signal from the color image data of the first correction image area; and a first correction coefficient calculation process for calculating a first correction coefficient to match the first correction color signal to the first reference color signal.

[0007] In an image processing device according to another aspect of the present invention, it is preferable that the one or more processors are further configured to execute a first color correction process that applies the first correction coefficient calculated by the first correction coefficient calculation process to the correction frame to perform color correction of the correction frame.

[0008] In an image processing device according to yet another aspect of the present invention, it is preferable that the first reference image area and the first corrected image area determined by the first image area determination process are each a plurality of image areas, and that each image area is an image area of ​​a specific pixel or surrounding pixels including the specific pixel.

[0009] In the image processing device according to still another aspect of the present invention, it is preferable that the first reference color signal and the first corrected color signal each be made up of a plurality of color signals.

[0010] In an image processing device according to still another aspect of the present invention, a plurality of color signals of the first reference color signal are ref , green G ref , blue B ref and the plurality of color signals of the first correction color signal are red R, green G, and blue B, the first correction coefficient calculation process is performed in the plurality of image regions using the following equation: R ref =α1·R+β1·G+γ1·B+Δ1 G ref =α2·R+β2·G+γ2·B+Δ2 B ref =α3·R+β3·G+γ3·B+Δ3 It is preferable to calculate first correction coefficients α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, and Δ3 that satisfy the following equation.

[0011] In an image processing device according to still another aspect of the present invention, the first correction coefficient calculation process is performed by: ref and R.G. ref and G, and B ref It is preferable to calculate α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, and Δ3 such that the difference between and B is minimum, respectively.

[0012] In an image processing device according to yet another aspect of the present invention, the moving image data is captured by sequentially exposing at least one or more pixels or scanning lines of pixels formed on an image sensor, and it is preferable that the first image area determination process determines a first reference image area and a first corrected image area, respectively, from image areas in the reference frame and the corrected frame in which the exposure start and end timing of the image sensor are the same.

[0013] In the image processing device according to yet another aspect of the present invention, it is preferable that the first image area determination process determines the first reference image area in the reference frame and the first corrected image area in the corrected frame to be at the same position within the frame.

[0014] In an image processing device according to yet another aspect of the present invention, the one or more processors are configured to perform a flicker phase detection process for detecting the phase of a light source flicker from moving image data, and it is preferable that the frame identification process identifies a reference frame using the phase of the flicker.

[0015] In the image processing device according to yet another aspect of the present invention, it is preferable that the frame identification process identifies a corrected frame that has been color corrected by the first color correction process as a reference frame for the next corrected frame.

[0016] In the image processing device according to still another aspect of the present invention, the moving image acquisition process preferably acquires moving image data made up of a plurality of frames, each frame having an exposure time shorter than the frame interval.

[0017] In an image processing device according to still another aspect of the present invention, moving image data is captured by sequentially exposing pixels formed on an imaging element under a light source having flicker for at least one pixel or scan line, and the moving image acquisition process acquires moving image data consisting of a plurality of frames, each frame having an exposure time shorter than the frame interval and shorter than the period of the flicker of the light source, and the one or more processors perform a line image specification process for specifying a line image on a reference scan line as a reference line image and a line image on a scan line adjacent to the reference scan line as a correction line image in each frame constituting the moving image data, and a line image specification process for determining an image area to be used in calculating a second correction coefficient within the line image, and The image processing system is configured to execute a second image area determination process that determines a second reference image area from a reference line image and determines a second corrected image area corresponding to the second reference image area from the corrected line image; a second color signal acquisition process that acquires a second reference color signal from the color image data of the second reference image area and acquires a second corrected color signal from the color image data of the second corrected image area; a second correction coefficient calculation process that calculates a second correction coefficient that aligns the second corrected color signal with the second reference color signal; and a second color correction process that applies the second correction coefficient calculated by the second correction coefficient calculation process to the corrected line image to perform color correction of the corrected line image, and it is preferable that the line image identification process identifies the corrected line image color-corrected by the second color correction process as a reference line image for the next corrected line image.

[0018] In an image processing device according to still another aspect of the present invention, moving image data is captured by sequentially exposing pixels formed on an imaging element under a light source having flicker for at least one pixel or scan line, and the moving image acquisition process acquires moving image data consisting of a plurality of frames, each frame having an exposure time shorter than the frame interval and shorter than the period of the flicker of the light source, and the one or more processors perform a line image specification process for specifying, in each frame constituting the moving image data, a line image on a reference scan line as a reference line image and a line image on a scan line other than the reference scan line as a correction line image, and a process for determining an image area within the line image to be used for calculating a second correction coefficient. The image processing system is configured to execute the following processes: a second image area determination process that determines a second reference image area from the reference line image and determines a second corrected image area corresponding to the second reference image area for each correction line image; a second color signal acquisition process that acquires a second reference color signal from the color image data of the second reference image area and acquires a second corrected color signal from the color image data of the second corrected image area; a second correction coefficient calculation process that calculates a second correction coefficient that aligns the second corrected color signal with the second reference color signal; and a second color correction process that applies the calculated second correction coefficient to the correction line image to perform color correction of the correction line image, and it is preferable that the second image area determination process determines an image area in the correction line image whose color is similar to the color of the second reference image area of ​​the reference line image as the second corrected image area.

[0019] In an image processing device according to yet another aspect of the present invention, it is preferable that one or more processors are configured to perform an object detection process to detect an object from a frame, and that the second image area determination process determines, as the second corrected image area, an image area of ​​the corrected line image that includes an object that is the same as an object detected by the object detection process and that is included in the second reference image area of ​​the reference line image.

[0020] An imaging device according to yet another aspect of the present invention comprises an imaging element for capturing moving images and any of the image processing devices described above, and the moving image acquisition process acquires moving image data indicating the moving images captured by the imaging element.

[0021] An image processing method according to yet another aspect of the present invention includes the steps of acquiring moving image data captured by an imaging element, identifying a reference frame and a correction frame for the reference frame from a plurality of frames constituting the moving image data, determining an image area to be used in calculating a first correction coefficient within the frame, determining a plurality of first reference image areas from the reference frame, and determining a plurality of first correction image areas corresponding to the plurality of first reference image areas from the correction frame, acquiring a plurality of first reference color signals from color image data for each of the plurality of first reference image areas, and acquiring a plurality of first correction color signals from the color image data for each of the plurality of first correction image areas, calculating first correction coefficients that align the plurality of first correction color signals with the plurality of first reference color signals, and applying the calculated first correction coefficients to the correction frame to perform color correction of the correction frame.

[0022] In an image processing method according to still another aspect of the present invention, the first reference color signal is ref , green G ref , blue B refと When the first correction color signals are red R, green G, and blue B, the step of calculating the first correction coefficients is carried out by using the following equations for the plurality of image regions: R ref =α1·R+β1·G+γ1·B+Δ1 G ref =α2·R+β2·G+γ2·B+Δ2 B ref =α3·R+β3·G+γ3·B+Δ3 It is preferable to calculate first correction coefficients α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, and Δ3 that satisfy the following equation.

[0023] In an image processing method according to still another aspect of the present invention, the step of calculating the first correction coefficients includes: ref and R.G. ref and G, and B ref It is preferable to calculate α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, and Δ3 such that the difference between and B is minimum, respectively.

[0024] An image processing program according to yet another aspect of the present invention causes a computer to realize the following functions: acquiring moving image data captured by an imaging element; identifying a reference frame and a correction frame relative to the reference frame from multiple frames constituting the moving image data; determining an image area to be used in calculating a first correction coefficient within the frame, determining multiple first reference image areas from the reference frame, and determining multiple first correction image areas corresponding to the multiple first reference image areas from the correction frame; acquiring multiple first reference color signals from color image data for each of the multiple first reference image areas and acquiring multiple first correction color signals from color image data for each of the multiple first correction image areas; calculating first correction coefficients that align the multiple first correction color signals with the multiple first reference color signals; and applying the calculated first correction coefficients to the correction frame to perform color correction of the correction frame. [Brief explanation of the drawings]

[0025] [Figure 1] FIG. 1 is a perspective view of an imaging device including a shake correction device according to the present invention, seen obliquely from the front. [Figure 2] FIG. 2 is a rear view of the imaging device shown in FIG. [Figure 3] FIG. 3 is a block diagram showing an embodiment of the internal configuration of the imaging device shown in FIG. [Figure 4] FIG. 4 is a block diagram showing a first embodiment of an image processing apparatus according to the present invention. [Figure 5] FIG. 5 is a diagram showing an embodiment of a stacked CMOS, showing an example of the hardware configuration of an image processing device. [Figure 6] FIG. 6 is a block diagram showing a second embodiment of an image processing apparatus according to the present invention. [Figure 7] FIG. 7 is a diagram showing an example of a reference frame Fref of a moving image captured outdoors and a corrected frame F of a moving image captured indoors. [Figure 8] FIG. 8 is a diagram showing an example of a reference frame Fref and a correction frame F prepared in advance. [Figure 9]FIG. 9 is a diagram showing a schematic diagram of the relationship between the flicker of the light source, the exposure operation of the image sensor, and the moving image data. [Figure 10] FIG. 10 is a block diagram showing a third embodiment of an image processing apparatus according to the present invention. [Figure 11] FIG. 11 is a block diagram showing a fourth embodiment of an image processing apparatus according to the present invention. [Figure 12] FIG. 12 is a flowchart showing an embodiment of an image processing method according to the present invention. [Figure 13] FIG. 13 is an external view of a smartphone which is an embodiment of the imaging device according to the present invention. [Figure 14] FIG. 14 is a block diagram showing the configuration of a smartphone. DETAILED DESCRIPTION OF THE INVENTION

[0026] Hereinafter, preferred embodiments of an image processing device, an imaging device, an image processing method, and an image processing program according to the present invention will be described with reference to the accompanying drawings.

[0027] <External view of the imaging device> FIG. 1 is a perspective view of an imaging device including an image processing device according to the present invention, seen obliquely from the front, and FIG. 2 is a rear view of the imaging device shown in FIG.

[0028] As shown in FIG. 1, the imaging device 10 is a mirrorless digital single-lens camera that is made up of an interchangeable lens 100 and a camera body 200 to which the interchangeable lens 100 can be attached or detached.

[0029] In FIG. 1, the front of the camera body 200 is provided with a body mount 260 to which the interchangeable lens 100 is attached, a viewfinder window 20 of the optical viewfinder, etc., and the top of the camera body 200 is mainly provided with a shutter release switch 22, a shutter speed dial 23, an exposure compensation dial 24, a power lever 25, and a built-in flash 30.

[0030] As shown in FIG. 2, the rear surface of the camera body 200 is provided with a liquid crystal monitor 216, an optical finder eyepiece 26, a MENU / OK key 27, a cross key 28, a playback button 29, and the like.

[0031] The LCD monitor 216 displays a live view image in the image capture mode and displays a captured image in the playback mode. In addition, it also functions as a display unit that displays various menu screens and as a notification unit that notifies the user of various information.

[0032] The MENU / OK key 27 is an operation key that functions both as a menu button for issuing a command to display a menu on the screen of the LCD monitor 216 and as an OK button for issuing a command to confirm and execute a selection.

[0033] The cross key 28 is an operation unit for inputting instructions in four directions (up, down, left, and right), and functions as a multi-function key for selecting an item from a menu screen and for instructing the selection of various setting items from each menu. The up and down keys of the cross key 28 also function as a zoom switch during image capture or a playback zoom switch in playback mode, and the left and right keys function as frame-by-frame (forward and backward) buttons in playback mode. The cross key 28 also functions as an operation unit for specifying a desired subject for focus adjustment or color correction from among multiple subjects displayed on the LCD monitor 216.

[0034] Furthermore, by using the MENU / OK key 27, the cross key 28, and the menu screen displayed on the LCD monitor 216, various imaging modes can be set, including a still image imaging mode for capturing a single still image, a moving image imaging mode (first moving image imaging mode) for capturing a normal moving image, and a moving image imaging mode (second moving image imaging mode), which is a type of moving image imaging mode, for capturing a moving image for extracting a still image.

[0035] The second moving image capturing mode is a mode for setting capturing conditions suitable for extracting still images and extracting high-quality still images from moving images, and for example, the exposure time per frame is set shorter than the frame interval of the moving image.

[0036] The playback button 29 is a button for switching to a playback mode in which captured and recorded still images or moving images are displayed on the liquid crystal monitor 216.

[0037] <Internal configuration of the imaging device> [Interchangeable Lens] FIG. 3 is a block diagram showing an embodiment of the internal configuration of the imaging device 10. As shown in FIG.

[0038] The interchangeable lens 100 that functions as an imaging optical system that constitutes the imaging device 10 is manufactured in accordance with the communication standard of the camera body 200, and is an interchangeable lens that can communicate with the camera body 200 as described below. The interchangeable lens 100 includes an imaging optical system 102, a focus lens control unit 116, an aperture control unit 118, a lens-side CPU (Central Processing Unit) 120, a flash ROM (Read Only Memory) The lens unit 126 includes a lens-side communication unit 150 and a lens mount 160.

[0039] The imaging optical system 102 of the interchangeable lens 100 includes a lens group 104 including a focus lens, and an aperture 108 .

[0040] The focus lens control unit 116 moves the focus lens and controls the position (focus position) of the focus lens in accordance with a command from the lens side CPU 120. The iris control unit 118 controls the iris 108 in accordance with a command from the lens side CPU 120.

[0041] The lens-side CPU 120 controls the interchangeable lens 100 and includes a ROM 124 and a RAM (Random Access Memory) 122 built in.

[0042] The flash ROM 126 is a non-volatile memory that stores programs downloaded from the camera body 200 and the like.

[0043] The lens-side CPU 120 performs overall control of each section of the interchangeable lens 100 in accordance with a control program stored in a ROM 124 or a flash ROM 126, using a RAM 122 as a work area.

[0044] With the lens mount 160 attached to a body mount 260 of the camera body 200, the lens side communication unit 150 communicates with the camera body 200 via multiple signal terminals provided on the lens mount 160. That is, in accordance with commands from the lens side CPU 120, the lens side communication unit 150 transmits and receives (bidirectional communication) request signals and response signals to and from a body side communication unit 250 of the camera body 200 connected via the lens mount 160 and the body mount 260, and notifies the camera body 200 of lens information of each optical member of the imaging optical system 102 (position information of the focus lens, aperture information, etc.).

[0045] The interchangeable lens 100 also includes a detection unit (not shown) that detects position information of the focus lens and aperture information. Here, the aperture information is information indicating the aperture value (F-number) of the aperture 108, the opening diameter of the aperture 108, etc.

[0046] In order to respond to a request for lens information from the camera body 200, the lens side CPU 120 preferably stores various types of lens information, including detected focus lens position information and aperture information, in the RAM 122. Furthermore, the lens information can be detected when there is a request for lens information from the camera body 200, or detected when the optical member is driven, or detected at a fixed cycle (a cycle sufficiently shorter than the frame cycle of a moving image), and the detection results can be stored.

[0047] [Camera body] The camera body 200 constituting the imaging device 10 shown in FIG. 3 includes an image sensor 201, an image sensor control unit 202, an analog signal processing unit 203, an A / D (Analog to Digital) converter 204, an image input controller 205, a digital signal processing unit 206, a RAM 207, a compression / decompression processing unit 208, a media control unit 210, a memory card 212, a display control unit 214, a liquid crystal monitor 216, a main body CPU 220, an operation unit 222, a flash ROM 226, a ROM 228, an AF (Autofocus) control unit 230, an AE (Auto Exposure) control unit 232, a white balance correction unit 234, a wireless communication unit 236, a GPS (Global Positioning System) receiving unit 238, a power control unit 240, a battery 242, a main body communication unit 250, a main body mount 260, a flash light emission unit 270 and a flash control unit 272 constituting the built-in flash 30 (FIG. 1), a focal-plane shutter (FPS), a The controller 290 includes a shutter 280 and an FPS control unit 296.

[0048] The imaging element 201, which functions as an imaging unit, is configured by a CMOS (Complementary Metal-Oxide Semiconductor) color image sensor. Note that the imaging element 201 is not limited to a CMOS type, and may be a CCD (Charge Coupled Device) type imaging element.

[0049] In the image sensor 201, a color filter of one of the three primary colors of red (R), green (G), and blue (B) (R filter, G filter, B filter) is arranged in a predetermined color filter array on a plurality of pixels each made up of a photoelectric conversion element (photodiode) two-dimensionally arranged in the x direction (horizontal direction) and y direction (vertical direction), thereby forming pixels of each color of RGB. The color filter array may be a general Bayer array, but is not limited to this, and may be, for example, another color filter array such as a Trans (registered trademark) array.

[0050] An optical image of a subject formed on the light receiving surface of the image sensor 201 by the imaging optical system 102 of the interchangeable lens 100 is converted into an electrical signal by the image sensor 201. Charges corresponding to the amount of incident light are accumulated in each pixel formed on the image sensor 201, and an electrical signal corresponding to the amount of charge (signal charge) accumulated in each pixel is read out from the image sensor 201 as an image signal.

[0051] The image sensor control unit 202 controls the reading of image signals from the image sensor 201 in accordance with commands from the main body side CPU 220 .

[0052] Furthermore, the image sensor control unit 202 drives the image sensor 201 in a so-called global shutter system or rolling shutter system in response to a shutter control signal from the main body side CPU 220 .

[0053] Here, the global shutter method refers to a method in which, in a CMOS image sensor 201, all pixels in one screen are reset at once to start the exposure operation (i.e., charge accumulation is started for all pixels in one screen), and the rolling shutter method refers to a method in which, in a CMOS image sensor 201, the exposure operation is performed sequentially for at least one or more scanning lines or pixels (i.e., a method in which reset is performed sequentially for each scanning line or pixel to start charge accumulation and read out the accumulated charge, and is also called a focal plane shutter method).

[0054] The analog signal processing unit 203 performs various analog signal processing operations on the analog image signal obtained by capturing an image of a subject with the image sensor 201. The analog signal processing unit 203 includes a sample-and-hold circuit, a color separation circuit, an AGC (Automatic Gain Control) circuit, etc. The AGC circuit functions as a sensitivity adjustment unit that adjusts the sensitivity (ISO sensitivity (ISO: International Organization for Standardization)) during imaging, and adjusts the gain of an amplifier that amplifies the input image signal so that the signal level of the image signal falls within an appropriate range. The A / D converter 204 converts the analog image signal output from the analog signal processing unit 203 into a digital image signal.

[0055] When capturing a still image or a moving image, RGB image data for each pixel (RAW data having RGB density values ​​for each pixel) is output via the image sensor 201, analog signal processing unit 203, and A / D converter 204 and is input from the image input controller 205 to the RAM 207 and temporarily stored therein. Note that when the image sensor 201 is a CMOS type image sensor, the analog signal processing unit 203 and the A / D converter 204 are often built into the image sensor 201.

[0056] The digital signal processing unit 206 performs various types of digital signal processing on the RAW data stored in the RAM 207. The digital signal processing unit 206 reads out the RAW data stored in the RAM 207 as needed, and performs digital signal processing on the read-out RAW data, such as offset processing, linear matrix processing that color corrects the RGB RAW data to new RGB signals with accurate color reproducibility, gain control processing including sensitivity correction, gamma correction processing, demosaic processing (also called demosaicing processing or synchronization processing), and RGB / YCrCb conversion processing, and stores the image data after digital signal processing back into the RAM 207.

[0057] Note that demosaic processing is a process for calculating all RGB color information for each pixel from a mosaic image made up of RGB, for example, in the case of an imaging element having color filters of the three primary colors of RGB, and generating synchronized RGB three-plane image data from the mosaic data (dot-sequential RGB data). Also, RGB / YCrCb conversion processing is a process for converting the synchronized RGB data into luminance data (Y) and color difference data (Cr, Cb).

[0058] When recording still images or moving images, the compression / decompression processor 208 performs compression processing on uncompressed luminance data Y and color difference data Cb, Cr temporarily stored in the RAM 207. In the case of still images, compression is performed in, for example, JPEG (Joint Photographic Coding Experts Group) format, and in the case of moving images, compression is performed in, for example, H.264 format. The image data compressed by the compression / decompression processor 208 is recorded on a memory card 212 via a media control unit 210. In addition, in playback mode, the compression / decompression processor 208 performs decompression processing on compressed image data obtained from the memory card 212 via the media control unit 210 to generate uncompressed image data.

[0059] The media control unit 210 controls the recording of image data compressed by the compression / decompression processing unit 208 onto a memory card 212. The media control unit 210 also controls the reading of compressed image data from the memory card 212.

[0060] A display control unit 214 controls the display of uncompressed image data stored in the RAM 207 on a liquid crystal monitor 216. The liquid crystal monitor 216 is configured with a liquid crystal display device, but may be configured with a display device such as an organic electroluminescence device instead of the liquid crystal monitor 216.

[0061] When a live view image is to be displayed on the liquid crystal monitor 216, digital image signals continuously generated by the digital signal processing unit 206 are temporarily stored in the RAM 207. The display control unit 214 converts the digital image signals temporarily stored in the RAM 207 into a signal format for display and sequentially outputs them to the liquid crystal monitor 216. This allows the captured image to be displayed in real time on the liquid crystal monitor 216, allowing the liquid crystal monitor 216 to be used as an electronic viewfinder.

[0062] The shutter release switch 22 is an image capture instruction unit for inputting an instruction to capture a still image or a moving image, and is configured as a two-stroke switch having a so-called "half-press" and "full press."

[0063] In the still image capture mode, an S1 on signal is output when the shutter release switch 22 is pressed halfway, and an S2 on signal is output when the switch is pressed further from halfway to the full press. When the S1 on signal is output, the main body CPU 220 executes capture preparation processing such as AF control (automatic focus adjustment) and AE control (automatic exposure control), and when the S2 on signal is output, it executes still image capture processing and recording processing.

[0064] It goes without saying that AF control and AE control are performed automatically when the auto mode is set by the operation unit 222, and that AF control and AE control are not performed when the manual mode is set.

[0065] Also, in the moving image capture mode, when the shutter release switch 22 is fully pressed and an S2 on signal is output, the camera body 200 enters a moving image recording mode in which moving image recording begins, and performs image processing and recording processing of the moving image. After that, when the shutter release switch 22 is fully pressed again and an S2 on signal is output, the camera body 200 enters a standby state and temporarily suspends the moving image recording processing.

[0066] The shutter release switch 22 is not limited to a two-stroke switch consisting of a half-press and a full press, but may output an S1 on signal and an S2 on signal with a single operation, or separate switches may be provided to output the S1 on signal and the S2 on signal.

[0067] In addition, in a form in which operation instructions are given using a touch panel or the like, the operation instructions may be output by touching an area corresponding to the operation instructions displayed on the screen of the touch panel as the operation means, and the form of the operation means is not limited to these as long as it instructs the shooting preparation process or the imaging process.

[0068] The still or moving images obtained by capturing images are compressed by the compression / decompression processing unit 208, and the compressed image data is converted into an image file with the necessary additional information such as the image capture date and time, GPS information, and image capture conditions (F-number, shutter speed, ISO sensitivity, etc.) added to the header, and then recorded on the memory card 212 via the media control unit 210.

[0069] The body-side CPU 220 controls the overall operation of the camera body 200 and the driving of the optical components of the interchangeable lens 100, and controls each part of the camera body 200 and the interchangeable lens 100 based on input from an operating unit 222 including the shutter release switch 22.

[0070] The flash ROM 226 is a readable and writable non-volatile memory that stores setting information.

[0071] The ROM 228 stores a camera control program executed by the body-side CPU 220, an image processing program according to the present invention, defect information on the image sensor 201, and various parameters and tables used for image processing, etc. The body-side CPU 220 controls each part of the camera body 200 and the interchangeable lens 100 in accordance with the camera control program stored in ROM 228, using RAM 207 as a working area.

[0072] The AF detection unit 230 calculates values ​​necessary for AF control based on the digital image signal. In the case of so-called contrast AF, for example, it calculates an integrated value (focus evaluation value) of the high-frequency components of the G signal within a predetermined AF area. The body-side CPU 220 moves the focus lens included in the lens group 104 of the interchangeable lens 100 to a position where the focus evaluation value is maximized during AF control (i.e., a position where the contrast is maximized). Note that AF is not limited to contrast AF, and for example, phase-difference AF may be performed, in which the amount of defocus is detected based on pixel data from phase-difference detection pixels provided in the image sensor, and the focus lens is moved so that this defocus amount becomes zero.

[0073] The AE control unit 232 detects the brightness of the subject (subject luminance) and calculates a numerical value (exposure value (EV)) required for AE control and AWB (Auto White Balance) control corresponding to the subject luminance. The AE control unit 232 calculates the EV value based on the luminance of the image acquired via the image sensor 201, and the shutter speed and F-number when the luminance of the image was acquired.

[0074] The main body CPU 220 can determine the F-number, shutter speed, and ISO sensitivity from a predetermined program diagram based on the EV value obtained from the AE control unit 232, and perform AE control.

[0075] The white balance correction unit 234 calculates white balance gains (WB (White Balance) gains) Gr, Gg, Gb for each color data of the RGB data (R data, G data, and B data), and performs white balance correction by multiplying the R data, G data, and B data by the calculated WB gains Gr, Gg, Gb, respectively. Here, one possible method for calculating the WB gains Gr, Gg, Gb is to identify the type of light source illuminating the subject based on scene recognition (determination of outdoors, indoors, etc.) using the brightness (EV value) of the subject and the color temperature of the ambient light, and then read out the WB gain corresponding to the identified light source type from a storage unit in which appropriate WB gains for each light source type are stored in advance. However, other known methods are also possible that use at least the EV value to calculate the WB gains Gr, Gg, Gb.

[0076] The wireless communication unit 236 is a part that performs short-range wireless communication using standards such as Wi-Fi (Wireless Fidelity) (registered trademark) and Bluetooth (registered trademark), and sends and receives necessary information to and from surrounding digital devices (mobile terminals such as smartphones).

[0077] The GPS receiver 238 receives GPS signals transmitted from multiple GPS satellites in accordance with instructions from the main body CPU 220, executes positioning calculation processing based on the multiple received GPS signals, and acquires GPS information consisting of the latitude, longitude, and altitude of the camera main body 200. The acquired GPS information can be recorded in the header of the image file as additional information indicating the imaging position of the captured image.

[0078] In accordance with commands from the body-side CPU 220, the power supply control unit 240 supplies the power supply voltage supplied from the battery 242 to each unit of the camera body 200. In addition, in accordance with commands from the body-side CPU 220, the power supply control unit 240 supplies the power supply voltage supplied from the battery 242 to each unit of the interchangeable lens 100 via the body mount 248 and the lens mount 160.

[0079] The lens power switch 244 switches on and off and changes the level of the power supply voltage applied to the interchangeable lens 100 via the body mount 248 and the lens mount 160 in accordance with commands from the body-side CPU 220 .

[0080] In accordance with commands from the body-side CPU 220, the body-side communication unit 246 transmits and receives (bidirectional communication) request signals and response signals to and from the lens-side communication unit 150 of the interchangeable lens 100, which is connected via the body mount 248 and the lens mount 160. Note that the body mount 260 is provided with a plurality of terminals 260A as shown in Fig. 1, and when the interchangeable lens 100 is attached to the camera body 200 (the lens mount 160 and the body mount 260 are connected), the plurality of terminals 260A (Fig. 1) provided on the body mount 260 are electrically connected to a plurality of terminals (not shown) provided on the lens mount 160, enabling bidirectional communication between the body-side communication unit 250 and the lens-side communication unit 150.

[0081] The built-in flash 30 (FIG. 1) is, for example, a flash of a TTL (Through The Lens) automatic light control type, and is made up of a flash light emitting unit 270 and a flash control unit 272.

[0082] The flash control unit 272 has a function of adjusting the light emission amount (guide number) of the flash light emitted from the flash light emission unit 270. That is, the flash control unit 272 causes the flash light emission unit 270 to emit light in synchronization with a flash imaging instruction from the body-side CPU 220, starts metering the reflected light (including ambient light) incident via the imaging optical system 102 of the interchangeable lens 100, and stops the emission of the flash light from the flash light emission unit 270 when the photometric value reaches the standard exposure value.

[0083] The focal plane shutter 280 constitutes a mechanical shutter of the imaging device 10 and is disposed immediately in front of the imaging element 201 .

[0084] The FPS control unit 296 is a part that outputs control signals to the FPS drive unit (a charge motor, a leading curtain electromagnet, and a trailing curtain electromagnet, not shown) based on instruction input from the main body side CPU 220.

[0085] When the shutter release switch 22 is turned ON while the front and rear curtains of the focal plane shutter 280 are driven to the charge position by the charge motor and fixed by the front and rear electromagnets, the FPS control unit 296 turns off the front-curtain electromagnet that holds the front curtain at the charge position and causes the front curtain to travel in the opening direction using the front-curtain running spring, thereby opening the exposure opening. After a time corresponding to the shutter speed has elapsed, the FPS control unit 296 turns off the rear-curtain electromagnet that holds the rear curtain at the charge position and causes the rear-curtain running spring to travel in the closing direction, thereby closing the exposure opening. This exposes the image sensor 201 for a time corresponding to the shutter speed.

[0086] [First embodiment of image processing device] The camera body 200 is provided with an image processing device according to the present invention.

[0087] The image processing device according to the present invention is provided in the digital signal processing unit 206, or in the digital signal processing unit 206 and the main body side CPU 220, or is provided as dedicated hardware (not shown).

[0088] FIG. 4 is a block diagram showing a first embodiment of an image processing apparatus according to the present invention.

[0089] The image processing device shown in FIG. 4 is a device that performs different color corrections between frames of a moving image in accordance with changes in the light source when capturing a moving image in a moving image capturing mode (particularly the second moving image capturing mode for extracting still images), and is mainly composed of a moving image acquisition unit 302, a frame identification unit 310, a first image area determination unit 320, a first color signal acquisition unit 330, a first correction coefficient calculation unit 340, and a first color correction unit 350.

[0090] In addition, the moving image capturing mode in this example captures moving images using a rolling shutter method in which the image sensor 201 is reset for each scanning line, exposure operation (start of charge accumulation) is performed for each scanning line, and data corresponding to the charge accumulated for each scanning line is read out after the exposure time has elapsed.

[0091] The moving image acquisition unit 302 is a part that acquires moving image data captured by the interchangeable lens 100, the image sensor 201, etc. when the second moving image capturing mode is set, and in this example, acquires RAW data 300 that is temporarily stored in the RAM 207 as moving image data. The RAW data 300 acquired by the moving image acquisition unit 302 is output to the first color correction unit 350 and the frame identification unit 310.

[0092] The frame identification unit 310 identifies a reference frame and a correction frame for the reference frame from the multiple frames that make up the moving image acquired by the moving image acquisition unit 302. The frame identification unit 310 of this example has a flicker phase detection unit 312, which detects the flicker phase of the light source.

[0093] When capturing moving images under an indoor light source, artificial light sources such as fluorescent lights and LEDs (light emitting diodes) are affected by the frequency of the commercial power supply (50 Hz or 60 Hz), and generate flicker at a frequency corresponding to the frequency of the commercial power supply.

[0094] Therefore, when capturing moving images under an artificial light source, the flicker of the artificial light source causes the color to change between frames of the moving image, and the change in color between frames is particularly noticeable in the second moving image capturing mode, in which the exposure time per frame is shorter than the frame interval (shorter than the flicker period).

[0095] The flicker phase detection unit 312 detects the flicker frequency and flicker phase of the light source based on changes in brightness of each scanning line of one frame of RAW data 300 acquired by the video acquisition unit 302. The flicker phase can be detected, for example, based on the exposure start time of the first line of each frame, as the time from the exposure start time until the luminance of the light source reaches its maximum.

[0096] If the flicker frequency of the light source and the frame rate (fps: frames per second) of the video image match, there will be no flicker effect between frames, but if they do not match, the color will change between frames.

[0097] Even when the flicker frequency of the light source and the frame rate of the moving image are different, the same or nearly the same flicker phase appears at regular intervals.

[0098] The frame identification unit 310 determines a frame with a specific flicker phase that appears every certain period as a reference frame and other frames as correction frames based on the flicker phase detected by the flicker phase detection unit 312. The reference frame every certain period is preferably the brightest frame within the certain period.

[0099] The first image area determination unit 320 is a part that determines the image area to be used for calculating the first correction coefficient within each frame from the reference frame and correction frame identified by the frame identification unit 310, and determines a first reference image area from the reference frame within each calculation area of ​​the reference frame and correction frame, and determines a first corrected image area corresponding to the first reference image area from the correction frame.

[0100] The calculation region of each of the reference frame and the correction frame may be the entire region of the frame, a focus region of the frame, or a central region of the frame.

[0101] The first reference image area can be, for example, an image area of ​​a specific pixel at a plurality of predetermined pixel positions within the calculation area of ​​the reference frame or a peripheral pixel including the specific pixel. In this example, a region of one pixel at a specific position can also be an area constituting the first reference image area.

[0102] The first corrected image area is an area within the calculation area of ​​the correction frame, and corresponds to the first reference image area. In this example, the first reference image area and the first corrected image area are image areas on the same scanning line where the exposure start and end timing of the image sensor 201 is the same, and are areas at the same position.

[0103] The first color signal acquisition unit 330 calculates a first reference color signal (R) for each of the first reference image regions based on the color image data (RGB RAW data in this example) of the first reference image region determined by the first image region determination unit 320, by calculating a representative value of the pixel data of the R pixels, a representative value of the pixel data of the G pixels, and a representative value of the pixel data of the B pixels of the first reference image region in the RAW data. ref , G ref , B ref ) is obtained.

[0104] Here, if the RAW data in the first reference image area is any one of R, G, and B pixels, the pixel data of that pixel is the representative value. Furthermore, if there are multiple R pixels in the first reference image area, the representative value can be the average value of the pixel data of the multiple R pixels, or a weighted average of the pixel data of the multiple R pixels according to the distance from the center of the first reference image area. If there are multiple G pixels and multiple B pixels in the first reference image area, the representative values ​​of the G pixels and B pixels can also be calculated in the same manner as above.

[0105] Similarly, the first color signal acquisition unit 330 acquires, from the RGB RAW data of the first corrected image area determined by the first image area determination unit 320, a representative value of the pixel data of the R pixels, a representative value of the pixel data of the G pixels, and a representative value of the pixel data of the B pixels of the first corrected image area of ​​the RAW data as first corrected color signals (R, G, B) for each of the multiple first corrected image areas.

[0106] The first correction coefficient calculation unit 340 converts the first corrected color signals (R, G, B) into the first reference color signals (R ref , G ref , B ref ) to calculate the first correction coefficient.

[0107] Specifically, the first correction coefficient calculation unit 340 calculates the following equation for a plurality of image regions (a plurality of pairs of image regions each consisting of a first reference image region and a first correction image region): [Number 1] R ref =α1·R+β1·G+γ1·B+Δ1 [Number 2] G ref =α2·R+β2·G+γ2·B+Δ2 [Number 3] B ref =α3·R+β3·G+γ3·B+Δ3 First correction coefficients α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, and Δ3 that satisfy the above equation are calculated.

[0108] When calculating the first correction coefficients (α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, Δ3) that satisfy the formulas [Formula 1] to [Formula 3], the first correction coefficient calculation unit 340 calculates R ref and R.G. ref and G, and B ref Calculate α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, and Δ3 that minimize the difference between and B, respectively.

[0109] The first correction coefficients (α1, α2, α3, β1, β2, β3, γ1, γ2, γ3) shown in [Equation 1] to [Equation 3] are multipliers for the RGB RAW data of the correction frame, and the first correction coefficients (Δ1, Δ2, Δ3) are offset values ​​for the RGB RAW data of the correction frame.

[0110] The offset value is a correction coefficient required to improve the color reproducibility of the correction frame in the color correction described below when the light source during the exposure time of the correction frame has a biased spectrum, such as monochromatic light, or when the light source during the exposure time of the reference frame includes all RGB wavelength bands but the light source during the exposure time of the correction frame is deficient in one or more of the RGB wavelength bands. Therefore, depending on the wavelength band of the light source, the first correction coefficients (Δ1, Δ2, Δ3) may be set to zero, or the first correction coefficients (Δ1, Δ2, Δ3) for offset may not be provided.

[0111] The first color correction unit 350 applies the first correction coefficients (α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, Δ3) calculated by the first correction coefficient calculation unit 340 to the correction frame, and performs color correction on the correction frame.

[0112] That is, when a frame of RAW data 300 input from the moving image acquisition unit 302 is a correction frame and the first color correction unit 350 corrects the pixel data of an R pixel in the RAW data of that correction frame, the first color correction unit 350 multiplies the pixel data (R) of the R pixel, the pixel data (G) of the G pixels surrounding the R pixel, and the pixel data (B) of the B pixels surrounding the R pixel by first correction coefficients (α1, β1, γ1) as shown in equation 1, and adds an offset value Δ1 to obtain the color-corrected pixel data of the R pixel.

[0113] When correcting pixel data of a G pixel in the RAW data of a correction frame, the first color correction unit 350 multiplies the pixel data (R) of the R pixel surrounding the G pixel, the pixel data (G) of the G pixel, and the pixel data (B) of the B pixel surrounding the G pixel by first correction coefficients (α2, β2, γ2) as shown in equation (2), and adds an offset value Δ2 to obtain color-corrected pixel data of the G pixel.When correcting pixel data of a B pixel in the RAW data of a correction frame, the first color correction unit 350 multiplies the pixel data (R) of the R pixel surrounding the B pixel, the pixel data (G) of the G pixel surrounding the B pixel, and the pixel data (B) of the B pixel by first correction coefficients (α3, β3, γ3) as shown in equation (3), and adds an offset value Δ3 to obtain color-corrected pixel data of the G pixel.

[0114] According to the image processing device having the above configuration, the color tone of the RAW data of the correction frame can be matched to or made closer to the color tone of the reference frame.

[0115] Furthermore, unlike the white balance correction unit 234 that performs white balance correction by multiplying R data, G data, and B data by WB gains Gr, Gg, and Gb according to the light source type, color temperature, etc., this image processing device performs image processing that is independent of the light source type, color temperature, etc., and can match the color of the correction frame to the color of the reference frame regardless of light source flicker.

[0116] The linear matrix processing unit of the digital signal processing unit 206 performs matrix operations on RGB color image data using 3 × 3 fixed coefficients, and can apply values ​​obtained by multiplying these fixed coefficients by first correction coefficients (α1, α2, α3, β1, β2, β3, γ1, γ2, γ3). The offset processing unit of the digital signal processing unit 206 corrects the influence of dark current by subtracting a signal component (offset value) generated in the optical black region of the image sensor 201, and can apply values ​​obtained by adding the first correction coefficients (Δ1, Δ2, Δ3) to the offset value of the offset processing unit. This allows the linear matrix processing unit and offset processing unit to function as the first color correction unit 350.

[0117] Fig. 5 is a diagram showing an embodiment of a stacked CMOS showing an example of the hardware configuration of an image processing device. In Fig. 5, parts common to those in the image processing device shown in Fig. 4 are given the same reference numerals, and detailed descriptions thereof will be omitted.

[0118] 5, pixel data of a reference frame is temporarily stored in the reference frame storage area 314, and pixel data of a correction frame is temporarily stored in the correction frame storage area 316. The pixel data of the first reference image area (first reference color signal (R ref , G ref , B ref )) and the pixel data of the first correction image area (first correction color signals (R, G, B)) are read out in parallel and applied to the first correction coefficient calculation unit 340.

[0119] The first correction coefficient calculation unit 340 calculates the first reference color signals (R ref , G ref , B ref ) and the first corrected color signals (R, G, B) of the first corrected image area, a first correction coefficient is calculated to match the first corrected color signals of the first corrected image area with the first reference color signals of the first reference image area.

[0120] The first color correction unit 350 performs correction processing on pixel data of a correction frame read out from the correction frame storage area 316 using the first correction coefficient calculated by the first correction coefficient calculation unit 340, and outputs the result.

[0121] In this way, by performing calculations for each corresponding first image region of the reference frame and the correction frame in parallel, the calculation time for the first correction coefficients is shortened, and color correction of the correction frame can be performed in real time.

[0122] [Second embodiment of image processing device] FIG. 6 is a block diagram showing a second embodiment of an image processing apparatus according to the present invention.

[0123] 6 does not identify a reference frame and a correction frame in accordance with the flicker phase of a light source as in the first embodiment, but can, for example, identify the first frame when capturing a moving image as the reference frame and identify subsequent frames as correction frames, or can identify a frame specified by the user using the operation unit 222 as the reference frame and identify other frames as correction frames. The method of identifying the reference frame and the correction frame is not limited to this, and, for example, the frame in which a main subject such as a person is first detected can be set as the reference frame, and subsequent frames in which the main subject is captured can be set as correction frames.

[0124] The first image region determination unit 321 has an object detection unit 324, and determines a calculation region corresponding to an object in a frame detected by the object detection unit 324. The object detection unit 324 extracts image features and detects one or more objects in the frame, such as a person, a building, the sky, or the ground, based on the extracted features, and can use a machine-learned learner for object detection.

[0125] The first image area determination unit 321 determines the calculation area based on the object in the frame detected by the object detection unit 324. When multiple objects are detected, the object closest to the center of the screen or the object with the highest priority can be selected.

[0126] Fig. 7 shows the reference frame F of the video captured outdoors. ref 10A and 10B are diagrams showing an example of a corrected frame F of a moving image captured indoors.

[0127] In the example shown in FIG. ref A person is detected as an object in the correction frame F, and the first image area determination unit 321 determines the reference frame F using the detected person as a reference. refA calculation area A in the correction frame F and a calculation area C in the correction frame F can be determined.

[0128] The first image region determining unit 321 determines the reference frame F ref In the calculation area A of the corrected frame F, n first reference image areas B1 to Bn are determined, and n first corrected image areas D1 to Dn are determined in the calculation area C of the corrected frame F. In the example shown in Fig. 7, the first reference image areas B1 to Bn and the first corrected image areas D1 to Dn are located in the same positions. In Fig. 7, only one first reference image area B1 and one first corrected image area D1 are shown.

[0129] Returning to FIG. 6, the first color signal acquisition unit 330 acquires a first reference color signal (R ref , G ref , B ref ), and similarly, first corrected color signals (R, G, B) are obtained for each of the first corrected image regions D1 to Dn from the color image data of the first corrected image regions D1 to Dn determined by the first image region determination unit 321.

[0130] The first correction coefficient calculation unit 340 calculates a plurality of pairs of first reference color signals (R ref , G ref , B ref ) and the first correction color signal (R, G, B), the first correction color signal (R, G, B) is calculated based on the first reference color signal (R ref , G ref , B ref ) to calculate first correction coefficients (α1 to α3, β1 to β3, γ1 to γ3, Δ1 to Δ3 in equations [1] to [3]).

[0131] The first color correction unit 350 applies the first correction coefficients (α1, α2, α3, β1, β2, β3, γ1, γ2, γ3, Δ1, Δ2, Δ3) calculated by the first correction coefficient calculation unit 340 to the correction frame, and performs color correction on the correction frame.

[0132] When the light source is switched from natural light outdoors to monochromatic lighting indoors, the color of the subject will be significantly different even if white balance correction is performed by multiplying the R data, G data, and B data by WB gains Gr, Gg, and Gb corresponding to these light sources.However, according to the second embodiment, the moving image captured indoors (subject in calculation area C) can be made to approach the color of the subject in calculation area A captured outdoors.

[0133] The color of the background area other than the calculation area C of the correction frame F is the same as that of the reference frame F. ref Although the color of the calculation area C including the subject (main subject) of the correction frame F does not necessarily approach the color of the background area other than the calculation area A of the reference frame F, ref The color can be made closer to the color of calculation area A.

[0134] Figure 8 shows the reference frame F ref 10A and 10B are diagrams illustrating an example of a correction frame F.

[0135] The reference frame F shown in FIG. ref The face image is included as an object image to be color-matched, and the face region including this face image is set as the calculation region A.

[0136] The object detection unit 324 (FIG. 6) detects a face image from the corrected frame F, and the first image area determination unit 321 determines the face area including the face image detected by the object detection unit 324 as the calculation area C.

[0137] As shown in Figure 8, the reference frame F ref When the position and size of the calculation area A in the reference frame F differs from the position and size of the calculation area C in the correction frame F, the object detection unit 324 ref Facial parts (e.g., eyes, nose, mouth, cheeks, forehead, chin, eyebrows, hair, etc.) are detected from the face area included in the calculation area A of the correction frame F, and similarly, facial parts are detected from the face area included in the calculation area C of the correction frame F.

[0138] The first image region determination unit 321 determines the reference frame F detected by the object detection unit 324. refThe area of ​​each facial feature included in the calculation area A of the correction frame F can be set as the first reference image area, and the area of ​​each facial feature included in the calculation area C of the correction frame F can be set as the first corrected image area.

[0139] By determining the first reference image area and the first correction image area for calculating the first correction coefficient in this manner, the color of a specific subject in the correction frame can be made closer to the preferred color of the specific subject that has been determined in advance.

[0140] [Third embodiment of image processing device] FIG. 9 is a diagram showing a schematic diagram of the relationship between the flicker of the light source, the exposure operation of the image sensor, and the moving image data.

[0141] 9 repeatedly blinks in accordance with the frequency of the commercial power supply. Furthermore, the image sensor 201 is driven by a rolling shutter system when capturing moving images, and the timing of the start of exposure is shifted for each scanning line of one frame.

[0142] Therefore, due to flickering of the light source, the amount of incident light from the light source and the spectral ratio of RGB during exposure time differ depending on the position of the scan line, even within the same frame, and the color changes depending on the scan line, even within the same frame.

[0143] FIG. 10 is a block diagram showing a third embodiment of an image processing apparatus according to the present invention.

[0144] The image processing device of the third embodiment shown in FIG. 10 corrects changes in color within the same frame, and is mainly composed of a moving image acquisition unit 303, a line image identification unit 410, a second image area determination unit 420, a second color signal acquisition unit 430, a second correction coefficient calculation unit 440, and a second color correction unit 450.

[0145] 9, the image sensor 201 is driven by a rolling shutter system under a light source having flicker. The image sensor 201 acquires the RAW data 301, and outputs the acquired RAW data 301 to the line image specification unit 410 and the second color correction unit 450.

[0146] The line image specification unit 410 specifies, in each frame of the RAW data 301, an image on a reference scan line as a reference line image, and specifies an image on a scan line adjacent to the reference scan line image as a correction line image.

[0147] The first reference scan line of each frame can be, for example, the first scan line in one frame. The first reference scan line of each frame can also be a scan line that is less affected by flicker. A scan line that is less affected by flicker is a scan line that has less change in light source luminance during the exposure time of the scan line.

[0148] Furthermore, the line image specifying unit 410 specifies a corrected line image that has been color-corrected by a second color correcting unit 450 (described later) as a reference line image for the next adjacent corrected line image.

[0149] The second image area determination unit 420 is a part that determines the image area to be used for calculating the second correction coefficient within each line image from the reference line image and the correction line image identified by the line image identification unit 410, and determines a second reference image area from the reference line image within each calculation area of ​​the reference line image and the correction line image, and determines a second corrected image area corresponding to the second reference image area from the correction line image.

[0150] In this example, the second reference image area and the second corrected image area can be areas at the same position within the line image.

[0151] The second reference image area can be, for example, an image area of ​​specific pixels at a plurality of pixel positions set in advance within the calculation area of ​​the reference line image, or surrounding pixels including the specific pixels. It is desirable to select a plurality of specific pixels so that the pixels in the entire area of ​​the line image are included.

[0152] The second corrected image area is an area within the calculation area of ​​the corrected line image, and corresponds to the second reference image area.

[0153] The reference line image and correction line image in this example are line images on adjacent scanning lines, have similar exposure times, and have a high correlation of subjects, making it easy to extract color changes.

[0154] The second color signal acquisition unit 430 calculates a representative value of pixel data of R pixels, a representative value of pixel data of G pixels, and a representative value of pixel data of B pixels of the second reference image region of the RAW data from the color image data of the second reference image region determined by the second image region determination unit 420, for each of the plurality of second reference image regions, as a second reference color signal (R ref , G ref , B ref ) is obtained.

[0155] Similarly, the second color signal acquisition unit 430 acquires, from the RGB RAW data of the second corrected image area determined by the second image area determination unit 420, a representative value of the pixel data of the R pixels, a representative value of the pixel data of the G pixels, and a representative value of the pixel data of the B pixels of the second corrected image area of ​​the RAW data as second corrected color signals (R, G, B) for each of the multiple second corrected image areas.

[0156] It is desirable to increase the number of pairs of second reference color signals and second correction color signals to be compared, because this increases the number of pairs of second reference color signals and second correction color signals used in the calculation, thereby reducing errors in the second correction coefficients that are caused by low correlation between local lines of the subject.

[0157] The second correction coefficient calculation unit 440 converts the second corrected color signals (R, G, B) into the second reference color signals (Rref , G ref , B ref The calculation of the second correction coefficient can be performed in the same way as the calculation of the first correction coefficient between frames.

[0158] The second color correction unit 450 applies the second correction coefficients (α1 to α3, β1 to β3, γ1 to γ3, Δ1 to Δ3) calculated by the second correction coefficient calculation unit 440 to the correction line image, and performs color correction on the correction line image. Note that the offset values ​​(Δ1 to Δ3) of the second correction coefficients can be omitted.

[0159] The second color correction unit 450 applies the second correction coefficient calculated for each line to the correction line image of each line of the RAW data 300 input from the moving image acquisition unit 303 to perform color correction, and outputs the color-corrected correction line image. The line image identification unit 410 acquires the correction line image color-corrected by the second color correction unit 450 as a reference line image for the next correction line image.

[0160] By sequentially correcting the color of the corrected line images in this way, it is possible to correct changes in color (color unevenness, brightness unevenness) within the same frame.

[0161] Fig. 11 is a block diagram showing a fourth embodiment of an image processing device according to the present invention. In Fig. 11, parts common to those of the image processing device of the third embodiment shown in Fig. 10 are given the same reference numerals, and detailed description thereof will be omitted.

[0162] The image processing device of the fourth embodiment shown in Figure 11 corrects color changes within the same frame, similar to the third embodiment, but differs mainly in the method by which the line image identification unit 411 identifies the reference line image and the correction line image, and the method by which the second image area determination unit 421 determines the image area to be compared.

[0163] In each frame of the moving image data, the line image specifying unit 411 specifies a line image on a reference scan line as a reference line image, and specifies a line image on a scan line other than the reference scan line as a correction line image. Here, the reference scan line can be a scan line that is less affected by flicker.

[0164] The second image area determination unit 421 is a part that determines the image area to be used for calculating the second correction coefficient within each line image from the reference line image and the correction line image identified by the line image identification unit 411, and determines a second reference image area from the reference line image within each calculation area of ​​the reference line image and the correction line image, and determines a second corrected image area corresponding to the second reference image area for each correction line image.

[0165] Here, since it is assumed that the reference line image and the correction line image are separated within the frame, the second image area determination unit 421 determines a plurality of image areas including pixels in the scanning line direction (horizontal direction) that have similar colors from the reference line image and the correction line image. Whether the colors are similar can be determined, for example, by calculating (RG) / (BG) from pixel data (R, G, B) and determining whether the difference in the calculation result is equal to or less than a reference value. Note that in this example, since RAW data is used as input data, for example, if the specific pixel to be compared is pixel data of an R pixel, the pixel data of the G pixels and B pixels adjacent to the R pixel is used to determine whether the colors are similar.

[0166] The second image area determination unit 421 also includes an object detection unit 424 that detects an object such as a face image from the frame. The second image area determination unit 421 preferably determines, as the second corrected image area, an image area of ​​the corrected line image that includes the same object as the object included in the second reference image area of ​​the reference line image, detected by the object detection unit 424, and that includes pixels of similar color.

[0167] According to the fourth embodiment, even if the reference line image and the correction line image are separated from each other within a frame, it is possible to determine the specific pixel to be used for color correction.

[0168] In the case of moving image data acquired by driving an image sensor using a rolling shutter method under a light source having flicker, it is desirable to correct color changes between frames using the image processing device of the first or second embodiment, and to correct color changes within the same frame using the image processing device of the third or fourth embodiment. In this case, it is desirable to first correct color changes within the same frame, and then correct color changes between frames.

[0169] Furthermore, in this example, a case has been described in which RAW data is used as moving image data to be corrected, but this is not limiting. For example, moving image data to be corrected may be moving image data after linear matrix processing, synchronized RGB data after demosaic processing, or YCrCb data obtained by converting synchronized RGB data through RGB / YCrCb conversion processing into luminance data (Y) and color difference data (Cr, Cb).

[0170] When YCrCb data is used as the moving image data to be corrected, it goes without saying that the YCrCb data is used instead of the RGB data in equations 1 to 3. Furthermore, the imaging element is not limited to one equipped with RGB color filters, and may include other colors such as yellow and emerald.

[0171] [Image processing method] 12 is a flowchart showing an embodiment of an image processing method according to the present invention. The image processing method shown in FIG. 12 corresponds to the image processing device of the first embodiment shown in FIG.

[0172] 12, the main body CPU 220 is set to a second moving image capturing mode for capturing moving images for extracting still images, and determines whether or not a command to start capturing moving images has been input from the shutter release switch 22 during standby for capturing moving images (step S10). Upon receiving the command to start capturing moving images, the main body CPU 220 drives the image sensor 201 via the image sensor control unit 202 to capture moving images for extracting still images, and causes the digital signal processing unit 206, which functions as an image processing device, to perform image processing on the RAW data, which is the captured moving image data.

[0173] 4 acquires one frame of RAW data 300 (step S12). The frame identification unit 310 identifies whether the one frame acquired by the video acquisition unit 302 is a reference frame or a correction frame among the multiple frames that make up the video, and acquires the identified reference frame or correction frame (step S14). The reference frame or correction frame can be identified based on the flicker phase detected by the flicker phase detection unit 312.

[0174] The first image area determination unit 320 determines a plurality of first reference image areas to be used in calculating the first correction coefficient from the reference frame identified by the frame identification unit 310, and determines a plurality of first corrected image areas that respectively correspond to the plurality of first reference image areas from the corrected frame identified by the frame identification unit 310 (step S16).

[0175] The first color signal acquisition unit 330 acquires a first reference color signal (R ref , G ref , B ref ), and obtains first corrected color signals (R, G, B) for each of the first corrected image regions from the RAW data of the plurality of first corrected image regions (step S18).

[0176] The first correction coefficient calculation unit 340 calculates the first reference color signals (R ref , G ref , B ref ) and the first correction color signal (R, G, B), the first correction color signal (R, G, B) is calculated based on the first reference color signal (R ref , G ref , B ref ) are calculated (step S20).

[0177] The first color corrector 350 applies the first correction coefficient calculated by the first correction coefficient calculator 340 to the corrected frame input from the video acquirer 302, and performs color correction on the corrected frame (step S22). By performing this color correction on the corrected frame, the color of the corrected frame can be made closer to the color of the reference frame.

[0178] Next, the main body CPU 220 determines whether or not an instruction to end video capture has been input from the shutter release switch 22 (step S24). If an instruction to end video capture has not been input, the main body CPU 220 transitions to step S12. This causes the processing of steps S12 to S24 to be executed for the next frame. On the other hand, if an instruction to end video capture has been received, video capture and image processing in the second video capture mode are terminated.

[0179] In the image processing method of the above embodiment, the RAW data acquired by capturing a moving image is corrected in real time, but for example, the RAW data may be temporarily recorded in a storage medium (memory card 212 or flash ROM 226), and then the RAW data may be read from the storage medium and corrected.

[0180] The imaging device 10 of this embodiment is a mirrorless digital single-lens camera, but is not limited to this and may be a single-lens reflex camera, an imaging device with an integrated lens, a digital video camera, etc. It can also be applied to mobile devices that have imaging functions as well as other functions (call functions, communication functions, other computer functions). Other aspects to which the present invention can be applied include, for example, mobile phones and smartphones with camera functions, PDAs (Personal Digital Assistants), and portable game consoles. An example of a smartphone to which the present invention can be applied is described below.

[0181] <Smartphone configuration> Fig. 13 shows the appearance of a smartphone 500, which is one embodiment of the imaging device of the present invention. The smartphone 500 shown in Fig. 13 has a flat housing 502, and is provided with a display input unit 520, which is an integrated unit of a display panel 521 serving as a display unit and an operation panel 522 serving as an input unit, on one side of the housing 502. The housing 502 also has a speaker 531, a microphone 532, an operation unit 540, and a camera unit 541. Note that the configuration of the housing 502 is not limited to this, and for example, a configuration in which the display unit and the input unit are independent, or a configuration having a foldable structure or a sliding mechanism, can also be used.

[0182] Fig. 14 is a block diagram showing the configuration of smartphone 500 shown in Fig. 13. As shown in Fig. 14, the smartphone includes, as main components, a wireless communication unit 510, a display input unit 520, a call unit 530, an operation unit 540, a camera unit 541, a storage unit 550, an external input / output unit 560, a GPS receiving unit 570 (GPS: Global Positioning System), a motion sensor unit 580, a power supply unit 590, and a main control unit 501. Main control unit 501 functions as a moving image acquisition unit, a frame identification unit, a first image region determination unit, a first color signal acquisition unit, a first correction coefficient calculation unit, and a first color correction unit.

[0183] The wireless communication unit 510 performs wireless communication with a base station included in the mobile communication network in accordance with instructions from the main control unit 501. Using this wireless communication, it transmits and receives various file data such as audio data and image data, e-mail data, and the like, and receives web data, streaming data, and the like.

[0184] The display input unit 520 is a so-called touch panel that, under the control of the main control unit 501, displays images (still images and moving images), text information, etc. to visually convey information to the user and detects user operations on the displayed information, and is equipped with a display panel 521 and an operation panel 522.

[0185] An LCD (Liquid Crystal Display), an OLED (Organic Electro-Luminescence Display), or the like is used as the display device for the display panel 521. The operation panel 522 is placed so that an image displayed on the display surface of the display panel 521 can be seen, and is a device that detects one or more coordinates operated by a user's finger, pen, or the like. When such a device is operated by a user's finger, pen, or the like, a detection signal generated by the operation is output to the main control unit 501. Next, the main control unit 501 detects the operation position (coordinates) on the display panel 521 based on the received detection signal.

[0186] 13 , display panel 521 and operation panel 522 of smartphone 500, exemplified as an embodiment of the imaging device of the present invention, are integrated to form display input unit 520, with operation panel 522 being arranged so as to completely cover display panel 521. When such an arrangement is adopted, operation panel 522 may also have a function for detecting user operations in an area outside display panel 521. In other words, operation panel 522 may have a detection area for the overlapping portion that overlaps display panel 521 (hereinafter referred to as a display area), and a detection area for the remaining outer edge portion that does not overlap display panel 521 (hereinafter referred to as a non-display area).

[0187] The call unit 530 includes a speaker 531 and a microphone 532, and can convert the user's voice input through the microphone 532 into voice data that can be processed by the main control unit 501 and output the voice data to the main control unit 501, and can decode voice data received by the wireless communication unit 510 or the external input / output unit 560 and output the decoded voice data from the speaker 531. Also, as shown in FIG. 13 , for example, the speaker 531 can be mounted on the same surface as the surface on which the display input unit 520 is provided, and the microphone 532 can be mounted on the side surface of the housing 502.

[0188] The operation unit 540 is a hardware key using a key switch or the like, and is a device that accepts instructions from a user. For example, as shown in Fig. 13, the operation unit 540 is a push-button switch that is mounted on the side of the housing 502 of the smartphone 500, turns on when pressed with a finger or the like, and turns off when the finger is released by the restoring force of a spring or the like.

[0189] The storage unit 550 stores the control program of the main control unit 501, control data, application software (including a program for realizing the image processing method according to the present invention), address data associating names and telephone numbers of communication partners, data of emails sent and received, web data downloaded through web browsing, and downloaded content data, and also temporarily stores streaming data, etc. The storage unit 550 is composed of an internal storage unit 551 built into the smartphone and an external storage unit 562 having a removable external memory slot. The internal storage unit 551 and the external storage unit 552 constituting the storage unit 550 are realized using known storage media.

[0190] The external input / output unit 560 serves as an interface with all external devices connected to the smartphone 500. The smartphone 500 is directly or indirectly connected to other external devices via the external input / output unit 560 through communication or the like. Examples of communication or the like include a universal serial bus (USB), IEEE1394, and a network (e.g., the Internet, a wireless local area network (LAN)). Other examples of communication or the like include Bluetooth (registered trademark), RFID (Radio Frequency Identification), Infrared Data Association (IrDA) (registered trademark), UWB (Ultra Wideband) (registered trademark), and ZigBee (registered trademark).

[0191] Examples of external devices connected to the smartphone 500 include a wired / wireless headset, a wired / wireless external charger, a memory card connected via a wired / wireless data port or a card socket, and a Subscriber Identity Module Card (SIM) / User Identity Module Card (UIM) card. In addition, external devices such as external audio and video devices connected via an audio and video I / O (Input / Output) terminal, wirelessly connected external audio and video devices, a wired / wireless connected smartphone, a wired / wireless connected PDA, a wired / wireless connected personal computer, and earphones can also be connected. The external input / output unit 560 can transfer data received from such external devices to various internal components of the smartphone 500 and can transmit data from within the smartphone 500 to external devices.

[0192] The motion sensor unit 580 includes, for example, a three-axis acceleration sensor and a tilt sensor, and detects the physical movement of the smartphone 500 in accordance with instructions from the main control unit 501, thereby detecting the direction of movement, acceleration, and attitude of the smartphone 500. The detection results are output to the main control unit 501. The power supply unit 590 supplies power stored in a battery (not shown) to each unit of the smartphone 500 in accordance with instructions from the main control unit 501.

[0193] The main control unit 501 includes a microprocessor, operates according to the control program and control data stored in the storage unit 550, and controls all the units of the smartphone 500. The main control unit 501 also includes a mobile communication control function that controls all the units of the communication system to perform voice communication and data communication via the wireless communication unit 510, and an application processing function.

[0194] The main control unit 501 also has an image processing function for displaying video on the display input unit 520 based on image data (still image or moving image data) such as received data or downloaded streaming data. The image processing function refers to a function in which the main control unit 501 decodes the image data, performs image processing on the decoding result, and displays the image on the display input unit 520.

[0195] The camera unit 541 is a digital camera that takes electronic photographs using an imaging element such as a CMOS or a CCD, and corresponds to the imaging device 10 shown in Fig. 1. Furthermore, under the control of the main control unit 501, the camera unit 541 can convert image data (moving images and still images) obtained by imaging into compressed image data such as MPEG or JPEG, and record the converted image data in the storage unit 550 or output the converted image data via the external input / output unit 560 or the wireless communication unit 510.

[0196] Furthermore, the camera unit 541 can be used for various functions of the smartphone 500. For example, an image acquired by the camera unit 541 can be displayed on the display panel 521, or an image from the camera unit 541 can be used as one of the operation inputs for the operation panel 522. Furthermore, when the GPS receiver 570 detects a location, the location can be detected by referring to an image from the camera unit 541. Furthermore, by referring to the image from the camera unit 541, it is possible to determine the direction of the optical axis of the camera unit 541 of the smartphone 500 or the current usage environment without using a triaxial acceleration sensor or by using the image in combination with a triaxial acceleration sensor. Of course, the image from the camera unit 541 can also be used in application software.

[0197] In addition, image data of a still image or a moving image can be added with location information acquired by the GPS receiving unit 570, audio information acquired by the microphone 532 (which may be converted to text information by the main control unit, etc.), posture information acquired by the motion sensor unit 580, etc., and recorded in the memory unit 550, or output via the external input / output unit 560 or wireless communication unit 510.

[0198] 13, in smartphone 500, camera unit 541 is mounted on the same surface as display input unit 520, but the mounting position of camera unit 541 is not limited to this, and camera unit 541 may be mounted on the back surface of display input unit 520, or multiple camera units 541 may be mounted. Note that when multiple camera units 541 are mounted, it is possible to shoot an image by switching between camera units 541 used for shooting and to shoot an image individually, or to use multiple camera units 541 simultaneously.

[0199] [others] The image processing device of this embodiment is built into the imaging device 10, but may also be an image processing device configured by an external computer or the like.

[0200] The present invention also includes an image processing program that is installed in a computer within an imaging device or an external computer to cause the computer to function as an image processing device according to the present invention, and a storage medium on which this image processing program is recorded.

[0201] Furthermore, in this embodiment, the first correction coefficient and the second correction coefficient are calculated within the imaging device, and the calculated first correction coefficient and the second correction coefficient are applied to the correction frame to correct the color tone between frames or within the same frame in real time. However, this is not limited to this. For example, the imaging device may acquire video data (RAW data) and calculate a correction coefficient (the first correction coefficient, or the first correction coefficient and the second correction coefficient), associate the video data with the correction coefficient and save it, and the imaging device may acquire the video data and the correction coefficient in a correction mode, or an external device such as a computer may acquire the video data and the correction coefficient to correct the color tone between frames of the video data or within the same frame. In this case, the correction coefficient may be recorded in the header of the video file or in a separate file associated with the video file.

[0202] Furthermore, when the image sensor is driven by the global shutter method, color correction between line images within the same frame is not necessary, and color correction is performed between the reference frame and the correction frame. In this case, color correction can be performed between the same area in the reference frame and the correction frame, or between the same subject in the reference frame and the correction frame detected by object detection.

[0203] In this embodiment, the hardware structure of the processing units that execute various processes, such as the main body CPU 220 and the lens CPU 120, is made up of the following various processors: The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electrical circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing specific processes.

[0204] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a form in which a single processor is configured with a combination of one or more CPUs and software, as typified by computers such as client and server, and this processor functions as multiple processing units. Second, a form in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a system on chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0205] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0206] Furthermore, it goes without saying that the present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the present invention. [Explanation of symbols]

[0207] 10. Imaging device 20 Viewfinder window 22 Shutter release switch 23 Shutter speed dial 24 Exposure compensation dial 25 Power Lever 26 Eyepiece 27 MENU / OK key 28 D-pad 29 Play button 30 Built-in flash 100 interchangeable lenses 102 Imaging optical system 104 Lens Group 108 aperture 116 Focus lens control unit 118 Aperture control section 120 Lens side CPU 122,207 RAM 124, 228 ROM 126 Flash ROM 150 Lens communication unit 160 lens mount 200 camera body 201 Image sensor 202 Image sensor control unit 203 Analog signal processing section 204 A / D converter 205 Image Input Controller 206 Digital Signal Processing Unit 208 Compression / Expansion Processing Unit 210 Media Control Unit 212 Memory Card 214 Display control unit 216 LCD monitor 220 Main unit CPU 222 Operation section 226 Flash ROM 230 AF detection unit 232 AE control section 234 White balance correction section 236 Radio Communication Department 238 GPS receiver 240 Power supply control unit 242 Battery 244 Lens power switch 246 Main unit communication unit 248 Body Mount 250 Main unit communication unit 259 FPS control section 260 Body Mount 260A terminal 270 Flash unit 272 Flash control unit 280 focal plane shutter 296 FPS control unit 300 RAW data 301 RAW data 302 Video acquisition unit 303 Video acquisition unit 310 Frame Identification Part 311 Frame Identification Part 312 Flicker phase detector 314 Reference Frame Storage Area 316 Correction Frame Storage Area 320, 321 First image area determination unit 322 First Image Area 324 Object Detection Unit 330 First color signal acquisition unit 340 First correction coefficient calculation unit 350 1st color correction unit 410 Line image identification unit 411 Line image identification unit 420 Second image area determination unit 421 Second image area determination unit 424 Object Detection Unit 430 Second color signal acquisition unit 440 Second correction coefficient calculation unit 450 Second Color Correction Unit 500 smartphones 501 Main control unit 502 Case 510 Wireless Communication Department 520 Display and input section 521 Display Panel 522 Operation Panel 530 Telephone section 531 Speaker 532 Microphone 540 Operation section 541 Camera Club 550 Storage section 551 Internal storage 552 External Memory Unit 560 External input / output section 562 External Memory Unit 570 GPS receiver 580 Motion Sensor Unit 590 Power supply section A calculation area B1 First reference image area C calculation area D1 First corrected image area F Correction Frame F ref Reference Frame S10~S24 steps

Claims

1. 1. An image processing device including one or more processors, The one or more processors: a video image acquisition process for acquiring video image data; a frame identification process for identifying a reference frame and a correction frame for the reference frame from a plurality of time-series frames constituting the moving image data acquired by the moving image acquisition process when the acquired moving image data is captured in an environment where a light source environment fluctuates within a specific time period; a first reference signal acquisition process for acquiring a first reference color signal related to color from at least a partial region of the reference frame; a first correction signal acquisition process for acquiring a first correction color signal related to color from an area of ​​the correction frame corresponding to at least a part of an area of ​​the reference frame; a process of correcting the correction frame based on the first reference color signal and the first correction color signal; 1. An image processing device configured to perform the

2. the moving image data is captured by sequentially exposing pixels formed on an imaging element for at least one pixel or scanning line, 2. The image processing device according to claim 1, wherein at least a portion of the reference frame and an area of ​​the correction frame corresponding to at least a portion of the reference frame are image areas in which the exposure start and end timing of the imaging element are the same in the reference frame and the correction frame.

3. 3 . The image processing device according to claim 1 , wherein at least a part of the reference frame and an area of ​​the correction frame corresponding to at least a part of the reference frame are positioned at the same position within the frame.

4. The image processing device according to claim 1 , wherein the moving image acquisition process acquires the moving image data made up of a plurality of frames, each frame having an exposure time shorter than a frame interval.

5. an imaging element for capturing moving images; The image processing device according to any one of claims 1 to 4, The moving image acquisition process is an imaging device that acquires moving image data representing the moving image captured by the imaging element.

6. acquiring video data; a step of identifying a reference frame and a correction frame for the reference frame from a plurality of time-series frames constituting the moving image data when the moving image data is captured in an environment where a light source environment fluctuates within a specific time period; obtaining a first reference color signal relating to color from at least a partial region of the reference frame; obtaining a first corrected color signal related to color from an area of ​​the correction frame corresponding to at least a portion of an area of ​​the reference frame; correcting the correction frame based on the first reference color signal and the first correction color signal; An image processing method comprising:

7. A function for acquiring video data; a function of identifying a reference frame and a correction frame for the reference frame from a plurality of time-series frames constituting the moving image data when the moving image data is captured in an environment where a light source environment fluctuates within a specific time period; a function of acquiring a first reference color signal relating to color from at least a partial region of the reference frame; a function of acquiring a first corrected color signal related to color from an area of ​​the correction frame corresponding to at least a part of an area of ​​the reference frame; a function of correcting the correction frame based on the first reference color signal and the first correction color signal; An image processing program that realizes this on a computer.

8. A non-transitory computer-readable recording medium that causes a computer to execute the image processing program according to claim 7 when instructions stored in the recording medium are read by the computer.

Images