Image processing apparatus, image processing method, and camera device

Abstract

This invention provides an image processing apparatus, an image processing method, an image processing program, and a camera device capable of generating two moving image data sets that have undergone different image processing. The processor performs the following processes: calculating first image processing parameters from first moving image data obtained by capturing an image; generating second moving image data from the first moving image data based on the first image processing parameters; storing the first moving image data in a memory; and generating third moving image data from the first moving image data stored in the memory based on second image processing parameters calculated from the first moving image data after a time (T) has elapsed since the first moving image data was first stored in the memory.

Description

Technical Field

[0001] This invention relates to an image processing apparatus, an image processing method, an image processing program, and a camera device, and particularly to an image processing apparatus, an image processing method, an image processing program, and a camera device for processing dynamic image data. Background Technology

[0002] Patent document 1 describes a technique for processing white balance (WB) of a camera image, which evaluates the reliability of the color temperature detected from a scene or camera image, and changes the update rate of the WB gain based on the evaluated reliability.

[0003] Previous technical documents

[0004] Patent documents

[0005] Patent Document 1: Japanese Patent Application Publication No. 2008-172422 Summary of the Invention

[0006] One embodiment of the present invention provides an image processing apparatus, an image processing method, an image processing program, and a camera device capable of generating two dynamic image data that have undergone different image processing.

[0007] means for solving technical problems

[0008] (1) An image processing apparatus comprising a memory and a processor, wherein the processor performs the following processes: processing to calculate a first image processing parameter from first moving image data acquired by capturing an image; processing to generate second moving image data from the first moving image data based on the first image processing parameter; processing to store the first moving image data in the memory; and processing to generate third moving image data from the first moving image data stored in the memory based on a second image processing parameter calculated from the first moving image data after a time T has elapsed since the first moving image data was first stored in the memory.

[0009] (2) The image processing apparatus according to (1), wherein the processor further performs the following processes: processing of outputting the second dynamic image data to a display destination; and processing of outputting the third dynamic image data to a recording destination.

[0010] (3) The image processing apparatus according to (1) or (2), wherein the processor further performs processing to detect scene changes from the first dynamic image data obtained by shooting, and in the generation of the third dynamic image data, the processor generates a frame image captured at the moment the scene change begins, based on the second image processing parameters calculated from the first dynamic image data captured before the scene change.

[0011] (4) The image processing apparatus according to any one of (1) to (3), wherein the processor further performs processing to detect scene changes from the first dynamic image data obtained by shooting, and in the generation of the third dynamic image data, the processor generates a frame image captured at the time when the scene change ends, based on the second image processing parameters calculated from the first dynamic image data captured after the scene change.

[0012] (5) The image processing apparatus according to (3) or (4), wherein the processor causes the value of the second image processing parameter used in the generation of the frame image captured during the scene change to change continuously or in stages during the generation of the third dynamic image data.

[0013] (6) The image processing apparatus according to (5), wherein the processor causes the value of the second image processing parameter to change before the scene change begins during the generation of the third dynamic image data.

[0014] (7) The image processing apparatus according to (5) or (6), wherein the processor, in the generation of the third dynamic image data, causes the value of the second image processing parameter to stop changing after the scene change ends.

[0015] (8) The image processing apparatus according to (3) or (4), wherein, in the generation of the third dynamic image data, when the time of scene change exceeds time T, the value of the second image processing parameter used in the generation of frame images captured during the period from the start of the scene change to the time before the end of the scene change is fixed to the second image processing parameter calculated from the first dynamic image data captured at the start of the scene change.

[0016] (9) The image processing apparatus according to (8), wherein the processor, in the generation of the third dynamic image data, causes the value of the second image processing parameter used in the generation of frame images captured during the period from the time T before the end of the scene change to the end of the scene change to change continuously or in stages.

[0017] (10) The image processing apparatus according to (3) or (4), wherein, in the generation of the third dynamic image data, when the time of scene change exceeds time T, the processor fixes the value of the second image processing parameter used in the generation of frame images captured during the period from the start of the scene change to the end of the scene change to the second image processing parameter calculated from the first dynamic image data captured at the time of the start of the scene change.

[0018] (11) The image processing apparatus according to (10), wherein the processor further performs image processing on the generated third dynamic image data to smoothly change the tone before and after the scene change.

[0019] (12) The image processing apparatus according to (3) or (4), wherein when the processor generates the third dynamic image data at a variable frame rate, the value of the second image processing parameter used in the generation of frame images captured during scene changes varies equally between frames.

[0020] (13) An image processing apparatus according to any one of (1) to (12), wherein the image processing apparatus further comprises a terminal for connecting to an external device, and the processor further performs the following processing: processing for detecting the connection between the external recorder and the terminal; and processing for outputting third dynamic image data to the external recorder when the connection between the external recorder and the terminal is detected.

[0021] (14) An image processing apparatus according to any one of (1) to (12), wherein the image processing apparatus further comprises a terminal for connecting to an external display, and the processor further performs the following processing: a process for detecting the connection between the external display and the terminal; and a process for outputting second dynamic image data to an external recorder when the connection between the external display and the terminal is detected.

[0022] (15) The image processing apparatus according to any one of (1) to (14), wherein the processor further performs processing to notify that the time of generating the second moving image data and the time of generating the third moving image data are different.

[0023] (16) An image processing apparatus according to any one of (1) to (15), wherein the first image processing parameter and the second image processing parameter are image processing parameters required for white balance processing.

[0024] (17) An image processing apparatus comprising a memory and a processor, wherein the processor performs the following processes: receiving a switching instruction for image processing parameters applicable to a first moving image data acquired by shooting; detecting a scene change from the first moving image data; generating a second moving image data from the first moving image data according to the image processing parameters; storing the first moving image data in the memory; and generating a third moving image data from the first moving image data stored in the memory according to the image processing parameters, wherein in the generation of the third moving image data, when the image processing parameters are switched after a scene change, the processor generates a frame image captured at the start of the scene change according to the image processing parameters before the scene change, and generates a frame image captured at the end of the scene change according to the image processing parameters switched after the scene change.

[0025] (18) A camera device comprising: a camera element; and an image processing device of any one of (1) to (17) for processing moving image data output from the camera element by means of shooting.

[0026] (19) An image processing method comprising the following processes: processing of calculating a first image processing parameter from first moving image data obtained by shooting; processing of generating a second moving image data from the first moving image data according to the first image processing parameter; processing of storing the first moving image data in a memory; and processing of generating a third moving image data from the first moving image data stored in the memory according to a second image processing parameter calculated from the first moving image data after a time T has elapsed since the first moving image data was first stored in the memory.

[0027] (20) An image processing program for causing a computer to perform the following processes: processing to calculate a first image processing parameter from first moving image data obtained by shooting; processing to generate second moving image data from the first moving image data according to the first image processing parameter; processing to store the first moving image data in a memory; and processing to generate third moving image data from the first moving image data stored in the memory according to a second image processing parameter calculated from the first moving image data after a time T has elapsed since the first moving image data was first stored in the memory. Attached Figure Description

[0028] Figure 1 This is a block diagram illustrating an example of the hardware structure of a camera device.

[0029] Figure 2 This is a functional block diagram of WB processing performed by the image processing unit.

[0030] Figure 3 This is a conceptual diagram recording the settings for WB gain.

[0031] Figure 4 It is a conceptual diagram that records the generation of dynamic image data.

[0032] Figure 5 It is a conceptual diagram showing the generation of dynamic image data.

[0033] Figure 6 This is a conceptual diagram of the WB gain used for recording, which is set to begin before the scene change begins and end after the scene change ends.

[0034] Figure 7 This is a block diagram of a function that automatically outputs and records dynamic image data.

[0035] Figure 8 This is a block diagram of a function that automatically outputs and displays dynamic image data.

[0036] Figure 9 It is a conceptual diagram that records the generation of dynamic image data.

[0037] Figure 10 It is a conceptual diagram that records the generation of dynamic image data.

[0038] Figure 11 This is a diagram that schematically represents fixed frame rate and variable frame rate. Detailed Implementation

[0039] Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

[0040] [First Implementation]

[0041] As one of the image processing techniques in camera devices, AWB (Auto White Balance) is known. AWB estimates the color of the light source from the image obtained through shooting and corrects it to make the light source color closer to achromatic (gray). Therefore, if the exposure of the captured image deviates significantly from proper exposure, there is a problem that the light source color cannot be accurately estimated and cannot be properly corrected. For example, in scenes with large changes in brightness, such as from indoors to outdoors (scene change), it sometimes takes about one second to achieve proper exposure using AE (Automatic Exposure). During this time, the exposure becomes incorrect. AWB processes the image based on the properly exposed image, determining the color of the light source, etc., so it sometimes takes several seconds until the appropriate hue (color, hue) is achieved.

[0042] In this embodiment, an AWB (Audio-Visual Image) camera device that can record dynamic image data without being affected by scene changes will be described.

[0043] [Hardware structure of the camera device]

[0044] Figure 1 This is a block diagram illustrating an example of the hardware structure of a camera device.

[0045] The camera device 10 of this embodiment is a camera device capable of capturing moving images, and includes a camera optical system 12, a camera unit 14, a display unit 16, a storage unit 18, an input / output (I / F) interface 20, an operation unit 22, a CPU (Central Processing Unit) 24, a ROM (Read Only Memory) 26, and a RAM (Random Access Memory) 28.

[0046] The camera optical system 12 includes multiple lens groups for imaging the subject onto the camera element and an aperture for adjusting the amount of light. It may also include, or replace the aperture, an ND filter (Neutral Density Filter) as a mechanism for adjusting the amount of light.

[0047] The imaging unit 14 includes an imaging element that converts an optical image into an electrical signal. As the imaging element, a color CMOS (Complementary Metal Oxide Semiconductor) image sensor with a predetermined color filter arrangement (e.g., Bayer arrangement) can be used. In the imaging device 10 of this embodiment, the imaging element provided in the imaging unit 14 is configured including a driving unit, an ADC (Analog to Digital Converter), and a signal processing unit. At this time, the imaging element is driven by the built-in driving unit to operate. Furthermore, the signal of each pixel is converted into a digital signal by the built-in ADC. Moreover, the signal of each pixel is subjected to signal processing such as correlation double sampling, gain processing, and correction processing by the built-in signal processing unit. It is also possible to configure the system to perform signal processing on the analog signals of each pixel. Furthermore, it is also possible to configure the system to perform signal processing on the digital signals of each pixel. As the imaging element, in addition to the aforementioned CMOS image sensor, known image sensors such as CCD (Charged Coupled Device) image sensors can also be used. The camera operates at a preset frame rate. The frame rate can be fixed or arbitrarily set by the user.

[0048] Display unit 16 includes displays such as LCD (Liquid Crystal Display) and OLED (Organic Light Emitting Diode). During shooting, live view is displayed on display unit 16. Furthermore, display unit 16 also serves as a user interface when making various settings. Display unit 16 also includes an EVF (Electronic View Finder) display.

[0049] Storage unit 18 stores the captured video data (recorded video data). Storage unit 18 may be composed of non-volatile semiconductor memory such as EEPROM (Electrically Erasable Programmable Read-only Memory) or SSD (Solid State Drive) that includes flash memory. Storage unit 18 may be an integral part of the device body (so-called built-in memory) or a removable structure (so-called memory card).

[0050] The I / F20 input / output port has terminals for connecting external devices. As a communication standard, it can adopt, for example, HDMI (High-Definition Multimedia Interface) (HDMI is a registered trademark).

[0051] The operation unit 22 is equipped with various operating components for operating the camera device 10. In addition to the power button and the recording button, the operating components include various types of operation buttons, touch panels, etc.

[0052] The CPU 24 functions as the control unit and image processing unit of the imaging device 10 by executing a prescribed program. The control performed by the CPU 24 includes not only camera controls such as AE and AF (Auto Focus), but also display control of the display unit 16 and recording control of the storage unit 18. Furthermore, the image processing performed by the CPU 24 includes the generation of recording and display (live view) motion picture data and the compression of the recording motion picture data. The recording and display motion picture data is generated by performing prescribed image processing (so-called development processing) on ​​RAW data (unprocessed motion picture data output from the imaging unit 14). The development processing includes offset processing, gamma correction processing, depigmentation processing, RGB / YCrCb conversion processing, and WB processing (white balance processing). The development processing itself is a well-known process, so its detailed description is omitted. In the imaging device 10 of this embodiment, the WB processing differs between the recording motion picture data and the display motion picture data. This will be described later. Compression processing is the process of generating moving image files in a specified compressed format. The encoder used during compression can be a known encoder. For example, encoders standardized by MPEG (Moving Picture Experts Group) (MPEG-1, MPEG-2, MPEG-4, etc.), ProRes, etc., can be used. CPU24 is an example of a processor. Furthermore, in addition to being executed by CPU-based software, the above processing can also be partially or entirely performed using hardware located within the CPU.

[0053] ROM26 stores various data required for programs and control executed by CPU24. ROM includes EEPROM, which includes Flash Memory.

[0054] RAM28 serves as the working area for CPU24 to perform various processes. RAM28 is an example of a memory.

[0055] [WB Processing]

[0056] Figure 2 This is a functional block diagram of WB processing performed by the CPU.

[0057] Regarding WB processing, the CPU 24 functions as a WB gain calculation unit 24A, a delay processing unit 24B, a scene change detection unit 24C, a display WB gain setting unit 24D, a display development processing unit 24E, a recording WB gain setting unit 24F, and a recording development processing unit 24G. Each function is implemented by the CPU 24 executing a predetermined program (image processing program). WB processing is an example of image processing.

[0058] The WB gain calculation unit 24A calculates the WB gain based on the motion picture data (RAW data) output from the camera unit 14. In the case where the color filter arrangement of the camera element is, for example, a Bayer arrangement, the motion picture data output from the camera unit 14 becomes a mosaic image data of R (Red), G (Green), and B (Blue). The WB gain calculation unit 24A calculates the WB gain for each frame based on the mosaic image data. The motion picture data (RAW data) output from the camera unit 14 is an example of the first motion picture data.

[0059] WB gain can be calculated using known methods. As an example, WB gain is calculated using the following method: First, for each of a pre-defined plurality of segmented regions, the cumulative average value of each color signal (R, G, B) is calculated. For example, in 8×8 or 16×16 segments, the segmentation region is defined by equally dividing one frame. Next, based on the calculation results of the cumulative average values ​​of each color signal, color information is calculated for each segmented region. The color information is calculated as the ratio of the calculated cumulative average value of color information G to the cumulative average value of color information R (R / G) and the ratio of the cumulative average value of color information G to the cumulative average value of color information B (B / G). Next, the light source color (color temperature) is estimated based on the calculated color information (R / G, B / G) of each segmented region. Specifically, the light source color is estimated based on the distribution of the R / G and B / G axis coordinates in the color space. Next, the WB gain corresponding to the estimated light source color is read out, and the WB gain of that frame image is obtained. The WB gain information corresponding to the light source color is stored, for example, in ROM26.

[0060] The delay processing unit 24B uses RAM 28 as a buffer memory, temporarily storing the dynamic image data (RAW data) sequentially output from the camera unit 14 in RAM 28 and delaying it. The delay processing unit 24B causes RAM 28 to store a preset number of frame images. The number of stored frame images is set to n. The delay processing unit 24B retrieves the frame images after a time (T) of n frames has elapsed since they were stored in RAM 28. Corresponding to the amount freed up by the retrieval, one new frame image is stored in RAM 28. The delay processing unit 24B performs delay processing by repeating this series of actions. Therefore, the delayed time T is determined by the number of frames (n) stored in RAM 28 and the frame rate.

[0061] The scene change detection unit 24C detects scene changes from the dynamic image data (RAW data) acquired through shooting. In this embodiment, scene changes are detected by detecting changes in brightness from the dynamic image data (RAW data) acquired through shooting.

[0062] The display WB gain setting unit 24D sets the WB gain required for WB processing when generating display motion picture data. Hereinafter, the WB gain required for WB processing of the display motion picture data is referred to as the display WB gain. The display WB gain setting unit 24D sets the display WB gain based on the WB gain calculated by the WB gain calculation unit 24A. In this embodiment, for each frame of the display motion picture data, the WB gain obtained by weighted averaging with the previous frame is set as the display WB gain. Therefore, for example, for the Nth frame, the WB gain obtained by weighted averaging the WB gain calculated from the Nth frame and the WB gain calculated from the (N-1)th frame is set as the display WB gain. The display WB gain setting unit 24D sets the WB gain calculated by the WB gain calculation unit 24A (in this embodiment, the WB gain obtained by weighted averaging with the previous frame) as the display WB gain without delay. This example shows an image processing parameter with WB gain set to the first parameter. Furthermore, in this specification, the term "no latency" includes not only cases of strictly zero latency but also cases of very low latency.

[0063] The display developing unit 24E generates display motion picture data by developing motion picture data (RAW data) acquired through shooting. Regarding WB processing, the display developing unit 24E performs WB processing using the display WB gain set by the display WB gain setting unit 24D. As described above, the WB gain calculated by the WB gain calculation unit 24A (in this embodiment, the WB gain obtained by weighted averaging with the previous frame image) is applied sequentially to the display motion picture data without delay. The display motion picture data generated by the display developing unit 24E is an example of the second motion picture data.

[0064] The display dynamic image data generated by the display developing unit 24E is output to the display unit 16 and displayed on the monitor of the display unit 16. Thus, a live view is displayed on the monitor of the display unit 16. The display unit 16 is an example of the display destination for the display dynamic image data.

[0065] The recording WB gain setting unit 24F sets the WB gain required for WB processing when generating recording motion picture data. Hereinafter, the WB gain required for WB processing of the recording motion picture data is referred to as the recording WB gain. The recording WB gain setting unit 24F sets the recording WB gain based on the WB gain calculated by the WB gain calculation unit 24A and the scene change detection result based on the scene change detection unit 24C. Specifically, it sets the recording WB gain applicable to each frame of the recording motion picture data so that the recording WB gain changes in conjunction with scene changes. Therefore, the method for setting the recording WB gain differs depending on whether there is a scene change.

[0066] When there is no scene change, the recording WB gain setting unit 24F sets the WB gain for each frame image to be obtained by weighted averaging between the previous frame image and the previous frame image. That is, the same as the display WB gain applicable to display motion picture data, for the Nth frame image, the WB gain obtained by weighted averaging the WB gain calculated from the Nth frame image and the WB gain calculated from the (N-1)th frame image is set as the recording WB gain.

[0067] Here, the RAW data developed by the recording development processing unit 24G is delayed by a certain time through delay processing. Therefore, the recording WB gain setting unit 24F sets the recording WB gain for each frame image by the amount of delay by the delay processing unit 24B. As a result, the WB gain calculated from each frame image (the WB gain calculated from the RAW data after time T) is applied to each frame image. More specifically, the WB gain obtained by weighted averaging between the frame image and the previous frame image is applied to each frame image.

[0068] On the other hand, when there is a scene change, the recording WB gain setting unit 24F sets the WB gain for each frame image so that the WB gain changes in conjunction with the scene change. Specifically, for a frame image captured at the start of a scene change, the WB gain calculated from the motion picture data (RAW data) captured before the scene change (in this embodiment, the WB gain obtained by weighted averaging with the previous frame image) is set as the recording WB gain. Furthermore, for a frame image captured at the end of a scene change, the WB gain calculated from the motion picture data (RAW data) captured after the scene change (the WB gain obtained by weighted averaging with the previous frame image) is set as the recording WB gain. Moreover, during a scene change, the applicable recording WB gain is set in a way that allows the WB to change smoothly between the recording WB gain applicable to the frame image at the start of the scene change and the recording WB gain applicable to the frame image at the end of the scene change. In this embodiment, a linearly changing WB gain is set as the recording WB gain.

[0069] As described above, the RAW data developed by the recording development processing unit 24G is delayed by time T through delay processing. Therefore, the recording WB gain delay of time T set by the recording WB gain setting unit 24F applies to each frame of the recorded moving image data. Therefore, the recording WB gain is equivalent to the WB gain calculated from the time T elapsed after the moving image data (RAW data) obtained by shooting was stored in RAM 28. The recording WB gain is an example of the second image processing parameter.

[0070] Figure 3 This is a conceptual diagram recording the settings for WB gain.

[0071] Figure 3 (A) is a diagram schematically showing the scene change from indoors to outdoors. The horizontal axis represents the passage of time, and the vertical axis represents the brightness. Figure 3 (A) illustrates that if one moves from an indoor space illuminated by a light bulb to an outdoor space illuminated by sunlight, the brightness changes dramatically within a short period of time. Figure 3 In (A), the symbol t1 represents the starting point of the scene change. And the symbol t2 represents the ending point of the scene change.

[0072] Figure 3 (B) is a schematic representation of the occurrence of Figure 3 (A) is a graph showing the brightness changes in live view under changing scene conditions. The horizontal axis represents time elapsed, and the vertical axis represents brightness. The camera's after-effects (AE) adjusts the shutter speed, aperture, and sensitivity based on the ambient brightness to control appropriate exposure. However, rapid brightness changes introduce a time lag, during which the live view will brighten ( Figure 3 (B) The mountain portion of the curve).

[0073] Figure 3 (C) is a diagram schematically showing the WB gain settings for recording. The horizontal axis represents the passage of time, and the vertical axis represents the set WB gain for recording. Generally, regarding WB gain for images, the higher the color temperature, the more likely the WB gain is to make blue appear gray, and the lower the color temperature, the more likely the WB gain is to make red appear gray. Figure 3 As shown in (C), in the camera device of this embodiment, the recording WB gain is set according to scene changes. That is, the change in the recording WB gain begins when the scene change starts (t1), and the WB gain for the new scene is set as the recording WB gain when the scene change ends (t2). During the scene change, the continuously changing WB gain is set as the recording WB gain.

[0074] The recording development unit 24G performs development processing on the RAW data (moving image data) acquired through shooting to generate recording moving image data. Regarding WB processing, the recording development unit 24G performs WB processing using the WB gain set by the recording WB gain setting unit 24F. As described above, if there is a scene change, the WB gain applicable to the recording moving image data changes in conjunction with that change. Therefore, the generated moving image data has WB corrected in conjunction with the scene change. The recording moving image data generated by the recording development unit 24G is an example of the third type of moving image data.

[0075] The recording motion image data generated by the recording developing unit 24G is compressed and then stored in the storage unit 18. The storage unit 18 is an example of the recording destination for the recording motion image data.

[0076] [The action of the camera device]

[0077] [Generation of recording dynamic image data]

[0078] Figure 4 It is a conceptual diagram that records the generation of dynamic image data.

[0079] exist Figure 4 In the diagram, the symbols F1, F2, ... represent the frame image numbers. The frame image numbers are assigned in chronological order.

[0080] Figure 4 The upper part of the diagram shows the dynamic image data (RAW data) output from the camera unit 14. The camera unit 14 outputs each frame of the image at a predetermined time interval (frame rate). Figure 4 This shows the status up to the 14th frame. That is, the latest frame image is the 14th frame image, F14.

[0081] Each frame of the RAW data output from the camera unit 14 is temporarily stored in RAM 28. In this example, for ease of explanation, an example of storing 4 frames of RAW data in RAM 28 (n=4) will be described. After 4 frames of time have elapsed, each frame of the RAW data stored in RAM 28 is retrieved from RAM 28 and added to the recording development processing unit 24G for development processing.

[0082] Figure 4 The lower section of the diagram shows the recording motion image data generated by the recording development processing unit 24G. Each frame of the image is generated with a certain time delay (T) relative to the output of the camera unit 14.

[0083] Assume the scene change begins at frame 9 (F9) and ends at frame 13 (F13). Frame 9 (F9) is the frame captured at the start of the scene change. Frame 13 (F13) is the frame captured at the end of the scene change. Frames 10 (F10) to 12 (F12) are the frames captured during the scene change.

[0084] At this point, up to the 9th frame (F9), the WB gain before the scene change begins is set to the recording WB gain for WB processing. Furthermore, starting from the 13th frame (F13), the WB gain after the scene change (the WB gain of the new scene) is set to the recording WB gain for WB processing. And, from the 10th frame (F10) to the 12th frame (F12), the WB gain that linearly varies between the WB gain before and after the scene change is set to the recording WB gain for WB processing.

[0085] For example, in a scene transitioning from indoors under bulb light to outdoors under sunlight, WB processing is applied up to the 9th frame (F9) using bulb WB gain. Furthermore, from the 10th frame (F10) to the 12th frame (F12), WB processing is applied using a combination of bulb WB gain and sunlight WB gain. Finally, from the 13th frame (F13), sunlight WB gain is applied.

[0086] The generated recordings of dynamic image data are compressed and then stored in storage unit 18.

[0087] Thus, in this embodiment, when there is a scene change, the WB gain is applied to the recorded motion picture data in conjunction with that scene change. This allows for the generation of high-quality recorded motion picture data unaffected by scene changes.

[0088] [Generation of dynamic image data for display]

[0089] Figure 5 It is a conceptual diagram showing the generation of dynamic image data.

[0090] exist Figure 5 In the diagram, the symbols F1, F2, ... represent the frame image numbers. The frame image numbers are assigned in chronological order.

[0091] Figure 5 The upper part of the diagram shows the dynamic image data (RAW data) output from the camera unit 14. Figure 5 This shows the status of the images captured up to the 14th frame.

[0092] Figure 5The lower section of the diagram shows display motion image data generated by the display developing unit 24E. For example... Figure 5 As shown, relative to the output of the camera unit 14, the display generates each frame of image using dynamic image data without delay.

[0093] Here, the WB gain for displaying each frame of the image is applied without delay by the WB gain calculation unit 24A (in this embodiment, the WB gain obtained by weighted averaging between the previous frame of the image is applied).

[0094] The generated display uses dynamic image data, which is output to the display unit 16 and displayed on the monitor of the display unit 16.

[0095] In this way, for display purposes (live view), dynamic image data is generated without delay. As a result, users can check the camera image in real time, and can check the focus status, field of view, etc.

[0096] As explained above, the camera device 10 according to this embodiment reflects an appropriate WB gain during recording, and the user can confirm the camera image in real time.

[0097] [Variation Example]

[0098] [A variation of setting the WB gain in the logbook]

[0099] As described above, when there is a scene change, the recording WB gain applicable to the recorded motion picture data changes according to the scene change. This recording WB gain applicable to the recorded motion picture data does not necessarily have to perfectly match the scene change. It can be configured to change the recording WB gain before the scene change begins, so that the WB of the generated recorded motion picture data changes more smoothly. Furthermore, it can also be configured to stop changing the recording WB gain after the scene change ends.

[0100] Figure 6 This is a conceptual diagram of the WB gain used for recording, which is set to begin before the scene change begins and end after the scene change ends.

[0101] Figure 6 (A) is a diagram schematically showing the scene change from indoors to outdoors. The horizontal axis represents the passage of time, and the vertical axis represents the brightness. Figure 6 (A) illustrates that if one moves from an indoor space illuminated by a light bulb to an outdoor space illuminated by sunlight, the brightness changes dramatically within a short period of time. Figure 6 In (A), the symbol t1 represents the starting point of the scene change. And the symbol t2 represents the ending point of the scene change.

[0102] Figure 6(B) is a diagram schematically showing the setting of the recording WB gain. The horizontal axis represents the passage of time, and the vertical axis represents the set recording WB gain (e.g., the recording WB gain of R). The symbol ta indicates the start point of the change in the recording WB gain. And the symbol tb indicates the end point of the change in the recording WB gain. Figure 6 As shown in (B), the change in the recording WB gain begins before the scene change starts (t1) and ends after the scene change ends (t2). This allows for a smoother change in the WB of the generated recording motion image data. The amount of displacement at the start and end points of the recording WB gain change can be preset. Furthermore, it can be configured to change according to the length of the scene change.

[0103] Furthermore, in the example above, both the start and end points of the change in the recording WB gain were changed, but it is also possible to configure it to change only one of them. For example, it is also possible to configure it to delay only the end point of the change in the recording WB gain from the end point of the scene change.

[0104] Furthermore, in the above example, the start point of the change in WB gain for recording is set before the start point of the scene change, and the end point of the change in WB gain for recording is set after the end point of the scene change. However, it is also possible to set the start point of the change in WB gain for recording after the start point of the scene change, and vice versa.

[0105] [A variation of setting WB gain for recording during scene changes]

[0106] In the above embodiments, the WB gain is linearly varied to apply to the frame image during scene changes, but the application of the WB gain is not limited to this. For example, the WB can be varied in stages between the start and end of the scene change. Furthermore, it can also be varied in a curved manner. The linear or curved variation of the WB gain between the start and end of the scene change is an example of a continuous variation of the WB gain. By making the WB gain change continuously, the WB can be varied smoothly.

[0107] In addition, it can also be configured as a WB gain that switches to a new scene from the center of the scene change or from the beginning of the scene change.

[0108] [Showing and recording variations of WB gain calculations]

[0109] In the above embodiments, regarding the display WB gain, the WB gain obtained by weighted averaging between the previous frame image is used as the display WB gain and is applied to each frame image of the display motion picture data. However, it is also possible to configure it to be applied without weighted averaging. For example, in cases where responsiveness is given priority, weighted averaging can be omitted, and the WB gain calculated by the WB gain calculation unit 24A can be directly applied to each frame image of the display motion picture data. However, under unstable light sources such as stroboscopic light sources, the determination of the light source color may fail. Therefore, it is preferable to use the WB gain obtained by weighted averaging between the previous frame image for the display WB gain.

[0110] The same applies to the recording WB gain for each frame of the recorded motion picture data when there are no scene changes. It can also be applied without weighted averaging between the frame and the previous frame.

[0111] [Second Implementation]

[0112] In the capture of moving images, an external recorder is typically used to record the captured moving image data. External recorders are compatible with high-speed recording media (e.g., SSDs) and can record using compression methods with minimal degradation (e.g., ProRes). Therefore, when using an external recorder to record captured moving image data, high-quality moving image data is required.

[0113] In the camera device of this embodiment, the connection of an external recorder is detected, and when a connection is detected, the recording motion image data is automatically output to the external recorder. Furthermore, the basic structure of the device is the same as that of the camera device of the first embodiment described above; therefore, only the function of automatically outputting recording motion image data will be described here.

[0114] Figure 7 This is a block diagram of a function that automatically outputs and records dynamic image data.

[0115] The CPU 24 functions as an external recorder connection detection unit 24H that detects the connection to the external recorder 100, and as a recording motion image data output control unit 24I that controls the output of recording motion image data to the external recorder 100. These functions are achieved by the CPU 24 executing a prescribed program.

[0116] The external recorder connection detection unit 24H detects the connection between the external recorder 100 and the terminals of the input / output I / F20.

[0117] The recording motion image data output control unit 24I controls the output of recording motion image data based on the connection detection result of the external recorder 100 based on the external recorder connection detection unit 24H. When the connection of the external recorder 100 is detected, the recording motion image data output control unit 24I outputs the recording motion image data to the external recorder 100 via the input / output I / F 20. That is, the recording motion image data generated by the recording development processing unit 24G is output to the external recorder 100. At this time, recording to the storage unit 18 is stopped.

[0118] The external recorder 100 performs prescribed compression processing (e.g., ProRes, etc.) on the recording video data output from the camera device 10 and records it on a built-in recording medium (e.g., SSD, etc.). The external recorder 100 is another example of a recording destination.

[0119] According to the camera device of this embodiment, when connected to an external recorder 100, it can automatically output recording video data to the external recorder 100. Therefore, high-quality video data can be automatically recorded using the external recorder 100.

[0120] In addition, in the above embodiment, when the external recorder 100 is connected, recording of the storage unit 18 is stopped, but it may also be configured such that recording continues in the storage unit 18.

[0121] [Third Implementation]

[0122] In shooting moving images, it is crucial to maintain focus on the main subject at all times. Therefore, sometimes a dedicated focusr is present in addition to the videographer (photographer) during the shooting of moving images. In this case, an external monitor is connected to confirm the focus status. Furthermore, sometimes focus adjustment is performed while using partial magnification for display. In such situations, real-time confirmation of the shooting results is necessary.

[0123] In the camera device of this embodiment, the connection to an external display is detected, and when a connection is detected, dynamic image data for display is automatically output to the external display. Furthermore, the basic structure of the device is the same as that of the camera device of the first embodiment described above; therefore, only the function of automatically outputting dynamic image data for display will be described here.

[0124] Figure 8 This is a block diagram of a function that automatically outputs and displays dynamic image data.

[0125] The CPU 24 functions as an external display connection detection unit 24J that detects the connection to the external display 200, and as a display dynamic image data output control unit 24K that controls the output of display dynamic image data to the external display 200. These functions are achieved by the CPU 24 executing a prescribed program.

[0126] The external display connection detection unit 24J detects the connection between the external display 200 and the input / output I / F20 terminals.

[0127] The display dynamic image data output control unit 24K controls the output of display dynamic image data based on the connection detection result of the external display 200 based on the external display connection detection unit 24J. When the connection of the external display 200 is detected, the display dynamic image data output control unit 24K outputs display dynamic image data to the external display 200 via the input / output I / F 20. That is, the display dynamic image data generated by the display developing unit 24E is output to the external display 200. At this time, the output (display) to the display unit 16 is stopped. The external display 200 is another example of a display destination.

[0128] According to the camera device of this embodiment, when an external display 200 is connected, display dynamic image data is automatically output to the external display 200. Therefore, when the external display 200 is connected, the camera can be used to view the captured dynamic image in real time.

[0129] Furthermore, in the above embodiment, when an external display 200 is connected, the output to the display unit 16 is stopped, but it can still be configured to be displayed on the display unit 16.

[0130] Furthermore, in the above embodiments, the connection to the external display 200 is used as a trigger to output display video data to the external display 200. However, it can also be configured to output display video data to the external display 200 upon user instruction. That is, it can also be configured to turn the output of display video data to the external display 200 on or off upon user instruction. Moreover, it can also be configured to output display video data to the external display 200 when a specific function of the camera device is used. For example, if the device has a zoom function for the displayed image, it can be configured to output display video data to the external display 200 according to the zoom operation of the displayed image. Furthermore, if the device has a peaking function (a function that colors the outline of the focused area), it can be configured to output display video data to the external display 200 in conjunction with the use of the peaking function. Moreover, if the device has a manual focus function, it can be configured to output display video data to the external display 200 when manual focus is set.

[0131] [Fourth Implementation]

[0132] As described above, recording image data is generated from RAW data that has been delayed by a certain time. Therefore, even when there are scene changes, the WB (wound image) can be changed according to the scene.

[0133] However, this processing can only be performed if the scene change is completed within the delay time. If the scene change time is longer than the delay time (T) of RAM28 (exceeding the so-called buffer capacity), the WB cannot be smoothly changed according to the scene change. For example, even when RAM28 can store four frames of image data, if the end of the scene change is not completed within four frames, the end point of the scene change is unclear, and therefore the WB cannot be smoothly changed according to the scene change.

[0134] In this embodiment, the recording of scene changes when the time exceeds the delay time (T) is explained using the setting of WB gain.

[0135] Figure 9 It is a conceptual diagram that records the generation of dynamic image data.

[0136] exist Figure 9 In the diagram, the symbols F1, F2, ... represent the frame image numbers. The frame image numbers are assigned in chronological order.

[0137] Figure 9 The upper part of the diagram shows the dynamic image data (RAW data) output from the camera unit 14. The 14th frame image F14 is the latest frame image. Each frame image of the RAW data output from the camera unit 14 is temporarily stored in RAM 28. In this example, for ease of explanation, an example of storing 4 frames of RAW data in RAM 28 will be used for illustration.

[0138] Figure 9 The lower section of the diagram shows the recording motion image data generated by the recording development processing unit 24G. Each frame of the image is generated with a certain time delay (T) relative to the output of the camera unit 14.

[0139] Assume the scene change begins at frame 4 (F4) and ends at frame 13 (F13). Frame 4 (F4) is the frame captured at the start of the scene change. Frame 13 (F13) is the frame captured at the end of the scene change. Frames 5 (F5) to 12 (F12) are the frames captured during the scene change.

[0140] As described above, RAM 28 stores 4 frames of RAW data. Therefore, it is not possible to determine the end within the delay time (T). Therefore, in the camera device of this embodiment, if the time of scene change exceeds the delay time (T), the recording WB gain is fixed to the WB gain calculated from the frame image where the scene change begins. Then, if the end of the scene change is detected, the set recording WB gain is continuously varied from the frame image where the development process begins to the frame image where the scene change ends.

[0141] Here, the frame image that begins development processing refers to the frame image retrieved from RAM 28 and used for development processing. This frame image is the frame image that has been stored in RAM 28 for a period of time T. Therefore, the frame image that begins development processing at the point when the end of the scene change is detected refers to the frame image captured before the end of the scene change for a period of time T.

[0142] Furthermore, the frame image at the end of the scene change refers to the frame image taken at the moment when the scene change ends.

[0143] Furthermore, the frame image at the start of the scene change refers to the frame image captured at the moment the scene change begins.

[0144] Therefore, in this embodiment, for frame images from the start of the scene change to time T before the end of the scene change (including frame images before time T before the end of the scene change), the WB gain calculated from the frame image at the start of the scene change is fixed as the recording WB gain. Then, for the frame image at the end of the scene change, the WB gain calculated from the frame image at the end of the scene change is set as the recording WB gain. From the frame image before time T before the end of the scene change (excluding frame images before time T before the end of the scene change) to the frame image at the end of the scene change, the WB gain that continuously varies between the WB gain calculated from the frame image at the start of the scene change and the WB gain calculated from the frame image at the end of the scene change is set as the recording WB gain.

[0145] exist Figure 9 In the example shown, the frame image at the start of the scene change is the 4th frame image, F4. Then, the frame image at the point where the end of the scene change is detected (frame images taken before time T when the scene change ends) is the 9th frame image, F9. The frame image at the end of the scene change is the 13th frame image, F13. Therefore, in Figure 9In the example shown, from frame 5 (F5) to frame 9 (F9), the WB gain calculated from frame 4 (F4) is set as the recording WB gain. For frame 13 (F13), which is the frame where the scene change ends, the WB gain calculated from frame 13 (F13) is set as the recording WB gain. For frame 10 (F10) to frame 12 (F12), the WB gain that continuously varies between the recording WB gain set for frame 9 (F9) and the recording WB gain set for frame 13 (F13) is set as the recording WB gain.

[0146] According to the imaging device of this embodiment, even when the time of scene change exceeds the delay time (T) (exceeding the buffer capacity), the WB gain of the new scene can be applied after the scene change ends. Therefore, high-quality recording motion picture data can be generated. Furthermore, the WB can be smoothly changed according to scene changes.

[0147] [Variation Example]

[0148] The change in WB gain for recording can be made to end after the scene change has ended, so that the WB of the generated recorded motion picture data changes smoothly. Alternatively, the change in WB gain for recording can end before the scene change has ended. Furthermore, it is not necessary to start the change in WB gain for recording from the frame image where development processing begins at the point when the end of the scene change is detected; it can also begin from a later frame image.

[0149] Furthermore, when the WB gain is changed during recording, it is made to change linearly in the above embodiment, but it can also be made to change in stages. Furthermore, it can also be made to change curvilinearly.

[0150] [Fifth Implementation]

[0151] In shooting moving images, sometimes AWB is not used, and the WB setting (Manual White Balance, MWB) is manually switched by the user according to the scene.

[0152] When manually switching WB settings, users may forget to switch or be performing other operations (such as aperture operation, zoom operation, etc.) and fail to switch WB settings at the appropriate time.

[0153] In the camera device of this embodiment, when the user switches the WB setting after a scene change, the system traces back to the past and applies the switched WB based on the time when the scene change ended.

[0154] The following explains the generation of recording motion image data when the WB setting is manually switched.

[0155] Figure 10 It is a conceptual diagram that records the generation of dynamic image data.

[0156] exist Figure 10 In the diagram, the symbols F1, F2, ... represent the frame image numbers. The frame image numbers are assigned in chronological order.

[0157] Figure 10 The upper part of the diagram shows the moving image data (RAW data) output from the camera unit 14. The 14th frame image F14 is the latest frame image. Each frame image of the RAW data output from the camera unit 14 is temporarily stored in RAM 28. In this example, for ease of explanation, an example of storing 4 frames of RAW data in RAM 28 will be described. Each frame image of the RAW data stored in RAM 28 is retrieved from RAM 28 after 4 frames of time and added to the recording and developing unit 24G for developing.

[0158] Figure 10 The lower section of the diagram shows the recording motion image data generated by the recording development processing unit 24G. Each frame of the image is generated with a certain time delay (T) relative to the output of the camera unit 14.

[0159] Assume the scene change begins at frame 6 (F6) and ends at frame 10 (F10). Frame 6 (F6) is the frame captured at the start of the scene change. Frame 10 (F10) is the frame captured at the end of the scene change. Frames 7 (F7) to 9 (F9) are the frames captured during the scene change.

[0160] Suppose that at the moment of capturing the 13th frame image F13, the user instructs the user to switch the WB setting. Furthermore, the WB setting switch is configured to be performed via the operation unit 22 (e.g., based on a toggle switch). The CPU 24 accepts the WB setting switch via the operation unit 22.

[0161] At this point, starting from the 10th frame image F10, which is the frame image captured at the moment the scene change ends, WB processing is performed by setting the WB gain based on the WB after the switch to the recording WB gain (until the 9th frame image F9, WB processing is performed by setting the WB gain based on the WB before the switch to the recording WB gain).

[0162] Thus, according to the camera device of this embodiment, even if the user forgets to set the switching or changes the WB setting after the scene changes, the appropriate WB can be applied according to the time when the scene change ends.

[0163] Furthermore, in the camera device of this embodiment, dynamic image data generated using dynamic image data is displayed, which switches in conjunction with the WB setting. Figure 10 In the example shown, dynamic image data is generated that switches the WB setting starting from the 13th frame image F13. This dynamic image data is displayed on the monitor of the display unit 16 without delay.

[0164] [Variation Example]

[0165] In the above embodiments, the WB gain based on the WB after the switch is applied according to the end of the scene change, but the WB can also change smoothly according to the end of the scene change. Furthermore, the switch can be configured to occur before or after the end of the scene change. Additionally, the user can be allowed to select these switching methods.

[0166] [Sixth Implementation]

[0167] As described above, in the camera device of the first embodiment, when the time of scene change is longer than the delay time (T), it is impossible to make WB change smoothly according to the scene change.

[0168] Therefore, in the camera device of this embodiment, when the time of scene change exceeds the delay time (T), dynamic image data for recording is generated as follows.

[0169] When a scene change occurs, the WB gain before the scene change is applied to generate recording motion picture data until the scene change ends. That is, during the period from the start to the end of the scene change, the WB gain at the start of the scene change is fixed to generate recording motion picture data. After the scene change ends, the WB gain after the scene change is applied to generate recording motion picture data. The generated recording motion picture data is recorded in storage unit 18. Then, image processing is performed on the recording motion picture data in storage unit 18 to smoothly change the tone before and after the scene change. That is, WB correction is performed. This image processing is performed by CPU 24. CPU 24 retrieves the motion picture data of the processing target from storage unit 18 and performs the above image processing. Furthermore, information about scene changes (information on the start and end times of the scene change) can be obtained by CPU 24 analyzing the motion picture data, or it can be configured to establish a correlation between the information of scene changes detected during shooting and the motion picture data and record it in advance.

[0170] The camera device according to this embodiment can acquire dynamic image data in which the hue changes smoothly according to the scene change, even when the time of scene change is longer than the delay time (T).

[0171] Furthermore, the motion picture data recorded in the storage unit 18 can be either compressed motion picture data or uncompressed motion picture data. Regarding compressed motion picture data, it is preferable to store frames during scene changes as I-frames. This is because I-frames do not require predicting preceding and following frames, thus reducing the computational load and memory consumption during decoding.

[0172] Furthermore, the motion image data to be processed is not limited to the motion image data stored in the storage unit 18, but can also include motion image data stored in an external storage device.

[0173] [Seventh Implementation]

[0174] There are cases where dynamic image data is recorded at a variable frame rate in order to reduce the amount of data.

[0175] Figure 11 This is a diagram that schematically represents fixed frame rate and variable frame rate.

[0176] exist Figure 11 The upper section of the diagram schematically represents motion image data captured at a constant frame rate (CFR). The lower section of the diagram schematically represents motion image data captured at a variable frame rate (VFR).

[0177] like Figure 11 As shown, at a fixed frame rate, shots are taken at a specified frame rate across all intervals. On the other hand, at a variable frame rate, since similar frame images are not saved, the frame rate varies depending on the interval. Figure 11 In the example shown, there are no 5th frame image F5 to 7th frame image F7, 9th frame image F9 to 10th frame image F10, and 12th frame image F12.

[0178] Here, we assume the scene change begins in the 3rd frame image F3 and ends in the 13th frame image F13. At this point, by varying the WB gain applicable between the 3rd and 13th frame images F3, the WB can be appropriately varied according to the scene change. That is, for the 4th frame image F4 to the 12th frame image F12, by varying the applicable WB gain, the WB can be appropriately varied according to the scene change.

[0179] However, in the case of variable frame rates, if the applied WB gain changes linearly with time, the tonal values ​​during playback will vary significantly, sometimes resulting in noticeable unnaturalness. For example, in Figure 11 In the example, the hue changes drastically in the fourth frame image F4 and the eighth frame image F8.

[0180] Therefore, in the camera device of this embodiment, when recording at a variable frame rate, the applicable WB gain is set on a frame-by-frame basis. Specifically, the applicable WB gain is set in a manner that makes the variation equal across all frames.

[0181] For example, in Figure 11 In the example, the applicable WB gain is varied for the 4th frame image F4 to the 12th frame image F12. At this time, the WB gain applicable to each frame image is set in such a way that the amount of change between the 3rd frame image F3 and the 4th frame image F4, the amount of change between the 4th frame image F4 and the 8th frame image F8, the amount of change between the 8th frame image F8 and the 11th frame image F11, and the amount of change between the 11th frame image F11 and the 13th frame image F13 are equal.

[0182] Therefore, when recording at a variable frame rate, it is possible to prevent the recorded dynamic image data from becoming unnatural in tone due to scene changes.

[0183] Furthermore, the operation unit 22 allows setting whether recording is performed at a fixed frame rate or a variable frame rate. The CPU 24 generates recording motion picture data according to the set mode.

[0184] [Other Implementation Methods]

[0185] [Scene Change Detection]

[0186] In the above embodiments, scene changes are detected by detecting changes in brightness from dynamic image data obtained through shooting, but the method for detecting scene changes is not limited to this. For example, it can be configured to detect user operations on the camera device and detect scene changes based on the detection results. For example, scene changes can be detected by detecting focusing operations, aperture operations, camera sensitivity operations, etc. Regarding focusing operations, aperture operations, and camera sensitivity operations, for example, operations exceeding a threshold are considered scene changes and detected accordingly. Furthermore, scene changes can also be detected by detecting the opening or closing of a neutral density filter. Moreover, when the camera device is equipped with a sensor such as an accelerometer, scene changes can also be detected based on the output of that sensor. In this case, for example, a significant change in the sensor output is considered a scene change and detected accordingly.

[0187] [Notification of Delayed Recording]

[0188] As described above, in the camera device to which the present invention is applied, two types of motion image data are generated: display motion image data and recording motion image data. The recording motion image data is generated and recorded with a delay compared to the display motion image data. Therefore, in use, it is preferable to inform the user that the display motion image data and the recording motion image data are generated at different times.

[0189] There is no particular limitation on the timing of the notification. For example, it can be sent at the start of camera recording. It can also be sent at the moment of scene change detection. The moment of scene change detection is also the moment of the applicable WB gain change.

[0190] There are no particular limitations on the notification method. For example, messages, icons, etc. can be displayed on the display of the display unit 16 to make notifications, or a display element such as an LED (Light Emitting Diode) can be set separately and displayed on that display element.

[0191] Image Processing

[0192] In the above embodiments, the example of performing WB processing was used, but the application of the present invention is not limited to this. It is also applicable when performing image processing using other image processing parameters.

[0193] [Image processing device]

[0194] In the above embodiments, the example described is based on a camera device that also functions as an image processing device; however, the image processing device can also be a standalone unit. Furthermore, the camera device can also be integrated into other devices such as smartphones or personal computers.

[0195] [Hardware Structure]

[0196] In this invention, the hardware structure of the processing unit performing each process is implemented through various processors. These processors include general-purpose processors that execute programs to function as various processors, such as CPUs (Central Processing Units) and GPUs (Graphics Processing Units), FPGAs (Field Programmable Gate Arrays), etc., which are processors whose circuit structure can be changed after manufacturing, i.e., Programmable Logic Devices (PLDs), and processors with circuit structures specifically designed for executing specific processes, such as ASICs (Application Specific Integrated Circuits), i.e., dedicated circuits. The term "program" has the same meaning as "software."

[0197] A processing unit can be composed of one of these various processors, or it can be composed of two or more processors of the same or different types. For example, a processing unit can be composed of multiple FPGAs or a combination of a CPU and an FPGA. Furthermore, a single processor can also constitute multiple processing units. Examples of a single processor constituting multiple processing units include: first, in computers such as client and server computers, a processor is composed of a combination of one or more CPUs and software, which functions as multiple processing units; second, in systems-on-chips (SoCs), a processor that implements the functions of the entire system containing multiple processing units is used through a single IC (integrated circuit) chip. Thus, various processing units are constructed as hardware structures using one or more of the aforementioned processors.

[0198] Symbol Explanation

[0199] 10-Camera device, 12-Camera optical system, 14-Camera unit, 16-Display unit, 18-Storage unit, 20-Input / Output (I / F), 22-Operation unit, 24-CPU, 24A-WB gain calculation unit, 24B-Delay processing unit, 24C-Scene change detection unit, 24D-Display WB gain setting unit, 24E-Display development processing unit, 24F-Recording WB gain setting unit, 24G-Recording development processing unit, 24H-External recorder connection detection unit, 24I-Recording motion image data output control unit, 24J-External display connection detection unit, 24K-Display motion image data output control unit, 26-ROM, 28-RAM, 100-External recorder, 200-External display, F1~F14-Frame images.

Claims

1. An image processing apparatus including a storage and a processor, wherein the processor performs the following processes: a process of calculating a first image processing parameter from first moving image data obtained by imaging; a process of generating second moving image data from the first moving image data in accordance with the first image processing parameter; a process of outputting the second moving image data to a display destination; a process of storing the first moving image data in the storage; a process of generating third moving image data from the first moving image data stored in the storage in accordance with a second image processing parameter calculated from the first moving image data after a time T has elapsed since the first moving image data was stored in the storage; and a process of outputting the third moving image data to a recording destination.

2. The image processing apparatus according to claim 1, wherein the processor further performs a process of detecting a scene change from the first moving image data obtained by imaging, the processor generates a frame image captured at a time when a scene change starts in the generation of the third moving image data in accordance with the second image processing parameter calculated from the first moving image data captured before the scene change.

3. The image processing apparatus according to claim 1 or 2, wherein the processor further performs a process of detecting a scene change from the first moving image data obtained by imaging, the processor generates a frame image captured at a time when a scene change ends in the generation of the third moving image data in accordance with the second image processing parameter calculated from the first moving image data captured after the scene change.

4. The image processing apparatus according to claim 2, wherein the processor changes a value of the second image processing parameter used in the generation of a frame image captured during a scene change continuously or in stages in the generation of the third moving image data.

5. The image processing apparatus according to claim 4, wherein the processor starts changing the value of the second image processing parameter before a scene change starts in the generation of the third moving image data.

6. The image processing apparatus according to claim 4 or 5, wherein the processor ends changing the value of the second image processing parameter after a scene change ends in the generation of the third moving image data.

7. The image processing apparatus according to claim 2, wherein the processor fixes a value of the second image processing parameter used in the generation of a frame image captured between a time T before a scene change ends from a time when a scene change starts when a time of a scene change exceeds the time T in the generation of the third moving image data to the second image processing parameter calculated from the first moving image data captured at the time when the scene change starts.

8. The image processing apparatus according to claim 7, wherein the processor changes the value of the second image processing parameter used in the generation of a frame image captured between a time T before a scene change ends from a time when a scene change starts when a time of a scene change exceeds the time T in the generation of the third moving image data to the second image processing parameter calculated from the first moving image data captured at the time when the scene change starts. ​ ​ The value of the second image processing parameter used in the generation of frame images taken from the time T before the end of the scene change to the end of the scene change is changed continuously or in stages.

9. The image processing apparatus according to claim 2, wherein the processor, in the generation of the third dynamic image data, changes the value of the second image processing parameter used in the generation of frame images taken when the time of the scene change exceeds the time T, The value of the second image processing parameter used in the generation of frame images taken from the time T before the end of the scene change to the end of the scene change is changed continuously or in stages.

10. The image processing apparatus according to claim 9, wherein the processor further performs image processing that smoothly changes the color tone before and after the scene change on the generated third dynamic image data.

11. The image processing apparatus according to claim 2, wherein the processor, in the generation of the third dynamic image data at a variable frame rate, The value of the second image processing parameter used in the generation of frame images taken during the scene change is changed equally between frames.

12. The image processing apparatus according to claim 1 or 2, wherein the image processing apparatus further has a terminal that connects to an external device, the processor further performs the following processing: processing that detects connection of an external recorder to the terminal; and processing that outputs the third dynamic image data to the external recorder when the connection of the external recorder to the terminal is detected.

13. The image processing apparatus according to claim 1 or 2, wherein the image processing apparatus further has a terminal that connects to an external device, the processor further performs the following processing: processing that detects connection of an external display to the terminal; and processing that outputs the second dynamic image data to the external display when the connection of the external display to the terminal is detected.

14. The image processing apparatus according to claim 1 or 2, wherein the processor further performs processing that notifies that the time of generation of the second dynamic image data and the time of generation of the third dynamic image data are different.

15. The image processing apparatus according to claim 1 or 2, wherein the first image processing parameter and the second image processing parameter are image processing parameters required for white balance processing.

16. An image pickup apparatus comprising: an image pickup element; and the image processing apparatus according to any one of claims 1 to 15 that processes dynamic image data output from the image pickup element by imaging.

17. An image processing apparatus comprising a memory and a processor, wherein the processor performs the following processing: processing that accepts a switching instruction of an image processing parameter applied to first dynamic image data obtained by imaging; processing that detects a scene change from the first dynamic image data; processing that generates second dynamic image data from the first dynamic image data in accordance with the image processing parameter; processing that outputs the second dynamic image data to a display destination; processing that stores the first dynamic image data in the memory; generating third dynamic image data from the first dynamic image data stored in the memory, according to the image processing parameter; and outputting the third dynamic image data to a recording destination, the processor, in the generating of the third dynamic image data, generates a frame image taken at a time when a scene change starts, according to the image processing parameter before the scene change, and generates a frame image taken at a time when the scene change ends, according to the image processing parameter switched after the scene change.

18. An image pickup apparatus comprising: an image pickup element; and the image processing apparatus according to claim 17, which processes dynamic image data output from the image pickup element by image pickup.

19. An image processing method comprising the processes of: calculating a first image processing parameter from first dynamic image data obtained by image pickup; generating second dynamic image data from the first dynamic image data, according to the first image processing parameter; outputting the second dynamic image data to a display destination; storing the first dynamic image data in a memory; generating third dynamic image data from the first dynamic image data stored in the memory, according to a second image processing parameter calculated from the first dynamic image data after a time T has elapsed since the first dynamic image data was stored in the memory; and outputting the third dynamic image data to a recording destination.

20. A recording medium which is non-transitory and computer-readable, and which records a program for causing a computer to execute the image processing method according to claim 19. ​

Images