Image processing device, imaging device, image processing method, image processing program, and recording medium

Abstract

To provide an image processing apparatus, an imaging apparatus, an image processing method, an image processing program, and a recording medium that allow recognition of a focus state of a specific subject during imaging and can display a live view image suitable for checking a main subject and checking framing.SOLUTION: An image processing apparatus comprises a processor 16 and a memory, and the processor 16 performs processing of generating first moving image data for recording based on input imaging data; processing of setting a specific subject area including a specific subject in an imaging area represented by the imaging data or the first moving image data; processing of evaluating a focus state based on the imaging data or the first moving image data; processing of, based on data of the specific subject area, a result of evaluation of the focus state, and the first moving image data, generating second moving image data for indication allowing visual recognition of a focus state of the specific subject area; and processing of outputting the second moving image data to a display destination.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing device, an imaging device, an image processing method, an image processing program, and a recording medium, and more particularly to a technique for generating moving image data suitable for assisting manual focus. [Background technology]

[0002] In recent years, mirrorless cameras and single-lens reflex cameras have become increasingly popular for video content production due to their enhanced video functions. While these cameras are small and lightweight, making them highly mobile when capturing images, the monitors used to check live view images and recorded images are also small, making it difficult to visually confirm whether the focus on the main subject is as intended by the user.

[0003] To solve this problem, focus assist functions have been developed, and some cameras are equipped with functions such as a "magnified display of the focused area" and a "peaking function that displays the outline of the focused subject in color" (Patent Documents 1 and 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-105768 [Patent Document 2] Japanese Patent Application Laid-Open No. 2017-169188 Summary of the Invention

[0005] One embodiment of the technology of the present disclosure provides an image processing device, an imaging device, an image processing method, an image processing program, and a recording medium that enable a user to recognize the focus state of a main subject on which the user wants to focus while capturing a moving image, and that can display a live view image suitable for confirming the main subject and checking the framing. [Means for solving the problem]

[0006] The invention according to a first aspect is an image processing device comprising a processor and a memory, wherein the processor performs the following processes: generating first moving image data based on input imaging data; setting a specific subject area including a specific subject within an imaging area represented by the imaging data or the first moving image data; evaluating a focus state based on the imaging data or the first moving image data; generating second moving image data in which the focus state of the specific subject area is visible based on data on the specific subject area, the evaluation result of the focus state, and the first moving image data; and outputting the second moving image data to a display destination.

[0007] In the image processing device pertaining to the second aspect of the present invention, the second moving image data is preferably moving image data in which the focus state of only the specific subject region is visible.

[0008] In the image processing device according to the third aspect of the present invention, it is preferable that the process of generating the second moving image data is performed by adding data based on the evaluation result of the focus state to the specific subject area represented by the data of the specific subject area within the imaging area represented by the first moving image data.

[0009] In the image processing device according to the fourth aspect of the present invention, it is preferable that the processor generates third moving image data including an evaluation result of the focus state of the specific subject based on the imaging data or the first moving image data, and generates second moving image data based on the specific subject area, the first moving image data, and the third moving image data.

[0010] In the image processing device pertaining to the fifth aspect of the present invention, it is preferable that the processor generates the second moving image data by combining the first moving image data excluding the specific subject region with the third moving image data of the specific subject region.

[0011] In the image processing device according to the sixth aspect of the present invention, it is preferable that the process of evaluating the focus state is a process of extracting contour components of the subject based on the imaging data or the first moving image data, and applying contour enhancement processing to the first moving image data according to the amplitude of the extracted contour components.

[0012] In the image processing device according to the seventh aspect of the present invention, it is preferable that the contour enhancement process is a process of adding a specific color to a portion where the amplitude of the extracted contour component exceeds a first threshold, and / or changing the density of the specific color according to the amplitude of the contour component.

[0013] In the image processing device according to the ninth aspect of the present invention, it is preferable that the processor sets the specific subject region based on an instruction input of the specific subject by the user.

[0014] In the image processing device according to the eighth aspect of the present invention, it is preferable that the processor detects the movement of the specific subject and changes the specific subject region in accordance with the movement of the specific subject.

[0015] In the image processing device according to the tenth aspect of the present invention, it is preferable that the image processing device is provided with a first connection unit that can be connected to a display unit or an external display device, and that the second moving image data is displayed on the display unit or the external display device connected to the first connection unit.

[0016] In the image processing device according to the eleventh aspect of the present invention, it is preferable that the image processing device further comprises a recording unit or a second connection unit connectable to an external recording device, and the processor outputs the first moving image data to the recording unit or the second connection unit.

[0017] In the image processing device according to the twelfth aspect of the present invention, it is preferable that the processor outputs the first moving image data and the second moving image data alternately to the display destination every first time.

[0018] In the image processing device according to the thirteenth aspect of the present invention, it is preferable that the processor acquires a focus evaluation value indicating the evaluation result of the focus state of a specific subject, and if the acquired focus evaluation value exceeds a second threshold, outputs the first moving image data to a display destination.

[0019] In the image processing device according to the fourteenth aspect of the present invention, it is preferable that the processor acquires a focus evaluation value indicating the evaluation result of the focus state of a specific subject, and if the acquired focus evaluation value is not on a decreasing trend and this state of not decreasing continues for a second time, output the first moving image data to a display destination.

[0020] In the image processing device according to the fifteenth aspect of the present invention, it is preferable that the processor obtains a focus evaluation value indicating an evaluation result of the focus state of a specific subject, obtains a maximum evaluation value which is the maximum value of the focus evaluation values ​​for the most recent multiple frames, and when the focus evaluation value of the most recent frame during the manual focus period exceeds an allowable value set based on the maximum evaluation value, outputs notification information to a display destination indicating that the focus state of the specific subject in the most recent frame is good.

[0021] An imaging device according to a sixteenth aspect of the present invention includes an imaging section that outputs imaging data, and the image processing device described above.

[0022] The invention of the 17th aspect is an image processing method for generating moving image data that supports manual focusing using an image processing device having a processor and a memory, and each process of the processor includes a step of generating first moving image data based on input imaging data, a step of setting a specific subject area including a specific subject within an imaging area represented by the imaging data or the first moving image data, a step of evaluating a focus state based on the imaging data or the first moving image data, a step of generating second moving image data in which the focus state of the specific subject area is visible based on data of the specific subject area, the evaluation result of the focus state, and the first moving image data, and a step of outputting the second moving image data to a display destination.

[0023] In the image processing method according to the 18th aspect of the present invention, it is preferable that the method further includes a step of generating third moving image data including an evaluation result of the focus state of the specific subject based on the imaging data or the first moving image data, and that the step of generating second moving image data generates the second moving image data based on the specific subject area, the first moving image data, and the third moving image data.

[0024] In the image processing method according to the 19th aspect of the present invention, it is preferable that the step of generating the second moving image data comprises generating the second moving image data by combining the first moving image data excluding the specific subject area with the third moving image data of the specific subject area.

[0025] An image processing program according to a twentieth aspect of the present invention enables a computer to perform the following functions: generate first moving image data based on input imaging data; set a specific subject area including a specific subject within an imaging area represented by the imaging data or the first moving image data; evaluate a focus state based on the imaging data or the first moving image data; generate second moving image data in which the focus state of the specific subject area is visible based on data on the specific subject area, the evaluation result of the focus state, and the first moving image data; and output the second moving image data to a display destination. [Brief explanation of the drawings]

[0026] [Figure 1] FIG. 1 is a diagram showing an embodiment of the internal configuration of an imaging device according to the present invention. [Figure 2] FIG. 2 is a block diagram showing an embodiment of an image processing apparatus according to the present invention. [Figure 3] FIG. 3 is a waveform diagram showing contour components and the like extracted from luminance data. [Figure 4] FIG. 4 is a waveform diagram showing an example of the first moving image data that has not been subjected to peaking processing. [Figure 5] FIG. 5 is a waveform diagram showing an example of the first moving image data (third moving image data) on which peaking processing has been performed. [Figure 6] FIG. 6 is a diagram showing an example of a specific subject and a peaking range. [Figure 7] FIG. 7 is a diagram showing the recording destination of the first moving image data and the display destination of the second moving image data. [Figure 8] FIG. 8 is a flowchart showing a first embodiment of the image processing method according to the present invention. [Figure 9] FIG. 9 is a flowchart showing a second embodiment of the image processing method according to the present invention. [Figure 10] FIG. 10 is a flowchart showing a third embodiment of the image processing method according to the present invention. [Figure 11] FIG. 11 is a diagram showing an example of the transition of the amplitude [Amp] of the contour component. [Figure 12] FIG. 12 is a flowchart showing a fourth embodiment of the image processing method according to the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0027] Hereinafter, preferred embodiments of an image processing program and a recording medium according to the present invention will be described with reference to the accompanying drawings.

[0028] [Configuration of imaging device] FIG. 1 is a diagram showing an embodiment of the internal configuration of an imaging device according to the present invention.

[0029] The imaging device 10 is a camera capable of capturing moving images, and includes an imaging lens 12, an imaging unit 14, a processor 16, a memory 18, a display unit 20, an input / output interface 22, an operation unit 24, and the like.

[0030] The imaging lens 12 includes a group of lenses including a focus lens that forms an image of a subject, an aperture that adjusts the amount of light, a focus ring that manually adjusts the position of the focus lens, etc. The imaging lens 12 may be an interchangeable lens that is detachable from the camera body, or may be integrated into the camera body.

[0031] The imaging unit 14 includes an imaging element that converts an optical image into an electrical signal, an analog signal processing unit, an A / D (Analog to Digital) converter, and the like.

[0032] The imaging element is configured by, for example, a CMOS (Complementary Metal-Oxide Semiconductor) color image sensor. Note that the imaging element is not limited to the CMOS type, and may be a CCD (Charge Coupled Device) type imaging element.

[0033] The imaging element has a plurality of pixels each made up of a photoelectric conversion element (photodiode) two-dimensionally arranged in the x direction (horizontal direction) and the y direction (vertical direction), on which a color filter of one of the three primary colors, red (R), green (G), and blue (B), is arranged in a predetermined color filter array to form a pixel of each color of RGB. The color filter array may be a common Bayer array, but is not limited to this, and may be, for example, an X-Trans (registered trademark) array or other color filter array.

[0034] An optical image of the subject formed on the light receiving surface of the imaging element of the imaging unit 14 by the imaging lens 12 is accumulated as an electric charge in each pixel of the imaging element according to the amount of incident light. An electric signal according to the amount of electric charge accumulated in each pixel is read out from the imaging element as an image signal and output to an analog signal processing unit.

[0035] The analog signal processing section includes a sample-and-hold circuit, a color separation circuit, an AGC (Automatic Gain Control) circuit, etc. The AGC circuit functions as a sensitivity adjustment section that adjusts the sensitivity (ISO sensitivity (ISO: International Organization for Standardization)) during imaging, adjusting the gain of the amplifier that amplifies the input image signal so that the signal level of the image signal falls within an appropriate range. The A / D converter converts the analog image signal output from the analog signal processing section into a digital image signal. Note that when the imaging element is a CMOS image sensor, the analog signal processing section and A / D converter are often built into the CMOS image sensor.

[0036] When capturing a moving image, the imaging unit 14 outputs imaging data (RAW data) at a set frame rate (30 fps, 60 fps, etc.).

[0037] The processor 16 includes a CPU (Central Processing Unit), a dedicated electric circuit for performing specific signal processing on the imaging data, and functions as an image processing device for generating various types of moving image data based on the imaging data input from the imaging unit 14. The process of generating various types of moving image data by the processor 16 will be described in detail later.

[0038] The memory 18 includes a flash memory, a ROM (Read-only Memory), a RAM (Random Access Memory), etc. The flash memory and the ROM are non-volatile memories that store the camera control program, the image processing program according to the present invention, various data necessary for camera control, etc. The RAM temporarily stores captured image data and functions as a working area for processing by the processor 16. It also temporarily stores the camera control program, the image processing program, etc. stored in the flash memory, etc. Note that the processor 16 may have a portion of the memory 18 (RAM) built in.

[0039] The processor 16 controls each part of the camera body and processes images in accordance with a camera control program or an image processing program, using the RAM as a working area.

[0040] The display unit 20 is a display such as an LCD (Liquid Crystal Display) or an OLED (Organic Light Emitting Diode), and may have a touch panel on the display.

[0041] The display unit 20 is provided on the back of the camera body and displays a live view image during image capture. The user can check the focus state of the subject and confirm the framing using the live view image displayed on the display unit 20.

[0042] The display unit 20 also functions as a notification unit that notifies (displays) various types of notification information to the user. Furthermore, the display unit 20 is used as a user interface for displaying a menu screen and accepting instructions from the user when various settings are made. The display unit 20 also includes an EVF (Electronic View Finder) type.

[0043] The input / output interface 22 includes a first connection section connectable to an external display device, a second connection section connectable to an external recording device, a card connection section for inserting and removing a memory card, and a communication section connectable to a network. For example, the input / output interface 22 may be a Universal Serial Bus (USB), a High-Definition Multimedia Interface (HDMI) (HDMI is a registered trademark), or the like. Note that the USB and HDMI can be commonly used as the first connection section and the second connection section, respectively.

[0044] The card connector and the memory card connected to the card connector correspond to a recording section within the camera in relation to the external recording device.

[0045] The operation unit 24 includes various operation members for operating the imaging device 10. The operation members include a power button, a recording button, various operation buttons, a touch panel, and the like.

[0046] The processor 16 functions as a control unit and an image processing device of the imaging device 10 by executing predetermined programs (camera control program, image processing program, etc.).

[0047] The control performed by the processor 16 includes image processing, display control on the display unit 20, recording control on the recording unit, and the like, in addition to imaging control such as AE (Automatic Exposure) and AF (Auto Focus).

[0048] In addition, the image processing performed by the processor 16 includes processing for generating moving image data (first moving image data) for recording based on the imaging data (RAW data) input from the imaging unit 14, processing for generating moving image data (second moving image data) for display (live view), and compression processing of the first moving image data for recording.

[0049] [Image processing device] 2 is a block diagram showing an embodiment of an image processing device according to the present invention. The image processing device according to the present invention comprises a processor 16 and a memory 18.

[0050] The processor 16 shown in FIG. 2 includes a white balance correction processing unit 16A, a demosaic processing unit 16B, a luminance / color difference conversion processing unit 16C, a contour component extraction unit 16D, a contour enhancement processing unit 16E, a specific subject region setting unit 16F, and a synthesis processing unit 16G.

[0051] The white balance correction processing unit 16A, the demosaic processing unit 16B, and the luminance / color difference conversion processing unit 16C are development processing units that generate first moving image data for recording from input unprocessed imaging data (RAW data).

[0052] The white balance correction processor 16A calculates white balance gains (WB (White Balance) gains) Gr, Gg, Gb for each color data of the input imaging data (R data, G data, and B data), and performs white balance correction on the imaging data by multiplying the R data, G data, and B data by the calculated WB gains Gr, Gg, Gb, respectively. Here, methods for calculating the WB gains Gr, Gg, Gb include identifying the type of light source illuminating the subject based on scene recognition (e.g., determination of indoors or outdoors) using the brightness (EV value) of the subject and the color temperature of the ambient light, and reading out the WB gain corresponding to the identified light source type from memory 18, in which appropriate WB gains for each light source type are stored in advance.

[0053] The demosaic processing unit 16B is a processing unit that calculates all RGB color information for each pixel using interpolation processing from point-sequential mosaic image data consisting of RGB, and generates image data of three RGB planes that has been demosaic processed (also called demosaicing processing or synchronization processing) from the mosaic image data.

[0054] The luminance / color difference conversion processing unit 16C converts the demosaic processed RGB data into luminance data (Y) and color difference data (Cb, Cr).

[0055] In this manner, the first moving image data for recording is generated from the unprocessed imaging data. Note that the development process for the unprocessed imaging data also includes gamma correction, edge enhancement for the luminance data (Y), and color difference matrix processing for the color difference data (Cb, Cr), but these are well-known processes and therefore will not be described in detail here.

[0056] The contour component extraction unit 16D receives the luminance data (Y) from the first moving image data and extracts contour components of the subject in the captured image. The contour components can be extracted by applying a contour extraction filter (for example, a Sobel filter) to the luminance data (Y).

[0057] FIG. 3 is a waveform diagram showing contour components and the like extracted from luminance data.

[0058] 3A is a waveform diagram of the brightness data (Y) of the edge portion. As shown in the figure, the brightness data (A) changes significantly in the edge portion.

[0059] Fig. 3(B) is a waveform diagram showing the contour components extracted from the luminance data (Y) of Fig. 3(A). The contour component extraction unit 16D extracts the portion (contour portion) where the luminance data (Y) changes as shown in Fig. 3(B) and outputs the contour components having an amplitude [Amp] corresponding to the magnitude of the change. This amplitude [Amp] corresponds to a focus evaluation value that indicates the evaluation result of the focus state of the subject, and the larger the amplitude [Amp] (focus evaluation value), the better the focus state.

[0060] The contour enhancement processing unit 16E receives the color difference data (Cb, Cr) from the first moving image data and the contour components of the subject extracted by the contour component extraction unit 16D, and applies contour enhancement processing to the first moving image data (in this example, the color difference data (Cb, Cr)) according to the amplitude [Amp] of the contour components.

[0061] The contour enhancement processing unit 16E in this example is a unit that performs a process (peaking process) to add a specific color (e.g., red) to the contour of the subject in the first moving image data, and adds the specific color to the portion where the amplitude [Amp] of the contour component extracted by the contour component extraction unit 16D exceeds the first threshold.

[0062] Here, the first threshold value is preferably set to the minimum amplitude of the contour components of a subject that is obtained when the subject is in proper focus, so that a subject in a proper focus state will have its contour component amplitude exceeding the first threshold value.

[0063] Since the amplitude [Amp] of the extracted contour components differs depending on the type of contour extraction filter, it is preferable to set the first threshold value taking into consideration the type of contour extraction filter. Also, since the amplitude of the contour components of a subject varies depending on the intensity of ambient light, it is preferable to set the first threshold value appropriately depending on the brightness of the subject.

[0064] In addition, the contour enhancement processing unit 16E may perform peaking processing to change the density of a specific color according to the amplitude [Amp] of the contour component extracted by the contour component extraction unit 16D, or may further perform peaking processing to change the density of a specific color according to the amplitude of the contour component for parts where the amplitude [Amp] of the contour component exceeds a first threshold.

[0065] The first moving image data (third moving image data) that has undergone peaking processing has a specific color added to the contours of the subject (i.e., areas with high contrast), and therefore is data that includes the evaluation result of the focus state of the imaging area, and the peaking processing is a process for evaluating the focus state.

[0066] FIG. 4 is a waveform diagram showing an example of the first moving image data to which peaking processing has not been applied, and FIG. 5 is a waveform diagram showing an example of the first moving image data (third moving image data) to which peaking processing has been applied.

[0067] If the first moving image data that has not undergone peaking processing has the luminance data (Y) and color difference data (Cb, Cr) shown in Figure 4, in the first moving image data (third moving image data) that has undergone peaking processing shown in Figure 5, the values ​​of the color difference data (Cb, Cr) are replaced so that pixels (contours) where the luminance data (Y) changes become red.

[0068] The specific subject region setting unit 16F performs processing to set a specific subject region that includes a specific subject within the imaging region represented by the imaging data or the first moving image data.

[0069] Here, the specific subject in this example refers to a subject that is part of the main subject within the imaging area.

[0070] FIG. 6 is a diagram showing an example of a specific subject and a peaking range.

[0071] In the case of a moving image in which a person's face is captured prominently in the imaging area as shown in Fig. 6(A), the person's face is the main subject, and the person's facial parts (in this example, the "mouth") are the specific subject. The specific subject area R shown in Fig. 6(A) is a rectangular area that includes the "mouth".

[0072] Furthermore, the specific subject region R is a peaking range in which peaking processing is performed, and is a region that includes the specific subject. In the moving image shown in Fig. 6(B), peaking processing is performed only on the "mouth".

[0073] When the main subject is a person's face, the specific subject is not limited to the "mouth," but may also be the "eyes" (including only one eye), the "nose," or the "hair," etc. Also, in the case of a close-up shot of a flower, the "pistil" or "stamen" may be the specific subject.

[0074] Furthermore, if the person in the imaging area is small, the entire face of the person can be the specific subject, and if there are multiple flowers in the imaging area, one of the multiple flowers can be the specific subject.

[0075] The specific subject is a part of the main subject, and it is preferable that when the specific subject is in focus, the main subject is also in focus. If the area of ​​the main subject is sufficiently small compared to the imaging area, the main subject may be the specific subject.

[0076] The specific subject area setting unit 16F can set a specific subject area based on a user's instruction input for a specific subject. In this case, the user can specify the specific subject using a touch panel or the like while viewing the live view image displayed on the display unit 20.

[0077] Furthermore, the specific subject area setting unit 16F can automatically set the central area of ​​the imaging area at the start of capturing a moving image as the specific subject area. In this case, it is preferable that the user frames the imaging area so that the specific subject is included in the central area of ​​the imaging area at the start of capturing a moving image.

[0078] Furthermore, it is preferable that the specific subject region setting unit 16F detects the movement of the specific subject and changes the specific subject region (position, size) according to the movement of the specific subject. The movement of the specific subject can be detected by tracking feature points of the specific subject in each frame constituting the first moving image data.

[0079] The synthesis processing unit 16G inputs the first moving image data, the third moving image data, and the data of the specific subject area set by the specific subject area setting unit 16F, and generates moving image data (second moving image data) based on this data in which the focus state of the specific subject area is visible.

[0080] The synthesis processor 16G synthesizes the first moving image data excluding the specific subject region with the third moving image data of the specific subject region to generate the second moving image data. The synthesis processor 16G can erase the portion of the first moving image data corresponding to the specific subject region, and fit the third moving image data of the specific subject region into that portion to synthesize the second moving image data.

[0081] Note that the generation of the second moving image data is not limited to the case where the second moving image data is generated by combining the first moving image data excluding the specific subject region with the third moving image data of the specific subject region, but may also be generated by applying peaking processing to only the specific subject region of the first moving image data based on the data of the specific subject region, the focus state evaluation result, and the first moving image data, in which case there is no need to generate third moving image data in which peaking processing has been applied to the entire screen.

[0082] Furthermore, the second moving image data may not necessarily be generated by performing a peaking process that adds a specific color to the contours of a specific subject based on the evaluation results of the focus state of the specific subject area, but may also be generated by performing an edge enhancement process that emphasizes the contours of a specific subject.In short, any process may be used to generate the second moving image data by performing a process that adds data based on the evaluation results of the focus state.

[0083] FIG. 6B shows an example of a live view image corresponding to the second moving image data in which peaking processing has been performed only on the specific subject region R.

[0084] The second video data is video data in which peaking processing has been applied only to the specific subject region R in the first video data, and as shown in Figure 6(B), peaking processing has been applied only to the specific subject region R which includes the specific subject, "mouth."

[0085] When operating the focus ring to focus the imaging lens 12 on a main subject (specific subject), the user operates the focus ring while viewing the live view image displayed on the display unit 20 (while checking the focus state of the specific subject).

[0086] That is, while viewing a live view image in which peaking processing has been applied only to the specific subject area R, the user can operate the focus ring so that a specific color that emphasizes the outline of the specific subject within the specific subject area R appears, or so that the density of the specific color is maximized, thereby enabling good manual focusing even if the screen size of the display unit 20 is small.

[0087] Furthermore, because peaking processing is applied only to the portion of the image that the user wants to focus on (specific subject area R) rather than to the entire image capture area, the user can check the focus state of the specific subject area R while viewing the live view image even while capturing a moving image. Furthermore, because peaking processing is not applied to the entire image capture area, it is possible to properly check, for example, the main subject, such as a person's facial expression, and the framing.

[0088] FIG. 7 is a diagram showing the recording destination of the first moving image data and the display destination of the second moving image data.

[0089] The processor 16 can output the first moving image data that has not been subjected to peaking processing to an external recording device connected to the recording unit 22A or the second connection unit 22B as a recording destination.

[0090] Recording unit 22A includes, for example, a memory card attached to a card connector of input / output interface 22, and records the first moving image data input from processor 16 onto the memory card.

[0091] The second connection unit 22B is, for example, a USB, HDMI, etc. of the input / output interface 22, and when an external recording device is connected to the second connection unit 22B, the external recording device can record the first moving image data input from the processor 16.

[0092] The processor 16 preferably includes a compression processing unit 16H, and outputs the first moving image data compressed by the compression processing unit 16H to the recording unit 22A or the second connection unit 22B.

[0093] The compression processing in the compression processing unit 16H is processing to generate a moving image file in a predetermined compressed format. A known codec can be used for compression. For example, a codec standardized by the Moving Picture Experts Group (MPEG) (MPEG-1, MPEG-2, MPEG-4, etc.), H.264, etc. can be used.

[0094] Furthermore, if the external recording device connected to the second connection unit 22B has a large storage capacity, the processor 16 can output the uncompressed first moving image data to the second connection unit 22B.

[0095] The processor 16 outputs the first moving image data (second moving image data) that has been subjected to the peaking process to the display unit 20 or an external display device connected to the first connection unit 22C as a display destination.

[0096] The display unit 20 can display a live view image based on the second moving image data input from the processor 16 during standby and moving image capture.

[0097] The first connection unit 22C is, for example, a USB, HDMI, etc. of the input / output interface 22, and when an external display device is connected to the first connection unit 22C, the external display device can display a live view image based on the second moving image data input from the processor 16.

[0098] In this example, the first moving image data and the second moving image data are luminance data (Y) and color difference data (Cb, Cr), but they may also be moving image data in which the luminance data (Y) and color difference data (Cb, Cr) are converted into RGB data.

[0099] [Image processing method] The image processing method according to the present invention is a method for generating moving image data that supports manual focusing, and the processor 16 shown in FIG. 2 is the main body of each process for generating moving image data.

[0100] First Embodiment FIG. 8 is a flowchart showing a first embodiment of the image processing method according to the present invention.

[0101] 8, when starting to capture a moving image, the user selects an arbitrary specific subject (step S10). This is to check the focus state for main subjects including the specific subject. The arbitrary specific subject can be selected using a user interface such as a touch panel and the display unit 20 that displays a live view image.

[0102] When the specific subject region setting unit 16F of the processor 16 receives a command input of a specific subject from the user, it sets a specific subject region R within the imaging region (step S12). This specific subject region R is a peaking range in which peaking processing is performed.

[0103] The processor 16 acquires the imaging data from the imaging unit 14 (step S14).

[0104] The specific subject area setting unit 16F tracks the specific subject selected by the user based on the imaging data and updates the specific subject area R (step S16). This is to allow the user to check the focus state of the main subject that the user wants to check even if the specific subject moves. Note that tracking of the specific subject may be performed based on the first moving image data generated from the imaging data.

[0105] The processor 16 develops the imaging data (RAW data) acquired in step S14, and generates first moving image data for recording (first moving image data with peaking OFF) (step S18).

[0106] Next, the processor 16 (contour component extraction unit 16D, contour enhancement processing unit 16E) generates third moving image data (first moving image data with peaking ON) that has been subjected to peaking processing based on the imaging data or the first moving image data (step S20). The third moving image data is moving image data that has been subjected to peaking processing of the first moving image data, with the entire range of the imaging area being the peaking range.

[0107] The synthesis processing unit 16G generates second moving image data for display, which allows the focus state of the specific subject area to be visually recognized, based on the first moving image data, the third moving image data, and the data of the specific subject area set by the specific subject area setting unit 16F (step S22). That is, the synthesis processing unit 16G generates the first moving image data excluding the specific subject area based on the first moving image data and the data of the specific subject area, and generates the third moving image data of the specific subject area based on the third moving image data and the data of the specific subject area, and synthesizes the first moving image data excluding the specific subject area with the third moving image data of the specific subject area to generate the second moving image data.

[0108] The processor 16 outputs the second moving image data for display to the display unit 20 of the camera as the display destination, and displays an image (live view image) in which the focus state of the specific subject area is visible (step S24).

[0109] Processor 16 also determines whether moving image recording is in progress (step S26), and if moving image recording is in progress (if "Yes"), records the first moving image data for recording generated in step S18 in recording unit 22A (step S28).

[0110] If the moving image is not being recorded (if "No"), the process skips step S28 and proceeds to step S30. For example, a situation where a live view image is being displayed before moving image shooting starts corresponds to a situation where moving image recording is not being performed.

[0111] Next, processor 16 determines whether or not to end the peaking display (step S30), and if it determines not to end the peaking display (in the case of "No"), it transitions to step S14 and repeats the processes of steps S14 to S30 for the imaging data of the next frame. The determination of whether or not to end the peaking display can be made based on, for example, whether or not the user has input an instruction to change the peaking setting to off.

[0112] When processor 16 determines that the peaking display should be ended (if "Yes"), it ends this process.

[0113] According to the first embodiment of the image processing method, peaking processing is applied only to a portion of the main subject (specific subject area R) on which the user wants to focus, rather than to the entire imaging area, so that the main subject, such as a person's facial expression, and framing can be properly confirmed.

[0114] Furthermore, as a modified example of the first embodiment, the processor 16 may output the first moving image data to the display destination (display unit 20 or first connection unit 22C) instead of the second moving image data when the focus evaluation value (amplitude [Amp]) of the specific subject in the specific subject area R exceeds a threshold value (second threshold value) at which the specific subject is considered to be in focus.

[0115] This allows the user to view a live view image without peaking when the focus lens is focused on the main subject, making it even easier to check the main subject.

[0116] Second Embodiment FIG. 9 is a flowchart showing a second embodiment of the image processing method according to the present invention.

[0117] The second embodiment of the image processing method differs from the first embodiment shown in FIG. 8 in the way the peaking display is performed, and therefore a description of the method for generating the first and second moving image data will be omitted.

[0118] In the second embodiment of the image processing method, first moving image data (without peaking) and second moving image data (with peaking) are alternately output to a display destination every first time.

[0119] In FIG. 9, the frame counter variable "count" is reset to "0" and the image selector variable "sel" is set to "1" (step S100).

[0120] Subsequently, the processor 16 determines whether the image selector variable "sel" is greater than "0" (sel>0) (step S102). If sel>0 (in the case of "Yes"), the process proceeds to step S104, where a live view image with "peaking" is displayed on the display unit 20 using the second moving image data.

[0121] On the other hand, if sel>0 is not satisfied (in the case of "No"), the process proceeds to step S106, where a live view image without "peaking" is displayed on the display unit 20 using the first moving image data. At the start of imaging, since the image selector variable "sel" is set to "1", a live view image with "peaking" is displayed on the display unit 20.

[0122] Next, the processor 16 increments the frame counter variable "count" by 1 (step S108) and determines whether the frame counter variable "count" is less than N (count<N) (step S110).

[0123] Now, when the frame rate is 30fps and a live view image with "peaking" and a live view image without "peaking" are alternately displayed every 1 to 2 seconds, it is preferable to set N to a value within the range of 30 to 60.

[0124] In step S110, if it is determined that count<N (in the case of "Yes"), the process transitions to step S114, and if it is determined that count<N is not satisfied (in the case of "No"), the process transitions to step S112. From the start of imaging until N frames are captured, since count<N is satisfied, the process transitions to step S114.

[0125] Next, the processor 16 determines whether to end the peaking display (step S114). If it is determined not to end the peaking display (in the case of "No"), the process transitions to step S102, and the processing from step S102 to step S114 is repeated for the imaging data of the next frame.

[0126] On the other hand, when the display of the live view image with peaking continues for the first hour (N frames), in step S110, it is determined that count < N (in the case of "No"), and the process proceeds to step S112. In step S112, the frame counter variable "count" is reset to "0", and sel = sel × (-1), reversing the sign of the image selector variable "sel". At the start of imaging, since the image selector variable "sel" is set to "1", the image selector variable "sel" after the first hour (after N frames) from the start of imaging has its sign reversed to "-1".

[0127] When the image selector variable "sel" is "-1", in step S102, it is determined that sel > 0 is not true, and the process proceeds to step S106, where a live view image without peaking is displayed on the display unit 20 using the first moving image data.

[0128] As described above, the processor 16 switches and displays on the display unit 20 the live view image with peaking using the second moving image data and the live view image without peaking using the first moving image data every first hour (N frames).

[0129] According to the second embodiment of the image processing method, since the peaking display disappears every first hour, it becomes even easier to confirm the main subject.

[0130] <Third Embodiment> FIG. 10 is a flowchart showing a third embodiment of the image processing method according to the present invention.

[0131] Since the display method of the peaking display is different in the third embodiment of the image processing method compared to the first embodiment shown in FIG. 8, the description of the generation method of the first moving image data and the second moving image data is omitted.

[0132] The third embodiment of the image processing method displays a live view image "without peaking" when manual focus is not required, and displays a live view image "with peaking" when manual focus is required.

[0133] In FIG. 10, the frame counter variable "count" is reset to "0" (step S200).

[0134] Next, the contour component extraction unit 16D of the processor 16 extracts contour components of the specific object from the luminance data (Y) of the specific object region R, and obtains the amplitude [Amp_n] of the contour components (step S202).

[0135] The amplitude [Amp_n] indicates the amplitude [Amp] of the current frame n (FIG. 3B). It is desirable that the amplitude [Amp_n] is the average value of the amplitudes of pixels whose contour components of a specific subject have amplitudes equal to or greater than the threshold value th.

[0136] Processor 16 acquires the increase / decrease trend of amplitude [Amp] from amplitudes [Amp_n] to [Amp_n-k] for the past (k+1) frames (step S204). The increase / decrease trend of amplitude can be determined, for example, by calculating the difference ΔAmp_n from the amplitude of the immediately preceding frame for each frame, and determining that if the average value of ΔAmp_n to ΔAmp_n-k for k+1 frames is less than 0, there is a decreasing trend, and if it is 0 or greater, there is an increasing trend.

[0137] Next, the processor 16 determines whether the increase / decrease trend of the amplitude [Amp] is decreasing (step S206).

[0138] 11 is a diagram showing an example of the transition of the amplitude [Amp] of the contour component. The example shown in FIG. 11 shows a case where the increase / decrease trend of the amplitude [Amp] is a decreasing trend.

[0139] In step S206, when it is determined that the increasing or decreasing trend of the amplitude [Amp] is a decreasing trend (in the case of "Yes"), the processor 16 outputs the second moving image data to the display unit 20 and causes the display unit 20 to display a live view image with "peaking" (step S208).

[0140] Subsequently, the frame counter variable "count" is reset to "0" (step S210), and the process proceeds to step S222.

[0141] In step S222, it is determined whether to end the peaking display. If it is determined not to end the peaking display (in the case of "No"), the process proceeds to step S202, and the processing from step S202 to step S222 is repeated for the imaging data of the next frame.

[0142] On the other hand, in step S206, when it is determined that the increasing or decreasing trend of the amplitude [Amp] is not a decreasing trend (in the case of "No"), the process proceeds to step S212, where it is determined whether the frame counter variable "count" is less than M (count < M).

[0143] Here, it is determined whether the state where the decreasing trend of the amplitude [Amp] corresponding to the focus evaluation value does not exist continues for M frames (second continuation). The second time is preferably about 3 to 5 seconds, and when the frame rate is 30 fps, M is preferably set to a value within the range of 90 to 150.

[0144] In step S212, when it is determined that count < M (in the case of "Yes"), the process proceeds to step S214. In step S214, a live view image with "peaking" is displayed on the display unit 20 based on the second moving image data (step S208).

[0145] Subsequently, the frame counter variable "count" is incremented by 1 (step S216), and the process proceeds to step S222.

[0146] On the other hand, in step S212, if it is determined that count < M (in the case of "No"), that is, if the amplitude [Amp] does not show a decreasing trend and the state without a decreasing trend continues for M frames or more, the processor 16 outputs the first moving image data to the display unit 20, causes the display unit 20 to display a live view image of "no peaking" (step S218), and increments the frame counter variable "count" by 1 (step S220).

[0147] As described above, when the amplitude [Amp] does not show a decreasing trend and the state without a decreasing trend continues for M frames or more (for the second time or more), the processor 16 determines that the focus state is stable and manual focusing is unnecessary, turns off the peaking display, and makes it easier for the user to confirm the main subject.

[0148] On the other hand, when the increasing and decreasing trend of the amplitude [Amp] changes to a decreasing trend (when the focus state deteriorates), the processor 16 makes a transition from step S206 to step S208 to resume the peaking display, so that the user can easily grasp that the focus state has deteriorated.

[0149] Although not shown in FIG. 10, when the user operates the focus ring and the focus position of the focus lens moves, it is desirable to resume the peaking display.

[0150] According to the third embodiment of the image processing method, when the focus state is stable and manual focusing is unnecessary, the peaking display is turned off, making it even easier for the user to confirm the main subject.

[0151] <Fourth Embodiment> FIG. 12 is a flowchart showing a fourth embodiment of the image processing method according to the present invention.

[0152] The fourth embodiment of the image processing method differs from the first to third embodiments in that, in addition to peaking display, it also notifies the user whether the focus evaluation value is good or not, so a description of the peaking display method will be omitted.

[0153] When a user operates the focus ring to move the focus position of the focus lens while viewing a live view image with peaking, they often move the focus lens too far or move the focus lens back and forth repeatedly.

[0154] The fourth embodiment of the image processing method notifies the user that the focus evaluation value becomes high during manual focusing and that there is no need to move the focus lens, thereby preventing excessive movement of the focus lens.

[0155] In FIG. 12, the contour component extraction unit 16D of the processor 16 extracts contour components of a specific object from the luminance data (Y) of the specific object region R of the latest frame, and obtains the amplitude [Amp_n] of the contour components (step S300).

[0156] Processor 16 acquires the maximum evaluation value, which is the maximum value among the focus evaluation values ​​for the most recent multiple frames. That is, processor 16 acquires the highest amplitude [Amp_max] among the amplitudes [Amp_n-1] to [Amp_n-h] of the contour components for the most recent h frames (step S302). It is preferable that h frames is the number of frames corresponding to the time it takes to rotate the focus ring in one direction during manual focusing.

[0157] Next, processor 16 determines whether the user has operated the focus ring to change the focus position of the focus lens (step S304). If it is determined that the focus position has changed (if "Yes"), processor 16 determines whether the focus evaluation value (amplitude [Amp_n]) of the latest frame acquired in step S300 exceeds a tolerance value set based on the maximum evaluation value (amplitude [Amp_max]) acquired in step S302. Here, the tolerance value is a value (Amp_max * ratio) obtained by multiplying the maximum amplitude [Amp_max] by a certain ratio.

[0158] If the amplitude [Amp_n] of the latest frame exceeds the allowable value (Amp_max * ratio) (if "Yes"), the processor 16 displays notification information (e.g., an icon) indicating that the focus evaluation value is high on the display unit 20, thereby notifying the user that the latest frame is in focus (step S308).

[0159] On the other hand, if the amplitude [Amp_n] of the latest frame does not exceed the allowable value (Amp_max * ratio) (if "No"), the processor 16 skips the process of step S308 and proceeds to step S310.

[0160] In step S310, it is determined whether or not to end the peaking display. If it is determined not to end the peaking display (in the case of "No"), the process proceeds to step S300, and the processes from step S300 to step S310 are repeated for the next frame.

[0161] According to the fourth embodiment of the image processing method, if the focus evaluation value of the latest frame during a manual focus period exceeds the allowable value, the user can know that the focus evaluation value is high, thereby preventing excessive movement of the focus lens during manual focus.

[0162] [others] The second moving image data for display in this embodiment is moving image data in which a peaking process is performed on the specific subject area, making the focus state of the specific subject visible, but is not limited to this, and moving image data in which the focus state of the specific subject is made visible by a method other than peaking process may also be used. Methods other than peaking process include, for example, intentionally blurring the specific subject area depending on the evaluation result of the focus state, or changing the display form of the frame surrounding the specific subject, and the like. In short, it is sufficient to generate moving image data in which the focus state of the specific subject can be easily confirmed.

[0163] Furthermore, the focus evaluation value indicating the evaluation result of the focus state of a specific subject is not limited to the amplitude of the contour components of the specific subject, but can also be, for example, the integrated value of the absolute values ​​of the high-frequency components of the moving image data of the specific subject area, or, in the case of an image sensor having phase difference pixels, the phase difference data obtained from the phase difference pixels of the specific subject area.

[0164] In this embodiment, for example, the hardware structure of the processing unit that executes various processes of the processor 16 is various processors as shown below. The various processors include a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) and functions as various processing units, a programmable logic device (PLD), such as an FPGA (Field Programmable Gate Array), whose circuit configuration can be changed after manufacture, and a dedicated electric circuit, such as an ASIC (Application Specific Integrated Circuit), which is a processor having a circuit configuration designed specifically for executing specific processes.

[0165] A single processing unit may be configured with one of these various processors, or may be configured with two or more processors of the same or different types (for example, multiple FPGAs, or a combination of a CPU and an FPGA). Also, multiple processing units may be configured with a single processor. Examples of multiple processing units configured with a single processor include, first, a configuration in which one processor is configured with a combination of one or more CPUs and software, as typified by client or server computers, and this processor functions as multiple processing units. Second, a configuration in which a processor is used to realize the functions of an entire system including multiple processing units on a single IC (Integrated Circuit) chip, as typified by a System on Chip (SoC). In this way, the various processing units are configured with one or more of the above-mentioned various processors as a hardware structure.

[0166] Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit made up of a combination of circuit elements such as semiconductor elements.

[0167] The present invention also includes an image processing program that, when installed in a computer, causes the computer to function as the image processing device according to the present invention, and a non-volatile storage medium on which this image processing program is recorded.

[0168] Furthermore, the present invention is not limited to the above-described embodiment, and it goes without saying that various modifications are possible without departing from the spirit of the present invention. [Explanation of symbols]

[0169] 10. Imaging device 12 Imaging lens 14 Imaging unit 16 processors 16A White balance correction processing unit 16B Demosaic Processing Unit 16C Luminance and chrominance conversion processing section 16D contour component extraction unit 16E Edge enhancement processing section 16F Specific subject area setting section 16G synthesis processing unit 16H compression processing section 18 Memory 20 Display section 22 Input / Output Interface 22A Recording section 22B Second connection part 22C First connection part 24 Control section R Specific subject area S10~S30, S100~S114, S200~S222, S300~S310 steps

Claims

1. An image processing device including a processor and a memory, The processor: generating first moving image data based on the input imaging data; a process of setting a specific subject area including a specific subject within an imaging area represented by the imaging data or the first moving image data; a process of evaluating a focus state based on the imaging data or the first moving image data; generating second moving image data in which the focus state of the specific subject region is visible based on the data of the specific subject region, the evaluation result of the focus state, and the first moving image data; outputting the second moving image data to a display destination; outputting the first moving image data to a recording destination; The process of generating the second moving image data includes: generating third moving image data including an evaluation result of a focus state of the specific subject based on the imaging data or the first moving image data; generating the second moving image data by combining the first moving image data excluding the specific subject region and the third moving image data of the specific subject region based on the specific subject region, the first moving image data, and the third moving image data; Image processing device.

2. the display destination is a first connection unit connectable to a display unit or an external display device, the recording destination is a recording unit or a second connection unit connectable to an external recording device; The image processing device according to claim 1 .

3. the second moving image data is moving image data in which a focus state of only the specific subject region is visible; 3. The image processing device according to claim 1.

4. the process of generating the second moving image data includes performing a process of adding data based on the evaluation result of the focus state to the specific subject area represented by data of the specific subject area within the imaging area represented by the first moving image data, thereby generating the second moving image data. The image processing device according to claim 1 .

5. the process of evaluating the focus state is a process of extracting contour components of a subject based on the imaging data or the first moving image data, and performing contour enhancement processing on the first moving image data in accordance with the amplitude of the extracted contour components. The image processing device according to claim 1 .

6. the edge enhancement processing is processing of adding a specific color to a portion where the amplitude of the extracted edge component exceeds a first threshold, and / or changing the density of the specific color according to the amplitude of the edge component; The image processing device according to claim 5 .

7. the processor sets the specific object region based on an instruction input of a specific object by a user; The image processing device according to claim 1 .

8. the processor detects a movement of the specific subject and changes the specific subject region in accordance with the movement of the specific subject. The image processing device according to claim 7 .

9. the processor outputs the first moving image data and the second moving image data alternately to the display device every first time; The image processing device according to claim 1 .

10. the processor acquires a focus evaluation value indicating an evaluation result of a focus state of the specific subject, and if the acquired focus evaluation value exceeds a second threshold, outputs the first moving image data to the display destination. The image processing device according to claim 1 .

11. the processor acquires a focus evaluation value indicating an evaluation result of a focus state of the specific subject, and outputs the first moving image data to the display destination when the acquired focus evaluation value is not on a decreasing trend and the state where the focus evaluation value is not on a decreasing trend continues for a second time. The image processing device according to claim 1 .

12. the processor acquires a focus evaluation value indicating an evaluation result of a focus state of the specific subject, and acquires a maximum evaluation value that is a maximum value among the focus evaluation values ​​for a plurality of most recent frames; When the focus evaluation value of the latest frame during a manual focus period exceeds an allowable value set based on the maximum evaluation value, notification information indicating that the focus state of the specific subject in the latest frame is good is output to the display destination. The image processing device according to claim 1 .

13. an imaging unit that outputs the imaging data; An image processing device according to any one of claims 1 to 12; An imaging device comprising:

14. An image processing method for generating moving image data that supports manual focusing by an image processing device including a processor and a memory, comprising: Each process of the processor is generating first moving image data based on the input imaging data; setting a specific subject area including a specific subject within an imaging area represented by the imaging data or the first video data; evaluating a focus state based on the imaging data or the first moving image data; generating second moving image data in which the focus state of the specific subject region is visible based on the data of the specific subject region, the evaluation result of the focus state, and the first moving image data; outputting the second moving image data to a display destination; outputting the first moving image data to a recording destination; generating third moving image data including an evaluation result of the focus state of the specific subject based on the imaging data or the first moving image data, The step of generating the second moving image data includes: generating the second moving image data by combining the first moving image data excluding the specific subject region and the third moving image data of the specific subject region based on the specific subject region, the first moving image data, and the third moving image data; Image processing methods.

15. a function of generating first moving image data based on input imaging data; a function of setting a specific subject area including a specific subject within an imaging area represented by the imaging data or the first moving image data; a function of evaluating a focus state based on the imaging data or the first moving image data; a function of generating second moving image data in which the focus state of the specific subject area is visible based on the data of the specific subject area, the evaluation result of the focus state, and the first moving image data; a function of outputting the second moving image data to a display destination; a function of outputting the first moving image data to a recording destination; a function of generating third moving image data including an evaluation result of the focus state of the specific subject based on the imaging data or the first moving image data; It is an image processing program that enables a computer to realize the above. The function of generating the second moving image data is generating the second moving image data by combining the first moving image data excluding the specific subject region and the third moving image data of the specific subject region based on the specific subject region, the first moving image data, and the third moving image data; Image processing program.

16. A non-transitory computer-readable recording medium having the program according to claim 15 recorded thereon.

Images