Imaging apparatus, method for controlling imaging apparatus, and program

Abstract

To provide an imaging apparatus, a method for controlling an imaging apparatus, and a program for improving convenience of a focus assist function.SOLUTION: A video camera 100 acquiring an image in an imaging optical system comprises a camera control unit 104 that sets a peaking detection frequency PeakFq, and when a frequency extracted from an image exceeds the peaking detection frequency PeakFq, superimposes a colored mask edge image on the image. When there is no movement of the video camera 100, the camera control unit 104 sets a base frequency Fpx to the peaking detection frequency PeakFq, and when there is a movement of the video camera 100, sets a frequency obtained by multiplying the base frequency Fpx by a Z motion coefficient α1, an XY motion coefficient α2, or a coefficient α3, to the peaking detection frequency PeakFq.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an imaging device, a control method for the imaging device, and a program.

Background Art

[0002] When a user performs shooting with a digital camera or a video camera in manual focus, the user moves the focus lens while observing the monitoring image to find the focus position where the subject becomes the sharpest, that is, performs focusing. In Patent Document 1 and the like, a focus assist function called peaking has been proposed to improve the accuracy of focusing. The peaking function detects sharp portions of the monitoring image using a pre-specified frequency, and displays a marker by superimposing an auxiliary signal on the detected sharp portions. When the user moves the focus lens while visually recognizing the movement of the marker, the amount and position of the marker change, so that the user can visually recognize the approximate focus position. Further, in Patent Document 2, since the ability of the frequency used in the detection of the peaking function (hereinafter referred to as "peaking detection frequency") changes depending on the subject frequency in the monitoring image, a technique for changing the peaking detection frequency according to the focused state has been proposed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in a manual focus shooting scene where a user with a camera approaches or moves away from a subject in a focused state, when the camera moves in the direction of the subject, even if the marker display of the peaking function changes, it may be difficult to tell if the focus state has changed. Therefore, there was a risk that the user with the camera would get too close to or move too far away from the focused subject, causing the subject to become out of focus. Also, when the camera moves, the subject in the image may be enlarged or reduced, causing the subject frequency to change, or blurring may occur in the image, causing the frequency of the edges and contours of the subject to fluctuate. Therefore, there was a problem in that the peaking function could not be fully utilized due to the inability to detect edges at a pre-specified peaking detection frequency.

[0005] The present invention has been made in view of the above problems. An object of the present invention is to provide an imaging device, a control method for the imaging device, and a program that improve the convenience of the focus assist function.

Means for Solving the Problems

[0006] In order to achieve the above object, an imaging device of the present invention is an imaging device that acquires an image with an imaging optical system, and includes a setting means for setting a detection frequency, and a display means for superimposing a specific display on the image when a frequency extracted from the image exceeds the detection frequency. The setting means sets a base frequency as the detection frequency when there is no movement of the imaging device, and sets a frequency obtained by multiplying the base frequency by a first weight as the detection frequency when there is movement of the imaging device.

Effects of the Invention

[0007] According to the present invention, the convenience of the focus assist function can be improved.

Brief Description of the Drawings

[0008]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Mode for Carrying Out the Invention

[0009] Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. However, the configurations described in this embodiment are merely examples, and the scope of the present invention is not limited by the configurations described in this embodiment. For example, each part constituting the present invention can be replaced with any configuration that can exhibit the same function. Also, any components may be added. Further, any two or more configurations (features) in this embodiment can be combined.

[0010] <Imaging Device> Hereinafter, this embodiment will be described. FIG. 1 is a block diagram showing the overall configuration of a lens-exchangeable video camera 100 as an example of an imaging device according to this embodiment. The video camera 100 (imaging device) of this embodiment is composed of an exchangeable lens unit 101 and a video camera body 102. A lens control unit 103 that comprehensively controls the operation of the entire lens and a camera control unit 104 (setting means) (display means) that comprehensively controls the operation of the entire camera system including the lens unit 101 can communicate with each other through terminals provided on the lens mount. First, the configuration of the lens unit 101 will be described. The fixed lens 105, the aperture 106, and the focus lens 107 together with a configuration not shown constitute an imaging optical system.

[0011] The aperture 106 is driven by an aperture drive unit 108 to control the amount of incident light to the image sensor 111 described later. The focus lens 107 is driven by a focus lens drive unit 109 composed of a DC motor and its control circuit. Thereby, according to the position of the focus lens 107, the focal position of the imaging optical system and the subject conjugate focal length change. In the following description, the focus lens 107 may be abbreviated as "lens". The aperture drive unit 108 and the focus lens drive unit 109 are controlled by the lens control unit 103 to determine the aperture amount of the aperture 106 and the position of the focus lens 107.

[0012] The lens operation unit 110 is a group of input devices for the user to set the operations of the lens unit 101. The settings related to the operations of the lens unit 101 include switching between AF (auto focus) / MF (manual focus) modes, adjusting the position of the focus lens 107 by MF, setting the operation range of the focus lens 107, etc. When the lens operation unit 110 is operated, the lens control unit 103 performs control corresponding to the operation. The lens control unit 103 controls the aperture drive unit 108 and the focus lens drive unit 109 according to control commands and control information including drive target position information and lens drive speed setting information received from the camera control unit 104. Further, as a result of the control, the lens control unit 103 transmits the position information of each unit in the lens unit 101 and the optical information at that time to the camera control unit 104.

[0013] Next, the configuration of the video camera body 102 will be described. The video camera body 102 is configured to be able to acquire an imaging signal from the light beam that has passed through the imaging optical system of the lens unit 101. The image sensor 111 is composed of a CMOS sensor. The light beam that has passed through the imaging optical system of the lens unit 101 forms an image on the light receiving surface of the image sensor 111, and is converted into signal charges corresponding to the incident light amount by the photodiodes provided in each pixel arranged in the image sensor 111. The signal charges accumulated in each photodiode are sequentially read out from the image sensor 111 as voltage signals corresponding to the signal charges by the drive pulses output by the timing generator 112 according to the commands of the camera control unit 104. Each process is repeated with the period of full-frame reading from the image sensor 111 as the camera control period.

[0014] In addition, each pixel of the image sensor 111 used in the present embodiment is composed of a pair of photodiodes A and B and one microlens. Each pixel of the image sensor 111 divides the incident light with the microlens to form a pair of optical images on the pair of photodiodes A and B, and outputs a pair of pixel signals (A signal and B signal) used for the AF signal described later from the pair of photodiodes A and B. By adding the outputs of the A signal and the B signal, an imaging signal (A + B signal) is obtained. Further, a plurality of A signals output from a plurality of pixels are synthesized among the A signals, and a plurality of B signals are synthesized among the B signals. As a result, a pair of image signals as an AF signal (in other words, a focus detection signal) used for AF by the imaging surface phase difference detection method is obtained.

[0015] The AF signal processing unit 113 performs a correlation operation on the pair of image signals, calculates the phase difference (hereinafter referred to as "image shift amount"), which is the shift amount of the pair of image signals, and further calculates the defocus amount and defocus direction of the imaging optical system from the image shift amount. The CDS / AGC / AD converter 114 performs correlation double sampling, gain adjustment, and AD conversion on the AF signal and the imaging signal read from the image sensor 111 to remove reset noise. The CDS / AGC / AD converter 114 outputs the processed imaging signal and AF signal to the image input controller 115 and the AF signal processing unit 113, respectively.

[0016] The image input controller 115 stores the imaging signal output from the CDS / AGC / AD converter 114 as an image signal in the SDRAM 117 via the bus 116. The image signal stored in the SDRAM 117 is read out by the display control unit 118 via the bus 116 and displayed on the display unit 119. Also, in the recording mode for recording the image signal, the image signal stored in the SDRAM 117 is recorded by the recording medium control unit 120 on a recording medium 121 such as a semiconductor memory. The ROM 122 stores control programs and processing programs executed by the camera control unit 104 and various data necessary for the execution of these programs. The flash ROM 123 stores various setting information related to the operation of the video camera main body 102 set by the user.

[0017] Here, an outline description of the peaking process will be given. When the user performs an on-set operation for peaking display with the camera operation unit 124, the peaking detection frequency designation screen is displayed on the display unit 119 to accept the user operation. The image processing control unit 125 transfers the user-specified peaking detection frequency and the image signal stored in the SDRAM 117 to the image processing unit 126. The image processing unit 126 performs band-pass filter processing on the image based on the user-specified peaking detection frequency, and performs highlighting processing to enhance the display in blue or the like based on the rules described later on the image after the band-pass filter processing. The image processing control unit 125 stores the image signal subjected to such highlighting processing again in the SDRAM 117. The image signal stored again in the DRAM 117 is read out by the display control unit 118 via the bus 116 and displayed on the display unit 119 as a peaking image.

[0018] In addition, the image processing unit 126 is equipped with circuits for performing various image processes. The circuits mounted in the image processing unit 126 include a motion vector calculation circuit that calculates motion vectors using images before and after a frame, a codec circuit for recording captured images, a circuit that extracts a subject with specific features from an image, and the like. The subject detection unit 127 in the camera control unit 104 determines the coordinate position of a specific subject in the imaging signal based on information on the detection of a specific subject based on the imaging signal and the evaluation value of its feature amount. Further, the subject detection unit 127 uses the feature information of imaging indicated by the continuously incoming imaging signals in the video mode or live mode, and determines the coordinate position of the destination in the imaging signal and the predicted position in the next frame when the detected specific subject moves. By this determination, the subject detection unit 127 detects the coordinate position of the moved specific subject. Note that the specific subject is, for example, a face subject, a human body, an animal, or the like.

[0019] On the other hand, the subject tracking unit 128 is used for the purpose of continuously and tenaciously tracking the same subject when the subject detected by the subject detection unit 127 is to be the AF target. The subject tracking unit 128 performs discrimination processing as to whether it is the same subject by a method different from that of the subject detection unit 127, for example, using color information, edge shape, or the like, and tracks the identified subject. When the subject tracking unit 128 captures a subject with a high probability of being the same as the subject detected by the subject detection unit 127, it feeds back to the detection range of the subject detection unit 127. As a result, in the subject detection unit 127, the subject is used as the center of the detection range in the next frame.

[0020] The AF signal processing unit 113 for focus detection performs a correlation operation on a pair of image signals that are AF signals output from the CDS / AGC / AD converter 114, and calculates the amount of image shift and its reliability for these pair of image signals. The reliability is calculated using the degree of coincidence when the two images are correlated, the steepness of the correlation change amount, and the contrast signals of the two images. The AF signal processing unit 113 sets the position and size of the distance measurement area, which is the area where focus detection and AF are performed within the captured image. The AF signal processing unit 113 outputs the information on the amount of image shift and reliability calculated in the distance measurement area to the camera control unit 104.

[0021] The AF control unit 129 within the camera control unit 104 performs a focus adjustment operation by converting the amount of image shift detected by the AF signal processing unit 113 into a defocus amount and sending an instruction to move the focus position to the lens control unit 103. The AF control switching unit 130 switches the focus adjustment method performed by the AF control unit 129 based on the operations of the motion determination unit 131 and the camera operation unit 124. Also, the camera control unit 104 has a prediction unit 132 that has a function of predicting the next detected defocus position from the temporal motion of the detected defocus information, thereby realizing a subject tracking function. The storage unit 133 holds a focusing trajectory curve indicating how to move the focus lens 107 according to the detected defocus position. Furthermore, the storage unit 133 holds various determination information for focus control in time series. The data held in the storage unit 133 is temporarily held in the memory circuit 134.

[0022] The movement determination unit 131 has a function of calculating an estimated value of how much the focus lens 107 moves by integrating the difference between the operation instruction amount instructed for the focus lens 107 and the actual operation amount by which the focus lens 107 actually moves in the time series direction. Note that the movement determination unit 131 can be controlled by the AF control unit 129 for these functions. The acceleration sensor 135 is attached to the video camera main body 102 and is a sensor capable of detecting accelerations in three axial directions. Here, assuming that the video camera 100 (sensor surface) is facing the subject directly, the horizontal direction, the vertical direction, and the direction of the distance between the video camera 100 and the subject are respectively referred to as the X-axis, the Y-axis, and the Z-axis. Therefore, the X-axis and the Y-axis are directions orthogonal to the Z-axis. The acceleration sensor 135 can also detect the speed of the video camera 100 by integrating the acceleration information detected every hour in the time axis direction.

[0023] Hereinafter, operations and processes particularly related to the autofocus function of the video camera 100 having the above-described configuration will be described in detail. FIG. 2 is a flowchart mainly describing the operations of the camera control unit 104 when the video camera 100 is activated, particularly the operations related to the peaking function. The operations (control method of the imaging device) shown in the flowchart of FIG. 2 are realized by the camera control unit 104 (computer) executing the program stored in the ROM 122. Note that hereinafter, the video camera 100 may be abbreviated as the "camera". When an activation operation is performed on the camera operation unit 124 of the video camera main body 102, this flowchart is started.

[0024] When camera startup begins, in step S201, the camera control unit 104 performs startup processing. The details of the startup processing are further described in the flowchart of FIG. 3. As shown in FIG. 3, when the startup processing begins, in step S301, after confirming the attachment / detachment state of the lens, if the lens is attached, the camera control unit 104 performs a lens reset operation. Specifically, the camera control unit 104 performs communication processing with the lens unit 101, acquires lens type information and lens drive speed information that can be set for the lens, issues a lens drive command, moves the lens to the initial position, and adjusts the lens position. By such a lens reset operation, the lens is driven to the held aperture state and zoom position.

[0025] In step S302, the camera control unit 104 resets each part of the video camera body 102. Thereby, calibration processing for improving the detection accuracy of the direction and the magnitude of movement obtained from the acceleration sensor 135 and sensor reset processing including reset processing of the imaging element 111 are performed. In step S303, the camera control unit 104 performs initial setting of focus control. Specifically, the camera control unit 104 acquires information related to focus control held in the camera in advance and sets it as the initial setting value. Thereby, driving the focus lens position, AF frame setting, etc. are performed.

[0026] In step S304, the camera control unit 104 reads the mode setting related to the peaking function and performs the setting inside the camera based on the held data in the ROM 122. Here, reading of the peaking mode as to whether the peaking function is on or off, and further, when the peaking function is on, the peaking detection frequency (detection frequency) and settings related to camera motion detection to be described in detail later are performed. Note that the description of the frame rate, recording mode, etc. in the shooting mode is omitted from the detailed description. Thereafter, the startup processing shown in the flowchart of FIG. 3 ends.

[0027] Return to FIG. 2. When the startup process of step S201 ends, in step S202, the camera control unit 104 determines whether the peaking function is on. Thereby, the process branches according to the previous mode setting of the peaking function. When the camera control unit 104 determines that the peaking function is not on, that is, when the peaking function is off in the user setting, the process returns to step 202 itself. Thereby, it becomes a loop process and enters a standby state. On the other hand, when the camera control unit 104 determines that the peaking function is on, the process proceeds to step S203. In step S203, the camera control unit 104 sets the peaking detection frequency and the like. Here, the initial setting value or the previous held value obtained and held previously is set for the peaking detection frequency and the like.

[0028] In step S204, the camera control unit 104 performs camera motion detection and setting processing (setting step). Here, the camera control unit 104 re-sets the peaking detection frequency and the like set in step S203 according to the result of the camera motion detection. A series of processes in step S204 will be described in more detail using the flowchart of FIG. 4. As an overview, first, information such as whether the camera is not moving, is in the middle of moving, or has stopped again after moving, and in which direction and at what speed it is moving in the space between the camera and the subject is obtained. Then, the peaking detection frequency is re-set according to the motion state of the camera determined by that information.

[0029] Hereinafter, the camera movement detection and setting process in step S204 will be described in detail with reference to FIG. 4. When the camera movement detection and setting process is started, in step S401, the camera control unit 104 checks the data of the acceleration sensor 135 in the camera and detects at what speed and in which direction of the X, Y, and Z axes the camera is moving. Note that the camera control unit 104 (movement detection means) detects the movement of the camera as a frequency variation of the captured image. The frequency variation of the captured image is detected based on the frequency of the captured image calculated from the movement amount and the optical parameters of the imaging optical system. In step S402, the camera control unit 104 determines the movement state Cs of the camera according to the state transition process of extracting the movement state Cs of the camera according to the movement state Cs of the camera at the previous frame time.

[0030] Here, the movement state Cs of the camera will be described in detail with reference to FIG. 5. Regarding the movement of the camera, the movement in the Z direction and the movement in the XY direction are separated, and the detection and state transition process are performed. The camera holds the movement state in the Z direction of the camera as CZs and the movement state in the XY direction of the camera as CXYs. FIG. 5(a) is a diagram showing the movement in the Z direction among the movements of the camera. When the camera is just started or there is no movement in the Z direction of the camera, the camera is in the stop state of state ST501. When the movement in the Z direction of the camera is detected in the stop state of state ST501, the camera transitions to state ST502 and enters the moving state in the Z direction.

[0031] When the camera stops moving in the Z direction during the movement state of State ST502, the camera transitions to State ST503 and enters the post-movement re-stop state. In the post-movement re-stop state of State ST503, if the camera remains stopped and not moving for a predetermined period of time, the camera times out and transitions to the stop state of the initial State ST501. On the other hand, in the post-movement re-stop state of State ST503, if movement in the Z direction of the camera is detected, the camera transitions back to the moving state in the Z direction of State ST502. In this way, the above-described state transition determination process is repeated. Also, in Fig. 5(a), the transition events between each of States ST501, ST502, and ST503 are those indicated by ST504, ST505, ST506, and ST507 and are held in the camera as CZs.

[0032] Fig. 5(b) is a diagram showing the movement of the camera in the XY direction. Here, if either movement detection in the X direction or movement detection in the Y direction occurs, the camera is determined to be moving in the XY direction. That is, movement detection in the XY direction is the logical sum of the movement in the X direction and the movement in the Y direction, and the movement state of the camera in the XY direction detected by that logical sum is held as CXYs. The states of each of States ST511, ST512, and ST513 are the same as the states of States ST501, ST502, and ST503 in the movement in the Z direction, so the description is omitted. What is specific to the movement of the camera in the XY direction is the panning / tilting state as the state of State ST514. The transition events between each of States ST511, ST512, and ST513 are those indicated by ST515, ST516, ST517, and ST518, but since they are the same as the transition events of ST504 to ST507 in the movement of the camera in the Z direction, the description is omitted.

[0033] When it is detected that the camera is moving in the same direction and at the same speed as the movement direction during the state of moving in the XY direction of state ST512, as the transition event of ST519, the camera transitions to the panning / tilting state of state ST514. In the panning / tilting state of state ST514, when it is detected that the movement direction or speed of the camera has changed as the transition event of ST520, it transitions back to the state of moving in the XY direction of state ST512. Note that the stop states of state ST501 and state ST511 correspond to "when there is no movement of the imaging device". Also, the post-movement re-stop states of state ST503 and state ST513 correspond to "when the movement of the imaging device has stopped".

[0034] Figure 6 is a diagram showing a state transition table summarizing how the peaking detection frequency is switched based on the state transition of camera movement detection. Hereinafter, the switching of the peaking detection frequency will be described in detail with reference to Figure 6, including the state transition by mode transition. On the vertical axis of Figure 6, the peaking detection state before the event and the set value of the peaking detection frequency are arranged. Note that the set value of the peaking detection frequency is composed of a base frequency and a coefficient, and among the coefficients, the speed coefficient described later is omitted. State A is the peaking detection off state, state B is the state where the peaking detection frequency is set to a high frequency, state C is the state where the peaking detection frequency is set to a low frequency, and state D is the image extraction setting state where the peaking detection frequency is set based on the frequency extracted from the image. State E is the peaking detection frequency setting state in the movement state of the Z direction of the camera, state F is the peaking detection frequency setting state in the movement state of the XY direction of the camera, and state G is the peaking detection frequency setting state in the movement states of the Z direction and XY direction of the camera. State H is the peaking detection frequency setting state in the state where the camera is moving towards a stop from the state of having movement, that is, in the transition state from movement to stop.

[0035] On the horizontal axis, as events according to the mode, the events of (1) peaking off, (2) peaking on in the high-frequency mode, (3) peaking on in the low-frequency mode, and (4) peaking on in the image extraction mode are arranged. Also, as events according to the movement state of the camera, the events of (5) detection of movement of the camera in the Z direction, (6) detection of movement of the camera in the XY direction, (7) stop of movement of the camera in the Z direction, (8) stop of movement of the camera in the XY direction, and (9) movement timeout of the camera are arranged. Further, as events according to the result of image contrast evaluation, the events of (10) low image contrast and (11) high image contrast are arranged. The state transition table in FIG. 6 shows that when an event on the horizontal axis occurs in the current state on the vertical axis, it transitions to the process shown at the intersection of the vertical axis and the horizontal axis. Hereinafter, in the state transition table of FIG. 6, for example, when indicating the intersection of the state A on the horizontal axis and the event (1) on the vertical axis, it is denoted as "A-(1)".

[0036] Note that in the initial setting of the camera, it is peaking off, and the initial mode when peaking on is the high-frequency mode. Also, in the transition between the state of moving in the XY direction in state ST512 and the panning / tilting state in state ST514, there is no change in the peaking detection frequency setting. Therefore, since the panning / tilting state in state ST514 is equivalent to the state of moving in the XY direction in state ST512, it is not shown in the state transition table of FIG. 6.

[0037] <First Use Case> From here, based on the state transition table of FIG. 6, the present embodiment will be described in detail by limiting it to a use case where the peaking detection frequency is switched according to the movement state Cs of the camera. Here, as one such use case, a use case will be described in which a camera that is not moving in the peaking-off state is activated, the high-frequency mode is turned on, and after the movement of the camera occurs in the Z direction, the first use case that has timed out. In the first use case, in step S304 that constitutes the activation process of step S201, the camera starts up with the peaking mode off. In step S202, until the peaking mode is turned on, it becomes a mode check state, that is, a standby state.

[0038] When the peaking mode is set to on by the user at a certain timing, in step S203, based on the default of A-(2) in FIG. 6, the frequency Fp1 is extracted as the base frequency. Note that the frequency Fp1 is a predetermined frequency, which is designed in advance and held by the camera. This is the same for the frequency Fp2 described later. Note that the frequency Fp3 described later is used as the base frequency from the frame after the image frequency evaluation process for extracting the frequency Fp3 from the image. However, the frequency Fp3 may be configured to be used as the base frequency from the frame of the image in which the frequency Fp3 is extracted.

[0039] In step S402 that constitutes the movement detection and setting process of the camera in step S204, a state where no movement of the camera has occurred is determined as the movement state Cs of the camera. Hereinafter, the continuation of the movement detection and setting process of the camera in step S204 will be described with reference to FIG. 4. In step S403, the camera control unit 104 branches the process according to the movement state Cs of the camera. When the movement of the camera is not in both the Z direction and the XY direction, the camera control unit 104 advances the process to step S404. In step S404, the camera control unit 104 extracts the base frequency Fpx. In the first use case, as described above, the frequency Fp1 is extracted as the base frequency Fpx.

[0040] In step S405, the camera control unit 104 calculates the peaking detection frequency PeakFq using the formula PeakFq = Fpx. Here, the frequency Fp1 is calculated as the peaking detection frequency PeakFq. In step S406, the camera control unit 104 re - sets the peaking detection frequency PeakFq calculated in step S405 to the peaking detection function of the image processing control unit 125. After that, the camera motion detection and setting process shown in the flowchart of FIG. 4 ends. When returning to FIG. 2, each process after step S205 is performed, and further, the process returns to step S203, and the loop process from step S203 to step S209 is repeated. Note that each process after step S205 is assumed to have no difference depending on the presence or absence of camera motion. Therefore, the description of each process after step S205 will be given together with the description of the case where there is camera motion, which will be described later.

[0041] In the first use case, at a certain timing thereafter, a movement in the Z - direction occurs in the camera. In this case, in step S401 which constitutes the camera motion detection and setting process of step S204, the movement of the camera in the Z - direction is detected. Therefore, it is considered that the event of camera Z - motion detection in ST504 has occurred, and a transition is made from the stop state of state ST501 to the moving - in - the - Z - direction state of state ST502. As a result, in the state transition table of FIG. 6, a transition is made to state E (the Z - direction motion state of the camera) by the process of B - (5). In step S402, the state in which the camera has a movement in the Z - direction is determined as the camera motion state Cs.

[0042] Hereinafter, the continuation of the camera motion detection and setting process of step S204 will be described with reference to FIG. 4. As described above, in step S403, the camera control unit 104 branches the process according to the camera motion state Cs. When the camera has a movement in either the Z - direction or the XY - direction, the camera control unit 104 advances the process to step S407. In step S407, the camera control unit 104 branches the process according to whether there is a movement of the camera in the Z - direction. When there is a movement of the camera in the Z - direction, the camera control unit 104 advances the process to step S408.

[0043] In step S408, the camera control unit 104 extracts a Z movement coefficient α1 (first weight), which is a coefficient during movement in the Z direction, a base frequency Fpx corresponding to the mode, and a Z movement speed Vz, which is the speed during movement in the Z direction. Note that the Z movement coefficient α1 is less than 1 and is held in advance. In the first use case, as the base frequency Fpx, the frequency Fp1 of the high-frequency mode is extracted. In step S409, the camera control unit 104 calculates a Z movement speed coefficient according to the Z movement speed Vz. The Z movement speed coefficient (speed coefficient) is a coefficient for calculation to lower the peaking detection frequency PeakFq because the faster the speed of the camera in the Z direction, the earlier the acquired captured image becomes blurred. The Z movement speed coefficient is calculated in consideration of the Z movement speed Vz based on a coefficient obtained experimentally in advance. Note that the Z movement speed coefficient is made less than 1.

[0044] In step S410, the camera control unit 104 calculates the peaking detection frequency PeakFq using the formula PeakFq = Fpx × α1 × Z movement speed coefficient. Thereby, in the first use case, the peaking detection frequency PeakFq is calculated using the formula Fp1 × α1 × Z movement speed coefficient. In step S411, the camera control unit 104 holds Z movement information indicating that the movement of the camera is in the Z direction as a Z movement flag so that it can be used later. In step S412, the camera control unit 104 branches the process according to whether there is movement of the camera in the XY direction. When there is no movement of the camera in the XY direction, the camera control unit 104 advances the process to step S406. In step S406, the camera control unit 104 resets the peaking detection frequency PeakFq calculated in step S411 to the peaking detection function of the image processing control unit 125. Thereafter, the camera movement detection and setting process shown in the flowchart of FIG. 4 ends.

[0045] Returning to FIG. 2, in step S205, the camera control unit 104 performs edge extraction processing for peak display of the captured image (image) based on the re-set peaking detection frequency PeakFq. Details of the edge extraction processing are further described in the flowchart of FIG. 7. As shown in FIG. 7, when the edge extraction processing starts, in step S701, the camera control unit 104 sets the re-set peaking detection frequency PeakFq as the filter coefficient of the band-pass filter through the image processing control unit 125 to the image processing unit 126. In step S702, the camera control unit 104 performs band-pass filter processing on the captured image. In step S703, the camera control unit 104 counts the number of peaks (number of pixels) for each region where pixels exceeding a predetermined peak value are consecutive in the image after the band-pass filter and holds it as Pnum. In step S704, the camera control unit 104 generates an edge image in which the pixels in the region where Pnum is equal to or more than a predetermined number are converted to white and the other pixels are converted to black in the image after the band-pass filter. Thereafter, the edge extraction processing shown in the flowchart of FIG. 7 ends.

[0046] Returning to FIG. 2, in step S206, the camera control unit 104 evaluates what high-frequency components are included in the high-frequency components of the captured image by performing image frequency evaluation processing. Details of the image frequency evaluation processing are further described in the flowchart of FIG. 8. Note that in the image frequency evaluation processing shown in the flowchart of FIG. 8, processing using two-dimensional discrete cosine transform (hereinafter referred to as "DCT") is performed, but other methods such as FFT may also be used. As shown in FIG. 8, when the image frequency evaluation processing starts, in step S801, the camera control unit 104 divides the captured image into predetermined image blocks and performs DCT processing for each of the divided image blocks. In step S802, the camera control unit 104 extracts one frequency with high-frequency components for each image block, excluding the high-frequency components that are noise components.

[0047] In step S803, the camera control unit 104 branches the process according to whether there is an area designation. Area designation generally depends on a mode setting such as zone AF known to the camera. In addition to zone AF, area designation refers to, for example, something that is not explicitly shown to the user such as image enlargement but is internally limited in area by the camera. Furthermore, area designation refers to something that is limited based on the execution result of the camera control unit 104 (focus detection means) in the focus evaluation process of step S207 described later. For example, the camera control unit 104 extracts an area determined to be in focus and designates an area where the extracted area overlaps with the zone AF area. In this designation, the defocus information held in the defocus image generated in the previous frame is used. In particular, among the defocus images, the area is designated limited to the in-focus area. If there is no such area designation, the camera control unit 104 advances the process to step S804. On the other hand, if there is any of these area designations, the camera control unit 104 advances the process to step S805.

[0048] In step S804, while generating a histogram that counts how many blocks there are for the highest frequency component in the entire image, the camera control unit 104 extracts a high frequency having the same high frequency component with a block count equal to or more than a predetermined number of blocks. As a standard for how many blocks, it is designed to be equivalent to 0.1% of the image. For example, when the image size is equivalent to 4K (8M), and 0.1% of 8M, that is, equivalent to 8K pixels, and the DCT unit pixel block is 8×8 pixels, if all 64 pixels corresponding to 125 blocks are high frequency, the calculation is such that the 8K pixel equivalent becomes the high frequency target. In reality, since not all pixel blocks have the same high frequency, it becomes smaller than this value, but about 0.1% of the high frequency components in the image can be extracted. In this way, after extracting the high frequency component by DCT, the camera control unit 104 converts it to the frequency for peak detection and then holds it as the frequency Fp3.

[0049] In step S805, the camera control unit 104 (extraction means) extracts a high frequency in the specified area. At this time, when the camera control unit 104 extracts high frequency components based on about 1% to 5% of the size of the specified area as a guide, after converting them into frequencies for peak detection, it holds them as the frequency Fp3. Note that the above numerical values of 1% to 5% are changed according to the size of the specified area. Also, the frequency Fp3 held here is used as the base frequency Fpx after the next frame and subsequent frames. For example, if the camera is not moving, in step S404, the frequency Fp3 is set as the base frequency Fpx. After the processing of step S804 or step S805 is performed, the image frequency evaluation process shown in the flowchart of FIG. 8 ends.

[0050] Returning to FIG. 2, in step S207, the camera control unit 104 performs focus evaluation processing. Details of the focus evaluation processing are further described in the flowchart of FIG. 9. As shown in FIG. 9, when the focus evaluation processing starts, in step S901, the camera control unit 104 acquires lens information. In step S902, the camera control unit 104 designates a detection area for the focus state and notifies the AF signal processing unit 113. Note that when there is the above area designation, the camera control unit 104 sets the designated area in terms of the size of a rectangle, the start point, the end point, and the number of divisions of the frame inside.

[0051] In step S903, the camera control unit 104 detects the focus state for each frame based on the information regarding the set frame, and generates a defocus image mainly as the distribution of defocus information in units of frames. Note that the minimum frame size unit is equivalent to 16×16 pixels. That is, for a 4K image, the defocus image is generated with a pixel size equivalent to 256×128. In step S904, the camera control unit 104 (classification means) performs classification image generation based on the sign and magnitude of the defocus amount. Specifically, the camera control unit 104 calculates Fδ from the aperture value F and the allowable confusion circle δ of the lens information acquired in step S901 above. Further, the camera control unit 104 generates six images with a pixel size equivalent to 256×128 in the manner described later.

[0052] Specifically, the camera control unit 104 extracts frames within ±1Fδ from the defocus image, and generates an image in which the pixels of those frames are white and the rest are black as the in-focus image. In this way, the camera control unit 104 generates the first image as the in-focus image. Also, the camera control unit 104 generates the second and subsequent images in the same manner as the first image. However, the second image is an image of the frames from -1Fδ to -3Fδ, and is generated as a slightly blurred image. The third image is an image of the frames from +1Fδ to +3Fδ, and is generated as a slightly blurred image. The fourth image is an image of the frames from -3Fδ to -8Fδ, and is generated as a blurred image. The fifth image is an image of the frames from +3Fδ to +8Fδ, and is generated as a blurred image. The sixth image is an image of the frames below -8Fδ or above +8Fδ, and is generated as a highly blurred image.

[0053] Also, in the following description, each of the first to sixth images described above is referred to as MaskDefImage0 to MaskDefImage5. Further, MaskDefImage0 is referred to as the "in-focus area mask image", and each of MaskDefImage1 to MaskDefImage5 is referred to as the "out-of-focus area mask image". FIG. 10 is an image diagram showing these six mask images and the conditions of the in-focus state. In FIG. 10, the horizontal axis shows the defocus amount indicated based on the allowable confusion circle and the relative relationship with the six mask images from MaskDefImage0 to MaskDefImage5. Among the signs of the defocus amount here, + represents a state where the focus is on the front side and the subject is on the side closer to the in-focus position of the camera, indicating that the focus approaches as the camera and the subject move away from each other. On the contrary, - represents a state where the focus is on the rear side and the subject is on the side farther from the in-focus position of the camera, indicating that the focus approaches as the camera and the subject move closer to each other.

[0054] Return to FIG. 9. In step S905, the camera control unit 104 performs an enlargement process of converting each of the six images with a pixel size equivalent to 256×128 generated in step 904 to the original image size. As a result, each of the six mask images classified as defocus from the defocus image is generated in 4K size. Then, the focus evaluation process shown in the flowchart of FIG. 9 ends.

[0055] Returning to FIG. 2, in step S208, the camera control unit 104 performs a peaking image generation process. The peaking image generation process uses the defocus image and its mask image generated in the focus evaluation process and the edge image generated in the edge extraction process to generate a masked edge image, and synthesizes the generated image with the captured image to generate a peaking image. The details of the peaking image generation process are further described in the flowchart of FIG. 11.

[0056] As shown in FIG. 11, when the peaking image generation process starts, in step S1101, the camera control unit 104 acquires the defocused image and six mask images generated by the focus evaluation process. In step S1102, the camera control unit 104 acquires the edge image generated by the edge extraction image process. In step S1103, the camera control unit 104 acquires the movement direction and speed of the camera detected in step S401. In step S1104, the camera control unit 104 branches the process according to the movement of the camera. When there is no movement of the camera, the camera control unit 104 advances the process to step S1105. On the other hand, when there is movement of the camera, the camera control unit 104 advances the process to step S1106. In step S1105, the camera control unit 104 uses only the in-focus area mask image as the mask image.

[0057] In step S1106, the camera control unit 104 selects at least one or more out-of-focus area mask images from the five out-of-focus area mask images, that is, from MaskDefImage1 to MaskDefImage5, according to the movement direction and speed of the camera. The selected out-of-focus area mask image is used in step S1107 described later. When the direction of the camera movement is the direction in which the camera and the subject approach each other and the speed of the camera is small, if MaskDefImage1 is used in step S1107 described later, the mask image becomes an image including an out-of-focus area that will be in focus when approaching further. The out-of-focus area is an area that will be in focus in the next frame or a slightly later frame if it is a display in the state where the camera is moving, and can be a reference display for the user to stop the movement of the camera as a display function including preview or prediction using the defocus result.

[0058] On the one hand, when MaskDefImage2 is used in step S1107 described below, the mask image will become an image that includes a defocused area that blurs more clearly than the current state if the camera moves closer to the subject. This defocused area can serve as a reference display for the user to stop the camera movement by warning the user that if the camera moves closer to the subject, it will become strongly blurred. The selection of MaskDefImage1 or MaskDefImage2 is based on the use case and thus depends on the mode setting. That is, whether to emphasize the defocused area that will be in focus in the future or the defocused area that will become clearly blurred in the future when the camera moves depends on the user's mode setting. Of course, the selection of the defocused area mask image does not necessarily have to clearly depend on the dedicated peaking mode. For example, in the default mode such as auto shooting, it may be associated with a shooting mode that requires emphasizing the defocused area that will become clearly blurred in the future.

[0059] Also, when the direction of the camera movement is the direction in which the camera and the subject approach each other and the camera speed is high, it is advisable to use MaskDefImage3 in step S1107 described below. In this case, the mask image is similar to the above, but becomes an image that includes a defocused area where a larger defocus detected when the camera movement is large is detected. Also, when the direction of the camera movement is the direction in which the camera and the subject move away from each other and the camera speed is low, it is advisable to use MaskDefImage2 in step S1107 described below. On the other hand, when the direction of the camera movement is the direction in which the camera and the subject move away from each other and the camera speed is high, it is advisable to use MaskDefImage4 in step S1107 described below.

[0060] In this way, when the out-of-focus area mask image is selected, in step S1107, the camera control unit 104 takes the logical sum of the in-focus area mask image and the selected out-of-focus area mask image to obtain a mask image of defocus information. In step S1108, the camera control unit 104 processes the edge image generated by the edge extraction process with the mask image of step S1105 or step S1107 to generate a colored mask edge image. In this way, in the colored mask edge image, the edges in the in-focus area are colored blue, and the edges in the out-of-focus area are colored yellow or the like. Further, the edges in the area not included in the mask image of the defocus information are masked so that the edges disappear.

[0061] In step S1109, the camera control unit 104 synthesizes the colored mask edge image and the captured image to generate a peaking image. In this way, in the peaking image, through the colored mask edge image, color separation is performed according to the four states of in-focus, slightly out of focus, out of focus, and severely out of focus. Thereafter, the peaking image generation process shown in the flowchart of FIG. 11 ends. Returning to FIG. 2, in step S209, the camera control unit 104 displays the generated peaking image on the display unit 119 (display step). That is, the camera control unit 104 superimposes a colored mask edge image (specific display) on the captured image and displays it on the display unit 119. Thereby, a series of processes related to peaking are completed, and when the process returns to step S202, the process for the next imaging frame is performed.

[0062] Here, an additional explanation will be given for the base frequency Fpx in the case of the second frame and subsequent frames that are not the first imaging frame. When the base frequency Fpx is the frequency Fp1 and the processing of the first frame is completed, in the next second frame, the image frequency extraction result for the entire image is in a state of being available. Therefore, if the base frequency Fpx is the frequency Fp1, when extracting the base frequency Fpx such as in step S404, step S408, and step S417 (to be described later), the previously obtained image frequency can be actively utilized. For example, if the edge detection count is small when the base frequency Fpx is set to the frequency Fp1, it is possible to change the base frequency Fpx to the frequency Fp2 on the lower frequency side than the frequency Fp1. Conversely, if the edge detection count is large when the base frequency Fpx is set to the frequency Fp2, it is possible to change the base frequency Fpx to the frequency Fp1 on the higher frequency side than the frequency Fp2. Such processing corresponds to events (10) and (11) based on the result of image contrast evaluation in the state transition table of FIG. 6.

[0063] In the first use case, after a movement in the Z direction occurs in the camera, a timeout occurs. Therefore, hereinafter, the case where, after the above-described processing is continuously performed due to the presence of a movement in the Z direction in the camera over a plurality of frames, the movement of the camera stops (that is, when it enters the post-movement re-stopped state) will be described. The processes of step S202 and step S203 are the same as those in the case where there is a movement in the Z direction in the camera. In step S403, which constitutes the movement detection and setting process of the camera in step S204, the camera control unit 104 advances the process to step S413 because the movement of the camera changes from being present to absent.

[0064] In step S413, the camera control unit 104 calculates the peaking detection frequency PeakFq. The camera control unit 104 calculates the peaking detection frequency PeakFq using the formula PeakFq = Fpx × β, where the base frequency Fpx and the coefficient β (second weight) held in advance are used. The base frequency Fpx is the base frequency Fpx (frequency Fp1 in this first use case) that was used to calculate the peaking detection frequency PeakFq when the camera was moving immediately before. Here, the coefficient β is 1.

[0065] In step S414, the camera control unit 104 increments a re - stop counter for time - managing the transition of the camera from the moving state to the stopped state. In step S415, the camera control unit 104 determines whether the counter value of the re - stop counter is equal to or greater than a predetermined time. When the camera control unit 104 determines that the counter value of the re - stop counter is equal to or greater than a predetermined time, that is, when the camera has timed out and transitioned from the post - movement re - stop state to the stopped state, the process proceeds to step S416. On the other hand, when the camera control unit 104 determines that the counter value of the re - stop counter is less than a predetermined time, the process proceeds to step S406.

[0066] In step S416, the camera control unit 104 sets the movement state Cs of the camera to the stopped state. Then, the process proceeds to step S406. In step S406, the camera control unit 104 resets the peaking detection frequency PeakFq calculated in step S413 to the peaking detection function of the image processing control unit 125. As a result, when the movement of the camera stops (i.e., when it enters the post - movement re - stop state), the base frequency Fpx is reset to the peaking detection frequency PeakFq. Then, the camera movement detection and setting process shown in the flowchart of FIG. 4 ends.

[0067] Next, each image when the camera control unit 104 operates based on each of the above-described flowcharts will be described with reference to FIG. 12. FIG. 12 is a diagram showing a scene where there are two people in front and behind the subject facing the camera, and the camera is taking pictures while moving on a dolly. In FIG. 12, the camera moves on a dolly as time indicated by (A) to (D) on the horizontal axis elapses. Further, in FIG. 12, as the camera moves, the size of the subject entering the imaging angle of view of the camera changes, and the subject frequency changes. In FIG. 12, the camera is set to manual focus, there is no focus lens drive, and the change in the focus subject occurs when the camera moves closer to the two subjects.

[0068] At (A) on the horizontal axis, the camera is in a stationary state. At that time, the focus is adjusted to the subject on the front side. At (B) and (C) on the horizontal axis, the camera is in a state of approaching the two subjects. At (D) on the horizontal axis, the camera is in a state of having stopped after approaching until the focus is adjusted to the subject on the back side. In FIG. 12, the imaging images at each of the times (A) to (D) on the horizontal axis are shown at (1) on the vertical axis. The peak detection frequencies at each of the times (A) to (D) on the horizontal axis are shown at (2) on the vertical axis by the base frequency and coefficients. In addition to the frequency Fp1 for high-frequency setting and the Z movement coefficient α1 which is a coefficient during movement in the Z direction, there is a coefficient β for the base frequency and coefficients. Note that the Z movement speed coefficient corresponding to the Z movement speed Vz is omitted at (2) on the vertical axis.

[0069] On the vertical axis (3), edge images at times (A) to (D) on the horizontal axis are shown. That is, on the vertical axis (3), edge images at the respective peaking detection frequencies on the vertical axis (2) are shown. On the vertical axis (4), as a conventional example, a fixed peaking detection frequency is shown. Note that in the vertical axis (4), the peaking detection frequency is fixed at the frequency Fp1. On the vertical axis (5), as a conventional example, edge images at times (A) to (D) on the horizontal axis are shown when the peaking detection frequency is fixed at the frequency Fp1 shown on the vertical axis (4). On the vertical axis (6), defocus images at times (A) to (D) on the horizontal axis are shown. That is, on the vertical axis (6), defocus images at the respective peaking detection frequencies on the vertical axis (2) are shown. On the vertical axis (7), color-coded mask edge images at times (A) to (D) on the horizontal axis are shown. In the vertical axis (7), each color-coded mask edge image is generated from the edge image on the vertical axis (3) and the defocus image on the vertical axis (6). Note that hereinafter, among the respective images described in FIG. 12, for example, when showing an image with the horizontal axis being (A) and the vertical axis being (1), it is denoted as "(A)-(1).

[0070] In the captured image of (A)-(1), 1201 is the imaging angle of view. This is the same for each image other than (A)-(1), although no reference numeral is attached. 1202 is a person who is close to the camera, that is, a subject on the front side (hereinafter referred to as "the first person"). 1203 is a person who is slightly farther from the camera, that is, a subject on the back side (hereinafter referred to as "the second person"). In the captured image of (A)-(1), the first person 1202 is in a focused state. Also, the second person 1203 is in a defocused state. At (B) on the horizontal axis, as described above, the camera is moved closer to each of the two subjects. Therefore, in the captured image of (B)-(1), the first person 1204 is in a state where it has started to become slightly blurred and the sharpness has decreased. Also, the second person 1205 is in a state where the sharpness has slightly increased as it approaches focus and the blurring has been slightly reduced.

[0071] On the horizontal axis (C), as described above, the camera is in a state where it has been moved closer to each of the two subjects than the horizontal axis (B). Therefore, in the captured image of (C)-(1), the first person 1206 is slightly blurred and in a state where the sharpness has further decreased. Also, the second person 1207 is in a state where the sharpness is further increasing and the blurring is being resolved as it approaches focus more closely than the horizontal axis (B). On the horizontal axis (D), as described above, the camera is in a state where it has been moved closer to each of the two subjects than the horizontal axis (C) and then stopped. Therefore, in the captured image of (D)-(1), the first person 1208 is completely blurred. Also, the second person 1209 is in a state of being in focus. Note that hereinafter, both the first person and the second person will be described assuming that the face part has a higher sharpness than the body and limb parts.

[0072] (In the edge image of (A)-(3), since the peaking detection frequency is the frequency Fp1 for high-frequency setting, both the face part and the body and limb parts of the first person on the front side, which is in a focused state, are edge-detected. In the edge image of (B)-(3), since the peaking detection frequency is set low by multiplying the frequency Fp1 by the Z movement coefficient α1, although the first person on the front side is starting to get slightly blurred, it is edge-detected, and the face part of the second person on the back side is also edge-detected.)

[0073] (In the edge image of (C)-(3), the first person on the front side is slightly blurred, and the body and limb parts are not edge-detected, only the face part is edge-detected. Also, the second person on the back side has increased sharpness as it approaches focus more closely, and both the face part and the body and limb parts are edge-detected. In the edge image of (D)-(3), the peaking detection frequency has returned to the frequency Fp1 for high-frequency setting by multiplying the frequency Fp1 by the coefficient β1. The first person on the front side is completely blurred and has a low subject frequency, so it is not edge-detected. On the other hand, the second person on the back side is in focus and has a high subject frequency, so both the face part and the body and limb parts are edge-detected.)

[0074] In the defocused image of (A)-(6), the first person 1210 is in focus. In this case, the area of the first person 1210 is classified as being in the focused state. Here, the area in the focused state is shown in black. Furthermore, the area in the focused state is simply represented as matching the subject shape. However, since the focus detection frame is rectangular, the actual area in the focused state is detected as an aggregate of rectangles. Also, the second person 1211 is in a medium defocus state slightly away from the focused state. Here, the area in the medium defocus state is shown by hatching. In the defocused image of (B)-(6), the first person 1212 is starting to blur slightly and is in a small defocus state. Here, the small defocus state is shown by a grid. Also, the second person 1213 is closer to the focused state compared to the above-mentioned second person 1211, but is still in a medium defocus state. Therefore, the area in the medium defocus state is shown by hatching.

[0075] In the defocused image of (C)-(6), the first person 1214 is more blurred compared to the above-mentioned first person 1212, but is still in a small defocus state. Therefore, the small defocus state is shown by a grid. Also, the second person 1215 is closer to the focused state compared to the above-mentioned second person 1213 and is in a small defocus state. Therefore, the small defocus state is shown by a grid. In the defocused image of (D)-(6), the first person 1216 is even more blurred compared to the above-mentioned first person 1214 and is in a medium defocus state. Therefore, the area in the medium defocus state is shown by hatching. Also, the second person 1217 is in the focused state. Therefore, the area in the focused state is shown in black.

[0076] The color-coded mask edge image of (7) on the vertical axis is generated from the edge image of (3) on the vertical axis and the defocused image of (6) on the vertical axis. In the color-coded mask edge image of (A)-(7), the first person 1218 has edges and is in a focused state. Here, the edges in the focused state are generated in blue. In the color-coded mask edge image of (B)-(7), there is a focus position between the first person 1219 and the second person 1220. Therefore, the first person 1219 is in a state where it will become more blurred as the camera moves further in the future. Here, the edges in the state where it will become more blurred are generated in yellow. On the other hand, the second person 1220 is in a slightly blurred state but is in the direction of camera movement, that is, the direction of movement of the focus position. Here, the edges in the direction of movement of the focus position and in a slightly blurred state are generated in red.

[0077] (C) - In the color-coded mask edge image of (7), the first person 1221 is in a slightly blurred state. Here, the edges in the slightly blurred state are also generated in yellow. On the other hand, the second person 1222 is in the direction of camera movement, that is, the direction of movement of the focus position, and is about to enter the focused state. Here, the edges in the state of about to enter the focused state are generated in green. In the color-coded mask edge image of (D)-(7), the second person 1223 has edges and is in a focused state. Therefore, the edges in the focused state are generated in blue. Although not shown in the figure, when the color-coded mask edge image of (7) on the vertical axis is superimposed on the captured image, the captured image becomes an image that peaks by adding color to some of its edges. In the example shown in Fig. 12, it can convey to the user that at the stage of (C) on the horizontal axis, the first person on the near side starts to become slightly blurred and the second person on the far side is about to enter the focused state, which can draw attention to how the camera is moved. On the other hand, in the edge image of (C)-(5) of the conventional example, only a part of the edges of the second person on the far side is shown, and it can be seen that there is little edge information including the blurred area.

[0078] As described above, in the video camera 100, even if the user moves during camera work using a dolly or the like, it is possible to detect the edges of a subject with deteriorated sharpness, so the convenience of the focus assist function has been improved.

[0079] <Second Use Case> In the above first use case, the camera that was not moving in peaking mode was activated, the high-frequency mode was turned on, and after the camera movement occurred in the Z direction, it timed out. In the following, until the camera is activated, it is the same as the first use case. A second use case will be described in which the camera moves in the Z direction and the XY direction, and several seconds later, the camera stops in both the Z direction and the XY direction. The description of the second use case will focus on the differences and supplements from the first use case.

[0080] In the second use case, in the flowchart shown in FIG. 2, each process from step S201 to step S203 is the same as that in the first use case, but the camera movement detection and setting process in step S204 is different from that in the first use case. Specifically, in step S412 that constitutes the camera movement detection and setting process in step S204, the camera control unit 104 determines that there is movement of the camera in the XY direction, so the process proceeds to step S417. In step S417, the camera control unit 104 extracts the XY movement coefficient α2 (first weight), which is the coefficient during movement in the XY direction, the base frequency Fpx according to the mode, and the XY movement speed Vxy, which is the speed during movement in the XY direction. In the second use case, as the base frequency Fpx, the frequency Fp1 of the high-frequency mode is extracted. Note that the XY movement coefficient α2 is less than 1 and is held in advance.

[0081] In step S418, the camera control unit 104 calculates an XY movement speed coefficient according to the XY movement speed Vxy. The XY movement speed coefficient (speed coefficient) is a coefficient for calculation to lower the peaking detection frequency PeakFq because the faster the speed of the camera in the XY direction, the more blurred the acquired captured image becomes and the image becomes a low-frequency image. The XY movement speed coefficient is calculated in consideration of the XY movement speed Vxy based on a coefficient obtained experimentally in advance. Note that the XY movement speed coefficient is set to be less than 1.

[0082] In step S419, the camera control unit 104 calculates the peaking detection frequency PeakFq using the formula PeakFq = Fpx × α2 × XY movement speed coefficient. Thus, in the second use case, the peaking detection frequency PeakFq is calculated using the formula Fp1 × α2 × XY movement speed coefficient. Note that if the XY movement coefficient α2 is large, peaking detection cannot be performed on the captured image. Therefore, by setting the XY movement coefficient α2 to be large, it is also possible to prevent peaking detection when there is movement in the XY direction. As a result, when there is movement in the Z direction of the camera, the camera control unit 104 can display a peaking image in which the colored mask edge image and the captured image are combined on the display unit 119, and when there is movement in the XY direction of the camera, the camera control unit 104 can display only the captured image on the display unit 119.

[0083] In step S420, the camera control unit 104 holds the XY movement information indicating that the movement of the camera is in the XY direction as an XY movement flag so that it can be used later. In step S421, the camera control unit 104 branches the process according to whether there is movement of the camera in the Z direction and the XY direction. When there is no movement of the camera in the Z direction or the XY direction, the camera control unit 104 advances the process to step S406. On the other hand, when there is movement of the camera in the Z direction and the XY direction, the camera control unit 104 advances the process to step S422. In step S422, the camera control unit 104 recalculates the peaking detection frequency PeakFq by the formula PeakFq = Fpx × α3. This formula is composed of the base frequency Fpx and the coefficient α3 (first weight) used when there is movement of the camera in the Z direction and the XY direction. The coefficient α3 is designed not by synthesizing the Z movement coefficient α1 and the XY movement coefficient α2 by simple multiplication or the like, but mainly by calculating with weights attached to the data in the Z direction.

[0084] In step S406, the camera control unit 104 resets the peaking detection frequency PeakFq calculated in step S419 or step S422 to the peaking detection function of the image processing control unit 125. Thereafter, the camera movement detection and setting process shown in the flowchart of FIG. 4 ends. When returning to FIG. 2, the process advances to step S205, but each process from step S205 to step S209 thereafter has no difference depending on the presence or absence of camera movement. In this way, even when the camera moves in the XY direction, by changing the coefficient multiplied by the base frequency Fpx, the peaking detection frequency PeakFq suitable for the XY direction movement of the camera is set.

[0085] As described above, in the video camera 100, also in the second use case, the convenience of the focus assist function is improved in the same manner as in the first use case.

[0086] <Modification example> As described above, the preferred embodiments of the present invention have been explained. However, the present invention is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist thereof. For example, in the present embodiment, in step S904, the camera control unit 104 generates a plurality of mask images classified into four states: in-focus, slightly out of focus, out of focus, and highly out of focus. In this regard, the camera control unit 104 may generate a plurality of mask images in which the in-focus state and the defocus state are classified into three or five or more states.

[0087] Further, as the movement of the camera in the Z direction becomes faster, the camera control unit 104 may increase the width of the defocus amount that determines the in-focus state, that is, the width of the defocus amount indicating MaskDefImage0 (in FIG. 10, -1Fδ to +1Fδ). Similarly, as the movement of the camera in the Z direction becomes faster, the camera control unit 104 may increase the defocus amount that determines the defocus state, for example, the width of the defocus amount indicating MaskDefImage1 (in FIG. 10, -1Fδ to -3Fδ). This also applies to MaskDefImage2 to MaskDefImage5.

[0088] Further, the camera control unit 104 may change the defocus amount that determines the in-focus state or the defocus state according to the direction of the movement of the camera in the Z direction, that is, according to whether the camera approaches or moves away from the subject. Specifically, assume a case where the focus position is in front of the subject when the camera with manual focus setting approaches the subject. In this case, if the camera control unit 104 provides more defocus amounts on the - sign side of the defocus amount that determines the defocus state, the state of the subject coming into focus can be displayed with fine color separation. On the other hand, if the camera control unit 104 provides more defocus amounts on the + sign side of the defocus amount that determines the defocus state, the state of the subject going out of focus can be displayed with fine color separation.

[0089] Also, assume a case where, when a camera with manual focus setting moves away from a subject, the focus position is behind the subject. In this case, if the camera control unit 104 provides a larger defocus amount that determines the defocus state on the + sign side, the state of the subject coming into focus can be finely displayed in color-coding. On the other hand, if the camera control unit 104 provides a larger defocus amount that determines the defocus state on the - sign side, the state of the subject becoming out of focus can be displayed in fine color-coding.

[0090] Also, in step S410, the camera control unit 104 may set a frequency on the lower frequency side or the higher frequency side than the base frequency Fpx as the peaking detection frequency PeakFq according to whether the movement of the camera approaches or moves away from the subject. Specifically, when the movement of the camera approaches the subject, for example, by making the Z movement coefficient α1 smaller than 1, a frequency on the lower frequency side than the base frequency Fpx is set as the peaking detection frequency PeakFq. In this case, even if the subject frequency decreases as the camera approaches, the subject can be edge-detected. Conversely, when the movement of the camera moves away from the subject, for example, by making the Z movement coefficient α1 larger than 1, a frequency on the higher frequency side than the base frequency Fpx is set as the peaking detection frequency PeakFq. In this case, even if the subject frequency increases as the camera moves away, the subject can be edge-detected.

[0091] Also, the camera control unit 104 may blink the color-coded display in the peaking image. In this embodiment, the case where the present invention is applied to the video camera 100 has been described as an example, but the present invention is not limited to this example, and the present invention may be applied to a digital camera. Further, the present invention is applicable even to a camera without manual focus setting, and is also applicable to any electronic device having a camera function, such as a mobile phone with a camera function or a computer with a camera.

[0092] The present invention can also be realized by supplying a program that implements one or more functions of the above-described embodiments to a system or apparatus via a network or a storage medium, and causing one or more processors of a computer of the system or apparatus to read and execute the program. Further, the present invention can also be realized by a circuit (for example, an ASIC) that implements one or more functions.

[0093] The disclosure of the present embodiment includes the following configurations, methods, and programs. (Configuration 1) An imaging device that acquires an image with an imaging optical system, setting means for setting a detection frequency, display means for superimposing a specific display on the image when the frequency extracted from the image exceeds the detection frequency, and the setting means sets a base frequency as the detection frequency when the imaging device is not moving, and sets a frequency obtained by multiplying the base frequency by a first weight as the detection frequency when the imaging device is moving. An imaging device characterized by this. (Configuration 2) The imaging device according to Configuration 1, wherein the setting means sets, as the base frequency, a frequency obtained from the highest frequency component extracted from the image when the imaging device is not moving. (Configuration 3) The imaging device according to Configuration 1 or 2, wherein the setting means sets the base frequency as the detection frequency when the movement of the imaging device stops. (Configuration 4) Focus detection means for detecting a focused state or a defocused state of the imaging optical system, extraction means for extracting a high-frequency component from a region detected to be in a focused state by the focus detection means, and the setting means sets, as the base frequency, a frequency obtained from the high-frequency component extracted by the extraction means. The imaging device according to claim 1, characterized by this. (Configuration 5) The imaging device according to Configuration 4, wherein the setting means uses the same base frequency as when the imaging device is not moving when the movement of the imaging device stops. (Configuration 6) The imaging device according to Configuration 4 or 5, characterized by comprising classification means for classifying the in-focus state and the defocus state detected by the focus detection means into at least three states. (Configuration 7) The imaging device according to Configuration 6, characterized in that the display means performs superimposition of the specific display with color separation according to the state classified by the classification means. (Configuration 8) The imaging device according to Configuration 6 or 7, characterized in that the classification means changes the magnitude of the defocus amount that determines the in-focus state or the defocus state according to the speed of movement of the imaging device in the distance direction from the subject of the image. (Configuration 9) The imaging device according to any one of Configurations 6 to 8, characterized in that the classification means changes the defocus amount that determines the in-focus state or the defocus state according to the direction of movement of the imaging device in the distance direction from the subject of the image. (Configuration 10) The imaging device according to any one of Configurations 1 to 9, characterized in that the display means performs superimposition of the specific display when there is movement of the imaging device in the distance direction between the subject of the image and the imaging device, and does not perform superimposition of the specific display when there is movement of the imaging device in a direction orthogonal to the distance direction. (Configuration 11) The imaging device according to any one of Configurations 1 to 10, characterized by comprising movement detection means for detecting the movement of the imaging device as frequency variation of the image. (Configuration 12) The imaging device according to any one of Configurations 1 to 11, characterized in that when there is movement of the imaging device, the setting means sets, as the detection frequency, a frequency obtained by multiplying the base frequency by the first weight and a speed coefficient corresponding to the speed of the imaging device. (Configuration 13) The imaging device according to Configuration 1, characterized in that when the movement of the imaging device stops, the setting means sets, as the detection frequency, a frequency obtained by multiplying the base frequency by the second weight. (Configuration 14) The imaging device according to Configuration 13, characterized in that the second weight is 1. Configuration 15: The setting means is configured to set a frequency on the lower frequency side or the higher frequency side than the base frequency as the detection frequency according to whether the movement of the imaging device approaches or moves away from the subject in the image. The imaging device according to any one of Configurations 1 to 14. Method 1: A control method for an imaging device that acquires an image with an imaging optical system, A setting step of setting a detection frequency, A display step of superimposing a specific display on the image when the frequency extracted from the image exceeds the detection frequency. The control method for an imaging device includes: In the setting step, when there is no movement of the imaging device, the base frequency is set as the detection frequency, and when there is movement of the imaging device, the frequency obtained by multiplying the base frequency by a first weight is set as the detection frequency. The control method for an imaging device is characterized by this. Program 1: A program for causing a computer to execute each means of the imaging device according to any one of Configurations 1 to 15.

Description of Reference Numerals

[0094] 100 Video camera (imaging device) 104 Camera control unit (setting means) (display means) PeakFq Peak detection frequency (detection frequency) Fpx Base frequency α1 Z movement coefficient (first weight) α2 XY movement coefficient (first weight) α3 Coefficient (first weight)

Claims

1. An imaging device that acquires an image with an imaging optical system, comprising: setting means for setting a detection frequency; display means for superimposing a specific display on the image when the frequency extracted from the image exceeds the detection frequency, wherein the setting means sets a base frequency as the detection frequency when the imaging device is not moving, and sets a frequency obtained by multiplying the base frequency by a first weight as the detection frequency when the imaging device is moving. The imaging device is characterized by this.

2. The imaging device according to claim 1, wherein the setting means sets, as the base frequency, a frequency obtained from the highest frequency component extracted from the image when the imaging device is not moving.

3. The imaging device according to claim 1, wherein the setting means sets the base frequency as the detection frequency when the movement of the imaging device stops.

4. focus detection means for detecting a focused state or a defocused state of the imaging optical system; extraction means for extracting a high-frequency component from a region detected as being in a focused state by the focus detection means, wherein the setting means sets, as the base frequency, a frequency obtained from the high-frequency component extracted by the extraction means. The imaging device according to claim 1 is characterized by this.

5. The imaging device according to claim 4, wherein the setting means uses the same base frequency as when the imaging device is not moving when the movement of the imaging device stops.

6. The imaging device according to claim 4, further comprising classification means for classifying the focused state and the defocused state detected by the focus detection means into at least three states.

7. The imaging device according to claim 6, wherein the display means superimposes the specific display with color separation according to the state classified by the classification means.

8. The imaging device according to claim 6, wherein the classification means changes the width of the defocus amount that determines the focused state or the defocused state according to the speed of movement of the imaging device in the distance direction from the subject of the image.

9. The imaging device according to claim 6, wherein the classification means changes the defocus amount that determines the focused state or the defocused state according to the direction of movement of the imaging device in the distance direction from the subject of the image.

10. The imaging device according to claim 1, wherein the display means performs superimposition of the specific display when there is movement of the imaging device in the distance direction between the subject of the image and the imaging device, and does not perform superimposition of the specific display when there is movement of the imaging device in a direction orthogonal to the distance direction.

11. The imaging device according to claim 1, further comprising movement detection means for detecting movement of the imaging device as frequency variation of the image.

12. The imaging device according to claim 1, wherein when there is movement of the imaging device, the setting means sets, as the detection frequency, a frequency obtained by multiplying the base frequency by the first weight and a speed coefficient corresponding to the speed of the imaging device.

13. The imaging device according to claim 1, wherein when the movement of the imaging device stops, the setting means sets, as the detection frequency, a frequency obtained by multiplying the base frequency by a second weight.

14. The imaging device according to claim 13, wherein the second weight is 1.

15. The imaging device according to claim 1, wherein when the imaging device moves closer to or away from the subject of the image, the setting means sets, as the detection frequency, a frequency on the lower frequency side or the higher frequency side than the base frequency.

16. A control method for an imaging device that acquires an image with an imaging optical system, the method comprising: a setting step of setting a detection frequency; and a display step of superimposing a specific display on the image when a frequency extracted from the image exceeds the detection frequency, wherein in the setting step, when there is no movement of the imaging device, the base frequency is set as the detection frequency, and when there is movement of the imaging device, a frequency obtained by multiplying the base frequency by a first weight is set as the detection frequency.

17. A program for causing a computer to execute each means of the imaging device according to Configuration 1.

Images