Image processing device, imaging apparatus, control method for image processing device, and program

Abstract

To provide an image processing device that makes it possible for a photographer to facilitate checking a subject shake.SOLUTION: The image processing device includes a subject detection unit 306 for detecting a subject of an image, a subject distance detection unit 305 for detecting distance information indicating the distance to the subject, a subject shake detection unit 302 for detecting a subject shake, and a reporting plane generation unit 303 and an image superimposition unit 304 for generating a shake reporting image for reporting a subject shake of a specific subject on the basis of a detected subject shake and distance information.SELECTED DRAWING: Figure 7

Description

Technical Field

[0001] The present invention relates to a technique for notifying a subject blur occurring in an imaged image.

Background Art

[0002] In order to capture an image with a camera without causing subject blur, the following two points are important. The first point is to synchronize the camera with the movement of the subject and make it follow. At this time, if a relative deviation occurs between the subject and the camera, the deviation will appear as subject blur in the image. The second point is the setting of an appropriate shutter speed. Similar to camera shake, as the shutter speed becomes longer with respect to subject blur, the influence of subject blur on the captured image is more likely to be显著 shown (it is highly possible that subject blur can be visually confirmed by the human eye). Patent Document 1 discloses a technique for visualizing subject blur for a photographer by notifying subject blur on an electronic viewfinder or a rear liquid crystal screen of an imaging device.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1, as a means for detecting subject blur, subject blur is detected using motion vector detection means that detects the motion between frames of an image as a vector value. Depending on the subject size captured by the imaging device and the detection frame size of the motion vector, there may be a subject and something other than the subject in the vector frame, and if vectors other than the subject are detected, the edge of the subject cannot be accurately extracted. Therefore, the edges of regions different from the subject may also overlap, which may reduce the quality.

[0005] The present invention aims to provide an image processing device that generates images that allow the photographer to easily check for subject blur. [Means for solving the problem]

[0006] To solve the above problems, the image processing apparatus of the present invention comprises: subject detection means for detecting an image subject; distance detection means for detecting distance information indicating the distance to the subject; blur detection means for detecting subject blur; notification image generation means for generating a blur notification image for notifying subject blur of a specific subject based on the detected subject blur and the distance information; and display means for displaying the blur notification image. [Effects of the Invention]

[0007] According to the present invention, it is possible to generate images that allow the photographer to easily check for subject blur. [Brief explanation of the drawing]

[0008] [Figure 1] This is a block diagram of an imaging device. [Figure 2] This is a flowchart showing the imaging process. [Figure 3] This figure shows an example of the configuration of the blur notification image generation unit in the first embodiment. [Figure 4] This figure shows the processing flow for generating blur notification images in the first embodiment. [Figure 5] This is a diagram explaining how to crop a subject. [Figure 6] This diagram explains the changes in the display's colors. [Figure 7] This diagram shows the processing of the notification plane generation unit. [Figure 8] This figure shows an example of a blur-reporting image. [Figure 9] This figure shows an example of the configuration of the blur notification image generation unit in the second embodiment. [Figure 10] This figure shows the processing of the subject blur detection unit in the second embodiment. [Modes for carrying out the invention]

[0009] (First Embodiment) Figure 1 is a block diagram showing the imaging device in this embodiment. The imaging device 100 is, for example, a digital camera. In this embodiment, the imaging device 100 is described as an imaging device equipped with a lens device including an optical system 104, but it is not limited to this, and it may be an imaging device in which the lens device can be attached to and detached from a main body equipped with an imaging unit 105. Furthermore, the imaging device can be applied to any electronic device capable of processing captured images. These electronic devices may include, for example, information terminals such as mobile phones and tablet terminals.

[0010] The imaging device 100 comprises a control unit 101, a ROM 102, a RAM 103, an optical system 104, an imaging unit 105, an A / D conversion unit 106, an image processing unit 107, a recording unit 108, an angular velocity detection unit 111, a display unit 109, and an instruction input unit 110. The control unit 101 is, for example, a CPU (Central Processing Unit) and controls the entire imaging device 100 by controlling the operation of each block that comprises the imaging device 100.

[0011] ROM 102 is an electrically erasable and recordable non-volatile memory that stores the operation programs for each block of the imaging device 100, as well as parameters necessary for the operation of each block. RAM 103 is a rewritable volatile memory used for deploying programs executed by the control unit 101, etc., and for temporarily recording data generated by the operation of each block of the imaging device 100. The control unit 101 controls the imaging device 100 by reading the control programs for each block of the imaging device 100 from ROM 102, deploying them in RAM 103, and executing them.

[0012] The optical system 104 forms an image of the subject onto the imaging surface of the imaging unit 105. The optical system 104 has a lens group including a zoom lens, a focus lens, an image stabilization lens, etc. The imaging unit 105 acquires an image (image signal) by photoelectric conversion of the optical image formed on the imaging surface of the imaging unit 105 via the optical system 104. Specifically, the imaging unit 105 is an image sensor such as a CCD or CMOS sensor, and the optical system 104 photoelectrically converts the optical image formed on the imaging surface of the imaging unit 105 and outputs the obtained analog image signal to the A / D conversion unit 106. In this embodiment, an example is described in which the imaging unit 105 has an image sensor that outputs a signal capable of measuring distance using the imaging surface phase difference method. The A / D conversion unit 106 converts the analog image signal input from the imaging unit 105 into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily recorded in the RAM 103.

[0013] The image processing unit 107 performs various image processing operations on the image data recorded in the RAM 103. Specifically, the image processing unit 107 applies various image processing operations for developing, displaying, and recording digital image data, such as pixel defect correction processing caused by the optical system 104 or image sensor, demosaicing processing, white balance correction processing, color interpolation processing, and gamma processing. The image processing unit 107 also has a blur notification image generation unit 300. The blur notification image generation unit 300 generates a blur notification image to notify the photographer of blur of the subject in the image. The blur notification image is an image that notifies the photographer of motion blur occurring in the subject. In addition, the blur notification image may also be an image that notifies that motion blur does not occur in the subject. The blur notification image generation unit 300 generates a blur notification image by generating and superimposing an image plane on the image data output from the A / D conversion unit 106 or the image data recorded in the RAM 103, based on subject blur information and distance information, which allows for easy confirmation of subject blur. Furthermore, the blur notification image generation unit 300 calculates subject blur and detects subject edges in order to generate an image plane corresponding to subject blur. Note that the image processing performed by the image processing unit 107 is not limited to image data recorded in the RAM 103, but may also be performed on, for example, digital image data output from the A / D conversion unit 106 or image data recorded in the recording unit 108.

[0014] The recording unit 108 records data including image data onto a removable memory card or other recording medium, or outputs image data to an external device via an external interface. The recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103.

[0015] The display unit 109 includes a display device such as an LCD (Liquid Crystal Display), and displays the images recorded in the RAM 103 and the images recorded in the recording unit 108 on the display device. The display unit 109 also performs operations such as displaying an operation user interface (operation UI) for receiving instructions from the user. Further, the display unit 109 may have a plurality of display devices, such as an EVF (Electronic Viewfinder) and a rear monitor provided on the photographer side (rear). The display unit 109 may be capable of simultaneous output to the plurality of display devices, or may be configured to selectively display by switching. The instruction input unit 110 is an input interface including various physical operation members such as a touch panel and a shutter button, and receives input of instructions from the user. The rear monitor provided on the photographer side (rear) may be composed of a touch panel having the functions of both the display unit 109 and the instruction input unit 110. By associating the input coordinates and the display coordinates on the touch panel, an operation UI can be configured as if the photographer can directly operate the screen displayed on the touch panel.

[0016] Also, the imaging device 100 performs live view shooting in which the analog image signal sequentially output from the imaging unit 105 is displayed on the sequential display device via the A / D conversion unit 106, the RAM 103, the image processing unit 107, and the display unit 109 under the control of the control unit 101. During live view shooting, it is possible to adjust the composition for the actual shooting assuming recording on a recording medium or output to an external device, and to prepare for shooting by changing shooting parameters such as the exposure time (Tv value), aperture value (Av value), and ISO sensitivity at the time of actual shooting.

[0017] The angular velocity detection unit 111 detects the shake applied to the imaging device 100. The angular velocity detection unit 111 is, for example, an angular velocity sensor or the like, and detects the angular velocity in the pitch direction, yaw direction, and roll direction applied to the imaging device 100 due to hand shake or camera work. Note that the imaging device 100 may be realized as an image processing device that performs processing until an output signal is acquired from the imaging unit 105 and an image is displayed on the display unit 109 or recorded in the recording unit 108. The image processing device generates an image for notifying subject blur, which will be described later, in the image processing unit 107.

[0018] Next, the imaging process by the imaging device 100 in the present embodiment will be described in detail with reference to the flowchart of FIG. 2. FIG. 2 is a flowchart showing the imaging process. The processing of each step in this flowchart is realized by the control unit 101 instructing each part of the imaging device 100 according to a predetermined program read from the ROM 102 and executed. In the present embodiment, a case where live view shooting is performed in which the photographer performs imaging while checking a real-time image displayed on a rear monitor provided on the photographer side (rear surface) will be described as an example.

[0019] The imaging process starts when the photographer turns on the power of the imaging device 100. In step S201, the control unit 101 starts preparatory imaging in response to the power of the imaging device 100 being turned on. Specifically, the control unit 101 controls the optical system 104 and the imaging unit 105 to start live view shooting. During the live view period, the imaging unit 105 sequentially captures images (preparatory imaging images) and displays the acquired images on the display device of the display unit 109. The photographer can perform shooting preparations such as composition adjustment while checking the live view images sequentially displayed on the display device. Note that the processing of steps S202 to S206, which will be described later, is performed during the live view shooting period.

[0020] In step S202, the control unit 101 determines the shooting parameters for this image capture. The shooting parameters are parameters such as shutter speed, aperture value, and ISO sensitivity. The shooting parameters may be determined based on instructions from the user input via the instruction input unit 110, or the control unit 101 of the imaging device 100 may automatically set the shooting parameters.

[0021] In step S203, the control unit 101 determines whether motion blur notification is set to ON or OFF. The motion blur notification setting can be set by the user using the instruction input unit 110, or the imaging device 100 can automatically set it to ON or OFF based on predetermined shooting conditions. When the user sets the motion blur notification, the setting can be made using one physical operating element (button, bar, etc.) or one icon on the touch panel, and ON or OFF can be set at any timing during preparation shooting. It may also be possible to set the display of blur notification to be switched ON / OFF periodically. If the control unit 101 determines that the motion blur notification setting is ON, the process proceeds to step S204. On the other hand, if the control unit 101 determines that the motion blur notification setting is OFF, the process proceeds to step S205.

[0022] In step S204, the blur notification image generation unit 300 generates a blur notification image. The blur notification image is generated by superimposing a notification plane onto the preparation image that notifies the motion blur (or lack thereof) that occurs in the subject when captured with the shooting parameters used in the main image capture. Details of the process in step S204 will be described later with reference to Figure 4.

[0023] In step S205, the control unit 101 displays the blur notification image on the display device of the display unit 109 if a blur notification image was generated in step S204, or a preparatory image without the motion blur notification plane superimposed if motion blur notification is set to OFF. If motion blur notification is set to ON, the user can set the shooting parameters such as the shutter speed (exposure time) for the main image while checking the displayed blur notification image.

[0024] In step S206, the control unit 101 determines whether the shutter button on the instruction input unit 110 has been pressed by the user. Here, pressing the shutter button is an instruction to take an image. Therefore, if the shutter button input is configured to accept two-stage input methods, such as a half-press to instruct the preparation operation for taking an image and a full press to instruct the actual image, the control unit 101 determines whether a full press has been performed. If only a simple single-stage input is accepted, the control unit 101 determines whether that single-stage input has been performed. If the control unit 101 determines that the shutter button has not been pressed, it returns to step S202 and repeats the process from step S202 to step S206. This allows the user to easily check the motion blur that would occur in the subject if the actual image were taken with the currently set shooting parameters, even while preparation shooting is in progress. If motion blur is confirmed and it is not the motion blur the user prefers (motion blur is not desired), the user can simply reset the shooting parameters such as the shutter speed (exposure time) for the actual image without pressing the shutter button. On the other hand, if the control unit 101 determines that the shutter button has been pressed, it proceeds to step S207.

[0025] In step S207, the control unit 101 controls the optical system 104, imaging unit 105, etc., to perform actual imaging based on the set shooting parameters and acquire an image (actual image). In step S208, the control unit 101 outputs the actual image acquired by the actual imaging to the display unit 109 and the recording unit 108. The display unit 109 displays the actual image on a display device. The recording unit 108 records the actual image on a recording medium or outputs it to an external device.

[0026] Thus, in the imaging process of this embodiment, motion blur of the subject is notified to the user by displaying a blur notification image during the preparation shooting. This allows the user to set an appropriate exposure time for the main shoot while checking the blur notification image displayed on the display unit 109. Then, the user can take the shot (main image capture) when the exposure time has been set to appropriately address the motion blur.

[0027] Next, an example of the configuration of the blur notification image generation unit 300, which is part of the image processing unit 107, will be described with reference to Figure 3. Figure 3 is a diagram showing an example of the configuration of the blur notification image generation unit 300. The blur notification image generation unit 300 generates a blur notification image based on the prepared image and shooting conditions. The blur notification image generation unit 300 includes an image cropping unit 301, a subject blur detection unit 302, a notification plane generation unit 303, an image superposition unit 304, a subject distance detection unit 305, and a subject detection unit 306.

[0028] The subject distance detection unit 305 detects the distance to the subject as subject distance data. The subject detection unit 306 detects the subject in the image. Furthermore, the subject detection unit 306 can detect characteristic parts of the subject, such as the type of subject or organs. The image cropping unit 301 crops the subject detected by the subject detection unit 306 from the image. In this embodiment, the image cropping unit 301 crops the prepared image in units of characteristic parts of the subject detected by the subject detection unit 306. The subject blur detection unit 302 detects the subject blur of the subject detected by the subject detection unit 306. The notification plane generation unit 303 and the image superposition unit 304 are notification image generation means that generate a blur notification image for notifying subject blur. First, the notification plane generation unit 303 generates a notification plane according to the subject blur, and the image superposition unit 304 generates a blur notification image by superimposing the notification plane onto the image. The notification plane generation unit 303 generates a motion notification plane, which is data for notifying subject blur based on subject blur. This embodiment also makes it possible to notify only the subject blur of a specific subject, rather than all subjects. Therefore, the notification plane generation unit 303 of this embodiment determines a specific subject for which subject blur will be notified based on the distance information of the subject, and generates a notification plane for notifying the blur of that specific subject. The image superposition unit 304 generates a blur notification image by superimposing the motion blur notification plane onto the captured image.

[0029] Furthermore, one or more of the functional blocks shown in Figure 3 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. They may also be implemented by a combination of software and hardware. Therefore, even if different functional blocks are described as the primary operating components in the following explanation, the same hardware may be the primary implementing component.

[0030] Next, the process by which the blur notification image generation unit 300 generates a blur notification image, which is performed in step S204 of Figure 2, will be explained in detail with reference to the flowchart in Figure 4. Figure 4 is a flowchart of the blur notification image generation process. The processing of each step in this flowchart is realized by the control unit 101 issuing instructions to each part of the imaging device 100, such as the blur notification image generation unit 300, according to a predetermined program that is read from the ROM 102 and executed by the control unit 101.

[0031] In step S401, the control unit 101 inputs the preparatory images sequentially captured by the imaging unit 105, the shooting parameters to be used during the actual shooting determined in step S202, and the angular velocity data detected by the angular velocity detection unit 111 to the blur notification image generation unit 300. In step S402, the subject detection unit 306 of the blur notification image generation unit 300 detects the subject and acquires subject information. The subject detection unit 306 detects subject information by comparing the captured video with data that has been learned, for example, based on deep learning. Subject information here refers to the type of subject and its organs. The type of subject is, for example, a person, an animal (dog, cat, bird, horse, etc.), a vehicle (car, bicycle, motorcycle, formula car, etc.). The organs of the subject are, for example, if the type of subject is a person, the eyes, hands, feet, head, chest, etc. If the type of subject is an animal, the organs of the subject are the eyes, torso, limbs, etc. If the subject is a vehicle, the subject's components include the helmet, headlights, taillights, etc. Note that machine learning-based subject information detection is just one example; other methods may also be used to detect subject information.

[0032] In step S403, the subject blur detection unit 302 acquires the shooting conditions. Here, the shooting conditions acquired in step S403 are the exposure time (shutter speed) for the main shoot and the time interval between images in the preparation shoot (frame rate), which were determined in step S202 and acquired in step S401.

[0033] In step S404, the subject blur detection unit 302 acquires angular velocity data indicating the shaking of the imaging device 100 from the angular velocity detection unit 111. The angular velocity detection unit 111 may be an inertial sensor such as a gyro sensor, or it may calculate angular velocity data using a motion vector, a vector value outside the subject area from the difference between different preparatory imaging images, the focal length, and the frame rate.

[0034] In step S405, the subject distance detection unit 305 acquires distance information (distance data) to the subject. The subject distance detection unit 305 uses, for example, an image sensor with pixels for performing phase-difference distance measurement, which is capable of on-sensor phase-difference distance measurement and focus detection on the imaging surface of the imaging unit 105, to detect distance information from the defocus value at each pixel level. By using an image sensor capable of on-sensor phase-difference AF, the subject distance detection unit 305 can detect the amount of defocus based on the phase difference of the output image signals of the image sensor obtained from different regions of the exit pupil of the optical system 104. Alternatively, the subject distance detection unit 305 may also detect subject distance data by mounting distance measuring sensors such as a laser displacement sensor, an ultrasonic sensor, or a millimeter-wave radar inside the imaging device 100. The subject distance detection unit 305 outputs the acquired distance information to the subject to the subject blur detection unit 302.

[0035] In step S406, the image extraction unit 301 extracts an image of the subject from the prepared image based on the subject information detected by the subject detection unit 306 in step S402. The image extraction unit 301, for example, detects characteristic parts of the subject within the same subject and extracts an image for each part of the subject. The extraction of the subject image will be explained using Figure 5. Figure 5 is a diagram illustrating the extraction of a subject. The example shown in Figure 5 represents a panning shot in a motorsport scene as an example of a shooting scene, and there are multiple subjects (subject 501, subject 502, subject 503) in the shooting angle of view of the prepared image 500. The subject detection unit 306 generates an extracted image for each part (subject characteristic region) within each subject based on subject information such as the type of subject.

[0036] If there are multiple subjects, the main subject may be selected from among them. In this case, the image cropping unit 301 may automatically determine the main subject based on the detection result of the subject detection unit 306, or the subject at a position arbitrarily selected by the photographer may be prioritized as the main subject. Furthermore, when identifying the main subject, the image cropping unit 301 may use the distance information to the subject detected by the subject distance detection unit 305. Once the main subject is determined, the subject detection unit 306 detects characteristic parts of the main subject. For example, if subject 501 is the main subject, the characteristic parts are the person 501a, the helmet 501b, and the motorcycle 501c. The image cropping unit 301 crops the image in the respective regions (subject characteristic regions) of the person 501a, the helmet 501b, and the motorcycle 501c. The purpose of cropping the subject from the image here is to eliminate as much noise as possible and improve the accuracy of detecting subject blur, which will be described later. For example, in panning shots like those shown in Figure 5, or in portrait photography where the aperture is wide open to blur the background and make the subject stand out, the background is noticeably blurred. If this area is also included in the subject blur detection input, the accuracy of subject blur detection will decrease. In addition to the composition of the background, if another subject overlaps the main subject and each subject moves differently, the accuracy of subject blur detection will similarly decrease. Therefore, it is important to detect the main subject as accurately as possible.

[0037] Returning to the explanation of Figure 4, in step S407, the subject blur detection unit 302 detects subject blur from the cropped image input from the image cropping unit 301. In this embodiment, subject blur (amount of subject blur) is detected from the degree of blurring (clarity) of the subject's edges. This is because when subject blur occurs, the edges become blurry and indistinct, and when there is no subject blur, the edges become sharp and clear without blur. Therefore, if the edges are clear, it is determined that the subject blur is small, and if the edges are indistinct, it is determined that the subject blur is large. For example, a Laplacian filter is used to detect the clarity of the edges corresponding to subject blur. Furthermore, by using a Laplacian filter, it is possible to detect subject blur and the subject's edges simultaneously.

[0038] This section describes an example of detecting motion blur using a Laplacian filter. The Laplacian filter is a spatial filter that extracts edges from an image using the second derivative. Since the input image is digital data and discrete data, the derivative can be calculated using differences. The first derivatives Ix and Iy of the horizontal and vertical pixel values ​​of the Laplacian filter are given by equation (1).

number

[0039] The Laplacian filter is a second derivative. Therefore, by taking the difference of the first derivatives, the Laplacian filter can be expressed by equation (2).

number

number

[0040] The Laplacian ∇ shown in equation (3) 2 The kernel value, which indicates the strength (weighting) of the filter applied to I(x,y), is given by equation (4) for the 4-neighbor kernel value, and by equation (5) for the 8-neighbor kernel value.

number

number

[0041] The kernel value is the number of neighboring pixels of the pixel of interest. Equation (4) is the kernel value for 4 neighbors, showing that the second derivative is taken with respect to the four pixels above, below, left, and right of the pixel of interest. Equation (5) is the kernel value for 8 neighbors, showing that the second derivative is taken with respect to 8 pixels, including not only the top, bottom, left, and right directions but also the diagonal directions. In this embodiment, the kernel value is changed for each image extracted from the subject feature region, according to the type of subject (subject information) of the extracted image. For example, the kernel value is set to 4 neighbors for the helmet 501b and 8 neighbors for the motorcycle 501c. The number of neighboring pixels may also be greater than 8.

[0042] The clearer the edge, the greater the difference in pixel values ​​(luminance values) between the pixel of interest and neighboring pixels. Furthermore, a large difference in pixel values ​​(luminance values) between the pixel of interest and neighboring pixels results in a larger Laplacian value. Therefore, a larger Laplacian value indicates clearer edges and less motion blur, while a smaller Laplacian value indicates unclear edges and greater motion blur. In this way, the motion blur detection unit 302 determines the amount of motion blur based on the magnitude of the Laplacian value for each cropped image obtained using the set kernel value. The magnitude of the Laplacian value for each cropped image can be, for example, the average of the absolute values ​​of the Laplacian values ​​of each pixel in the cropped image, as long as it allows for the determination of edge clarity. While this embodiment describes an example using a Laplacian filter, it is not limited to this; any method for detecting motion blur according to edge clarity is acceptable.

[0043] Returning to the explanation of Figure 4, in step S408, the notification plane generation unit 303 generates a notification plane for notifying the motion blur of the subject in accordance with the subject blur detected in step S407. Here, an example of notifying the user of subject blur will be described. In this embodiment, peaking is performed to highlight the edges of the subject in the image displayed on the display device, and the magnitude of the subject blur is notified by changing the color of the peaking display according to the amount of subject blur. For example, if the subject blur is large enough to be visible on the image, the peaking display of the main subject is shown in red, and if the subject blur is small enough not to be visible on the image, the peaking display is shown in green. That is, if the Laplacian value is small and the edges of the subject are unclear, the peaking display is shown in red, and if the Laplacian value is large and the edges of the subject are clear, the peaking display is shown in green. Note that the number of color codings for the peaking display and the colors displayed are not limited to these.

[0044] Here, we will explain in more detail how the display color changes according to the amount of subject blur. Figure 6 is a diagram illustrating the change in display color. In Figure 6, the vertical axis represents the Laplacian value of the cropped image, and the horizontal axis represents the kernel value. Figure 6 shows the Laplacian values ​​when a Laplacian filter is applied with a kernel of 4 neighbors and 8 neighbors to the image cropped from the helmet 501b of subject 501 in Figure 5. In this way, the color displayed on the edges of the subject is changed to indicate subject blur according to the Laplacian value corresponding to the amount of subject blur. Even using the same image, the Laplacian value will differ depending on the kernel value. Therefore, the threshold for determining the color when visualizing the amount of subject blur is changed according to the kernel value.

[0045] For example, if a Laplacian filter is applied to the image size cropped by helmet 501b using a kernel value of 4 neighbors, the peaking display color will be green if the Laplacian value is 601 or greater, orange if it is less than 601 but 603 or greater, and red if it is less than 603. Similarly, if a Laplacian filter is applied using a kernel value of 8 neighbors, the peaking display color will be green if the Laplacian value is 602 or greater, orange if it is less than 602 but 604 or greater, and red if it is less than 604. In this way, the threshold for determining the displayed color is changed depending on the kernel value. In addition to the kernel value, the threshold for determining the display color may also be varied depending on the subject area cropped by the image cropping unit 301 and the image size. Furthermore, although two types of kernel values ​​are explained in Figure 6, the number may be increased to 16 neighbors, for example. Note that the method for notifying subject blur is not limited to changing the color of the peaking display. For example, the thickness of the peaking display edge lines may be changed, the lines may be made to blink, or the lines may be made to dashed lines. Alternatively, the system may be configured to display a notification of subject blur only when the amount of subject blur exceeds a predetermined threshold.

[0046] Next, the generation of a notification plane for notifying subject blur and the generation of a blur notification image by superimposing the generated notification plane onto the image will be explained in detail with reference to Figure 7. Figure 7 is a diagram showing the processing of the notification plane generation unit 303. Figure 7(A) is a diagram showing the arrangement of the distance detection frames 701. The distance detection frames 701 are detection frames for detecting the subject distance by the subject distance detection unit 305, and are arranged around approximately the entirety of the prepared image capture image 500, and have multiple detection frames (14 vertically × 24 horizontally in the example of Figure 7(A)). The main focus distance detection frames 703 corresponding to subject distances within a predetermined threshold range are shown semi-transparently, based on the subject distance in the main focus frame 702, which is the most in-focus frame among the distance detection frames 701. For example, the notification plane generation unit 303 identifies a subject that is within a depth of field within a threshold range (for example, ±2 depth) based on the depth of field of the main focus frame 702, using the depth of field of the main focus frame 702 as a reference, based on the distance information calculated by the subject distance detection unit 305. In this embodiment, an example was described in which the in-focus area is used as the reference area to determine a specific subject to be peaked based on distance information. However, this is not the only example; for instance, the area over which the AF frame (main distance measurement frame) is superimposed may be used as the reference area for a specific subject based on distance information.

[0047] Figure 7(B) shows an image to which the Laplacian filter of the subject blur detection unit 302 has been applied. The Laplacian filter is applied to the prepared image 500 to detect the edges of the subject. Figure 7(C) shows a notification plane. The notification plane is an image in which peaking displays such as color display corresponding to subject blur are applied to the edges of the subject within the main focus distance detection frame 703 that is in the best focus. Specifically, the notification plane generation unit 303 superimposes the main focus distance detection frame 703 (Figure 7(A)) detected based on the depth of field and the image of the subject edges detected by the Laplacian filter (Figure 7(B)), and applies peaking displays to the edges of the overlapping area. The color of the peaking display is determined according to the Laplacian value calculated for each cropped image, and if the cropped areas overlap, the color corresponding to the greater blur takes precedence. For example, if the motorcycle 501c is green, the person 501a is orange, and the helmet 501b is red, the edges within the helmet 501b area will be red.

[0048] Returning to the explanation of Figure 4, in step S409, the image superposition unit 304 superimposes the notification plane generated in step S406 onto the prepared image (Figure 7(C)) to generate a blur notification image. The image superposition unit 304 generates a blur notification image by superimposing the prepared image and the notification plane on, for example, RAM 103. By superimposing the prepared image and the notification plane, the main subject on the prepared image can be peaked and displayed according to the subject blur.

[0049] Figure 8 shows examples of blur-reporting images. Figures 8(A) and 8(B) feature motorsport (motorcycles) as the subject, while Figures 8(C) and 8(D) feature a ballerina as the subject. Examples of applying this embodiment are shown in Figures 8(A) and 8(C). In Figure 8(B), subjects 501 and 503 are detected as the same subject, and areas other than the main subject, subject 501, are also reported. On the other hand, according to this embodiment, it is possible to peak only a specific subject that is within a predetermined distance from the subject distance of the focus area that the photographer is trying to photograph. Therefore, according to this embodiment, as shown in Figure 8(A), an image is generated in which only the edge of the intended specific subject (subject 501) is peaked, and the photographer is notified of only the blur of subject 501. Furthermore, even if there is only one subject, as shown in Figure 8(D), there is a risk that the boundary line 802 between the stage and the back wall may be detected and peaked as part of the subject. On the other hand, according to this embodiment, as shown in Figure 8(C), only the subject 801 can be peaked and the photographer can be notified only of the motion blur of the subject 801.

[0050] In this embodiment, an example of live view shooting in which the image is displayed in real time on a display device provided on the photographer side of the imaging device 100 has been described, but it is not limited to this, and an image to notify subject blur may be displayed on another display device such as an EVF. Also, in the first embodiment, the process of acquiring angular velocity data (step S404) may not be performed.

[0051] As described above, this embodiment makes it possible to generate an image that allows the photographer to easily check for motion blur of the main subject they are trying to photograph. By checking this image, the photographer can easily set a shutter speed that does not cause motion blur, and thus obtain a high-quality image with suppressed motion blur in this shoot.

[0052] (Second Embodiment) In this embodiment, a different subject blur detection method from the first embodiment is used to detect subject blur and generate a blur notification image. In the first embodiment, subject blur was detected using a Laplacian filter on an image of the subject that had been cut out. On the other hand, in this embodiment, the main subject is identified based on motion vectors calculated from multiple images and subject blur is detected. Below, the detection method of the subject blur detection unit 302, which is the difference from the first embodiment, will be described, and descriptions of other common operations and processes will be omitted.

[0053] In the second embodiment, the image processing unit 107 has a blur notification image generation unit 900 instead of the blur notification image generation unit 300 of the first embodiment. Figure 9 shows the configuration of the blur notification image generation unit 900 in the second embodiment. The blur notification image generation unit 900 has a motion vector detection unit 901, a subject blur detection unit 902, a notification plane generation unit 303, an image superposition unit 304, and a subject distance detection unit 305. The operation and processing other than the motion vector detection unit 901, the subject blur detection unit 902, and the angular velocity detection unit 111 are the same as in the first embodiment, so their explanation is omitted.

[0054] The motion vector detection unit 901 detects motion vectors based on multiple preparatory captured images. Specifically, the motion vector detection unit 901 uses the multiple preparatory captured images as a reference frame and a reference frame, respectively, and performs a correlation calculation with each block of the target area in the reference frame using the reference block in the reference frame. As a result of the correlation calculation, the motion vector is calculated from the positional relationship between the block with the highest correlation and the reference block. The method for calculating the correlation value is not particularly limited and can be based on methods such as the sum of absolute differences, the sum of squared differences, or normal cross-correlation values. Furthermore, the method for calculating the motion vector itself is not limited to correlation calculation, and other methods such as the gradient method may be used. The motion vector detection unit 901 outputs the detected motion vector data (vector data) to the subject blur detection unit 902.

[0055] The subject blur detection unit 902 calculates subject vectors based on motion vectors and detects subject blur. Based on the image vector data, the angular velocity data of the imaging device 100, and the subject distance data, the subject blur detection unit 902 separates the vector data into vector data related to the subject and vector data not related to the subject. The vector data not related to the subject is, for example, the background vector data. The subject blur detection unit 902 then detects subject blur based on the vector data related to the subject. Image vector data (motion vectors) are input to the subject blur detection unit 902 from the motion vector detection unit 901, angular velocity data from the angular velocity detection unit 111, and subject distance data from the subject distance detection unit 305. Here, the method of separating the vector data will be explained in detail using Figure 10.

[0056] Figure 10 illustrates a method for separating vector data. Figure 10(A) illustrates motion vector detection. Figure 10(B) shows the frequency distribution of vector values. Figure 10(A) shows a panning scene of a motorsport as an example of filming, and there are multiple subjects (subject 1001, subject 1002, subject 1003) in the filming field of view. Motion vector detection frames 1004 are placed in the image, and for example, in each block within the motion vector detection frame 1004, the amount of movement compared to the previous frame is detected as a vector value. In Figure 10(B), which shows the frequency distribution, the horizontal axis shows the vector values ​​detected by the motion vector detection frame 1004, and the vertical axis shows the frequency. Note that if you want to detect the entire motion of the screen, it is desirable to cover the entire filming field of view with motion vector detection frames 1004, but if you want to detect the movement of only a part of the filming field of view (subject) as in this embodiment, it is desirable to arrange the detection frames in a small and dense arrangement. This is because, when the motion vector detection frame 1004 covers the entire field of view, the larger the motion vector detection frame 1004 is, the more computational processing becomes when the image sensor has a high pixel count or the frame rate cycle is fast. If the computational processing for motion vector detection becomes enormous, there is a risk of delay in the generation and display of motion blur notification images. In addition, if the computational processing for motion vector detection becomes enormous, power consumption will also increase, and the maximum number of shots that can be taken will decrease. For motion vector detection frames 1004 that do not cover the entire field of view, for example, the motion vector detection frame 1004 can be placed along the AF frame. This is because the AF frame is likely to be on the subject, and by placing the motion vector detection frame 1004 along the AF frame, the likelihood of the motion vector detection frame 1004 capturing the subject increases.

[0057] In Figure 10(B), the angular velocity data 1005 is the value obtained by converting the angular velocity data of the imaging device 100 detected by the angular velocity detection unit 111 into the amount of motion on the imaging surface. The subject detection unit 306 converts the angular velocity data into the amount of motion on the imaging surface using equation (6).

number

[0058] When shooting while moving the camera significantly (panning), as shown in Figure 10(A), the value obtained by converting the angular velocity data of the imaging device 100 into the amount of motion on the imaging plane coincides with the vector related to the background of the image on the imaging plane. In other words, the vector 1006 located near the angular velocity data 1005 converted to pixel units on the imaging plane corresponds to the background in the motion vector detection frame 1004. In reality, the angular velocity detection unit 111 experiences offset drift, where the reference value fluctuates due to disturbances such as temperature changes and shocks. Therefore, the subject blur detection unit 902 sets a predetermined range 1007 that takes into account the offset drift, with the angular velocity data 1005 converted to pixel units on the imaging plane as the starting point, and determines that the vectors within the predetermined range 1007 are background vectors. The range of the predetermined range 1007 based on the angular velocity data 1005 may be varied while observing the focal length and the amount of offset change.

[0059] The subject blur detection unit 902 then identifies vectors outside the predetermined range 1007 as subject candidates and determines that the vector closest to the predetermined conditions among the subject candidate vectors is the vector of the main subject. For example, the subject blur detection unit 902 determines that the vector with the highest frequency that is closest to 0 among the subject candidate vectors is the vector of the main subject. Alternatively, the subject blur detection unit 902 may preferentially determine that the vector frame at the position that the subject detection unit 306 has determined to be the main subject is the vector of the main subject among the subject candidate vectors.

[0060] Furthermore, the subject blur detection unit 902 may determine the vector frame to be adopted according to the subject detection results, for example, by using a subject detection function with deep learning capabilities, instead of using a frequency distribution. For example, in Figure 10(A), if the subject detection function detects subject 1001 as the main subject and also detects the position of the head, then the vector closest to the position of the head should be adopted.

[0061] Furthermore, as described in the first embodiment, subject distance data detected by the subject distance detection unit 305 may be used to detect the subject vector. For example, if the main focus frame (AF frame) is on the head of the subject 1001, the vector frame corresponding to the frame within a predetermined range (e.g., ±2 depth) based on the depth of field of the main focus frame is determined to be the subject, and the vector value of that vector frame is adopted as the subject vector. Alternatively, the subject vectors extracted from angular velocity data and subject distance data may be superimposed, and the region where both conditions overlap may be finally determined to be the subject vector.

[0062] The subject blur detection unit 902 outputs the detected subject blur to the notification plane generation unit 303. The notification plane generation unit 303 generates a notification plane corresponding to the subject blur detected by the subject blur detection unit 902. If the notification plane is a peaking display of the subject's edges, the notification plane generation unit 303 detects the subject's edges in the same manner as in the first embodiment. The notification plane generation unit 303 may also identify the area to notify of subject blur based on the subject's distance information, in the same manner as in the first embodiment. The image superimposition unit 304 then superimposes the notification plane onto the prepared image, in the same manner as in the first embodiment, to generate a blur notification image with edges highlighted according to the subject blur. The generated blur notification image is displayed on a display device, allowing the user to adjust settings such as shutter speed while checking the blur notification image.

[0063] As described above, this embodiment makes it possible to generate an image that allows the photographer to easily check for motion blur of the main subject they are trying to photograph. By checking this image, the photographer can easily set a shutter speed that does not cause motion blur, and thus obtain a high-quality image with suppressed motion blur in this shoot.

[0064] (Other embodiments) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions.

[0065] In the first and second embodiments described above, the magnitude of blur was determined based on the Laplacian value. However, to determine the magnitude of blur more accurately, the effect of blur reduction may be taken into consideration when determining the magnitude of blur. If the blur is small but the blur is large, the Laplacian value will be small. Therefore, if the Laplacian value is small, the blur and blur may be separated by analyzing the frequencies of blur and blur included in the cropped image, and a notification plane may be generated based on the amount of blur separated from the effect of blur. Frequency analysis can be performed, for example, by applying a point spreading function to the cropped image. Since blur has a specific frequency, while blur is a composite result of various frequencies, if a narrow peak appears at a specific frequency in the frequency space, the effect of blur can be considered large, and if a wide peak appears or if a specific peak does not appear, the effect of blur can be considered large. For example, as explained in Figure 6 and step S407, if the Laplacian value for displaying edges in red is calculated from the relationship with the threshold, and the frequency analysis determines that the effect of blur is large, the edges are displayed in red. On the other hand, even if the same Laplacian value is calculated, if the frequency analysis results indicate that the blur effect is significant, the Laplacian value is considered to have decreased due to the blur effect, and the blur is judged to be small, with the edges displayed in green.

[0066] Although preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of its gist. [Explanation of Symbols]

[0067] 100 Imaging device 101 Control Unit 102 ROM 103 RAM 105 Imaging Unit 107 Image Processing Unit 109 Display section 111 Angular velocity detection unit 300 Blur Notification Image Generation Unit 301 Image cropping section 302 Subject blur detection unit 303 Notification Plane Generation Unit 304 Image Overlay Area 306 Subject detection unit

Claims

1. A subject detection means for detecting a subject of an image; a distance detection means for detecting distance information indicating a distance to a subject; A blur detection means for detecting a subject blur; a notification image generating means for generating a blur notification image for notifying a user of the subject blur of a specific subject based on the detected subject blur and the distance information, 2. The image processing device according to claim 1, wherein the blur notification image changes color indicating subject blur according to the amount of subject blur.

2. 2. The image processing apparatus according to claim 1, wherein the notification image generating means determines a specific object for which a notification of object blur is to be given based on the distance information.

3. 3. The image processing device according to claim 1, wherein the specific subject is a subject at a subject distance within a predetermined range from the subject distance of the reference area, or a subject at a depth of field within a predetermined range from the depth of field corresponding to the subject distance of the reference area.

4. 4. The image processing device according to claim 3, wherein the reference area is an area corresponding to a main ranging frame or a focusing area.

5. further comprising an image cutout means for cutting out an image for each characteristic region of the subject based on the subject information; 5. The image processing apparatus according to claim 1, wherein the blur detection means detects subject blur for each cut-out image and changes a color indicating subject blur according to an amount of subject blur.

6. 6. The image processing apparatus according to claim 5, wherein the blur detection means detects subject blur based on clarity of an edge of the subject.

7. the blur detection means detects both the edge of the subject and the amount of subject blur simultaneously using a Laplacian filter; 7. The image processing apparatus according to claim 5, wherein in the blur notification image, a color indicating subject blur is changed according to a Laplacian value calculated using the Laplacian filter.

8. 8. The image processing device according to claim 7, wherein the blur detection means changes a kernel value of the Laplacian filter for each cut-out image in accordance with the subject information, thereby changing a threshold value for changing a color indicating subject blur in the blur notification image.

9. 5. The image processing apparatus according to claim 1, wherein the blur detection means detects subject blur based on a motion vector detected from a plurality of images.

10. 11. The image processing apparatus according to claim 1, wherein the distance detection means detects distance information based on distance measurement using a phase difference method.

11. an imaging element having pixels for performing distance measurement using a phase difference method; An image processing device according to any one of claims 1 to 10, and a display unit for displaying the blur alert image generated by the image processing device.

12. A control method for an image processing device, comprising: detecting a subject of the image; detecting distance information indicating a distance to a subject; Detecting subject blur; generating a blur notification image for notifying a user of the subject blur of a specific subject based on the detected subject blur and the distance information; displaying the blur alert image, 2. The method for controlling an image processing device, wherein the color of the blur notification image, which indicates subject blur, is changed according to the amount of subject blur.

13. A program for causing a computer to execute each step of the method for controlling an image processing apparatus according to claim 12.

Images