Image processing device, imaging device, and control method and program for image processing device

Abstract

To provide an image processing device that makes it possible for a photographer to facilitate checking a subject shake.SOLUTION: The image processing device includes a subject detection unit 306 for detecting a subject of an image, a subject distance detection unit 305 for detecting distance information indicating the distance to the subject, a subject shake detection unit 302 for detecting a subject shake, and a reporting plane generation unit 303 and an image superimposition unit 304 for generating a shake reporting image for reporting a subject shake of a specific subject on the basis of a detected subject shake and distance information.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to a technique for notifying a user of subject blurring that occurs in a captured image. [Background technology]

[0002] The following two points are important for capturing images with a camera without subject blur. The first is to synchronize the camera with the movement of the subject. If there is any relative misalignment between the subject and the camera, that misalignment will appear in the image as subject blur. The second is to set an appropriate shutter speed. As with camera shake, the longer the shutter speed is relative to subject blur, the more likely it is that the effects of subject blur will be more noticeable in the captured image (the subject blur will be visible to the human eye). Patent Document 1 discloses a technology that visualizes subject blur to the photographer by notifying the photographer of the subject blur on the electronic viewfinder or rear LCD screen of the imaging device. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2020-109948 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Document 1, a motion vector detection means is used to detect subject blur by detecting the movement between image frames as a vector value. Depending on the size of the subject captured by the image sensor and the size of the motion vector detection frame, the subject and other objects may exist within the vector frame, and if a vector other than the subject is detected, the subject edge cannot be accurately extracted. As a result, the edges of areas other than the subject may also be superimposed, which may reduce image quality.

[0005] An object of the present invention is to provide an image processing device that generates an image that allows the photographer to easily check for subject blur. [Means for solving the problem]

[0006] In order to solve the above problems, an image processing device of the present invention includes: a subject detection unit that detects a subject of an image; a distance detection unit that detects distance information indicating a distance to the subject; image cutting means for cutting out an image for each characteristic region of the subject based on the subject information; The camera includes a blur detection means for detecting subject blur, a notification image generation means for generating a blur notification image for notifying of subject blur of a specific subject based on the detected subject blur and the distance information, and a display means for displaying the blur notification image. The blur detection means detects subject blur by calculating a Laplacian value representing the edge of the subject and the amount of subject blur in each of the cut-out images using a Laplacian filter, and the notification image generation means changes a color representing subject blur in the blur notification image in accordance with the Laplacian value calculated for each of the cut-out images in a peaking display that emphasizes the edge of the subject, and the blur detection means sets a threshold value for changing the color representing subject blur in the blur notification image by changing a kernel value of the Laplacian filter in accordance with the subject information for each of the cut-out images. Change. [Effects of the Invention]

[0007] According to the present invention, it is possible to generate an image that allows the photographer to easily check for subject blur. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an imaging device. [Figure 2] 10 is a flowchart showing an imaging process. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of a blur notification image generating unit in the first embodiment. [Figure 4] FIG. 4 is a diagram showing a processing flow for generating a blur notification image in the first embodiment. [Figure 5] FIG. 2 is a diagram illustrating cutting out a subject. [Figure 6] FIG. 10 is a diagram illustrating a change in display color. [Figure 7] FIG. 10 is a diagram illustrating the processing of a notification plane generation unit. [Figure 8] FIG. 10 is a diagram showing an example of a blur notification image. [Figure 9] FIG. 10 is a diagram illustrating an example of the configuration of a blur notification image generating unit in the second embodiment. [Figure 10]10A to 10C are diagrams illustrating processing by a subject blur detection unit in the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] (First embodiment) FIG. 1 is a block diagram showing an imaging device according to this embodiment. The imaging device 100 is, for example, a digital camera. Note that, in this embodiment, an example of the imaging device 100 is described as an imaging device equipped with a lens device including an optical system 104, but the imaging device is not limited to this and may be an imaging device in which the lens device is detachable from a main body equipped with an imaging unit 105. The imaging device can also be applied to any electronic device capable of processing captured images. These electronic devices may include, for example, information terminals such as mobile phones and tablet terminals.

[0010] The imaging device 100 includes a control unit 101, a ROM 102, a RAM 103, an optical system 104, an imaging unit 105, an A / D conversion unit 106, an image processing unit 107, a recording unit 108, an angular velocity detection unit 111, a display unit 109, and an instruction input unit 110. The control unit 101 is, for example, a CPU (Central Processing Unit), and controls the operation of each block included in the imaging device 100 to thereby control the entire imaging device 100.

[0011] ROM 102 is an electrically erasable and recordable nonvolatile memory, and records operation programs for each block of imaging device 100 as well as parameters required for the operation of each block. RAM 103 is a rewritable volatile memory, and is used for loading programs executed by control unit 101 and the like, and for temporarily storing data generated by the operation of each block of imaging device 100. Control unit 101 reads out control programs for each block of imaging device 100 from ROM 102, loads them into RAM 103, and executes them to control imaging device 100.

[0012] The optical system 104 forms an image of a subject on the imaging surface of the imaging unit 105. The optical system 104 has a group of lenses including a zoom lens, a focus lens, an image stabilization lens, etc. The imaging unit 105 photoelectrically converts the optical image formed on the imaging surface of the imaging unit 105 via the optical system 104 to obtain an image (image signal). Specifically, the imaging unit 105 is an imaging element such as a CCD or CMOS sensor, and photoelectrically converts the optical image formed on the imaging surface of the imaging unit 105 by the optical system 104 and outputs the obtained analog image signal to the A / D conversion unit 106. In this embodiment, an example will be described in which the imaging unit 105 has an imaging element that outputs a signal capable of distance measurement using an imaging surface phase difference method. The A / D conversion unit 106 converts the analog image signal input from the imaging unit 105 into digital image data. The digital image data output from the A / D conversion unit 106 is temporarily recorded in the RAM 103.

[0013] The image processing unit 107 performs various image processing operations on the image data stored in the RAM 103. Specifically, the image processing unit 107 applies various image processing operations to develop, display, and record digital image data, such as pixel defect correction processing due to the optical system 104 or the image sensor, demosaicing processing, white balance correction processing, color interpolation processing, and gamma processing. The image processing unit 107 also includes a blur notification image generation unit 300. The blur notification image generation unit 300 generates a blur notification image to notify the photographer of blur of a subject in an image. The blur notification image is an image to notify the photographer of motion blur occurring in the subject. The blur notification image may also be an image that notifies the photographer of the absence of motion blur in the subject in addition to motion blur occurring in the subject. The blur notification image generation unit 300 generates and superimposes an image plane that allows the photographer to easily check subject blur based on subject blur information and distance information on the image data output from the A / D conversion unit 106 or image data stored in the RAM 103, thereby generating the blur notification image. Furthermore, the blur notification image generation unit 300 calculates the blur of the subject and detects the edge of the subject in order to generate an image plane according to the blur of the subject. Note that the target of image processing by the image processing unit 107 is not limited to image data recorded in the RAM 103, and may be, for example, digital image data output from the A / D conversion unit 106 or image data recorded in the recording unit 108.

[0014] The recording unit 108 records data including image data on a recording medium such as a removable memory card, and outputs the image data to an external device via an external interface. The recording unit 108 records the image data processed by the image processing unit 107 as a recorded image via the RAM 103.

[0015] The display unit 109 includes a display device such as an LCD (Liquid Crystal Display) and displays images stored in the RAM 103 and images stored in the recording unit 108 on the display device. The display unit 109 also displays an operation user interface (operation UI) for receiving instructions from the user. The display unit 109 may also have multiple display devices, such as an EVF (Electronic Viewfinder) and a rear monitor provided on the photographer's side (rear). The display unit 109 may be capable of simultaneously outputting to multiple display devices or may be configured to selectively display one display device at a time by switching between them. The instruction input unit 110 is an input interface including various physical operation members such as a touch panel and a shutter button, and receives instructions input by the user. The rear monitor provided on the photographer's side (rear) may be configured as a touch panel that has the functions of both the display unit 109 and the instruction input unit 110. By associating input coordinates on the touch panel with display coordinates, an operation UI can be configured that makes it appear as if the photographer can directly operate the screen displayed on the touch panel.

[0016] Furthermore, under the control of the control unit 101, the imaging device 100 performs live view shooting in which analog image signals sequentially output from the imaging unit 105 are sequentially displayed on a display device via the A / D conversion unit 106, RAM 103, image processing unit 107, and display unit 109. During live view shooting, it is possible to prepare for shooting by adjusting the composition for actual shooting, which is intended for recording on a recording medium or output to an external device, and by changing shooting parameters such as the exposure time (Tv value), aperture value (Av value), and ISO sensitivity for actual shooting.

[0017] The angular velocity detection unit 111 detects shake applied to the imaging device 100. The angular velocity detection unit 111 is, for example, an angular velocity sensor, and detects angular velocities in the pitch direction, yaw direction, and roll direction applied to the imaging device 100 due to camera shake or camera work. Note that the imaging device 100 may be realized as an image processing device that acquires an output signal from the imaging unit 105 and performs processing such as displaying an image on the display unit 109 and recording an image on the recording unit 108. The image processing device generates an image in the image processing unit 107 to notify of subject shake, which will be described later.

[0018] Next, the imaging process by the imaging device 100 in this embodiment will be described in detail with reference to the flowchart in Fig. 2. Fig. 2 is a flowchart showing the imaging process. The processing of each step in this flowchart is realized by the control unit 101 issuing instructions to each unit of the imaging device 100 in accordance with a predetermined program that is read out from the ROM 102 and executed. Note that this embodiment will be described taking as an example a case where live view imaging is performed in which the photographer captures an image while checking a real-time image displayed on a rear monitor provided on the photographer's side (rear).

[0019] The imaging process is started when the photographer turns on the power of the imaging device 100. In step S201, the control unit 101 starts preparatory imaging in response to the imaging device 100 being powered on. Specifically, the control unit 101 controls the optical system 104 and imaging unit 105 to start live view imaging. During the live view period, the imaging unit 105 sequentially captures images (preparatory captured images), and the acquired images are displayed on the display device of the display unit 109. The photographer can make preparations for imaging, such as adjusting the composition, while checking the live view images sequentially displayed on the display device. Note that the processes of steps S202 to S206, which will be described later, are performed during the live view imaging period.

[0020] In step S202, the control unit 101 determines shooting parameters for the actual image capture. The shooting parameters include the shutter speed, aperture value, ISO sensitivity, etc. The shooting parameters may be determined based on instructions from the user input via the instruction input unit 110, or the control unit 101 of the imaging device 100 may automatically set the shooting parameters.

[0021] In step S203, control unit 101 determines whether the motion blur notification is set to ON or OFF. The motion blur notification may be set by the user using instruction input unit 110, or may be automatically set to ON or OFF by imaging device 100 based on predetermined shooting conditions, for example. When the user sets the motion blur notification, the motion blur notification can be set using a single physical operation member (button, bar, etc.) or a single icon on a touch panel, and can be set to ON or OFF at any timing during preparatory shooting. Alternatively, the motion blur notification display may be set to be periodically switched ON / OFF. If control unit 101 determines that the motion blur notification is set to ON, the process proceeds to step S204. On the other hand, if control unit 101 determines that the motion blur notification is set to OFF, the process proceeds to step S205.

[0022] In step S204, the blur notification image generating unit 300 generates a blur notification image. The blur notification image is generated by superimposing, on the preparation captured image, an notification plane that notifies the user of motion blur that will occur in the subject when the preparation captured image is captured using the shooting parameters for main capture (or the absence of motion blur). Details of the processing in step S204 will be described later with reference to FIG. 4.

[0023] In step S205, the control unit 101 displays the blur notification image if one was generated in step S204, or displays the preparatory captured image without the motion blur notification plane superimposed on it on the display device of the display unit 109 if the motion blur notification is set to OFF. If the motion blur notification is set to ON, the user can set shooting parameters such as the shutter speed (exposure time) for actual capturing while checking the displayed blur notification image.

[0024] In step S206, the control unit 101 determines whether the shutter button of the instruction input unit 110 has been pressed by a user. Here, pressing the shutter button is an instruction to capture actual image capture. Therefore, if the shutter button is configured to accept two input methods, such as a half-press to instruct a preparatory operation for image capture and a full press to instruct actual image capture, the control unit 101 determines whether the button has been fully pressed. If only a simple one-step input is accepted, the control unit 101 determines whether the one-step input has been accepted. If the control unit 101 determines that the shutter button has not been pressed, the control unit 101 returns to step S202 and repeats the processes of steps S202 to S206. This allows the user to easily check the motion blur that will occur in the subject if actual image capture is performed with the currently set shooting parameters, even during preparatory image capture. If the motion blur is confirmed and is not what the user desires (if the user does not want motion blur to occur), the user can simply reset the shooting parameters, such as the shutter speed (exposure time), for actual image capture without pressing the shutter button. On the other hand, if the control unit 101 determines that the shutter button has been pressed, the process proceeds to step S207.

[0025] In step S207, the control unit 101 controls the optical system 104, the imaging unit 105, etc. to perform actual imaging based on the set imaging parameters and acquire an image (actually captured image). In step S208, the control unit 101 outputs the actual captured image acquired by actual imaging to the display unit 109 and the recording unit 108. The display unit 109 displays the actual captured image on a display device. In addition, the recording unit 108 records the actual captured image on a recording medium or outputs it to an external device.

[0026] In this way, the imaging process of this embodiment notifies the user of motion blur of the subject by displaying a blur alert image during preparatory imaging. This allows the user to set an appropriate exposure time for actual imaging while checking the blur alert image displayed on display unit 109. Then, the shutter chance (actual imaging) can be reached with the exposure time set to an appropriate level that corresponds to the motion blur.

[0027] Next, an example of the configuration of the blur alert image generation unit 300 included in the image processing unit 107 will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the configuration of the blur alert image generation unit 300. The blur alert image generation unit 300 generates a blur alert image based on a preparatory captured image and shooting conditions. The blur alert image generation unit 300 includes an image cropping unit 301, a subject blur detection unit 302, a notification plane generation unit 303, an image superimposition unit 304, a subject distance detection unit 305, and a subject detection unit 306.

[0028] The subject distance detection unit 305 detects the distance to the subject as subject distance data. The subject detection unit 306 detects the subject in the image. Furthermore, the subject detection unit 306 can detect characteristic parts of the subject, such as the type of subject or organs. The image cropping unit 301 crops the subject detected by the subject detection unit 306 from the image. In this embodiment, the image cropping unit 301 crops the preparatory capture image in units of characteristic parts of the subject detected by the subject detection unit 306. The subject blur detection unit 302 detects subject blur of the subject detected by the subject detection unit 306. The notification plane generation unit 303 and the image superimposition unit 304 are notification image generation means that generate a blur notification image to notify of subject blur. First, the notification plane generation unit 303 generates a notification plane according to the subject blur, and the image superimposition unit 304 superimposes the notification plane on the image to generate a blur notification image. The notification plane generation unit 303 generates a motion notification plane, which is data for notifying of subject blur based on the subject blur. This embodiment also makes it possible to notify of subject blur of only a specific subject, rather than all subjects. Therefore, the notification plane generation unit 303 of this embodiment determines the specific subject for which subject blur is to be notified based on the subject distance information, and generates a notification plane for notifying of the blur of the specific subject. The image superimposition unit 304 generates a blur notification image by superimposing the motion blur notification plane on the captured image.

[0029] 3 may be implemented by hardware such as an ASIC or a programmable logic array (PLA), or by a programmable processor such as a CPU or MPU executing software. Alternatively, one or more of the functional blocks shown in FIG. 3 may be implemented by a combination of software and hardware. Therefore, even when different functional blocks are described as being responsible for the operations in the following description, they may be implemented by the same hardware.

[0030] Next, the process of generating a blur warning image by the blur warning image generating unit 300, which is executed in step S204 in Fig. 2, will be described in detail with reference to the flowchart in Fig. 4. Fig. 4 is a flowchart showing the process of generating a blur warning image. The process of each step in this flowchart is realized by the control unit 101 issuing instructions to each unit of the imaging device 100, such as the blur warning image generating unit 300, in accordance with a predetermined program that is read out from the ROM 102 and executed.

[0031] In step S401, the control unit 101 inputs the preparatory captured images sequentially captured by the imaging unit 105, the shooting parameters to be used for actual shooting determined in step S202, and the angular velocity data detected by the angular velocity detection unit 111 to the blur warning image generation unit 300. In step S402, the subject detection unit 306 of the blur warning image generation unit 300 detects the subject and acquires subject information. The subject detection unit 306 detects the subject information by comparing the captured video with data learned, for example, based on deep learning. The subject information here refers to the type and organs of the subject. Examples of the type of subject include people, animals (dogs, cats, birds, horses, etc.), vehicles (cars, bicycles, motorcycles, formula cars, etc.), etc. Furthermore, the organs of the subject include, for example, the eyes, hands, feet, head, chest, etc. if the type of subject is a person. Furthermore, examples of the organs of the subject include the eyes, torso, limbs, etc. if the type of subject is an animal. If the type of subject is a vehicle, the organs of the subject are a helmet, a headlight, a taillight, etc. Note that detection of subject information based on machine learning is one example, and subject information may be detected by other methods.

[0032] In step S403, subject blur detection unit 302 acquires the shooting conditions. Here, the shooting conditions acquired in step S403 are the exposure time (shutter speed) for main shooting and the time interval between images (frame rate) in preparatory shooting, which were acquired in step S401 and determined in step S202.

[0033] In step S404, the subject blur detection unit 302 acquires angular velocity data indicating the shake of the imaging device 100 from the angular velocity detection unit 111. The angular velocity detection unit 111 may be an inertial sensor such as a gyro sensor, or may use a motion vector to calculate the angular velocity data from the difference between different preparatory capture images using a vector value outside the subject region, a focal length, and a frame rate.

[0034] In step S405, the subject distance detection unit 305 acquires distance information (distance data) to the subject. The subject distance detection unit 305 detects distance information from a defocus value for each pixel using, for example, an image sensor having pixels for performing phase-difference distance measurement on the image capturing surface of the image capturing unit 105, which is capable of image-surface phase-difference distance measurement and focus detection. By using an image sensor capable of image-surface phase-difference AF, the subject distance detection unit 305 can detect the defocus amount based on the phase difference between output image signals of the image sensor obtained from different regions of the exit pupil of the optical system 104. The subject distance detection unit 305 may also detect subject distance data using a sensor for measuring distance, such as a laser displacement sensor, an ultrasonic sensor, or a millimeter-wave radar, installed inside the image capturing device 100. The subject distance detection unit 305 outputs the acquired distance information to the subject to the subject blur detection unit 302.

[0035] In step S406, the image cropping unit 301 crops out an image of the subject from the preparatory captured image based on the subject information detected by the subject detection unit 306 in step S402. The image cropping unit 301, for example, detects characteristic parts of the subject within the same subject, and crops out an image for each part of the subject. The cropping of an image of a subject will be described with reference to FIG. 5. FIG. 5 is a diagram for explaining subject cropping. The example shown in FIG. 5 represents a panning shot of a motorsport scene as an example of a shooting scene, and multiple subjects (subject 501, subject 502, subject 503) are present within the shooting angle of view of the preparatory captured image 500. The subject detection unit 306 generates cropped images for each part (subject characteristic region) within each subject based on subject information such as the type of subject.

[0036] If there are multiple subjects, the main subject may be determined from among the multiple subjects. In this case, the image cropping unit 301 may automatically determine the main subject based on the detection result of the subject detection unit 306, or the photographer may prioritize a subject at a position arbitrarily selected as the main subject. Furthermore, when identifying the main subject, the image cropping unit 301 may use information about the distance to the subject detected by the subject distance detection unit 305. Once the main subject is determined, the subject detection unit 306 detects characteristic features of the subject within the main subject. For example, if subject 501 is the main subject, the characteristic features are person 501a, helmet 501b, and motorcycle 501c. The image cropping unit 301 crops the image into each region (subject characteristic region) of person 501a, helmet 501b, and motorcycle 501c. Note that the purpose of cropping the subject from the image here is to eliminate noise as much as possible and improve the accuracy of subject blur detection when detecting subject blur, which will be described later. For example, in panning shots like the one shown in Figure 5, or portraits where the aperture is wide open to blur the background and make the person stand out, the background is noticeably blurred and out of focus, so if this area is also input into subject blur detection, the accuracy of subject blur detection will decrease.Furthermore, in addition to the background composition, if a composition in which other subjects overlap the main subject and the movements of each subject differ, the accuracy of subject blur detection will also decrease, so it is important to detect the main subject as accurately as possible.

[0037] Returning to the explanation of FIG. 4, in step S407, subject blur detection unit 302 detects subject blur from the cropped image input from image cropping unit 301. In this embodiment, subject blur (amount of subject blur) is detected from the degree of blur (clarity) of the subject's edges. This is because when subject blur occurs, the edges appear blurred and unclear, while when there is no subject blur, the edges appear clear and unblurred. Therefore, when the edges are clear, it is determined that subject blur is small, and when the edges are unclear, it is determined that subject blur is large. To detect the clarity of the edges corresponding to subject blur, for example, a Laplacian filter is used. Furthermore, by using a Laplacian filter, it is possible to simultaneously detect subject blur and subject edges.

[0038] We will explain an example of detecting subject blur using a Laplacian filter. A Laplacian filter is a spatial filter that extracts edges from an image using second-order derivatives. Since the input image is digital and discrete data, derivatives can be calculated using differences. The first-order derivatives Ix and Iy of the pixel values ​​in the horizontal and vertical directions of the Laplacian filter are given by equation (1).

number

[0039] The Laplacian filter is a second-order derivative. Therefore, by taking the difference of the first-order derivative, the Laplacian filter can be expressed as equation (2).

number

number

[0040] The Laplacian ∇ shown in equation (3) 2 The kernel value indicating the strength of filtering (value used for weighting) from I(x, y) is expressed by equation (4) in the case of a kernel value of 4 neighbors, and by equation (5) in the case of a kernel value of 8 neighbors.

number

number

[0041] The kernel value is the number of neighboring pixels of the pixel of interest. Equation (4) is the kernel value for four neighbors, indicating that the quadratic derivative is taken for four pixels above, below, left, and right from the pixel of interest. Equation (5) is the kernel value for eight neighbors, indicating that the quadratic derivative is taken for eight pixels, including not only the pixels above, below, left, and right from the pixel of interest but also diagonal directions. In this embodiment, the kernel value is changed for each image cut out from the object feature region according to the type of object (object information) of the cut-out image. For example, the kernel value is set to four neighbors for helmet 501b and eight neighbors for motorcycle 501c. The number of neighboring pixels may also be more than eight pixels.

[0042] The clearer the edge, the greater the difference in pixel value (brightness value) between the pixel of interest and its neighboring pixels. Furthermore, when the difference in pixel value (brightness value) between the pixel of interest and its neighboring pixels is large, the magnitude of the Laplacian value also increases. Therefore, the larger the Laplacian value, the clearer the edge and the less subject blurring it is determined to be, and the smaller the Laplacian value, the less clear the edge and the more subject blurring it is determined to be. In this way, the subject blur detection unit 302 determines the magnitude of the Laplacian value for each cropped image obtained using the set kernel value as the amount of subject blurring. The magnitude of the Laplacian value for each cropped image may be any value that can determine the clarity of the edge, such as the average of the absolute values ​​of the Laplacian values ​​of each pixel in the cropped image. While the present embodiment has been described using an example of using a Laplacian filter, the present invention is not limited to this, and any method for detecting subject blurring based on the clarity of the edge may be used.

[0043] Returning to the description of FIG. 4, in step S408, the notification plane generation unit 303 generates a notification plane for notifying the user of motion blur of the subject in accordance with the subject blur detected in step S407. Here, an example of notifying the user of subject blur will be described. In this embodiment, a peaking display is performed in which the edges of the subject in the image displayed on the display device are highlighted, and the color of the peaking display is changed depending on the amount of subject blur to notify the user of the magnitude of the subject blur. For example, if the subject blur is large enough to be visible on the image, the peaking display of the main subject is performed in red, and if the subject blur is small enough to be invisible on the image, the peaking display is performed in green. In other words, if the Laplacian value is small and the subject edge is unclear, the peaking display is performed in red, and if the Laplacian value is large and the subject edge is clear, the peaking display is performed in green. Note that the number of different colors in the peaking display and the colors displayed are not limited to these.

[0044] Here, the change in display color according to the amount of subject blur will be explained in more detail. FIG. 6 is a diagram illustrating the change in display color. In FIG. 6, the vertical axis represents the Laplacian value of the cropped image, and the horizontal axis represents the kernel value. FIG. 6 shows the Laplacian value when a Laplacian filter with a 4-neighbor and 8-neighbor kernel is applied to an image cropped from helmet 501b of subject 501 in FIG. 5. In this way, the color displayed on the edge of the subject to notify subject blur is changed according to the Laplacian value corresponding to the amount of subject blur. Even when the same image is used, the Laplacian value differs depending on the kernel value. Therefore, the threshold value that determines the color when visualizing the amount of subject blur is changed depending on the kernel value.

[0045] For example, when a Laplacian filter is applied to the size of the image cropped by helmet 501b using a kernel value with four neighbors, the color of the peaking display is green when the Laplacian value is equal to or greater than threshold 601, orange when it is less than threshold 601 and equal to or greater than threshold 603, and red when it is less than threshold 603. Furthermore, when a Laplacian filter is applied to eight neighbors, the color of the peaking display is green when the Laplacian value is equal to or greater than threshold 602, orange when it is less than threshold 602 and equal to or greater than threshold 604, and red when it is less than threshold 604. In this way, the threshold for determining the display color is changed depending on the kernel value. Furthermore, in addition to the kernel value, the threshold for determining the display color may also be varied depending on the subject region cropped by image cropping unit 301 and the image size. Furthermore, although two types of kernel values ​​are described in FIG. 6, the number of kernel values ​​may be increased to sixteen neighbors. Note that the method for notifying subject blurring is not limited to changing the color of the peeing display. For example, the thickness of the line of the edge of the peaking display may be changed, the line may blink, or the line may be dashed. Alternatively, a display informing of subject blur may be displayed only when the amount of subject blur is equal to or greater than a predetermined threshold value.

[0046] Next, the generation of a notification plane for notifying subject blur and the generation of a blur notification image in which the generated notification plane is superimposed on an image will be described in detail with reference to FIG. 7. FIG. 7 illustrates the processing performed by the notification plane generation unit 303. FIG. 7(A) illustrates the arrangement of distance detection frames 701. The distance detection frames 701 are detection frames used by the subject distance detection unit 305 to detect the subject distance. They are arranged across substantially the entirety of the preparatory capture image 500 and include multiple detection frames (14 vertical × 24 horizontal in the example of FIG. 7(A)). Based on the subject distance in the main focus frame 702 that is most in focus among the distance detection frames 701, a main focus distance detection frame 703 corresponding to a subject distance within a predetermined threshold range is semi-transparently displayed. For example, the notification plane generation unit 303 identifies a subject whose depth of field is within a threshold range (e.g., ±2 depths) based on the depth of field of the main focus frame 702, based on the distance information calculated by the subject distance detection unit 305. In this embodiment, an example has been described in which a specific subject to be displayed in peaking mode is determined based on distance information using the in-focus area as the reference area, but this is not limited to this. For example, the area where the AF frame (main ranging frame) is superimposed may be used as the reference area for identifying the subject based on distance information.

[0047] FIG. 7B shows an image to which the Laplacian filter of the subject blur detection unit 302 has been applied. The Laplacian filter is applied to the preparatory capture image 500 to detect the subject's edges. FIG. 7C shows an information plane. The information plane is an image in which the edges of the subject within the main focus distance detection frame 703, which is the most in-focus area, are displayed with a peaking effect, such as a color display corresponding to the subject blur. Specifically, the information plane generation unit 303 overlays the main focus distance detection frame 703 (FIG. 7A) detected based on the depth of field with an image of the subject's edges detected with the Laplacian filter (FIG. 7B), and displays a peaking effect on the overlapping edges. The color of the peaking effect is determined according to the Laplacian value calculated for each cropped image. When cropped areas overlap, the color corresponding to the greater blur is prioritized. For example, if the motorcycle 501c is green, the person 501a is orange, and the helmet 501b is red, the edges within the area of ​​the helmet 501b will be displayed in red.

[0048] Returning to the description of Fig. 4, in step S409, the image superimposing unit 304 superimposes the notification plane generated in step S406 on the preparatory captured image (Fig. 7(C)) to generate a blur notification image. The image superimposing unit 304 generates the blur notification image by superimposing the preparatory captured image and the notification plane on the RAM 103, for example. By superimposing the preparatory captured image and the notification plane, it is possible to peak the main subject in the preparatory captured image in accordance with the subject blur.

[0049] FIG. 8 shows examples of blur notification images. FIGS. 8A and 8B depict motorsports (motorcycles), while FIGS. 8C and 8D depict a ballerina. Examples of application of this embodiment are shown in FIGS. 8A and 8C. In FIG. 8B, subjects 501 and 503 are detected as the same subject, and areas other than the main subject, subject 501, are also notified. On the other hand, this embodiment allows for peaking of only a specific subject within a predetermined distance from the subject distance of the in-focus area the photographer is attempting to capture. Therefore, this embodiment generates an image in which only the edge of the intended specific subject (subject 501) is peaked, as shown in FIG. 8A, thereby notifying the photographer of subject blur of subject 501 alone. Even when there is only one subject, as shown in FIG. 8D, there is a risk that the boundary line 802 between the stage and the rear wall may be detected as part of the subject and peaked. On the other hand, according to this embodiment, as shown in FIG. 8C, only the subject 801 is displayed in a peaking manner, and it is possible to notify the photographer of only the subject blur of the subject 801.

[0050] In this embodiment, an example of live view shooting has been described in which an image is displayed in real time on a display device provided on the photographer side of the imaging device 100, but the present invention is not limited to this, and an image for notifying of subject blur may be displayed on another display device such as an EVF. Also, in the first embodiment, the process of acquiring angular velocity data (step S404) does not need to be performed.

[0051] As described above, according to this embodiment, it is possible to generate an image that allows the photographer to easily check for subject blur in the main subject that the photographer is about to capture. By checking this image, the photographer can easily set a shutter speed that will not cause subject blur, and can obtain a high-quality image with reduced subject blur during actual shooting.

[0052] (Second embodiment) In this embodiment, a subject blur detection method different from that of the first embodiment is used to detect subject blur and generate a blur notification image. In the first embodiment, subject blur was detected using a Laplacian filter on an image extracted from the subject. In contrast, in this embodiment, the main subject is identified based on a motion vector calculated from multiple images, and subject blur is detected. The following describes the detection method of the subject blur detection unit 302, which is a difference from the first embodiment, and other common operations and processing are omitted.

[0053] The image processing unit 107 in the second embodiment has a blur notification image generation unit 900 instead of the blur notification image generation unit 300 in the first embodiment. Fig. 9 is a diagram showing the configuration of the blur notification image generation unit 900 in the second embodiment. The blur notification image generation unit 900 has a motion vector detection unit 901, a subject blur detection unit 902, a notification plane generation unit 303, an image superimposition unit 304, and a subject distance detection unit 305. The operations and processes other than those of the motion vector detection unit 901, the subject blur detection unit 902, and the angular velocity detection unit 111 are the same as those in the first embodiment, so a description thereof will be omitted.

[0054] The motion vector detection unit 901 detects motion vectors based on a plurality of preparatory captured images. Specifically, the motion vector detection unit 901 uses the plurality of preparatory captured images as a base frame and a reference frame, respectively, and performs a correlation calculation between a base block in the base frame and each block in the target area in the reference frame. As a result of the correlation calculation, the motion vector is calculated from the positional relationship between the block with the highest correlation and the base block. The method for calculating the correlation value is not limited to a method based on the sum of absolute differences, the sum of squared differences, or a normal cross-correlation value. Furthermore, the method for calculating the motion vector is not limited to a correlation calculation, and other methods such as a gradient method may also be used. The motion vector detection unit 901 outputs data of the detected motion vector (vector data) to the subject blur detection unit 902.

[0055] The subject blur detection unit 902 calculates a subject vector based on the motion vector and detects subject blur. Based on the vector data of the image, angular velocity data of the imaging device 100, and subject distance data, the subject blur detection unit 902 separates the vector data into vector data related to the subject and vector data unrelated to the subject. The vector data unrelated to the subject is, for example, vector data of the background. The subject blur detection unit 902 then detects subject blur based on the vector data related to the subject. The image vector data (motion vector) is input to the subject blur detection unit 902 from the motion vector detection unit 901, the angular velocity data from the angular velocity detection unit 111, and the subject distance data from the subject distance detection unit 305. Here, the vector data separation method will be described in detail using FIG. 10.

[0056] FIG. 10 is a diagram illustrating a method for separating vector data. FIG. 10(A) is a diagram illustrating motion vector detection. FIG. 10(B) shows a frequency distribution of vector values. FIG. 10(A) shows a panning shot of a motorsports scene as an example of photography, with multiple subjects (subject 1001, subject 1002, and subject 1003) present in the photography field of view. A motion vector detection frame 1004 is arranged within the image, and for example, in each block within the motion vector detection frame 1004, the amount of movement from the previous frame is detected as a vector value. In FIG. 10(B) showing the frequency distribution, the horizontal axis represents the vector value detected within the motion vector detection frame 1004, and the vertical axis represents the frequency. Note that, if the motion of the entire screen is to be detected, it is desirable for the motion vector detection frame 1004 to cover the entire photography field of view. However, if the motion of only a portion of the photography field of view (subject) is to be detected, as in this embodiment, it is desirable to arrange the detection frames small and densely. This is because, when the motion vector detection frame 1004 covers the entire shooting angle of view, the larger the motion vector detection frame 1004, the greater the amount of calculation processing required, especially when the image sensor has a high pixel count or the frame rate is high. If the calculation processing required for motion vector detection becomes too large, delays may occur in the generation and display of the motion blur warning image. Furthermore, if the calculation processing required for motion vector detection becomes too large, power consumption increases and the maximum number of images that can be captured decreases. For a motion vector detection frame 1004 that does not cover the entire angle of view, the motion vector detection frame 1004 may be positioned, for example, along the AF frame. This is because the AF frame is likely to be located on the subject, and positioning the motion vector detection frame 1004 along the AF frame increases the likelihood that the motion vector detection frame 1004 will capture the subject.

[0057] 10B, angular velocity data 1005 is a value obtained by converting the angular velocity data of the image capturing device 100 detected by the angular velocity detection unit 111 into the amount of movement on the imaging surface. The subject detection unit 306 converts the angular velocity data into the amount of movement on the imaging surface using equation (6).

number

[0058] When capturing an image while moving the camera significantly (panning) as shown in FIG. 10A, the angular velocity data of the image capturing device 100 converted into the amount of motion on the imaging surface coincides with the vector associated with the background of the image on the imaging surface. In other words, vector 1006 existing near angular velocity data 1005 converted into pixel units on the imaging surface corresponds to the background in motion vector detection frame 1004. In practice, the angular velocity detection unit 111 experiences offset drift, in which the reference value fluctuates due to disturbances such as temperature changes and impacts. Therefore, the subject blur detection unit 902 sets a predetermined range 1007 based on angular velocity data 1005 converted into pixel units on the imaging surface as a base point, taking into account the offset drift, and determines vectors within the predetermined range 1007 to be background vectors. Note that the range of predetermined range 1007 based on angular velocity data 1005 as a base point may be varied while considering the focal length and the amount of offset change.

[0059] Then, subject blur detection unit 902 determines vectors outside predetermined range 1007 as subject candidates, and determines the vector from among the subject candidate vectors that most closely meets predetermined conditions as the vector of the main subject. For example, subject blur detection unit 902 determines the vector from among the subject candidate vectors that is located at a position with the highest frequency close to 0 as the vector of the main subject. Furthermore, subject blur detection unit 902 may preferentially determine, from among the subject candidate vectors, the vector frame at the position determined by subject detection unit 306 to be the main subject as the vector of the main subject.

[0060] Furthermore, the subject blur detection unit 902 may determine the vector frame to be adopted based on the subject detection results using, for example, a subject detection function with a deep learning function, without using a frequency distribution. For example, in Fig. 10(A), if the subject 1001 is detected as the main subject by the subject detection function and the position of the head is also detected, the vector closest to the position of the head may be adopted.

[0061] Furthermore, as described in the first embodiment, subject distance data detected by subject distance detection unit 305 may be used to detect the subject vector. For example, if a main focus frame (AF frame) is located on the head of subject 1001, a vector frame that falls within a predetermined range (for example, ±2 depth) based on the depth of field of the main focus frame is determined to be the subject, and the vector value of that vector frame is used as the subject vector. Furthermore, subject vectors extracted using angular velocity data and subject distance data may be overlapped, and the area that overlaps under both conditions may be ultimately determined to be the subject vector.

[0062] The subject blur detection unit 902 outputs the detected subject blur to the notification plane generation unit 303. The notification plane generation unit 303 generates a notification plane according to the subject blur detected by the subject blur detection unit 902. If the notification plane is a peaking display of the edge of the subject, the notification plane generation unit 303 detects the edge of the subject, as in the first embodiment. Furthermore, the notification plane generation unit 303 may specify an area for notifying of subject blur based on distance information of the subject, as in the first embodiment. Then, the image superimposing unit 304 superimposes the notification plane on the preparatory capture image, as in the first embodiment, to generate a blur notification image in which the edge is highlighted according to the subject blur. The generated blur notification image is displayed on a display device, and the user can set the shutter speed and the like while checking the blur notification image.

[0063] As described above, according to this embodiment, it is possible to generate an image that allows the photographer to easily check for subject blur in the main subject that the photographer is about to capture. By checking this image, the photographer can easily set a shutter speed that will not cause subject blur, and can obtain a high-quality image with reduced subject blur during actual shooting.

[0064] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program.The present invention can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0065] In the first and second embodiments described above, the magnitude of blur is determined based on the Laplacian value. However, to determine the magnitude of blur more accurately, the magnitude of blur may be determined by taking into account the effect of a decrease in the Laplacian value due to blur. Even if the blur is small, if the blur is large, the Laplacian value will be small. Therefore, when the Laplacian value is small, the blur and the blur contained in the cropped image can be separated by analyzing the frequency of the blur and blur, and a notification plane may be generated based on the amount of blur separated from the blur. Frequency analysis can be performed, for example, by applying a point spread function to the cropped image. While blur has a specific frequency, blur is the result of a combination of various frequencies. Therefore, in frequency space, if a thin peak appears at a specific frequency, the influence of blur is considered to be large, while if a wide peak appears or no specific peak appears, the influence of blur is considered to be large. For example, as described in FIG. 6 and step S407, a Laplacian value that displays an edge in red is calculated based on its relationship with a threshold value, and if the influence of blur is determined to be large through frequency analysis, the edge is displayed in red. On the other hand, even if the same Laplacian value is calculated, if the frequency analysis determines that the influence of blur is large, it is assumed that the Laplacian value has decreased due to the influence of blur, and the blur is determined to be small, and the edge is displayed in green.

[0066] Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments and various modifications and changes are possible within the scope of the gist of the present invention. [Explanation of symbols]

[0067] 100 Imaging device 101 Control section 102 ROM 103 RAM 105 Imaging unit 107 Image processing section 109 Display section 111 Angular velocity detection unit 300 Blur notification image generation unit 301 Image Cutout Section 302 Subject blur detection unit 303 Notification plane generation unit 304 Image Overlay Unit 306 Subject detection unit

Claims

1. a subject detection means for detecting a subject of an image; a distance detection means for detecting distance information indicating the distance to a subject; an image cutting means for cutting out an image for each characteristic region of the subject based on the subject information; a blur detection means for detecting subject blur for each cut-out image; and notification image generating means for generating a blur notification image for notifying of subject blur of a specific subject based on the detected subject blur and the distance information, the blur detection means detects subject blur by calculating a Laplacian value representing an edge of a subject and an amount of subject blur in each of the cut-out images using a Laplacian filter; the notification image generating means changes a color indicating subject blur in the blur notification image in accordance with the Laplacian value calculated for each of the extracted images in a peaking display that emphasizes an edge of the subject; the blur detection means changes a kernel value of the Laplacian filter for each of the extracted images in accordance with the subject information, thereby changing a threshold value for changing a color indicating subject blur in the blur notification image.

2. 2. The image processing apparatus according to claim 1, wherein the notification image generating means determines a specific subject for which a notification of subject blur is to be given based on the distance information.

3. 3. The image processing device according to claim 1, wherein the specific subject is a subject at a subject distance within a predetermined range from the subject distance of the reference area, or a subject at a depth of field within a predetermined range from the depth of field corresponding to the subject distance of the reference area.

4. 4. The image processing device according to claim 3, wherein the reference area is an area corresponding to a main ranging frame or a focusing area.

5. 5. The image processing apparatus according to claim 1, wherein the distance detection means detects distance information based on distance measurement using a phase difference method.

6. an imaging element having pixels for performing phase difference ranging; An image processing device according to any one of claims 1 to 5; and a display unit that displays the blur alert image generated by the image processing device.

7. A control method for an image processing device, comprising: detecting a subject in the image; detecting distance information indicating a distance to a subject; A step of cutting out an image for each feature region of the subject based on the subject information; detecting subject blur for each extracted image; generating a blur notification image for notifying the user of the subject blur of a specific subject based on the detected subject blur and the distance information; and displaying the blur alert image. the step of detecting subject blur includes detecting subject blur by calculating a Laplacian value representing an edge of the subject and an amount of subject blur in each of the cut-out images using a Laplacian filter; In the step of displaying the blur alert image, a color indicating subject blur is changed in accordance with the Laplacian value calculated for each of the extracted images in a peaking display that emphasizes an edge of the subject in the blur alert image; a control method for an image processing device, characterized in that, in the step of detecting subject blur, a kernel value of the Laplacian filter is changed for each of the extracted images in accordance with the subject information, thereby changing a threshold value for changing a color indicating the subject blur in the blur notification image.

8. A program for causing a computer to perform each step of the method for controlling an image processing apparatus according to claim 7.

Images