Image processing apparatus, imaging apparatus, and image processing method

Abstract

To provide an image processing apparatus that can determine an appropriate subject as a main subject.SOLUTION: An image processing apparatus detects subject areas from an image and determines motion vectors related to the subject areas. The image processing apparatus determines, from the subject areas, a subject area in which a direction indicated by the motion vector is a predetermined preceding movement direction as a main subject area.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, and an image processing method. [Background technology]

[0002] There is known an imaging device that controls an imaging optical system and processes a captured image so that a detected main subject approaches the center of the angle of view and becomes a certain size or larger. Such an imaging device detects an area (subject area) in which a specific subject such as a person appears from the captured image, and selects the main subject area from the detected subject areas according to predetermined conditions.

[0003] For example, in Patent Document 1, in a group motion scene in which there are multiple subject regions having the same motion pattern, the subject region located at the front in the motion direction is determined to be the main subject region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2022-22767 Summary of the Invention [Problem to be solved by the invention]

[0005] However, in sports such as soccer and basketball, where many human subjects move in the same direction following the movement of the ball, the subject at the front of the multiple subjects moving in the same direction is not necessarily the main subject, and therefore it may happen that a range different from the range the user wants to see is captured.

[0006] In view of the problems with the conventional techniques, one aspect of the present invention provides an image processing device and an image processing method capable of determining an appropriate subject as a main subject. [Means for solving the problem]

[0007] The above-mentioned object can be achieved by an image processing device having a subject detection means for detecting a subject area from an image, a motion detection means for calculating a motion vector related to the subject area, and a determination means for determining, as a main subject area, a subject area among the subject areas whose motion vector indicates a predetermined preferred motion direction. Effect of the Invention

[0008] According to the present invention, it is possible to provide an image processing device and an image processing method capable of determining an appropriate subject as a main subject. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera as an example of an imaging apparatus according to an embodiment; [Diagram 2] FIG. 2 is a block diagram showing a schematic diagram of a main subject determination process according to an embodiment; [Diagram 3] 1 is a flowchart showing an example of a photographing operation according to an embodiment; [Figure 4] Schematic diagram for explaining specific examples and effects of the embodiment DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0011] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a digital camera as an example of an image processing device. However, the present invention can also be implemented in any electronic device capable of handling image data. Such electronic devices include not only imaging devices but also computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices. When the present invention is implemented in a camera, it can be implemented in a stationary camera that is not carried by the user, such as a surveillance camera, or in a wearable camera, such as an action camera.

[0012] (Digital Camera Configuration) FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera 100 as an example of an image processing device according to an embodiment. The digital camera 100 is capable of taking and recording moving images and still images. The functional blocks in the digital camera 100 are communicably connected to each other via a bus 160. The operation of the digital camera 100 is realized by one or more programmable processors in a main control unit 151 loading a program stored in a ROM 155, for example, into a RAM 154 and executing the program to control each functional block. Unless otherwise specified, each functional block can be realized by a hardware circuit such as an ASIC, or by one or more programmable processors loading a program into a memory and executing the program.

[0013] The photographing lens 101 (lens unit) has a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed third group lens 121, a focus lens 131, a zoom motor (ZM) 112, an aperture motor (AM) 104, and a focus motor (FM) 132. The fixed first group lens 102, the zoom lens 111, the aperture 103, the fixed third group lens 121, and the focus lens 131 constitute a photographing optical system. Other lenses, such as a lens for image blur correction, may be included. For convenience, the lenses 102, 111, 121, and 131 are illustrated as a single lens, but each may be composed of multiple lenses. The photographing lens 101 may be configured as an interchangeable lens that is removable from the digital camera 100.

[0014] The aperture control unit 105 controls the operation of the aperture motor 104 according to commands from the main control unit 151 , and changes the opening diameter of the aperture 103 . The zoom control unit 113 controls the operation of the zoom motor 112 according to commands from the main control unit 151 , and changes the focal length (angle of view) of the photographic lens 101 .

[0015] The focus control unit 133 calculates the defocus amount and defocus direction of the photographing lens 101 based on, for example, the phase difference between a pair of focus detection signals obtained from the image sensor 141. Then, the focus control unit 133 converts the defocus amount and defocus direction into a drive amount and drive direction of the focus motor 132. Based on the drive amount and drive direction, the focus control unit 133 controls the operation of the focus motor 132 and drives the focus lens 131, thereby controlling the focus state of the photographing lens 101.

[0016] In this way, the focus control unit 133 performs autofocus detection (AF) using the phase difference detection method, but the focus control unit 133 may also perform AF using the contrast detection method based on a contrast evaluation value of an image signal obtained from the image sensor 141. Also, the focus control unit 133 may perform AF using the phase difference detection method using a focus detection signal obtained from an AF sensor provided separately from the image sensor 141. Note that in the AF operation in the focus control unit 133, a focus detection area can be set in the area of ​​the main subject detected by the image processing unit 152 described later.

[0017] The subject image formed on the imaging plane of the image sensor 141 by the photographing lens 101 is converted into an electrical signal (image signal) by the photoelectric conversion element of each of the multiple pixels arranged on the image sensor 141 .

[0018] The imaging element 141 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element 141 may also be configured to be movable in a direction perpendicular to the optical axis and around the optical axis for image blur correction. The imaging element 141 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel accumulates electric charge according to the amount of incident light by photoelectric conversion. A group of pixel signals (analog image signals) representing the subject image formed on the imaging surface is obtained by reading out from each pixel a signal having a voltage according to the amount of electric charge accumulated during the exposure period.

[0019] In this embodiment, the pixel array of the image sensor 141 has m pixels arranged in the horizontal direction and n pixels arranged in the vertical direction (n and m are plural), and each pixel has two photoelectric conversion elements (photoelectric conversion regions). The sensor control unit 143 controls the signal reading from the image sensor 141 according to instructions from the main control unit 151.

[0020] The two photoelectric conversion areas of each pixel are called area A and area B, and an image made up of the image signals read out from area A of each pixel is called image A, and an image made up of the image signals read out from area B of each pixel is called image B. Additionally, an image obtained by adding images A and B on a pixel-by-pixel basis is called image A+B. Images A and B form a parallax image pair. Image A+B is used for display and recording. Images A and B are also used to generate focus detection signals used in phase difference detection AF and to generate distance maps.

[0021] The analog image signal read out from the imaging element 141 is supplied to a signal processing unit 142. The signal processing unit 142 applies signal processing such as noise reduction processing, A / D conversion processing, and automatic gain control processing to the image signal, and outputs the result as image data to a sensor control unit 143. The sensor control unit 143 accumulates the image data received from the signal processing unit 142 in a RAM (random access memory) 154.

[0022] When recording image data stored in RAM 154, main control unit 151 generates a data file according to the recording format by, for example, adding a predetermined header to the image data. At this time, main control unit 151 encodes the image data in compression / decompression unit 153 as necessary, and stores the encoded data in a data file. Main control unit 151 records the generated data file on recording medium 157, such as a memory card.

[0023] Furthermore, when displaying image data stored in RAM 154, main control unit 151 scales the image data in image processing unit 152 so as to fit the display size of display unit 150, and then writes the image data to an area of ​​RAM 154 used as a video memory (VRAM area). Display unit 150 reads out image data for display from the VRAM area of ​​RAM 154, and displays it on a display device such as an LCD or an organic EL display. Display unit 150 also displays the detection result of the main subject (moving object) detected by image processing unit 152 (such as a frame indicating the main subject area).

[0024] When shooting a video (in standby mode or while recording a video), digital camera 100 causes display unit 150 to function as an electronic viewfinder (EVF) by instantly displaying the shot video on display unit 150. The video and its frame images displayed when display unit 150 is functioning as an EVF are called live view images or through images. When digital camera 100 shoots a still image, it displays the still image shot immediately before on display unit 150 for a certain period of time so that the user can check the shooting result. These display operations are also realized under the control of main control unit 151.

[0025] The compression / decompression unit 153 encodes and decodes image data. For example, when recording still images or moving images, the image data and audio data are encoded by a predetermined encoding method. When playing back still image data files or moving image data files recorded on the recording medium 157, the compression / decompression unit 153 decodes the encoded data and stores it in the RAM 154.

[0026] The RAM 154 is used as a system memory for executing programs, a video memory, a buffer memory, and the like. The ROM 155 stores programs executable by the processor of the main control unit 151, various setting values, information specific to the digital camera 100, GUI data, etc. The ROM 155 may be electrically rewritable.

[0027] In this embodiment, it is assumed that a priority movement direction is associated with each type of photographic scene and is registered in advance in the ROM 155, for example. For example, in a basketball scene, the up and down direction is registered as the priority direction of movement. This is because in a basketball scene, a player shooting or a player jumping or crouching (before jumping) near the goal post is considered to be the main subject, and images are recorded or played focusing on players involved in the shooting or offense and defense near the goal post. The same is true for sports similar to basketball, such as handball. In field sports such as high jump, long jump, and hurdles, the priority movement direction can be set to diagonally upward. The priority movement direction can be determined not only for sports but also for other scenes, taking into account the posture of the subject to be focused on. Furthermore, when the movement direction is not a simple one, the preferential movement direction may be learned by actually capturing an image of a subject moving in the preferential movement direction.

[0028] Operation unit 156 is a general term for input devices (buttons, switches, dials, etc.) provided for the user to input various instructions to digital camera 100. The input devices constituting operation unit 156 have names according to the functions assigned to them. For example, operation unit 156 includes a release switch, a video recording switch, a shooting mode selection dial for selecting a shooting mode, a menu button, directional keys, an enter key, etc. In this embodiment, it is assumed that the shooting mode selection dial can be used to select a shooting mode for shooting a specific scene.

[0029] The release switch is a switch for recording still images, and the main control unit 151 recognizes the half-pressed state of the release switch as an instruction to prepare for shooting, and the full-pressed state as an instruction to start shooting. The main control unit 151 also recognizes the video recording switch pressed in a shooting standby state as an instruction to start recording a video, and recognizes the switch pressed during video recording as an instruction to stop recording. Note that the functions assigned to the same input device may be variable. The input device may be a software button or key using a touch display. The operation unit 156 may also include an input device compatible with a non-contact input method, such as voice input or gaze input. Furthermore, it may also include a receiver that accepts instructions from an external device, such as a remote control.

[0030] The main control unit 151 has one or more programmable processors such as a CPU or MPU, and controls each unit by loading a program stored in, for example, ROM 155 into RAM 154 and executing it, thereby realizing the functions of the digital camera 100. The main control unit 151 also executes AE processing that automatically determines exposure conditions (shutter speed or accumulation time, aperture value, sensitivity) based on information about the brightness of the subject. Information about the brightness of the subject can be obtained, for example, from the image processing unit 152. The main control unit 151 can also determine the exposure conditions based on brightness information of an area of ​​a specific subject, such as a person's face.

[0031] During video shooting, the main control unit 151 fixes the aperture 103 and controls the exposure by the electronic shutter speed (accumulation time) and the magnitude of the gain. The main control unit 151 notifies the sensor control unit 143 of the determined accumulation time and the magnitude of the gain. The sensor control unit 143 controls the operation of the image sensor 141 so that shooting is performed according to the notified exposure conditions.

[0032] The distance map generating unit 161 generates a distance map using image data stored in the RAM 154, for example. The distance map is information showing the distribution of distances (subject distances) from the digital camera 100 to objects present within the shooting range. The distance map may be in the form of a two-dimensional image in which the brightness value represents the subject distance, and may be called a depth map distance image, depth image, or the like. The distance map can be generated by a known method. For example, the distance map generating unit 161 can obtain the defocus amount (the amount of deviation from the in-focus position of the focus lens and the direction of deviation) at each pixel position from the image deviation amount of the parallax image (the above-mentioned A image and B image). Since the defocus amount represents the amount of deviation of the focus based on the current subject distance, the defocus amount can be regarded as distance information. Of course, the in-focus position of the focus lens may be obtained based on the defocus amount, and the subject distance corresponding to the in-focus position may be obtained. Note that the digital camera 100 may be a multi-lens camera such as a stereo camera to obtain the parallax image, or the parallax image may be obtained from a storage medium or an external device.

[0033] The distance map can also be generated without using parallax images. The focus lens position at which the contrast evaluation value is maximized can be obtained for each pixel, thereby obtaining the subject distance for each pixel. Distance information for each pixel can also be obtained based on the correlation between the amount of blur and distance from image data obtained by photographing the same scene multiple times with different focal distances and the point spread function (PSF) of the optical system. The distance map generating unit 161 may generate a distance map for the entire image, or may generate a distance map only for a partial region of the image that is necessary for moving object detection. The distance map generating unit 161 stores the generated distance map in the RAM 154. The image processing unit 152 refers to the distance map.

[0034] Furthermore, the distance map generating unit 161 can calculate the reliability for each region in the distance map and store it together with the distance map. There is no particular limit to the method of calculating the reliability. For example, when generating a distance map using a parallax image, in order to obtain the image shift amount between the parallax images, a correlation amount (similarity) such as SAD is calculated while changing the relative shift amount, and the shift amount at which the correlation is maximized (the correlation amount is minimized) is detected as the image shift amount. It is considered that the reliability of the detected image shift amount (defocus amount) is higher as the difference between the average value and the maximum value of the calculated correlation amount is larger. Therefore, the difference between the average value and the maximum value of the correlation amount calculated at each pixel position can be used as the pixel position reliability. Note that the distance map generating unit 161 may generate the distance map by obtaining the subject distance and its reliability for each small region (pixel block) instead of for each pixel.

[0035] The scene analysis unit 162 determines the type of photographic scene based on information (e.g., one or more of size, position, and movement) about the subject area detected by the image processing unit 152. The scene analysis unit 162 may use the distance map generated by the distance map generation unit 161 to determine the type of photographic scene. For example, information about the subject area and / or characteristics of the distance map that should be satisfied are registered in advance for each type of photographic scene to be determined. The scene analysis unit 162 may determine the photographic scene using a trained neural network prepared for each type of photographic scene. The scene analysis unit 162 notifies the main control unit 151 of the determined type of photographic scene.

[0036] The motion sensor 163 generates a signal according to the motion of the digital camera 100. The motion sensor 163 may be, for example, a combination of an acceleration sensor that outputs a signal according to the motion in each of the X, Y and Z axial directions and a gyro sensor that outputs a signal according to the motion around each axis. Optical image stabilization can be realized by moving the correction lens and / or the imaging element 141 so as to cancel out the motion of the digital camera 100 represented by the signal output by the motion sensor 163. Also, electronic image stabilization can be realized by cropping the image so as to cancel out the motion of the digital camera 100.

[0037] Image processing unit 152 applies predetermined image processing to image data stored in RAM 154, generates signals and image data according to the application, and acquires and / or generates various information. Image processing unit 152 outputs the acquired or generated information and data to main control unit 151, RAM 154, etc. according to the application.

[0038] The image processing applied by the image processing unit 152 can include, for example, pre-processing, color interpolation processing, correction processing, detection processing, data processing, evaluation value calculation processing, special effect processing, and the like. Pre-processing may include signal amplification, reference level adjustment, defective pixel correction, etc. Color interpolation processing is performed when a color filter is provided on an image sensor, and is a process for interpolating values ​​of color components that are not included in the individual pixel data that constitutes the image data. Color interpolation processing is also called demosaic processing. The correction processing can include white balance adjustment, gradation correction, correction of image degradation caused by optical aberration of the photographing lens 101 (image restoration), correction of the effects of peripheral light falloff of the photographing lens 101, color correction, and the like. The detection process may include detection of areas containing specific subjects (e.g., facial and body areas of people or animals, vehicles (two-wheeled vehicles, four-wheeled vehicles), aircraft, sports equipment and facilities (balls, rackets, nets, goals, etc.)) and their movements, as well as person recognition processing. The data processing may include processing such as cutting out an area (trimming), synthesis, scaling, etc. As the data processing, the image processing unit 152 may generate display image data or recording image data instead of the main control unit 151. The evaluation value calculation process can include processes such as generation of a signal and evaluation value used in automatic focus detection (AF) and generation of an evaluation value used in automatic exposure control (AE). The special effects processing may include adding a blur effect, changing color tones, relighting, and the like. Note that these are merely examples of processes that the image processing unit 152 can apply, and do not limit the processes that the image processing unit 152 can apply.

[0039] Information about the detected subject area may be used for other image processing (for example, white balance adjustment processing, generation processing of luminance information of the subject, etc.). When the focus control unit 133 performs AF using a contrast detection method, the image processing unit 152 can generate a contrast evaluation value and supply it to the focus control unit 133. The image processing unit 152 stores the processed image data, information about the subject area, and the like in the RAM 154.

[0040] Fig. 2 is a block diagram showing a schematic diagram of the process of determining the main subject by image processing unit 152. Each block in Fig. 2 corresponds to a main processing step executed by image processing unit 152 when determining the main subject.

[0041] Image processing unit 152 applies a main subject determination process to the input image data. The input image data is video data, and may be data generated in parallel with the imaging by image sensor 141, or data read from recording medium 157. Note that image processing unit 152 may reduce the resolution of the input image data or convert it to a grayscale image before inputting it to subject detection unit 210 in order to reduce the processing load.

[0042] The subject detection unit 210 detects a region (subject region) that is included in the image represented by the input image data and is likely to include a specific subject. The subject region can be detected using any known method such as template matching or machine learning, and therefore detailed description thereof will be omitted.

[0043] The type of subject for which the subject detection unit 210 detects the subject area may be fixed, or may change depending on, for example, the set shooting mode. Basically, the area of ​​the human subject (face area, pupil area, torso area, etc.) is always detected. Note that the detection of the subject area may be performed for each frame of the video data, or may be performed for each predetermined number of frames.

[0044] The subject detection unit 210 outputs the total number of detected subject regions, the position, size, detection reliability, etc. of each subject region as detection results for each type of subject to the motion detection unit 220. The subject detection unit 210 also outputs the detection results to the scene analysis unit 162.

[0045] The motion detection unit 220 obtains a motion vector for each object region based on the detection result of the object region. The motion vector can be obtained by any known method, such as using the object region as a template and searching for an area in the input image data that has the highest correlation with the template. Note that since the motion vector indicates the direction of movement of the object region within the screen, the motion vector of object regions with the same movement direction and amount within the screen is the same whether the image is shot in the horizontal position or the vertical position.

[0046] Note that motion detection section 220 can update the template using the detection result of the subject region for every predetermined number of frames. Motion detection section 220 outputs information on the motion vector found for each subject region to main subject determination section 230.

[0047] Main subject determination section 230 determines a main subject region from among the detected subject regions, based on the motion vector determined by motion detection section 220 and the priority motion direction provided by main control section 151. Main subject determination section 230 outputs information (at least the position and size) of the determined main subject region to main control section 151.

[0048] The priority movement direction is obtained, for example, by the main control unit 151 from the ROM 155 and supplied to the image processing unit 152 based on the type of shooting scene determined by the scene analysis unit 162 or the type of shooting scene that can be specified from the set shooting mode.

[0049] The main subject determination section 230 determines, as the main subject region, a subject region having a motion vector having the same direction as the priority motion direction (or a directional difference less than a threshold value) among the motion vectors found by the motion detection section 220. Multiple main subject regions may be determined. When there are multiple subject regions having motion vectors having the same direction as the priority motion direction (or a directional difference less than a threshold value), the main subject region may be determined taking into consideration the positional relationship with other types of subject regions. For example, when a ball subject or a goal subject is detected, a human subject region having a distance closest to the ball subject or the goal subject or less than a threshold value may be determined as the main subject region.

[0050] In addition, when subject regions are detected for multiple types of subjects, the main subject determination unit 230 can determine the main subject region from the subject region for a specific type of subject (e.g., a human) that is predetermined according to the type of shooting scene.

[0051] Main control unit 151 can use the main subject information obtained from image processing unit 152 for various processes. For example, if a video is being captured, the main control unit 151 Set the focus detection area to the main subject area, The angle of view of the photographic lens 101 is changed (zoomed up) so as to enlarge the main subject area, or the angle of view of the photographic lens 101 is changed (zoomed down) so that the entire subject area fits within the angle of view, and / or Change the shooting direction so that the main subject area is closer to the center of the shooting range. It is possible.

[0052] When there are multiple main subject regions, main control unit 151 can execute these controls by regarding the smallest region that encompasses all the main subject regions as one main subject region.

[0053] If the lens barrel of the lens unit 101 is configured to be rotatable in the pan and tilt directions, the shooting direction can be changed by changing the direction of the optical axis of the lens unit 101. If the lens barrel of the lens unit 101 is fixed, the shooting direction can be changed by driving (moving) the configuration for image blur correction (correction lens and / or image sensor). In this case, the main control unit 151 can change the shooting direction by regarding the motion vector of the main subject area as the movement of the digital camera 100 and executing image blur correction.

[0054] When digital camera 100 is not stationary or fixed, the shooting direction can be changed by determining the amount of image shake correction (the amount of drive of the image shake correction mechanism) in consideration of the movement of digital camera 100 itself obtained from sensor 163. For example, when trying to change the shooting direction to the left, if digital camera 100 is moving leftward, the amount of correction is decreased, and if digital camera 100 is moving rightward, the amount of correction is increased.

[0055] In addition, if a video is being played, the main control unit 151 Applying a process that reduces the sharpness or brightness of areas other than the main subject area or areas surrounding the main subject area, - Change the crop of the image to enlarge the main subject area or to fit the entire subject area within the field of view, and / or Change the cropping position of the image so that the main subject area is closer to the center of the image In this way, the image processing unit 152 can be controlled. Note that the image processor 152 may apply scaling to match the resolution of the cropped image to the display resolution.

[0056] In order to prevent degradation of video quality due to the presence or absence of a main subject area or switching between them, main control unit 151 may track a main subject area once determined for at least a predetermined time period. Alternatively, when a state where there is a main subject area changes to a state where there is no main subject area, main control unit 151 may gradually widen the angle of view of lens unit 101 to suppress a sudden change in the angle of view.

[0057] The main subject determination process is particularly useful when taking pictures automatically with digital camera 100 attached to the user's head or a tripod, rather than when the user takes pictures while holding digital camera 100. Furthermore, the user can set whether or not to enable the main subject determination process by operating a menu screen, for example.

[0058] Next, a video shooting operation using the above-mentioned main subject determination process will be described with reference to the flowchart shown in Fig. 3. Here, video shooting for recording is assumed, not video shooting for live view display.

[0059] For example, when an instruction to start recording a moving image is input from a remote control, main control unit 151 controls each unit to execute operations for recording a moving image. This starts shooting a moving image, generating moving image data for recording, and recording it. Since the same processes as in the past may be executed except for the process related to the main subject determination, the following description will focus on the process related to the main subject determination.

[0060] When video shooting starts, scene determination processing by scene analysis unit 162 also starts in S302. Note that if the shooting scene can be identified from settings for digital camera 100, such as the shooting mode, scene determination by scene analysis unit 162 does not need to be performed. When main control unit 151 has identified the scene being shot, it refers to ROM 155 and acquires the priority movement direction stored in association with the identified shooting scene. Main control unit 151 supplies the priority movement direction to image processing unit 152 (main subject determination unit 230) and then executes S303.

[0061] In S303, the main control unit 151 checks the angle of view (focal length) of the lens unit 101, and adjusts the angle of view so that, for example, the shooting range is maximized. Note that it is not necessarily required to set the angle of view to the maximum, and when the focal length of the shooting lens 101 is equal to or greater than a predetermined threshold, it may be adjusted to any focal length less than the threshold. The angle of view is widened in order to enable detection of a larger number of subject areas.

[0062] In S304, the image processing unit 152 (subject detection unit 210) detects a subject area within the captured frame image.

[0063] In S305, image processing unit 152 (motion detection unit 220) finds a motion vector for each of the detected subject regions.

[0064] In S306, image processing unit 152 (main subject determination unit 230) determines, as the main subject region, a subject region having a motion vector having the same direction as the priority motion direction or a directional difference less than a threshold value.

[0065] In S307, image processing section 152 (main subject determining section 230) outputs main subject information to main control section 151 regardless of the presence or absence of a determined main subject region.

[0066] Since the above is processing for one frame of a video, the processing from S304 onwards continues. When the angle of view of the photographing lens 101 is adjusted according to the main subject information, the subject area is detected within the range of the adjusted angle of view. When the main subject area goes from being detected to not being detected, the main control unit 151 widens the angle of view of the lens unit 101, so the range in which the subject area is detected becomes wider even without executing S303.

[0067] The effect of the present invention will be described with reference to FIG. FIG. 4 is a schematic diagram showing a portion of a sports video captured by a camera. Figures 4(a) and 4(b) are schematic diagrams showing an example of a basketball scene. Here, it is assumed that the scene in Figure 4(a) has changed to the scene in Figure 4(b).

[0068] In Fig. 4, 501A and 502A are players of team A, and 501B, 502B, and 503B are players of team B. 402 shows a schematic representation of the motion vectors found for the areas of each player. 403 shows a schematic representation of the preferred motion direction (up and down) for the basketball scene. The x-axis shows the left and right direction, and the y-axis shows the up and down direction.

[0069] In the scene in FIG. 4(a), all the players are moving to the right. In this case, in the method proposed in Patent Document 1, player 501B, who is at the front in the direction of movement, is recognized as the main subject. For example, when controlling to zoom in on the main subject, even if player 501B is shot up, the intention of the image will be unclear. In the main subject determination process of this embodiment, the main subject area is not determined in a scene such as FIG. 4(a), so the whole subject is photographed, and a more appropriate image can be recorded than if player 501B was shot up close.

[0070] In FIG. 4(b), player 501A is in a shooting position, and player 501B is in a position to block the shot, and both are in the midst of a jump. In this case, the motion vector for the areas of players 501A and 501B is pointing upward. As a result, the areas of players 501A and 501B are determined as the main subject area. 404 is a mark indicating the subject area determined as the main subject. Main control unit 151 may superimpose mark 404 on the live view image based on the main subject information.

[0071] The priority motion direction may be selectable by the user from among predetermined directions. When the user selects the priority motion direction, scene determination is not necessary. In addition, the priority motion direction may be dynamically determined according to, for example, the ratio or difference between the number of vertical motion vectors and the number of horizontal motion vectors among the motion vectors obtained by the motion detection unit 220. For example, the direction indicated by the motion vector with the smallest ratio or number may be set as the priority motion direction.

[0072] As described above, according to this embodiment, among the subject regions detected in an image, the subject region moving in a predetermined priority motion direction is determined as the main subject region. Therefore, by determining a characteristic motion direction in the scene to be photographed as the priority motion direction, the main subject region can be appropriately determined. The main subject determination process according to this embodiment is particularly useful in cases where photography is performed without the user operating the digital camera, such as automatic photography.

[0073] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0074] The disclosure of the present embodiment includes the following image processing device, imaging device, image processing method, and program. (Item 1) A subject detection means for detecting a subject area from an image; a motion detection means for determining a motion vector relating to the subject region; a determining means for determining, from among the subject regions, a subject region in which the direction indicated by the motion vector is a predetermined priority motion direction as a main subject region; 13. An image processing device comprising: (Item 2) 2. The image processing device according to item 1, wherein the priority movement direction is determined based on a user setting. (Item 3) 2. The image processing device according to item 1, wherein the priority motion direction is determined based on the number of motion vectors indicating a first direction and the number of motion vectors indicating a second direction. (Item 4) The method further comprises: determining a scene of the image; 2. The image processing device according to item 1, wherein the priority movement direction is determined according to a determined scene. (Item 5) further comprising image processing means for processing the image using information on the main subject area; 5. The image processing device according to any one of items 1 to 4, wherein the image processing means reduces sharpness or brightness of a region in the image that is not the main subject region. (Item 6) further comprising image processing means for processing the image using information on the main subject area; 6. The image processing device according to any one of items 1 to 5, wherein the image processing means changes a cropping range of the image so as to enlarge the main subject area. (Item 7) further comprising image processing means for processing the image using information on the main subject area; 7. The image processing device according to any one of items 1 to 6, wherein the image processing means changes a cropping range of the image so that the main subject area approaches the center of the image. (Item 8) An imaging means; The image processing device according to any one of items 1 to 7, which uses an image obtained by the imaging means; An imaging device comprising: (Item 9) An imaging means; The image processing device according to item 1, which uses an image obtained by the imaging means; having 11. An imaging apparatus, comprising: an imaging device, the imaging device being characterized in that the priority movement direction is determined based on a setting of an imaging mode for capturing an image of a specific photographic scene. (Item 10) An imaging means; The image processing device according to item 1, which uses an image obtained by the imaging means; a control means for controlling the imaging means based on information on the main subject area; An imaging device comprising: (Item 11) 11. The imaging apparatus according to item 10, wherein the control means sets a focus detection area in the main subject area. (Item 12) 11. The imaging apparatus according to item 10, wherein the control means changes the angle of view of the imaging means so as to enlarge the main subject area. (Item 13) 11. The imaging device according to item 10, wherein the control means changes the imaging direction of the imaging means so that the main subject area approaches the center of the imaging range. (Item 14) 14. The imaging device according to item 13, wherein the control means changes the photographing direction by changing the direction of the optical axis of the imaging means. (Item 15) Item 14. The imaging apparatus according to item 13, wherein the control means changes the shooting direction by driving an image blur correction mechanism included in the imaging means. (Item 16) Item 16. The imaging device according to item 15, wherein the control means determines the drive amount of the image stabilization mechanism in consideration of the movement of the imaging device. (Item 17) An image processing method executed by an image processing device, comprising: Detecting a subject region from an image; determining a motion vector for the object region; determining, as a main subject region, a subject region in which the direction indicated by the motion vector is a predetermined priority motion direction, from among the subject regions; 13. An image processing method comprising: (Item 18) A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 7. (Item 19) 17. A program for causing a computer included in an imaging apparatus to function as an image processing device and a control unit included in the imaging apparatus according to any one of items 8 to 16.

[0075] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0076] 100: digital camera, 101: lens unit, 115: main control unit, 141: image sensor, 152: image processing unit, 162: scene analysis unit

Claims

1. a subject detection means for detecting a subject area from an image; a motion detection means for determining a motion vector relating to the subject region; a determining means for determining, from among the subject regions, a subject region in which the direction indicated by the motion vector is a predetermined priority motion direction as a main subject region; 1. An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the priority direction of movement is determined based on a setting by a user.

3. 2. The image processing device according to claim 1, wherein the priority motion direction is determined based on the number of motion vectors indicating a first direction and the number of motion vectors indicating a second direction.

4. further comprising a determination means for determining a scene of the image; 2. The image processing device according to claim 1, wherein the priority movement direction is determined according to a determined scene.

5. further comprising image processing means for processing the image using information on the main subject area; 2. The image processing apparatus according to claim 1, wherein the image processing means reduces sharpness or brightness of an area other than the main subject area in the image.

6. further comprising image processing means for processing the image using information on the main subject area; 2. The image processing apparatus according to claim 1, wherein said image processing means changes a cropping range of said image so as to enlarge said main subject area.

7. further comprising image processing means for processing the image using information on the main subject area; 2. The image processing apparatus according to claim 1, wherein said image processing means changes a cropping range of said image so that said main subject area approaches the center of the image.

8. 2. The image processing device according to claim 1, wherein the determining means determines, as the main subject area, a subject area in which a difference between a direction indicated by the motion vector and the predetermined priority motion direction is equal to or smaller than a threshold value.

9. An imaging means; an image processing device according to any one of claims 1 to 7, which uses an image obtained by the imaging means; An imaging device comprising:

10. An imaging means; an image processing device according to claim 1, which uses an image obtained by the imaging means; and An imaging device, wherein the priority movement direction is determined based on a setting of an imaging mode for capturing a specific photographic scene.

11. An imaging means; an image processing device according to claim 1, which uses an image obtained by the imaging means; a control means for controlling the imaging means based on information about the main subject area; An imaging device comprising:

12. 12. The image pickup apparatus according to claim 11, wherein the control means sets a focus detection area in the main subject area.

13. 12. The imaging apparatus according to claim 11, wherein the control means changes the angle of view of the imaging means so as to enlarge the main subject area.

14. 12. The imaging apparatus according to claim 11, wherein the control means changes the imaging direction of the imaging means so that the main subject area approaches the center of an imaging range.

15. 15. The imaging device according to claim 14, wherein the control means changes the photographing direction by changing the direction of the optical axis of the imaging means.

16. 15. The imaging apparatus according to claim 14, wherein the control means changes the photographing direction by driving an image blur correction mechanism provided in the imaging means.

17. 17. The image pickup apparatus according to claim 16, wherein the control means determines the drive amount of the image stabilization mechanism in consideration of the movement of the image pickup apparatus.

18. An image processing method executed by an image processing device, Detecting a subject region from an image; determining a motion vector for the object region; determining, as a main subject area, a subject area in which the direction indicated by the motion vector is a predetermined priority motion direction, from among the subject areas; An image processing method comprising:

19. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 7.

20. 18. A program for causing a computer included in an imaging apparatus to function as the image processing device and control means included in the imaging apparatus according to claim 10.

Images