Image processing device, imaging apparatus, and control method for the same

Abstract

To solve the problem in which a user sometimes selects a subject that is unlikely to be selected when multiple types of detected subjects are selected while the user switches the main subject by means of an operating member that provides directional instructions.SOLUTION: An image processing device has acquisition means for sequentially acquiring images, detection means for detecting a plurality of subjects from the images acquired by the acquisition means, and main subject selection means for selecting a main subject from the plurality of subjects. The main subject selection means determines whether to permit reselection of any one of the plurality of subjects as a new main subject in accordance with whether an area of a subject of one of the plurality of subjects overlaps an area of another subject and a combination of the type of subject of one of the plurality of subjects and the type of the main subject.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, and a control method thereof that are capable of reselecting a main subject from among a plurality of subjects contained in an image. [Background technology]

[0002] There is a technique for changing a main subject among multiple subjects included in an image being captured. Patent Document 1 discloses that when multiple types of subjects can be detected, the main subject is changed by switching the type of subject to be selected. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-125735 Summary of the Invention [Problem to be solved by the invention]

[0004] In the method described in Patent Document 1, when multiple subjects of the same type are detected, the user cannot switch the intended subject. Also, when the user decides the main subject to be the AF target while reselecting the main subject with an operating member, the user may end up reselecting a subject that the user is unlikely to select as the main subject. [Means for solving the problem]

[0005] In order to solve the above-mentioned problems, the image processing device of the present invention has an acquisition means for sequentially acquiring images, a detection means for detecting multiple subjects from the images acquired by the acquisition means, and a main subject selection means for selecting a main subject from the multiple subjects, wherein the main subject selection means determines whether or not to allow re-selection of any of the multiple subjects as a new main subject depending on whether or not a subject area of ​​any of the multiple subjects overlaps with a subject area of ​​another subject and on a combination of a subject type of any of the multiple subjects and the type of the main subject. Effect of the Invention

[0006] According to the present invention, by determining whether or not to reselect a subject as a main subject depending on whether the subject to be reselected overlaps with another subject and the combination of the subject types of the subject to be reselected and the main subject, it is possible to prevent a subject that is unlikely to be selected by the user from being reselected as a main subject. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a diagram illustrating a configuration of a digital camera according to an embodiment of the present invention; [Diagram 2] FIG. 1 is a diagram illustrating a configuration of an image sensor according to an embodiment of the present invention; [Diagram 3] 1 is a flowchart showing an example of a flow for taking a still image with the digital camera of the present embodiment. [Figure 4] 1 is a flowchart showing a method for determining a main subject according to the present embodiment. [Diagram 5] 11 is a flowchart showing a main subject manual selection process according to the present embodiment. [Figure 6] 10 is a flowchart showing a process for determining a transition due to overlapping of subjects according to the present embodiment; [Figure 7] 1 is a table showing examples of combinations of main subjects and transitional subjects in this embodiment; [Figure 8] 11 is a flowchart showing a transition possibility determination based on the subject size in this embodiment; [Figure 9]FIG. 13 is a diagram illustrating an example of a display showing the main subject position in the present embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0009] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in a digital camera. However, the imaging function is not essential to the present invention, and the present invention can be implemented in any electronic device. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0010] <Camera configuration> FIG. 1 is a block diagram showing an example of the functional configuration of a digital camera according to a first embodiment of the present invention.

[0011] In Fig. 1, 100 is an imaging device (subject tracking device), 10 is a photographing lens, and 12 is a mechanical shutter equipped with an aperture function. 14 is an imaging element such as a CMOS sensor that converts an optical image into an electrical signal, and functions as an image acquisition means that successively acquires images of a subject. Also, 16 is an A / D converter that converts an analog signal output from the imaging element 14 into a digital signal.

[0012] Reference numeral 18 denotes a timing generation unit that supplies clock signals and control signals to the image sensor 14 and A / D converter 16, and is controlled by a memory control circuit 22 and a system control circuit 50. By controlling the reset timing of the image sensor 14 with the timing generation unit 18, it is possible to control the charge accumulation time, and it is possible to use it as an electronic shutter, separate from the mechanical shutter 12, for shooting moving images, etc.

[0013] The system control circuit 50 includes a CPU as a computer, and functions as a control unit that executes various operations of the entire apparatus based on computer programs stored in the non-volatile memory 31 .

[0014] An image processing circuit 20 performs pixel interpolation processing for image enlargement / reduction, color conversion processing, noise removal processing, edge emphasis processing, etc. on the data from the A / D converter 16 or the data from the memory control circuit 22. The image processing circuit 20 also functions as a detection means for detecting a specific subject from an image.

[0015] The image processing circuit 20 also has a face detection function that detects people, animals, and face regions by image recognition, a function that detects organs (parts) in the face such as the pupils, nose, and mouth, a whole body detection function that detects the whole body (whole object) of the subject, etc. Then, it performs calculation processing of the positions of the face, organs, and whole body from the results of the face detection, organ detection, and whole body detection.

[0016] In face detection and whole body detection in the image processing circuit 20, the shapes of the contours of the face and whole body are stored as feature data inside the image processing circuit 20, and an image area in the detected image that matches the feature data (template) is specified by pattern matching processing. In face detection, an image area that matches the feature data representing the shape of the face stored in advance inside the image processing circuit 20 is specified by pattern matching processing within the area found by whole body detection.

[0017] Also, the degree of match with feature data stored inside the image processing circuit 20 is calculated, and an area with a degree of match equal to or greater than a predetermined level is determined to be an area of ​​the face or the entire body.

[0018] Furthermore, in order to increase the number of opportunities to detect the face or the entire body and to improve the detection accuracy, pattern matching processing is performed using multiple feature data stored inside the image processing circuit 20. Note that pattern matching processing may be performed using feature data of only a portion of the shape of the face or the entire body. Furthermore, pattern matching processing may be performed by changing the size of the feature data in order to detect the face or the entire body regardless of its size.

[0019] Organ detection is performed by pattern matching processing to identify image areas that match feature data (templates) representing the shapes of organs that are stored in advance in the image processing circuit 20 within the areas found by face detection. The reliability of the organs is calculated by calculating the degree of match with the feature data.

[0020] As another detection method, detection by deep learning can also be performed. The image processing circuit 20 has a plurality of product-sum calculators and is also used as a processor that performs processing for deep learning.

[0021] The image processing circuit 20 applies subject detection processing to the image data using one learning model selected by the system control circuit 50 from among a plurality of learning models stored in a non-volatile memory 31, which will be described later.

[0022] Furthermore, the image processing circuit 20 may switch the learning models stored in the non-volatile memory 31 for one image data to perform a plurality of detection processes.

[0023] For example, the non-volatile memory 31 stores three learning models: a learning model capable of detecting parts of the body such as the eyes, face, and entire body of dogs and cats; a learning model capable of detecting parts of the body such as the eyes, face, and entire body of birds; and a learning model capable of detecting vehicles such as trains and cars.

[0024] The system control circuit 50 uses one of the three learning models and performs image detection processing in the image processing circuit 20 based on the learning model.

[0025] By the system control circuit 50 performing detection three times using three learning models for one image, it becomes possible to detect the pupils, faces, entire body parts, etc. of dogs and cats, the pupils, faces, entire body parts, etc. of birds, and train and car vehicles.

[0026] In this embodiment, the eyes, faces, entire body parts, etc. of dogs and cats, the eyes, faces, entire body parts, etc. of birds, and vehicles such as trains and cars are detected, but the subjects to be detected are not limited to these.

[0027] Also, as long as the subject can be detected, a method other than the method shown in this embodiment may be used.

[0028] The image processing circuit 20 also performs tracking processing between images during live view, etc. When the image processing circuit 20 detects a subject, it temporarily stores the subject in the memory 30 as a template of the detected subject.

[0029] In addition, based on the template information temporarily stored in the memory 30, the image processing circuit 20 searches for an area that matches the template from within the image generated at the next timing, and performs tracking processing on the matching area as the subject area.

[0030] In the tracking process, the image processing circuit 20 searches for an area that matches the template, and one method is to cut out the image for each area, calculate the difference between the image and the template, and determine the area with the least difference as the subject area. There is also a method to find the area based on the degree of match between the template and histogram, color data, etc., but other methods may be used as long as they can identify an area in the image that matches the template temporarily stored in memory 30.

[0031] Thus, in this embodiment, the image processing circuit 20 functions as a tracking means that tracks a specific subject (at least one part of the subject) detected from an image by comparing the images across a plurality of images sequentially acquired by the image acquisition means. The tracking means may be configured to include the system control circuit 50. The tracking means performs tracking based on a subject area corresponding to the specific subject (part of the subject) detected by the detection means.

[0032] When multiple subjects are detected, the system control circuit 50 determines the main subject from the subject selected by the user or from the size and position of the subject within the screen, and sets the area of ​​the main subject as the focus detection area described below.

[0033] In addition, when a specific subject (part) is no longer detected by the detection means, tracking of the specific subject (part) is continued by referring to the subject area in the image in which the specific subject (part) was detected.

[0034] When multiple subjects are detected, the system control circuit 50 determines the main subject from the subject selected by the user or from the size and position of the subject within the screen, and sets the area of ​​the main subject as the focus detection area described below.

[0035] When a subject detected during live view becomes undetectable, the system control circuit 50 switches to subject tracking processing using the image processing circuit 20, and sets the subject area determined by the subject tracking processing as the focus detection area. By performing subject tracking processing, it is possible to perform focus detection for a longer period of time on parts of the same subject, and to keep the subject in focus.

[0036] The system control circuit 50 executes focus detection processing if a part of the same subject is detected within the tracking duration time described below, and if tracking is no longer possible, sets a different part of the same subject or a different type of subject as the main subject (focus detection area).

[0037] That is, when the time during which the system control circuit 50 continues to track a part of the same subject has elapsed beyond the tracking duration period shown in FIG. 3, the system control circuit 50 sets a focus detection area to a different part of the same subject or a different type of subject as the main subject, as shown in FIG.

[0038] In addition, the image processing circuit 20 performs a predetermined calculation process using the captured image data for auto white balance (hereinafter referred to as AWB) processing. The obtained calculation result is calculated as a white balance (hereinafter referred to as WB) evaluation value. Also, color conversion of the image data is performed based on the calculated WB evaluation value.

[0039] Furthermore, the image processing circuit 20 performs a predetermined calculation process using the captured image data to calculate an AE evaluation value and an EF evaluation value for performing automatic exposure control (hereinafter referred to as AE) processing and strobe exposure control (hereinafter referred to as EF) processing.

[0040] Based on the obtained AE evaluation value and EF evaluation value, the system control circuit 50 controls the exposure control unit 40 and the flash 48 in accordance with a predetermined algorithm.

[0041] Reference numeral 22 denotes a memory control circuit, which controls the A / D converter 16, the timing generation circuit 18, the image processing circuit 20, the memory 30, the compression / decompression circuit 32, and the like.

[0042] The data from the A / D converter 16 is written into a memory 30 via an image processing circuit 20 and a memory control circuit 22 , or the data from the A / D converter 16 is written directly into a memory 30 via the memory control circuit 22 .

[0043] Reference numeral 28 denotes an image display unit consisting of a TFT, an LCD, etc., and the image data for display written in the memory 30 is displayed by the image display unit 28 under the control of the memory control circuit 22. By sequentially displaying captured image data using the image display unit 28, an electronic viewfinder function can be realized.

[0044] Furthermore, the image display unit 28 can turn the display ON / OFF according to an instruction from the system control circuit 50, and when the display is turned OFF, the power consumption of the imaging device 100 can be significantly reduced.

[0045] Reference numeral 30 denotes a memory for temporarily storing captured still images and moving images, and has a memory capacity sufficient to store a predetermined number of still images and a predetermined length of moving images. This allows a large amount of images to be written to the memory 30 at high speed even in the case of continuous shooting in which multiple still images are continuously shot. The memory 30 can also be used as an area for temporarily storing feature data for authentication and as a working area for the system control circuit 50.

[0046] Reference numeral 31 denotes a non-volatile memory configured with a Flash ROM or the like. The program code executed by the system control circuit 50 is written in the non-volatile memory 31, and the system control circuit 50 executes various processes while sequentially reading out the program code. The non-volatile memory also includes an area for storing face feature data for authentication as dictionary data, an area for storing system information, an area for storing user setting information, and the like, so that various information and settings can be read out and restored the next time the system is started.

[0047] Reference numeral 32 denotes a compression / expansion unit that compresses and expands image data using adaptive discrete cosine transform (ADCT) or the like, reads an image stored in memory 30, performs compression or expansion processing, and writes the processed data back into memory 30.

[0048] Reference numeral 40 denotes an exposure control unit that controls the mechanical shutter 12 having an aperture function, and is also capable of performing a flash light control function using a flash 48 .

[0049] Reference numeral 42 denotes a focus control unit that controls focusing of the photographic lens 10, and 44 denotes a zoom control unit that controls zooming of the photographic lens 10. Reference numeral 48 denotes a flash, which has a flash light control function.

[0050] Based on the results of calculations performed by the image processing circuit 20 on the captured image data, the system control circuit 50 controls the exposure control unit 40 and the focus control unit 42.

[0051] The system control circuit 50 also performs autofocus (hereinafter referred to as AF) processing based on pixel data for phase difference detection obtained from the image sensor 14. Here, AF refers to automatic focus detection that sets a main subject area selected by the user or a main subject area automatically set by the camera as a focus detection area and automatically detects the focus position.

[0052] The image sensor 14 is composed of a C-MOS sensor and its peripheral circuits, with one photoelectric conversion element arranged on each of m horizontal and n vertical light receiving pixels. The image sensor 14 is configured to enable independent output from all pixels. In addition, some of the pixels are focus detection pixels, enabling AF using the image sensor phase difference detection method (image sensor phase difference AF).

[0053] Each of the focus detection pixels 251 is configured to receive a light beam that passes through one of a pair of different exit pupil regions of the photographing optical system.

[0054] The imaging pixels 250 each receive a light beam that passes through the entire area of ​​the exit pupil of the imaging optical system that forms an image of the subject, and generate an image of the subject.

[0055] In front of the imaging pixels 250, a color filter having, for example, a Bayer array is disposed.

[0056] 2, in the image sensor 14, among the 2 rows x 2 columns of pixels belonging to the first and second rows, a pair of diagonally arranged G pixels (Gr pixel and Gb pixel) are configured as imaging pixels, and the B pixels are replaced with focus detection pixels. The 2 rows x 2 columns of pixels belonging to the fifth and sixth rows are configured in the same way.

[0057] The image sensor 14 has a plurality of imaging pixels 250, each of which receives a light beam that passes through the entire area of ​​an exit pupil of an imaging optical system that forms an image of a subject, to generate an image of the subject. The image sensor 14 also has a plurality of focus detection pixels 251, each of which receives a light beam that passes through a different area of ​​the exit pupil of the imaging optical system. The plurality of focus detection pixels as a whole can receive a light beam that passes through the entire area of ​​the exit pupil of the imaging optical system. For example, of the 2 rows x 2 columns of pixels in the image sensor 14, a pair of diagonally arranged G pixels remain as imaging pixels, and the R pixel and B pixel are replaced with focus detection pixels.

[0058] The system control circuit 50 performs focus detection processing using a phase difference AF method using image signals from focus detection pixels that are discretely arranged in the image sensor 14. That is, a pair of image signals is formed from a plurality of focus detection pixels by a light beam passing through a pair of pupil regions of the image pickup optical system. The focus detection pixel group in the second row of FIG. 2 forms, for example, a right-eye image signal, and the focus detection pixel group in the sixth row of FIG. 2 forms, for example, a left-eye image signal. Then, the amount of deviation between the pair of images is obtained by performing correlation calculation of the pair of image signals. Since the amount of deviation changes depending on the distance to the subject, focus detection is performed based on the amount of deviation.

[0059] In this embodiment, imaging surface phase difference AF is realized by replacing a part of the imaging pixel array arranged on the imaging surface with focus detection pixels 251, but the present invention is not limited to this method and any configuration capable of focus detection may be used. For example, phase difference focus detection using a known dedicated focus detection sensor or known contrast focus detection may be used.

[0060] Reference numerals 60, 62, 66, 70 and 72 constitute operating means for inputting various operational instructions to the system control circuit 50, and are comprised of one or a combination of switches, dials, touch panels, pointing by line of sight detection, voice recognition devices, etc.

[0061] 60 is a mode dial switch that can be used to switch between various function modes such as power off, automatic shooting mode, shooting mode, panoramic shooting mode, video shooting mode, playback mode, and PC connection mode.

[0062] Reference numeral 62 denotes a two-stroke (SW1, SW2) shutter switch, and SW1 turns ON while the shutter switch 62 is pressed, starting operations such as AF (autofocus) processing, AE (auto exposure) processing, and AWB (auto white balance) processing. In other words, the user can issue autofocus instructions and the like via the shutter switch 62.

[0063] When the shutter switch 62 is pressed all the way down, the shutter switch SW2 is turned on and still image shooting begins. In the case of flash shooting, after a flash pre-emission process for EF is performed, the mechanical shutter 12 is operated to expose the image sensor 14 for the exposure time determined by the AE process.

[0064] A flash is emitted during this exposure period, and when the exposure period ends, the exposure control unit 40 blocks the mechanical shutter 12 from light, thereby ending the exposure of the image sensor 14.

[0065] The signal read from the image sensor 14 is written as image data into the memory 30 via the A / D converter 16 and the memory control circuit 22, and development processing is performed using calculations in the image processing circuit 20 and the memory control circuit 22. In addition, the image data is read from the memory 30 and compressed in the compression / expansion unit 32.

[0066] Thereafter, a recording process is performed in which image data is written onto the recording medium 200. This series of operations of still image shooting and recording process is performed when the shutter switch SW2 is turned ON.

[0067] Reference numeral 66 denotes a display changeover switch that can change over (ON / OFF) the display of the image display unit 28. With this function, when taking a photograph using the optical viewfinder 104, it is possible to conserve power by turning off the image display unit 28, which is made up of an LCD or the like.

[0068] Reference numeral 70 denotes an operation section consisting of various buttons, a touch panel, a rotary dial, etc., including a menu button, a set button, a macro button, a multi-screen playback page change button, a flash setting button, and a single shot / continuous shot / self-timer switching button.

[0069] The operation unit 70 also includes a menu movement + (plus) button, a menu movement - (minus) button, a playback image movement + (plus) button, a playback image - (minus) button, a shooting image quality selection button, an exposure compensation button, a date / time setting button, and the like.

[0070] Reference numeral 72 denotes a zoom switch as a zoom operation means for instructing the user to change the magnification of a captured image. This zoom switch 72 includes a tele switch for changing the imaging angle of view to the telephoto side, and a wide switch for changing it to the wide-angle side. Using the zoom switch 72 triggers an optical zoom operation by instructing the zoom control unit 44 to change the imaging angle of view of the photographing lens 10. It also triggers image cropping by the image processing circuit 20, electronic zooming changes to the imaging angle of view by pixel interpolation processing, etc.

[0071] Reference numeral 86 denotes a power supply means consisting of a primary battery such as an alkaline battery, a secondary battery such as a Li-ion battery, an AC adapter, or the like.

[0072] Reference numeral 90 denotes an interface with a recording medium such as a memory card or a hard disk, and reference numeral 92 denotes a connector for electrically connecting with a recording medium 200 such as a memory card or a hard disk.

[0073] Reference numeral 104 denotes an optical finder, which is provided separately from the image display unit 28, and enables shooting to be performed using only the optical finder.

[0074] A communication unit 110 has various communication functions such as USB, IEEE1394, LAN, wireless communication, etc., and may also have a GPS built in. The communication unit 110 is capable of receiving various radio waves via a connector (antenna) 112.

[0075] Reference numeral 115 denotes a gyro sensor, which is an angular velocity detection device, and by detecting the change in angle per unit time, the system control circuit 50 can calculate the amount of movement in the yawing direction and the amount of movement in the pitching direction based on the angular velocity.

[0076] Reference numeral 200 denotes a recording medium, which includes a recording unit 202 configured from a semiconductor memory, a magnetic disk, or the like, an interface 204 for connecting to the image capturing device 100 , and a connector 206 for electrically connecting to the image capturing device 100 .

[0077] In the above-described configuration, when the user determines the main subject to be the AF target while reselecting the main subject using the operation unit 70 or the like, the user may end up reselecting as the main subject a subject that is unlikely to be selected by the user, such as a subject other than the actual subject or a subject that has been erroneously detected.

[0078] In this embodiment, in order to suppress such reselection, a determination is made as to whether or not the subject to be reselected overlaps with another subject, and whether or not to reselect it as the main subject depending on the combination of the subject types of the subject to be reselected and the main subject.

[0079] <Still image shooting flow> Next, an example of a flow for capturing a still image with the imaging device 100 will be described with reference to Fig. 3. Each step of this flow chart is executed by each section of the digital camera 100 in response to an instruction from the system control circuit 50 or the system control section circuit 50.

[0080] In FIG. 3, subject detection and determination of the main subject and AF target are performed during live view in still image shooting mode.

[0081] In step S301, it is determined whether the system control circuit 50 is set to still image shooting mode. If it is determined in S301 that the still image shooting mode is set, the process proceeds to step S302. If it is not determined in step S301 that the still image shooting mode is set, the process ends and the still image shooting flow ends. After the still image shooting flow, the system control circuit controls the imaging device 100 in accordance with the camera settings, such as playback mode and power OFF.

[0082] In step S302, the system control circuit 50 determines whether or not the live view start processing has been completed in the imaging device 100. Here, the live view start processing refers to processing in which the timing generation circuit 18 supplies a clock signal and a control signal for live view to the imaging element 14 and the A / D converter 16. If it is determined in step S302 that the live view start processing has been completed, the process proceeds to step S303, and if it is determined that the live view start processing has not been completed, the process waits in step S302.

[0083] In step S303, live view processing is started, and the process proceeds to step S304. The live view processing is a process in which image data generated by the image sensor 14 using the image processing circuit 20 is temporarily stored in the memory 30 and displayed on the image display unit 28. This processing is continuously performed according to the timing of the timing generation circuit 18, allowing the user to check the captured image in real time.

[0084] In step S304, the system control circuit 50 determines whether or not to continue the still image shooting mode. If it is not determined in step S304 that the still image shooting mode should be continued, the flow ends. Whether or not to continue the still image shooting mode is determined according to the settings of the imaging device 100, and if the camera setting is set to the shooting mode, it is determined that the still image shooting mode should be continued. On the other hand, if it is determined in step S304 that the still image shooting mode should be continued, the flow transitions to step S305.

[0085] In step S305, the image processing circuit 20 performs subject detection processing on the image data to determine whether or not a subject has been detected.

[0086] If it is determined in step S305 that no subject has been detected, the process proceeds to step S306, where it is determined that there is no main subject, and the process proceeds to step S310.

[0087] If it is determined in step S305 that a subject has been detected, the process proceeds to step S307.

[0088] In step S307, the system control circuit 50 performs automatic main subject selection processing to determine a main subject from among the detected subjects, and then proceeds to step S308. An example of the automatic main subject selection processing will be described later.

[0089] In step S308, the system control circuit 50 determines whether or not the right button or the left button of the operation unit 70 has been pressed. If it is determined in step S308 that the right button or the left button of the operation unit 70 has been pressed, the process proceeds to step S309.

[0090] In step S309, a main subject manual selection process is performed in which the main subject is reselected according to the direction of the button designated in step S308, and the process proceeds to step S310. An example of the main subject manual selection process will be described later.

[0091] In step S310, the system control circuit 50 determines whether or not SW1 of the shutter switch 62 is in the ON state. If it is determined in step S310 that SW1 of the shutter switch 62 is not in the ON state, the process proceeds to S311.

[0092] In step S311, the system control circuit 50 determines whether or not the image processing circuit 20 is tracking an object. If it is determined in step S311 that an object is being tracked, the process proceeds to step S308. If it is not determined in step S311 that an object is being tracked, the process proceeds to S304.

[0093] If the system control circuit 50 determines in step S310 that SW1 of the shutter switch 62 is ON, the process proceeds to step S312.

[0094] In step S312, system control circuit 50 determines whether or not there is a main subject. If it is determined that there is no main subject, the process proceeds to step S313.

[0095] In step S313, the system control circuit 50 controls each part of the imaging device 100 so that AF processing is performed on the center position of the screen (the center position of the image displayed in live view). The area on which AF processing is performed is not limited to the center of the screen. For example, AF processing may be performed on an area on the screen that is determined to be closest in the depth direction from the image sensor 14. Also, the image processing circuit 20 may determine a prominent area on the screen using color information and luminance information, and set the position on which AF processing is performed based on the result. When AF processing is completed in step S313, the process transitions to step S317.

[0096] If it is determined in step S312 that a main subject is present, the process proceeds to step S314.

[0097] In step S314, AF processing is performed on the area corresponding to the main subject, and the process proceeds to step S315.

[0098] In step S315, the system control circuit 50 determines whether or not the right button or the left button of the operation unit 70 has been pressed. If it is determined that the right button or the left button of the operation unit 70 has been pressed, the process proceeds to step S316.

[0099] In step S316, a main subject manual selection process is performed in which the main subject is reselected in accordance with the direction specified by the right button or the left button, and the process proceeds to step S314.

[0100] If it is determined in step S315 that the right button or the left button of the operation unit 70 has not been pressed, the process proceeds to step S317.

[0101] In step S317, the system control circuit 50 determines whether or not SW2 of the shutter switch 62 is in the ON state. If it is not determined that SW2 of the shutter switch 62 is in the ON state, the process proceeds to step S310.

[0102] If it is determined in step S317 that SW2 of the shutter switch 62 is in the ON state, the process proceeds to step S318.

[0103] In step S318, under the control of the system control circuit 50, each unit of the imaging device 100 performs a still image shooting process, and the process proceeds to step S319.

[0104] In step S319, the system control circuit 50 determines whether to end the still image shooting mode. Whether to end the still image shooting mode depends on the camera settings. If the camera is set to a mode other than shooting mode, such as power OFF or playback mode, the still image shooting mode is ended.

[0105] If it is not determined in step S319 that the still image shooting mode has ended, the process proceeds to step S310. If it is determined in step S319 that the still image shooting mode has ended, the process ends.

[0106] <Automatic main subject selection process> Next, the main subject automatic selection process in step S307 in Fig. 3 will be described in detail with reference to Fig. 4. The main subject automatic selection process is a process for determining a main subject to be used as an AF target from among subjects detected as a result of subject detection within the screen.

[0107] In step S401, the image processing circuit 20 determines whether a person has been detected within the screen as a result of the subject detection process, and if it is determined in step S401 that a person has been detected within the screen, the process proceeds to step S402.

[0108] In step S402, the image processing circuit 20 determines whether or not a plurality of people have been detected within the screen, and if it is determined in step S402 that a plurality of people have been detected within the screen, the process proceeds to step S403.

[0109] In step S403, the system control circuit 50 determines a human subject to be the main subject based on the position and size weight of the person's face. The position and size weight for determining the human subject to be the main subject are obtained by multiplying the evaluation values ​​of the position weight and size weight set by the system control circuit 50 for each person. The system control circuit 50 determines the subject with the largest evaluation value from the evaluation values ​​of the position weight and size weight for each person as the main subject. For position weight, a coefficient is set so that the weight is heavier the closer the center position of the human face is to the center of the AF frame. For size weight, a coefficient is set so that the weight is heavier the larger the face size of the subject.

[0110] In step S403, the position weight and size weight are used to determine the human subject to be the main subject, but the main subject may be determined using other criteria, such as using the direction of the person's face so that the weight is higher the more the person is facing forward.

[0111] When a person subject is determined as the main subject based on the position and size weight of the person's face in step S403, this flow ends.

[0112] If it is not determined in step S402 that a plurality of people are detected within the screen (that is, this means that only one person is detected), the process proceeds to step S404.

[0113] In step S404, the detected person is treated as the main subject, and this flow ends.

[0114] If it is determined in step S401 that a person has not been detected on the screen, the process proceeds to step S405.

[0115] In step S405, the image processing circuit 20 determines whether or not an animal such as a dog or a cat has been detected within the screen. If it is determined in step S405 that an animal has been detected within the screen, the process proceeds to step S406.

[0116] In step S406, the image processing circuit 20 determines whether or not a vehicle such as an automobile has been detected within the screen, and if it is determined in step S406 that a vehicle has been detected within the screen, the process proceeds to step S407.

[0117] In step S407, the system control circuit 50 uses the normalized size for each object type to determine the main object within the AF frame from the position and object size. For example, size normalization is performed by normalizing the size so that the object regions of a dog, a cat, and a car have the same area when they are equidistant from the image sensor 14. The object to be determined as the main object is determined using the position weight and the normalized size weight.

[0118] If it is determined in step S406 that a vehicle such as an automobile has not been detected within the screen, the process proceeds to step S408.

[0119] In step S408, the image processing circuit 20 determines whether or not a plurality of animals have been detected within the screen, and if it is determined in S408 that a plurality of animals have been detected, the process proceeds to step S409.

[0120] In step S409, system control circuit 50 determines an animal subject to be the main subject from among the animal subjects within the screen based on the position and size weight, and then ends the flow.

[0121] If it is not determined in step S408 that multiple animals have been detected, the process proceeds to step S410.

[0122] In step S410, the animal subject detected from the screen is treated as the main subject, and the flow ends.

[0123] If it is determined in step S405 that an animal has not been detected within the screen, the process proceeds to step S411.

[0124] In step S411, image processing circuit 20 determines whether or not a vehicle subject such as an automobile has been detected within the screen, and if it is determined in step S411 that a vehicle subject has been detected, the process proceeds to step S412.

[0125] In step S412, the image processing circuit 20 determines whether or not a plurality of vehicles have been detected within the screen, and if it is determined in step S412 that a plurality of vehicles have been detected, the process proceeds to step S413.

[0126] In step S413, system control circuit 50 determines which vehicle subject is to be the main subject from among the vehicle subjects within the screen based on the position and size weight, and then ends the flow.

[0127] If it is not determined in step S412 that a plurality of vehicles have been detected within the screen, the process proceeds to step S414.

[0128] In step S414, the flow ends with the vehicle object detected within the screen as the main object.

[0129] If it is not determined in S411 that a vehicle subject such as an automobile has been detected within the screen, it is determined that there is no main subject, and the flow ends.

[0130] <Manual main subject selection process> 5 shows the main subject manual selection process. The main subject manual selection process is a process for reselecting a subject that exists in a direction designated by the user on the screen as the main subject (main subject selection).

[0131] In step S501, it is determined whether or not there is a subject in a direction specified by the user on the screen with respect to the main subject. An example of determining whether or not there is a subject in a direction specified by the user will be described with reference to FIG. 9, which shows an example of photography.

[0132] 9 is a display example of an image for explaining the main subject manual selection process. 901 is an airplane that is the main subject, 902 is a detected four-wheeled vehicle, 903 is a detected airplane, and 904 is a detected bird.

[0133] Reference numeral 911 denotes a frame display indicating that the airplane of 901 is the main subject. Reference numerals 912 and 913 denote displays indicating that the main subject can be switched by pressing the right button or the left button of the operation unit 70.

[0134] Whether the subject is on the left or right side is determined by the center coordinates of the subject area. In FIG. 9, the subject to the left of 901 is 904, and the subject to the right of 901 is 902.

[0135] First, if it is determined in step S501 that the subject is not present in the direction specified by the user, this flow ends.

[0136] If it is determined in step S501 that the subject is present in the direction specified by the user, the process proceeds to step S502.

[0137] In step S502, the object adjacent to the main object in the direction designated by the user is set as the transition object, and the process proceeds to step S503.

[0138] In step S503, it is determined whether SW1 of the shutter switch 62 is in the ON state (autofocus instruction in progress). If it is determined that SW1 of the shutter switch 62 is not in the ON state, the process proceeds to step S504.

[0139] In step S504, it is determined whether the subject type of the transition subject is the same as that of the main subject. The subject type may be, for example, a classification such as a person, an animal, or a vehicle, or may be a classification such as a person, a dog, a cat, a horse, a two-wheeled vehicle, a four-wheeled vehicle, an airplane, or a train.

[0140] If it is determined in step S504 that the transition subject is of the same type as the main subject (people and animals, vehicles and vehicles, etc.), the process proceeds to step S510.

[0141] If it is not determined in step S504 that the transition object is of the same type as the main object, the process proceeds to step S501.

[0142] If it is determined in step S503 that SW1 of the shutter switch 62 is in the ON state, the process proceeds to step S505.

[0143] In step S505, a transition determination process due to overlapping of subjects is performed, and the process proceeds to step S506. The transition determination process due to overlapping of subjects is a process for determining whether or not it is necessary to determine the subject type in order to make the transition subject the main subject. A specific example will be described later.

[0144] In step S506, if it is determined that it is not necessary to determine the object type because the transition object is the main object, the process proceeds to step S510.

[0145] If it is not determined in step S506 that it is not necessary to determine the object type in order to set the transition object as the main object, the process proceeds to step S507.

[0146] In step S507, it is determined whether or not to permit a transition to the transition object based on the combination of object types of the main object and the transition object.

[0147] An example of determining whether to permit a transition to a transition subject according to a combination of subject types of a main subject and a transition subject will be described with reference to Fig. 7. In Fig. 7, the vertical axis represents the subject type of the main subject, and the horizontal axis represents the index of the subject type of the transition subject. If the overlapping portion of each subject combination is marked with a circle, the transition is permitted (determined as transition permitted), and if it is marked with an x, the transition is prohibited (determined as transition prohibited).

[0148] For example, as in the relationship between 910 and 902 in FIG. 9, if the main subject is an airplane and the transition subject is a four-wheeled vehicle, the transition is permitted, but if the main subject is a person and the transition subject is a four-wheeled vehicle, the transition is prohibited.

[0149] If it is determined in step S507 that the transition is not permitted based on the subject type combination of the main subject and the transition subject, the process proceeds to step S501.

[0150] If it is determined in step S507 that the transition is permitted based on the subject type combination of the main subject and the transition subject, the process proceeds to step S508.

[0151] In step S508, a transition permission determination is made based on the subject size, and the process proceeds to step S509. The transition permission determination based on the subject size is a process for determining whether or not to permit the transition depending on the size of the transition subject. A specific example will be described later.

[0152] In step S509, it is determined whether or not the transition is permitted based on the object size, and if it is not determined that the transition is permitted based on the object size, the process proceeds to step S501.

[0153] If it is determined in step S510 that the transition is permitted due to the subject size, the process transitions to step S510.

[0154] In step S510, the transition object is switched to the main object, and the flow ends.

[0155] According to this flow, when SW1 of the shutter switch 62 is OFF and 901 in FIG. 9 is the main subject, if the right button is pressed, the main subject is switched to 903.

[0156] 5 as an example, 902 is set as the transition subject in step S502, but it is not determined in step S505 that the subject type need not be determined (no determination required), and the transition is not determined to be permitted in step S507, so that the process transitions to step S501. After that, 903 is set as the transition subject in step S502, it is determined in step S505 that the subject type need not be determined, and the main subject is switched to 903 in step S510.

[0157] When SW1 of the shutter switch 62 is OFF and 901 in FIG. 9 is the main subject, if the left button is pressed, the main subject is switched to 904.

[0158] In the example of FIG. 5, 904 is determined as the transitional object in step S502, and it is determined in step S505 that it is not necessary to determine the object type, and the main object is switched to 904 in step S510.

[0159] When SW1 of the shutter switch 62 is ON and 901 in FIG. 9 is the main subject, the main subject can only be switched to 903 by pressing either the right button or the left button.

[0160] 5 as an example, when the right button is pressed, 902 is set as the transition subject in step S502, but depending on whether the transition subject is the same type as the main subject in step S504, the process transitions to step S501, and 903 is set as the transition subject in step S502. After that, it is determined in step S504 that they are the same type, and the main subject is switched to 903 in step S510. When the left button is pressed, 904 is set as the transition subject in step S502, but depending on whether the transition subject is the same type as the main subject in step S504, the process transitions to step S501, and 903 is set as the transition subject in step S502. After that, it is determined in step S504 that they are the same type, and the main subject is switched to 903 in step S510.

[0161] <Transition determination process due to overlapping subjects> FIG. 6 is a flow diagram showing an example of a transition determination process due to overlapping of subjects.

[0162] In step S601, it is determined whether there is an overlapping area between the main subject area and the transition subject area. The subject area for determination may be an area of ​​the subject that is the AF target, such as the pupil area of ​​a person or the cockpit area of ​​an airplane. If the AF target is the pupil area of ​​a person, the face or whole body area of ​​the person may be determined as the main subject area, and if the AF target is the cockpit area of ​​the airplane, the entire area of ​​the airplane may be determined as the main subject area. Alternatively, the face or whole body area of ​​the person may be estimated from the pupil area of ​​the person, or the entire area of ​​the airplane may be estimated from the cockpit area of ​​the airplane as the subject area for determination. If the body part of the two-wheeled vehicle and the head of the person riding on the two-wheeled vehicle are detected, respectively, an area that covers the body part of the two-wheeled vehicle and the area of ​​the person riding on the two-wheeled vehicle may be estimated as the subject area.

[0163] If it is not determined in step S601 that there is an overlapping area between the areas of the main subject and the transitional subject, the process proceeds to S602.

[0164] In step S602, it is determined that there is no need to determine the object type of the transition object, and the flow ends.

[0165] If it is determined in step S601 that there is an overlapping area between the main subject area and the transition subject area, the process proceeds to step S603.

[0166] In step S603, it is determined whether the overlapping area is greater than a predetermined value relative to the area of ​​the transition subject. For example, if the overlapping area is larger than a predetermined ratio to the area of ​​the transition subject, it is determined to be greater than the predetermined value. Alternatively, it may be determined to be greater than the predetermined value if the horizontal size or vertical size of the overlapping area exceeds the vertical size of the transition subject or a size multiplied by a predetermined ratio to the vertical size.

[0167] If it is determined in step S603 that the overlapping area is equal to or larger than a predetermined size for the transition object, the process proceeds to step S604.

[0168] In step S604, it is determined that the object type of the transition object needs to be determined, and the flow ends.

[0169] If it is not determined in step S603 that the overlapping area is equal to or larger than the predetermined size for the transition object, the process proceeds to step S605.

[0170] In step S605, it is determined whether the overlapping area is greater than a predetermined size relative to the main subject. For example, if the overlapping area is larger than a predetermined ratio to the area of ​​the main subject, it is determined to be greater than the predetermined size. Alternatively, it may be determined to be greater than the predetermined size if the horizontal or vertical size of the overlapping area exceeds the vertical size of the main subject or a size multiplied by a predetermined ratio to the vertical size.

[0171] If it is determined in step S605 that the overlapping area is equal to or larger than a predetermined size with respect to the main subject, the process proceeds to step S602. As an example of determining in step S605 that the overlapping area is equal to or larger than a predetermined size with respect to the main subject, let us consider a case in which the transition subject area encompasses the entire main subject area.

[0172] If it is not determined in step S605 that the overlapping area is equal to or larger than the predetermined size with respect to the main subject, the process proceeds to step S604.

[0173] <Determining whether transition is possible or not based on the subject size> Fig. 8 is a flow showing an example of a transition possibility determination based on the subject size. The transition possibility determination based on the subject size is performed to prevent the AF frame from transitioning to a subject such as a picture printed on the actual subject.

[0174] In step S801, it is determined whether the transition subject is a person.

[0175] If it is determined in step S801 that the transition subject is a person, the process proceeds to step S802.

[0176] In step S802, it is determined whether the transition subject area is within a predetermined range with respect to the entire screen. For example, if the ratio of the area of ​​the transition subject to the area of ​​the entire screen is within a predetermined range, it is determined that the transition subject area is within the predetermined range. The transition subject area may be the face area, the torso area, or the whole body area of ​​a person.

[0177] In step S802, if it is determined that the transition object region is within a predetermined range with respect to the entire screen, the process proceeds to step S803.

[0178] In step S803, the transition based on the subject size is permitted, and this flow ends.

[0179] In step S802, if it is not determined that the transition object region is within a predetermined range with respect to the entire screen, the process proceeds to step S804.

[0180] In step S804, transition due to subject size is prohibited and this flow ends.

[0181] If the transition subject is not determined to be a person in step S801, the process proceeds to step S805.

[0182] In step S805, it is determined whether or not a person has been detected within the screen regardless of whether the person is a transitional subject.

[0183] If it is determined in step S805 that a person has not been detected within the screen, the process proceeds to step S802.

[0184] If it is determined in step S805 that a person has been detected within the screen, the process proceeds to step S806.

[0185] In step S806, it is determined whether the ratio of the detected person to the transition subject is within a predetermined range. If multiple people are detected in the screen, it is acceptable to compare with the person who is most likely to be the main subject, or with the person who is closest to the transition subject, or to use the result of comparing with all detected people. For example, if the ratio of the area of ​​the transition subject area to the area of ​​the person area is within a predetermined range, it is determined that it is within the predetermined range.

[0186] In step S806, if it is determined that the ratio of the detected person to the transitional subject is not within a predetermined range, the process proceeds to step S804.

[0187] In step S806, if it is determined that the ratio of the detected person to the transitional subject is within a predetermined range, the process proceeds to step S807.

[0188] In step S807, the transition based on the subject size is permitted, and the flow ends.

[0189] <Effects of this embodiment> As described above, by determining whether or not to transition to a main subject based on the overlap of the transition subject with another subject or the combination of subject types, it is possible to avoid transitioning to a subject unintended by the user.

[0190] <Other embodiments> Although the preferred embodiments of the present invention have been described above, the present invention is not limited to these embodiments, and various modifications and changes are possible within the scope of the gist of the present invention.

[0191] For example, although the present embodiment has been described with respect to the transition of the subject by the user pressing the right or left button, the present embodiment may be applied to the transition of the main subject in response to another operating member. In addition, when the camera determines whether to automatically transition the main subject, the camera may determine whether to transition using the overlap of subject areas or a combination of subject types.

[0192] In this embodiment, in order to avoid the transition of the AF frame to a subject not intended by the user (transition to the main subject), the transition possibility is determined based on the size, but the transition possibility may be determined based on the amount of movement of the camera or the subject instead. As an example of determining whether or not the transition is possible based on the amount of movement of the camera or the subject, it is possible to determine not to transition to a subject that is likely to go outside the screen, or to transition to a subject that the user is panning.

[0193] In addition, by providing a new scene determination unit for determining the shooting scene, it can be utilized for determining whether or not to transition the main subject. For example, if a user intends to shoot an airplane flying in the sky, it is expected that the image to be shot will include a lot of sky. The scene determination unit determines that the subject to be shot is an airplane flying in the sky from the sky area that occupies in the shot image. Here, for example, consider a case where a picture of a four-wheeled vehicle is printed on the exterior of an airplane flying in the sky. In this case, the scene determination unit determines that the subject to be shot is an airplane flying in the sky. Then, even if a four-wheeled vehicle printed in addition to the airplane is detected, it is possible to control so as not to permit the transition from the airplane to the four-wheeled vehicle as the main subject.

[0194] On the other hand, even if a transition is not normally permitted for a combination of subject types, control can be performed to permit the transition depending on the shooting scene.

[0195] A computer program for implementing all or part of the control in this embodiment may be supplied to an imaging device or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the imaging device or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0196] 10. Shooting Lens 12 Shutter 14 Image sensor 16 A / D Converter 18 Timing generation circuit 20 Image processing circuit 22 Memory control circuit 28 Image display section 30 Memory 32 Image compression / expansion circuit 40 Exposure control section 42 Focus detection focus detection control section 44 Zoom control section 48 Flash 50 System control circuit 60 Mode dial switch 62 Shutter Switch 66 Display changeover switch 70 Operation section 72 Zoom switch 90 Interface 92 Connector 100 Image processing device 104 Optical Viewfinder 110 Communications Department 200 Recording media 202 Recording Department 204 Interface 206 Connector

Claims

1. an acquisition means for sequentially acquiring images; a detection means for detecting a plurality of subjects from the image acquired by the acquisition means; a main subject selection means for selecting a main subject from the plurality of subjects, The main subject selection means, when switching the main subject in response to a user instruction, reselects one of the plurality of subjects as a new main subject in accordance with a combination of the type of one of the plurality of subjects and the type of the main subject.

2. The image processing device described in Claim 1, characterized in that the main subject selection means re-selects one of the multiple subjects as a new main subject depending on whether or not the area of ​​any of the multiple subjects overlaps with the area of ​​another subject.

3. 2. The image processing apparatus for processing an image of a main subject according to claim 1, wherein said detecting means detects a subject including at least one of a person, an animal, and a vehicle.

4. 4. The image processing device according to claim 3, wherein, when a combination of the type of any one of the plurality of subjects and the type of the main subject is a combination of a person and an animal, reselection of any one of the plurality of subjects as a new main subject is permitted.

5. 4. The image processing device according to claim 3, wherein, in the case of a combination of a type of any one of the plurality of subjects and a type of the main subject, or a combination of a person and a vehicle, the image processing device allows re-selection of any one of the plurality of subjects as a new main subject.

6. 4. The image processing device according to claim 3, wherein, when a combination of the type of any of the plurality of subjects and the type of the main subject is a combination of animals, it is determined whether or not to allow re-selection of any of the plurality of subjects as a new main subject depending on the combination of subject types.

7. 4. The image processing device according to claim 3, wherein, when the combination of the type of any of the plurality of subjects and the type of the main subject is a combination of an animal and a vehicle, it is determined whether or not to allow re-selection of any of the plurality of subjects as a new main subject depending on the combination of subject types.

8. 4. The image processing device according to claim 3, wherein if the combination of the type of any of the plurality of subjects and the type of the main subject is a combination of an airplane and a car, reselection of any of the plurality of subjects as a new main subject is not permitted.

9. 4. The image processing device according to claim 3, wherein if the combination of the type of any of the plurality of subjects and the type of the main subject is a combination of a train and a car, reselection of any of the plurality of subjects as a new main subject is not permitted.

10. The image processing device described in Claim 1, characterized in that the main subject selection means changes the combination of subject types for which re-selection as the main subject is permitted or not, depending on the operation state of the operating member by the user.

11. 11. The image processing device according to claim 10, wherein, when the operation state of the operating member by the user is instructing autofocus, if the combination of the type of one of the plurality of subjects and the type of the main subject is the same type, the image processing device allows re-selection of one of the plurality of subjects as a new main subject.

12. further comprising a scene determination means for determining a photographed scene, 2. The image processing apparatus according to claim 1, wherein the main subject selection means changes a combination of subject types for which reselection as the main subject is permitted or not, depending on the photographic scene determined by the scene determination means.

13. 2. The image processing device according to claim 1, wherein whether or not to reselect any one of the plurality of subjects as a new main subject is determined depending on the size of any one of the plurality of subjects.

14. further comprising a movement amount detection means for detecting a movement amount of the image processing device; 2. The image processing device according to claim 1, wherein the main subject selection means allows re-selection of one of the plurality of subjects as a new main subject depending on at least one of the amount of movement of the image processing device and the amount of movement of one of the plurality of subjects.

15. The image processing device described in Claim 1, characterized in that when the main subject selection means determines that panning photography is being performed in which one of the multiple subjects is being tracked while being photographed, it allows one of the multiple subjects to be re-selected as a new main subject.

16. An imaging element; An image processing device according to any one of claims 1 to 15; An imaging device having the above configuration.

17. an acquisition step of sequentially acquiring images; a detecting step of detecting a plurality of subjects from the image acquired in the acquiring step; a main subject selection step of selecting a main subject from the plurality of subjects, a main subject selection step of reselecting one of the plurality of subjects as a new main subject in accordance with a combination of a type of one of the plurality of subjects and a type of the main subject when switching the main subject in accordance with a user's instruction;

18. 18. A program for causing a computer to function as each step of the method for controlling an image processing apparatus according to claim 17.

19. A computer-readable storage medium storing the program according to claim 18.

Images