IMAGE PROCESSING APPARATUS, IMAGING APPARATUS, CONTROL METHOD, AND PROG

Abstract

To provide a technology in which a balance between tracking performance of a detected subject and detecting performance of other subject can be adjusted.SOLUTION: An image processing apparatus that comprises: detecting means that performs subject detection processing using one or more dictionaries being a part of a plurality of dictionaries for each of a plurality of images which are obtained by imaging by imaging means, each of the plurality of dictionaries being for detecting a different type of subject; setting means that sets a tracking mode; and selecting means that selects the one or more dictionaries used in the detecting means, the selecting means selecting a second dictionary being different from a first dictionary corresponding to a type of a first subject detected in a previous image in a frequency corresponding to the tracking mode, is provided.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present invention relates to an image processing device, an imaging device, a control method, and a program. [Background technology]

[0002] Conventionally, digital cameras and the like have commercialized products equipped with a tracking AF mode that detects human faces and eyes from images continuously output from an image sensor and continuously optimizes focus and exposure conditions for the detected human faces or eyes.

[0003] In recent years, it has become possible to use machine learning techniques to detect not only human faces and eyes, but also various other types of subjects. In this case, the subject contained in the image can be detected by inputting to the detector both the image and dictionary data (hereinafter simply referred to as "dictionary") obtained by learning about the subject to be detected. By preparing multiple dictionaries corresponding to various types of subjects and switching the dictionary input to the detector, various types of subjects can be detected from within the image.

[0004] As the number of types of subjects to be detected increases, the number of dictionaries used also increases. In devices such as digital cameras in which the number of logical operation circuits constituting the detector and the processing power are limited, a control method for detecting multiple types of subjects by switching dictionaries for each frame is conceivable. Note that, here, one frame refers to the time or period until the execution and completion of operation processing and camera control for an image output from an image sensor.

[0005] For example, Patent Document 1 discloses a configuration in which, when detection of a subject is successful, the frequency of use of a dictionary corresponding to the detected subject is increased in subsequent frames, thereby achieving stable subject tracking. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2021-132369 A Summary of the Invention [Problem to be solved by the invention]

[0007] In Patent Document 1, as the frequency of use of a dictionary corresponding to a detected subject is increased, the frequency of use of other dictionaries is relatively decreased. However, Patent Document 1 does not take into consideration that the frequency of use of other dictionaries is a direct control target.

[0008] The present invention has been made in view of the above circumstances, and aims to provide a technology that enables adjustment of the balance between the tracking performance of a detected subject and the detection performance of other subjects by making it possible to change the frequency of use of a dictionary different from the dictionary corresponding to the type of detected subject. [Means for solving the problem]

[0009] In order to solve the above problem, the present invention provides an image processing device comprising: a detection means for performing subject detection processing using one or more dictionaries that are a part of a plurality of dictionaries for each of a plurality of images obtained by imaging using an imaging means, wherein each of the plurality of dictionaries is for detecting a different type of subject; a setting means for setting a tracking mode; and a selection means for selecting the one or more dictionaries to be used by the detection means, wherein the selection means selects a second dictionary different from a first dictionary corresponding to the type of a first subject detected in a previous image, with a frequency corresponding to the tracking mode. Effect of the Invention

[0010] According to the present invention, it is possible to adjust the balance between the tracking performance of a detected subject and the detection performance of other subjects.

[0011] Other features and advantages of the present invention will become apparent from the accompanying drawings and the following detailed description of the preferred embodiments of the present invention. [Brief description of the drawings]

[0012] [Figure 1] 1 is a side cross-sectional view showing the configuration of a camera 100 (digital single-lens camera). [Diagram 2] FIG. 1 is a functional block diagram of a camera 100. [Diagram 3] 4 is a flowchart of the operation of the camera 100 for one frame. [Figure 4] 4 is a flowchart showing details of the process of determining the priority of dictionaries in S308 of FIG. 3. [Diagram 5] 4 is a flowchart showing details of a process for determining a dictionary control schedule in S309 of FIG. 3 according to the first embodiment. [Figure 6] 5A to 5C are diagrams for explaining an example of a dictionary detection period and a control schedule in the case where a local dictionary does not exist according to the first embodiment. [Figure 7] 5A to 5C are diagrams for explaining an example of a dictionary detection period and a control schedule in the case where a local dictionary does not exist according to the first embodiment. [Figure 8] 5A to 5C are diagrams for explaining an example of a dictionary detection period and a control schedule in a case where a local dictionary exists according to the first embodiment. [Figure 9] 5A to 5C are diagrams for explaining an example of a dictionary detection period and a control schedule in a case where a local dictionary exists according to the first embodiment. [Figure 10] 10 is a flowchart showing details of a process for determining a dictionary control schedule in S309 of FIG. 3 according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing the calculation results of S1003 and S1004 in FIG. [Figure 12] 13A to 13C are diagrams for explaining two specific examples of a dictionary detection period and a control schedule according to the second embodiment. [Figure 13]13A to 13C are diagrams for explaining two specific examples of a dictionary detection period and a control schedule according to the second embodiment. [Figure 14] 13A to 13C are diagrams for explaining two specific examples of a dictionary detection period and a control schedule according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0014] [First embodiment] Fig. 1 is a side cross-sectional view showing the configuration of a camera 100 (digital single-lens camera) which is an example of an imaging device including an image processing device. Fig. 2 is a functional block diagram of the camera 100.

[0015] 1, in the camera 100 of this embodiment, a detachable and interchangeable lens unit 120 is attached to the front side (subject side) of a camera body 101. The lens unit 120 has a focus lens 121, an aperture 122, etc., and is electrically connected to the camera body 101 via a mount contact part 123. This makes it possible to adjust the amount of light taken into the camera body 101 and the focal position. The focus lens 121 can also be manually adjusted by the user.

[0016] The image sensor 104 is composed of a CCD sensor, a CMOS sensor, or the like, and includes an infrared cut filter, a low-pass filter, and the like. The image sensor 104 photoelectrically converts the subject image formed through the photographing optical system of the lens unit 120 during photographing, and transmits a signal for generating a photographed image to the arithmetic unit 102. The arithmetic unit 102 generates a photographed image from the received signal, stores the image in the image storage unit 107, and displays the image on the display unit 105 such as an LCD. The image storage unit 107 includes a storage medium such as an SD card. The shutter 103 blocks light from the image sensor 104 when photographing is not being performed, and opens the shutter 103 to expose the image sensor 104 to light when photographing is not being performed.

[0017] Next, the configuration related to the control of the camera 100 will be described with reference to the functional block diagram of Fig. 2. The arithmetic device 102 includes a multi-core CPU capable of parallel processing of multiple tasks, RAM, and ROM, as well as a dedicated circuit for executing specific arithmetic processing at high speed. With these hardware components, the arithmetic device 102 includes a control unit 201, a main subject calculation unit 202, a tracking calculation unit 203, a focus calculation unit 204, and an exposure calculation unit 205. The control unit 201 controls each unit of the camera body 101 and the lens unit 120.

[0018] The main subject calculation unit 202 is configured to include a dictionary priority determination unit 211, a dictionary switching control unit 212, a detector 213, and a main subject determination unit 214. The detector 213 performs a process of detecting a specific area of ​​the subject (for example, a human face or eyes, or a dog's face or eyes) from an image. The specific area of ​​the subject may not be detected or may be detected in multiple areas. As a detection method, any known method such as AdaBoost or a convolutional neural network may be used. In addition, the implementation form may be a program running on a CPU, dedicated hardware, or a combination of these. In addition, the type of subject to be detected can be changed by switching the dictionary input to the detector 213.

[0019] The dictionary is data in which, for example, the characteristics of the corresponding object are registered, and a control command for the logic circuit is described for each type of object. In this embodiment, a dictionary for each object is stored in the ROM in the arithmetic device 102 (multiple types of dictionaries are stored). Since a dictionary exists for each type of object, different types of objects can be detected by switching the dictionary. For example, the dictionary priority determination unit 211 calculates the priority of the dictionary at a predetermined timing as described later, and based on the calculation result, the dictionary switching control unit 212 determines the dictionary to be input to the detector 213. The object detection result obtained from the detector 213 is sent to the main object determination unit 214, and the main object is determined from the detected objects and the main object area is set. The main object is determined using a known determination method based on the size, position, reliability of the detection result, etc. If the detector 213 does not detect an object, the main object determination unit 214 can determine the main object area based on the past detection results, feature amounts such as edges of the target frame, object distance, etc.

[0020] For example, a detector that performs subject detection using a convolutional neural network (CNN) can be used as the detector 213. The dictionary that the detector 213 uses to detect each subject includes learned parameters that have been generated in advance by machine learning of the CNN in an external device (e.g., a PC) or the camera 100.

[0021] The machine learning of CNN may be performed by any method. For example, a predetermined computer such as a server may perform the machine learning of CNN, and the camera 100 may acquire the trained CNN from the predetermined computer. For example, the CNN may be trained by a predetermined computer performing supervised learning using training image data as input and the position of a subject corresponding to the training image data as training data. The CNN may also be trained by a predetermined computer performing supervised learning using training image data as input and a dictionary corresponding to the subject of the training image data as training data. Trained parameters of the CNN are generated by such training.

[0022] The detector 213 used in this embodiment has the processing capability to switch dictionaries within one frame period and perform subject detection multiple times, but does not always detect all types of subjects targeted by the camera 100 within one frame period. That is, this embodiment does not necessarily limit the processing capability of the detector 213, but in order to save processing speed and bus bandwidth, subject detection of a number of types less than all types is performed by switching dictionaries according to predetermined settings.

[0023] The tracking calculation unit 203 tracks the main subject region based on the detection information of the main subject. The focus calculation unit 204 calculates the control value of the focus lens 121 for focusing on the main subject region. In addition, the exposure calculation unit 205 calculates the control values ​​of the aperture 122 and the image sensor 104 for achieving proper exposure of the main subject region.

[0024] The operation unit 106 includes a release switch, a mode dial, and the like, and the control unit 201 can receive shooting instructions, mode change instructions, and the like from the user through the operation unit 106 .

[0025] Next, the operation of one frame of the camera 100 will be described with reference to Fig. 3. The operation of the flowchart in Fig. 3 is repeatedly executed for each frame.

[0026] In S301, the control unit 201 performs a process of reading out pixel signals from the image sensor 104, and generates image data from the read out pixel signals. The pixel signals and the generated image data are stored in the RAM in the arithmetic device 102.

[0027] In S302, the control unit 201 determines whether the current frame is the first frame. If the current frame is the first frame, the process proceeds to S306, otherwise the process proceeds to S303.

[0028] In S303, the tracking calculation unit 203 performs a tracking process to calculate the position of the main subject area set in the previous frame in the current frame using tracking reference information generated in S312 (described later) for the previous frame. Any known method may be used as the algorithm used in the tracking process. For example, a feature amount extracted from the main subject area set in the previous frame by a predetermined feature extraction process is used as the tracking reference information. In this case, the tracking calculation unit 203 can track the main subject area by searching for an area in the current frame where the feature amounts are close to each other. In addition, the implementation form of the algorithm may be a program running on a CPU, dedicated hardware, or a combination of these. After the tracking process is completed, the tracking calculation unit 203 stores the tracking result (the position and size of the main subject area in the current frame) in the RAM in the calculation device 102.

[0029] In S304, the focus calculation unit 204 calculates a control value for the focus lens 121 so as to focus on the main subject area using the tracking result generated in S303 and the pixel signal and image data obtained in S301. The control value for the focus lens 121 may be calculated using any known method such as a contrast method or a phase difference detection method. When the calculation of the control value for the focus lens 121 is completed, the focus calculation unit 204 notifies the control unit 201 of the control value of the focus lens 121. The control unit 201 controls the focus lens 121 through the mount contact unit 123 based on the notified control value.

[0030] In S305, the exposure calculation unit 205 uses the tracking result generated in S303 and the pixel signal and image data obtained in S301 to calculate control values ​​for the image sensor 104 and the aperture 122 so that the exposure of the main subject area is appropriate. Any known method may be used as a method for calculating the control values ​​for the image sensor 104 and the aperture 122. When the calculation of the control values ​​for the image sensor 104 and the aperture 122 is completed, the exposure calculation unit 205 notifies the control unit 201 of the control values ​​for the image sensor 104 and the aperture 122. The control unit 201 controls the image sensor 104 based on the notified control values ​​for the image sensor 104 and the aperture 122, and also controls the aperture 122 via the mount contact unit 123.

[0031] In S306, the control unit 201 reads the state of the operation unit 106 and determines whether or not the release switch is pressed. If the release switch is pressed, the process proceeds to S307; if not, the process proceeds to S308.

[0032] In S307, the control unit 201 performs imaging processing of a still image. The control unit 201 drives the shutter 103 based on the control value obtained in S305, exposes the image sensor 104, and generates image data from pixel signals read out from the exposed image sensor 104. The generated image data is stored in the image storage unit 107.

[0033] In S308, the dictionary priority determination unit 211 determines the priority of each dictionary. Details of the process in S308 will be described later with reference to FIG.

[0034] In S309, the dictionary switching control unit 212 determines a dictionary control schedule based on the priority of each dictionary determined in S308. Details of S309 will be described later with reference to FIG.

[0035] In S310, the dictionary switching control unit 212 selects a dictionary in accordance with the control schedule determined in S309, and inputs the selected dictionary and the image data generated in S301 to the detector 213. Then, the detector 213 performs a process of detecting the subject (subject detection process) by detecting a specific area of ​​a subject of a type corresponding to the dictionary from the input image data.

[0036] In S311, main subject determination unit 214 determines the main subject based on the detection result obtained in S310. The determination of the main subject can be performed using any known determination method based on, for example, the part of the detection result (e.g., the whole body, face, eyes, etc.), size, position, reliability of the detection result, etc. Main subject determination unit 214 stores information indicating the determined main subject and the area corresponding to the main subject (main subject area) in the RAM in the calculation device 102.

[0037] In S312, the tracking calculation unit 203 generates tracking reference information for use in the tracking process of S303 for the next frame based on the information on the main subject region obtained in S311 and the image data generated in S301. The generated tracking reference information is stored in the RAM in the calculation device 102.

[0038] When the process of S312 ends, the operation of one frame of the camera 100 ends. As long as the camera 100 continues to operate, the camera 100 continues continuous imaging at a predetermined frame rate by repeating the processes described up to this point, starting from S301.

[0039] Next, the process of determining the priority of dictionaries in S308 of FIG. 3 will be described in detail with reference to FIG.

[0040] In S400a, dictionary priority determination unit 211 determines whether the main subject or the tracking mode has changed. The tracking mode will be described later in S405. If the main subject or the tracking mode has changed, the process proceeds to S401, and if neither the main subject nor the tracking mode has changed, the process proceeds to S400b.

[0041] In S400b, dictionary priority determination unit 211 determines whether the current frame is the first frame. If the current frame is the first frame, the process proceeds to S407, otherwise the process of this flowchart ends. Thus, the process of determining the priority of the dictionary is performed in two cases: when the main subject or tracking mode has changed, and when the current frame is the first frame; otherwise, the priority of the dictionary previously determined is maintained.

[0042] In S401, dictionary priority determination unit 211 determines whether or not a main subject is present in the previous frame (i.e., whether or not a main subject has been determined by the processing of S311 for the previous frame) by referring to information stored in RAM in computing device 102. If a main subject is present in the previous frame, the processing proceeds to S402. If a main subject is not present in the previous frame, the processing proceeds to S407. For example, if no subject is detected by the processing of S310 for the previous frame, the main subject is not determined by the processing of S311, and therefore no main subject is present in the previous frame.

[0043] In S402, dictionary priority determination unit 211 sets the priority of the dictionary corresponding to the type of main subject detected in the previous frame to "high". In this embodiment, the priority of the dictionary is set to three levels: "high", "medium", and "low". However, the method of setting the priority can be changed as appropriate depending on the number of dictionaries, the processing capacity of detector 213 (the number of detection processes that can be performed within one frame period), etc.

[0044] In S403, the dictionary priority determination unit 211 determines whether or not there is a local dictionary corresponding to the main subject. Depending on the type of subject, a basic region (primary region) and a local region (secondary region) different from the basic region are defined as a detection region (a specific region to be detected). Specifically, the basic region and the local region are defined, for example, according to the following relationship.

[0045] (For humans) Basic area: Face Local area: Eyes (For dogs) Basic area: Face Local area: Eyes As in the above example, a local region is often a region inside a basic region, but is not limited to this. For example, it is possible to define the torso as a local region for a "person" whose face is defined as a basic region. Therefore, a local region is a "local" region in comparison with the whole subject, but is not necessarily "local" in comparison with the basic region.

[0046] In the following description, a dictionary for detecting a basic region of a subject and a dictionary for detecting a local region (secondary region) of a subject may be collectively referred to simply as a "dictionary." Furthermore, as necessary, a dictionary for detecting a basic region of a subject may be referred to as a "basic dictionary" or a "primary dictionary," and a dictionary for detecting a local region (secondary region) of a subject may be referred to as a "local dictionary" or a "secondary dictionary."

[0047] In this embodiment, local regions are detected based on the dictionary not from the entire captured image, but by inputting an image obtained by cutting out a portion including the basic region of the detected subject and its vicinity from the original captured image to detector 213. In other words, if the type of subject is a person (strictly speaking, a human face), detection processing of the pupils, which are local regions, is performed only after a human face is detected. If a local dictionary corresponding to the main subject exists, the processing proceeds to step S404, and if a local dictionary corresponding to the main subject does not exist, the processing proceeds to step S405.

[0048] In S404, the dictionary priority determination unit 211 sets the priority of the local dictionary corresponding to the main subject to "high."

[0049] In S405, the dictionary priority determination unit 211 determines whether the tracking mode is set to "tracking priority." If the tracking mode is set to "tracking priority," the process proceeds to S406, and if the tracking mode is set to "standard," the process proceeds to S407.

[0050] Here, the tracking mode will be described. The control unit 201 sets the tracking mode to "standard" (first mode) or "tracking priority" (second mode) depending on whether the tracking button included in the operation unit 106 is pressed by the user. The tracking button is a button for the user to input a user input (tracking instruction) indicating an intention to track a main subject to the camera 100. The control unit 201 sets the tracking mode to "standard" before receiving the tracking instruction (for example, when the camera 100 is started). When the user presses the tracking button, the control unit 201 receives the tracking instruction. In response to the reception of the tracking instruction, the control unit 201 sets the tracking mode to "tracking priority". When the user releases the tracking button, the reception of the tracking instruction stops, and the control unit 201 sets the tracking mode to "standard".

[0051] Note that the configuration for inputting a tracking instruction to camera 100 does not necessarily have to be a button. For example, a configuration may be adopted in which a tracking instruction is input by a user touching the touch panel display of display unit 105. Alternatively, a configuration may be adopted in which a tracking instruction is input according to the state of the eyeball of a user looking through a finder (not shown) (for example, a configuration in which a tracking instruction is input when the user's line of sight is directed toward a main subject).

[0052] In S406, the dictionary priority determination unit 211 sets the priority of all other dictionaries (basic dictionaries and local dictionaries for which a priority has not yet been set) to "low." Note that "priority has not yet been set" here means that a priority has not yet been set in the process of S308 for the current frame. Even if a dictionary had a priority set in a previous frame, if it does not have a priority set in the current frame, it corresponds to a dictionary for which a priority has not yet been set. The same applies to S407, which will be described later.

[0053] In S407, the dictionary priority determination unit 211 determines whether or not there is a dictionary for which a priority has not yet been set. If there is a dictionary for which a priority has not yet been set, the process proceeds to S408. If there is no dictionary for which a priority has not yet been set, the process of this flowchart ends.

[0054] In S408, the dictionary priority determination unit 211 sets the priority of the local dictionary to "low" and the priority of the base dictionary to "medium" among the dictionaries for which the priority has not yet been set.

[0055] Next, the process of determining the dictionary control schedule in S309 of FIG. 3 will be described in detail with reference to FIG.

[0056] In S500, dictionary switching control unit 212 determines whether or not the dictionary priority setting has been changed by the process of S308. If the dictionary priority setting has been changed, the process proceeds to S501, and if the dictionary priority setting has not been changed, the process of this flowchart ends. Therefore, similar to the process of determining the dictionary priority (S308), the process of determining the dictionary control schedule is performed in two cases: when the main subject or tracking mode has changed, and when the current frame is the first frame. In all other cases, the previously determined dictionary control schedule is maintained.

[0057] In S502, the dictionary switching control unit 212 sets the detection cycle of the dictionary with a "high" priority to 1 [frame / detection]. The detection cycle is a parameter [frame / detection] that indicates how many frames one detection result is obtained per. However, the detection cycle is not limited to the number of frames per detection [frame / detection], and may be specified as, for example, the time per detection [ms / detection].

[0058] In S503, the dictionary switching control unit 212 sets the detection cycles for the remaining dictionaries, that is, dictionaries with "medium" or "low" priority. A specific example of setting the detection cycle in S503 will be described later with reference to Figs. 6 to 9.

[0059] In S504, the dictionary switching control unit 212 determines a dictionary control schedule based on the detection cycle of each dictionary and the number of detection processes that the detector 213 can execute within one frame period. A specific example of determining the dictionary control schedule will be described later with reference to Figs. 6 to 9.

[0060] According to the example shown in Figures 4 and 5, the priority and detection period of each dictionary are set according to the presence or absence of a main subject, the type of the main subject, and the tracking mode, and the control schedule of the dictionary is determined based on the detection period of each dictionary. However, the method of determining the control schedule of the dictionary is not limited to the example shown in Figures 4 and 5. For example, a table that associates the presence or absence of a main subject, the type of the main subject, and the tracking mode with the control schedule of the dictionary may be stored in advance in a ROM in the calculation device 102. In this case, the dictionary switching control unit 212 can obtain the corresponding control schedule from the table based on the presence or absence of a main subject, the type of the main subject, and the tracking mode.

[0061] Next, examples of a dictionary detection period and a control schedule in the case where a local dictionary does not exist will be described with reference to FIGS.

[0062] FIG. 6(a) is a diagram showing the types of dictionaries and the processing constraints of the detector. The camera 100 uses five types of dictionaries corresponding to people, dogs, birds, four-wheeled vehicles, and two-wheeled vehicles, respectively. The detector 213 provided in the camera 100 includes one logic circuit block, and the number of times that detection processing can be executed per frame is three. Four-wheeled vehicles refer to racing vehicles such as four-wheeled rally cars and four-wheeled passenger cars, and two-wheeled vehicles refer to two-wheeled motorbikes and bicycles. In the processing of FIG. 4 and FIG. 5, the priority of each dictionary, the detection period, and the control schedule are determined based on the conditions of FIG. 6(a).

[0063] FIG. 6(b) shows the detection period of each dictionary when the tracking mode is "standard", and FIG. 6(c) shows the detection period of each dictionary when the tracking mode is "tracking priority". In FIG. 6(b) and FIG. 6(c), "0" indicates that the corresponding dictionary is not used. As shown in FIG. 6(b) and FIG. 6(c), when a main subject is not detected, the detection period of all dictionaries is set to 2 [frame / detection] regardless of the tracking mode. On the other hand, when a main subject is detected and the tracking mode is "standard", the detection period of the dictionary corresponding to the type of main subject is set to 1 [frame / detection], and the detection period of the other dictionaries is set to 2 [frame / detection], as shown in FIG. 6(b). Also, when a main subject is detected and the tracking mode is "tracking priority", the detection period of the dictionary corresponding to the type of main subject is set to 1 [frame / detection], and the detection period of the other dictionaries is set to 0 [frame / detection], as shown in FIG. 6(c). That is, in this case, dictionaries other than the dictionary corresponding to the type of main subject are not used.

[0064] Fig. 7 is a diagram showing an example of a dictionary control schedule and its changes. The example in Fig. 7 assumes a case where a dog 703 is detected in the middle of consecutive frames. In Fig. 7, reference numeral 701 indicates a dictionary scheduled for each frame (i.e., a dictionary input to the detector 213 and used in each frame). Reference numeral 702 indicates a captured image input to the detector 213.

[0065] In Fig. 7, the main subject is not detected at the timing of the first frame. Therefore, the detection cycle of all dictionaries is set to 2 [frames / detection]. As a result, a control schedule is determined such that, in units of two frames, the dictionaries corresponding to "person", "dog", and "bird" are used in the first frame, and the dictionaries corresponding to "four-wheel" and "two-wheel" are used in the next frame.

[0066] No main subject is detected in frames 1 to 4. Although a dog 703 appears in the captured image in frame 4, the dictionary corresponding to dogs is not used, so the dog 703 is not detected as a subject. Therefore, in frames 1 to 5, including the frame following frame 4, the dictionary to be used in each frame is selected according to the initially determined control schedule.

[0067] In the fifth frame, the detector 213 performs subject detection using a dictionary corresponding to a dog, and as a result, the face of the dog 704 is detected as the subject. Since the dog 704 is the only subject detected, the dog 704 is determined to be the main subject. At this point, the tracking mode is assumed to be "standard". Therefore, in the frame next to the fifth frame (sixth frame), the detection cycle of the dictionary corresponding to "dog" is set to 1 [frame / detection], and the detection cycle of the other dictionaries (person, bird, four-wheeled vehicle, two-wheeled vehicle) is set to 2 [frame / detection]. As a result, a control schedule is determined such that, in units of two frames, the dictionaries corresponding to "dog", "person", and "bird" are used in the first frame, and the dictionaries corresponding to "dog", "four-wheeled vehicle", and "two-wheeled vehicle" are used in the next frame.

[0068] Between the 6th to 9th frames, there is no change in the main subject and the tracking mode. Therefore, in the 6th to 9th frames, the dictionary to be used in each frame is selected according to the control schedule determined in the 6th frame. As a result of dog 704 being detected as the main subject, the frequency of use (selection frequency) of the dictionary corresponding to dogs increases compared to before the detection, so it becomes possible to detect the main subject with high accuracy (i.e., to track the main subject with high accuracy). In addition, since dictionaries other than the dictionary corresponding to dogs are still used, if a subject other than a dog appears in the captured image, the subject other than a dog can also be detected, and depending on the situation, the main subject may be switched to a type of subject other than a dog.

[0069] Assume that the tracking mode is changed to "tracking priority" in the 10th frame. In this case, the detection cycle of the dictionary corresponding to "dog" is set to 1 [frame / detection], and the detection cycle of the other dictionaries (people, birds, 4-wheeled vehicles, 2-wheeled vehicles) is set to 0 [frame / detection]. In other words, the detection cycle is set so that dictionaries other than the dictionary corresponding to dog are not used. As a result, a control schedule is determined so that only the dictionary corresponding to "dog" is used in units of one frame (i.e., in all frames).

[0070] In this way, when the tracking mode is "tracking priority," dictionaries other than the dictionary corresponding to the type of main subject are used less frequently (selected less frequently) compared to when the tracking mode is "normal." This reduces the possibility that a different type of subject will be erroneously detected, or that the main subject will be switched to a subject that has been erroneously detected.

[0071] In the examples of Figs. 6 and 7, the frequency of use (frequency of selection) of dictionaries other than the dictionary corresponding to the type of main subject is reduced to 0. That is, dictionaries other than the dictionary corresponding to the type of main subject are excluded from dictionary selection candidates by the dictionary switching control unit 212. However, the control of the frequency of use (frequency of selection) of dictionaries in this embodiment is not limited to the examples shown in Figs. 6 and 7. For example, when the detected main subject is a dog and the tracking mode is "tracking priority", a configuration may be adopted in which the detection cycle of dictionaries other than the dictionary corresponding to the dog is set to 4 [frames / detection]. Even in this case, since the frequency of use (frequency of selection) of dictionaries other than the dictionary corresponding to the type of main subject is reduced, the effect of reducing the possibility of erroneously detecting another type of subject or switching the main subject to a subject that has been erroneously detected can be obtained.

[0072] 7, when the main subject is detected and the tracking mode is "tracking priority", the number of dictionaries used (the number of dictionaries included in the selection candidates by dictionary switching control unit 212) is smaller than the number of detection processes that detector 213 can perform within one frame period. In this case, control unit 201 may improve tracking performance by controlling to increase the frame rate of the input image (image capture frame rate). Alternatively, a configuration may be adopted in which the frame rate of the input image is not changed, and a dictionary for a purpose other than subject detection, such as area estimation, is additionally used to improve tracking performance.

[0073] Next, examples of a dictionary detection period and a control schedule in the case where a local dictionary exists will be described with reference to FIGS.

[0074] Fig. 8(a) is a diagram showing dictionary types and processing constraints of a detector. Unlike the example of Fig. 6(a), local dictionaries exist for people, dogs, and birds. That is, in the example of Fig. 6(a), the local dictionary corresponding to people is associated with the basic dictionary corresponding to people, and a similar association exists for dogs and birds. In the processing of Figs. 4 and 5, the priority, detection period, and control schedule of each dictionary are determined based on the conditions of Fig. 8(a).

[0075] FIG. 8(b) shows the detection period of each dictionary when the tracking mode is "standard", and FIG. 8(c) shows the detection period of each dictionary when the tracking mode is "tracking priority". In FIG. 8(b) and FIG. 8(c), "0" indicates that the corresponding dictionary is not used. As shown in FIG. 8(b) and FIG. 8(c), when the main subject is not detected, the detection periods of all basic dictionaries are set to 2 [frame / detection] regardless of the tracking mode, and the detection periods of all local dictionaries are set to 0 [frame / detection].

[0076] On the other hand, when a main subject having a local dictionary is detected and the tracking mode is "standard", the detection cycles of the basic dictionary and local dictionary corresponding to the type of main subject are set to 1 [frame / detection], the detection cycles of the other basic dictionaries are set to 4 [frame / detection], and the detection cycles of the other local dictionaries are set to 0 [frame / detection], as shown in Fig. 8(b). When a main subject having no local dictionary is detected and the tracking mode is "standard", the detection cycle of the basic dictionary is set in the same way as in Fig. 6(b), and the detection cycles of all local dictionaries are set to 0 [frame / detection].

[0077] When a main subject having a local dictionary is detected and the tracking mode is "tracking priority", the detection cycle of the basic dictionary and local dictionary corresponding to the type of main subject is set to 1 [frame / detection], and the detection cycle of the other basic dictionaries and local dictionaries is set to 0 [frame / detection], as shown in Fig. 8(c). In other words, in this case, dictionaries other than the dictionary corresponding to the type of main subject are not used.

[0078] Fig. 9 is a diagram showing an example of a dictionary control schedule and its changes. In Fig. 9, reference numeral 901 indicates a dictionary scheduled for each frame (i.e., a dictionary input to the detector 213 and used in each frame). Reference numeral 902 indicates a captured image input to the detector 213.

[0079] As in the example of Fig. 7, a dog 904 is detected in the fifth frame, and the dictionary control schedule is changed in the sixth frame. However, unlike Fig. 7, in the example of Fig. 9, the detection cycle of the local dictionary corresponding to the dog is also set to 1 [frame / detection], so from the sixth frame onwards, the local dictionary corresponding to the dog is also used every frame, and in addition to the dog's face, the dog's eyes, indicated by reference numeral 905, are also detected.

[0080] Assume that the tracking mode is changed to "tracking priority" in the 10th frame. In this case, the priority of all dictionaries other than the basic dog dictionary and the local dog dictionary is set to "low." Therefore, in the following frames, only the basic dog dictionary and the local dog dictionary are input to the detector 213 and used.

[0081] 6 and 7 where no local dictionary exists, when the tracking mode is "tracking priority," the frequency of use (selection frequency) of basic dictionaries other than the dictionary corresponding to the type of main subject decreases compared to when the tracking mode is "normal." This reduces the possibility that a different type of subject will be erroneously detected, or that the main subject will be switched to a subject that has been erroneously detected.

[0082] As described above, according to the first embodiment, the camera 100 performs subject detection processing for each of a plurality of images obtained by imaging, using one or more dictionaries that are a part of the plurality of dictionaries. Each of the plurality of dictionaries is for detecting a different type of subject. The camera 100 also sets a tracking mode. The camera 100 selects one or more dictionaries to be used for the subject detection processing. With regard to this selection, the camera 100 selects a dictionary (second dictionary) different from a dictionary (first dictionary) corresponding to the type of subject detected in a previous image, with a frequency according to the tracking mode.

[0083] In this manner, according to the present embodiment, the frequency of use of a dictionary different from the dictionary corresponding to the type of detected subject can be changed through the setting of the tracking mode, making it possible to adjust the balance between the tracking performance of the detected subject and the detection performance of other subjects.

[0084] [Second embodiment] In the first embodiment, a case has been described as an example in which the number of dictionaries with a "high" priority is less than the number of times that the detector 213 can perform subject detection processing within one frame period. That is, in the first embodiment, in any frame, subject detection processing can be performed using all dictionaries with a "high" priority, and subject detection processing can be performed using at least one dictionary with a "medium" priority.

[0085] In the second embodiment, a configuration for dictionary switching control in a situation with stricter constraints will be described. In the second embodiment, the basic configuration and operation of the camera 100 are similar to those of the first embodiment. Below, differences from the first embodiment will be mainly described.

[0086] In the second embodiment, in S309 of Fig. 3, the process of Fig. 10 is executed instead of Fig. 5. Fig. 10 is a flowchart showing details of the process of determining the dictionary control schedule in S309 of Fig. 3 according to the second embodiment.

[0087] In S1000, dictionary switching control unit 212 determines whether or not the dictionary priority setting has been changed by the process of S308. If the dictionary priority setting has been changed, the process proceeds to S1002, and if the dictionary priority setting has not been changed, the process of this flowchart ends. Therefore, similar to the process of determining the dictionary priority (S308), the process of determining the dictionary control schedule is performed in two cases: when the main subject or tracking mode has changed, and when the current frame is the first frame. In all other cases, the previously determined dictionary control schedule is maintained.

[0088] In S1002, the dictionary switching control unit 212 determines whether or not a dictionary with a "medium" priority exists. If a dictionary with a "medium" priority exists, the process proceeds to S1003, and if a dictionary with a "medium" priority does not exist, the process proceeds to S1004.

[0089] In S1003, the dictionary switching control unit 212 sets the detection period for dictionaries with a "high" priority using the number of dictionaries with a "high" priority and the number of times that the detector 213 can detect per frame according to the following formula.

[0090] Detection cycle = ((number of dictionaries with "high" priority) ÷ (detection possible times per frame)) + 1 However, in the above formula, any decimals resulting from division shall be rounded down. For example, if the number of "high" priority dictionaries is 2 and the number of possible detections per frame is also 2, the detection period is set to 2÷2+1=2. Similarly, if the number of "high" priority dictionaries is 3 and the number of possible detections per frame is 2, the detection period is set to 3÷2+1=2 (decimals rounded down).

[0091] In S1004, the dictionary switching control unit 212 sets the detection period for dictionaries with a "high" priority using the number of dictionaries with a "high" priority and the number of times that the detector 213 can detect per frame according to the following formula.

[0092] Detection cycle = (number of dictionaries with "high" priority) ÷ (detection possible number per frame) However, in the above formula, unlike S1003, any decimals resulting from division are rounded up.

[0093] Fig. 11 is a diagram showing the calculation results of S1003 and S1004 in Fig. 10. Fig. 11(a) shows the number of times the detector can execute the detection process per frame (the number of times the detector can detect per frame). Fig. 11(b) shows the detection period of the "high" priority dictionary determined by the combination of the number of "high" priority dictionaries and the number of "medium" priority dictionaries.

[0094] In S1005, the dictionary switching control unit 212 sets the detection cycles for the remaining dictionaries, that is, dictionaries with "medium" or "low" priority. A specific example of setting the detection cycle in S1005 will be described later with reference to Figs. 12 to 14.

[0095] In S1006, the dictionary switching control unit 212 determines a dictionary control schedule based on the detection cycle of each dictionary and the number of detection processes that the detector 213 can execute within one frame period. A specific example of determining the dictionary control schedule will be described later with reference to Figs. 12 to 14.

[0096] Next, two specific examples of the dictionary detection cycle and control schedule according to the second embodiment will be described with reference to FIGS.

[0097] Fig. 12(a) is a diagram showing dictionary types and detector processing constraints. Fig. 12(b) is a diagram showing the detection cycle of each dictionary described in Fig. 10 when the tracking mode is "standard". Fig. 12(c) is a diagram showing the detection cycle of each dictionary described in Fig. 10 when the tracking mode is "tracking priority".

[0098] Here, referring to Fig. 13, a dictionary control schedule for a scenario in which a dog is detected as the main subject is shown. In Fig. 13, reference numeral 1301 indicates a dictionary scheduled for each frame (i.e., a dictionary input to the detector 213 and used in each frame). Reference numeral 1302 indicates a captured image input to the detector 213.

[0099] A dog 1303 appears in the third frame, and a dog 1304 is detected in the fourth frame. As a result, the dictionary control schedule is changed from the fifth frame. From the fifth frame, the priority of two dictionaries, the basic dictionary (face) and the local dictionary (eyes) for dogs, is set to "high", and the priority of dictionaries other than dogs is set to "medium". Considering that the number of detections possible per frame is two, the detection cycle of the basic dictionary and the local dictionary for dogs is determined as 2÷2+1=2 [frames / detection] in S1003 of FIG. 10. As a result, as shown in the fifth to ninth frames of FIG. 13, dictionaries related to subjects other than dogs can also be used by the detector 213 every other frame.

[0100] In this example, detector 213 has the processing capability to perform subject detection processing using two dictionaries within one frame period. In a situation where a main subject is detected and the tracking mode is "standard", the number of "high" priority dictionaries is equal to the number of dictionaries that detector 213 can use within one frame period. In this situation, dictionary switching control unit 212 alternates between selecting all "high" priority dictionaries and selecting dictionaries other than the "high" priority dictionaries for each frame (each image).

[0101] In addition, dictionaries with a "high" priority do not necessarily have to be used in the same frame. For example, a control schedule may be adopted in which the basic dictionary of dogs and people is used in the fifth frame, and the local dictionary of dogs and the basic dictionary of birds are used in the sixth frame.

[0102] Assume that the tracking mode is set to "tracking priority" in the 10th frame. At this time, the priority of dictionaries other than the basic dictionary and local dictionary for dogs is set to "low". As a result, the detection cycle of the basic dictionary and local dictionary for dogs is determined to be 2÷2=1 [frame / detection] in S1004 of FIG. 10. Therefore, in the subsequent frames, only the basic dictionary and local dictionary for dogs are input to the detector 213.

[0103] Next, a dictionary control schedule in a scenario in which a person is detected as the main subject will be shown with reference to Fig. 14. In Fig. 14, reference numeral 1401 denotes a dictionary scheduled for each frame (i.e., a dictionary input to the detector 213 and used in each frame). Reference numeral 1402 denotes a captured image input to the detector 213.

[0104] In the third frame, a person 1403 appears and is detected. As a result, the dictionary control schedule is changed from the fourth frame. From the fourth frame, the priorities of the three dictionaries, the basic human dictionary (face) and the local human dictionary (eyes and torso), are set to "high", and the priorities of the non-human dictionaries are set to "medium". Therefore, from S1003 in FIG. 10, the detection cycle of the basic human dictionary and the local human dictionary is determined to be 3÷2+1=2 [frames / detection] (decimals are rounded down). This makes it possible to use the "medium" priority dictionary in the detector 213, as shown in the fifth to ninth frames in FIG. 14.

[0105] In this example, detector 213 has the processing capability to perform subject detection processing using two dictionaries within one frame period. In a situation where a main subject is detected and the tracking mode is "standard", the number of "high" priority dictionaries is different from the number of dictionaries that detector 213 can use within one frame period by N times (N is a natural number). In this situation, dictionary switching control unit 212 selects dictionaries to be used by detector 213 so that at least one of the "high" priority dictionaries is included for each frame (each image).

[0106] Assume that the tracking mode is set to "tracking priority" in the 10th frame. At this time, the priority of dictionaries other than the basic human dictionary and local human dictionary is set to "low". As a result, the detection cycle of the basic human dictionary and local human dictionary is determined to be 3÷2=2 [frames / detection] (rounded up) by S1004 in FIG. 10. Therefore, in the subsequent frames, only the basic human dictionary and the local human dictionary are input to the detector 213.

[0107] 14, for the sake of explanation, the main subject is determined for each frame, but the main subject used by tracking calculation unit 203 does not necessarily need to be switched for each frame. For example, once an eye is detected, detection results other than the eye are not used until a predetermined number of frames have elapsed, and the main subject area is determined by tracking processing by tracking calculation unit 203.

[0108] As described above, according to the second embodiment, detector 213 has the processing capability to perform subject detection processing using a predetermined number of dictionaries (two in the examples of FIGS. 11 to 14) within one frame period. In this situation, even if the number of dictionaries with "high" priority, two or more, is equal to or greater than the number (predetermined number) of dictionaries that detector 213 can use within one frame period, it becomes possible to perform subject detection processing using dictionaries with "medium" priority.

[0109] [Other embodiments] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0110] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0111] 100 camera, 102 calculation device, 104 image sensor, 201 control unit, 202 main subject calculation unit, 211 dictionary priority determination unit, 212 dictionary switching control unit, 213 detector, 214 main subject determination unit

Claims

1. a detection means for performing subject detection processing for each of a plurality of images obtained by imaging using an imaging means, the subject detection processing using one or more dictionaries that are a part of a plurality of dictionaries, the plurality of dictionaries being each for detecting a different type of subject; A setting means for setting a tracking mode; a selection means for selecting the one or more dictionaries to be used by the detection means, the selection means selecting a second dictionary different from a first dictionary corresponding to a type of a first subject detected in a previous image with a frequency according to the tracking mode; An image processing device comprising:

2. a receiving means for receiving a user input indicating an intention to track the first object; the setting means sets the tracking mode to a first mode before receiving the user input, and sets the tracking mode to a second mode in response to receiving the user input; When the tracking mode is set to the second mode, the selection means selects the second dictionary less frequently than when the tracking mode is set to the first mode.

2. The image processing device according to claim 1,

3. When the tracking mode is set to the second mode, the selection means excludes the second dictionary from selection candidates.

3. The image processing device according to claim 2.

4. A control unit for controlling a frame rate of the image capturing is further provided. When the tracking mode is set to the second mode, the control means controls the frame rate to be higher than when the tracking mode is set to the first mode.

4. The image processing device according to claim 3.

5. the detection means has a processing capability of performing subject detection processing using a predetermined number of dictionaries within one frame period of the imaging performed at a predetermined frame rate; When the second dictionary is excluded from the selection candidates by the selection means and the number of dictionaries included in the selection candidates becomes smaller than the predetermined number, the detection means performs a predetermined detection process different from the subject detection process, using a predetermined dictionary not included in the selection candidates, in addition to the subject detection process using the dictionary included in the selection candidates.

4. The image processing device according to claim 3.

6. the plurality of dictionaries includes a plurality of primary dictionaries and one or more secondary dictionaries, each of which is associated with one of the plurality of primary dictionaries; the first dictionary corresponding to the type of the first subject is a first primary dictionary among the plurality of primary dictionaries; the second dictionary different from the first dictionary is a second primary dictionary different from the first primary dictionary among the plurality of primary dictionaries, Each of the plurality of primary dictionaries is configured to detect a particular region of a corresponding type of object; each of the one or more secondary dictionaries is configured to detect a region of an object of a type corresponding to a primary dictionary associated with the secondary dictionary that is different from the particular region; Before the detection of the first subject, the selection means selects the one or more dictionaries to be used by the detection means so as not to include a secondary dictionary associated with the first primary dictionary.

6. The image processing device according to claim 1,

7. In response to the first object being detected in the previous image, the selection means increases the selection frequency of the first dictionary.

7. The image processing device according to claim 1,

8. the detection means has a processing capability of performing subject detection processing using a predetermined number of dictionaries within one frame period of the imaging performed at a predetermined frame rate; In response to the first object being detected in the previous image, the selection means increases a selection frequency of two or more dictionaries including the first dictionary; When the number of the two or more dictionaries is equal to the predetermined number, the selection means alternately repeats, for each image, the selection of the two or more dictionaries and the selection of a dictionary other than the two or more dictionaries.

7. The image processing device according to claim 1,

9. the detection means has a processing capability of performing subject detection processing using a predetermined number of dictionaries within one frame period of the imaging performed at a predetermined frame rate; In response to the first object being detected in the previous image, the selection means increases a selection frequency of two or more dictionaries including the first dictionary; When the number of the two or more dictionaries is different from N times the predetermined number (N is a natural number), the selection means selects the one or more dictionaries to be used by the detection means so that at least one of the two or more dictionaries is included for each image.

7. The image processing device according to claim 1,

10. The first object is a main object among one or more objects detected in the previous image.

10. The image processing device according to claim 1,

11. The detection means is configured to select the main subject from the one or more objects detected in the previous image. The image processing device according to claim 10 .

12. The main subject is a subject on which the imaging means adjusts at least one of focus and exposure.

12. The image processing device according to claim 10, wherein the first and second inputs are input to the image processing device.

13. An image processing device according to any one of claims 1 to 12, The imaging means; An imaging device comprising:

14. A control method executed by an image processing device, comprising: a detection step for performing subject detection processing for each of a plurality of images obtained by imaging using an imaging means, using one or more dictionaries that are a part of a plurality of dictionaries, each of which is for detecting a different type of subject; A setting step for setting a tracking mode; a selection step of selecting the one or more dictionaries to be used in the detection step, in which a second dictionary different from a first dictionary corresponding to a type of a first subject detected in a previous image is selected with a frequency according to the tracking mode; A control method comprising:

15. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 12.

Images