Image processing device and control method thereof, imaging device, program, and storage medium

Abstract

To provide an imaging device capable of stably tracking a subject while being able to detect a plurality of types of subjects.SOLUTION: An imaging device includes a storage unit that stores a plurality of pieces of dictionary data for detecting a plurality of different subjects from an image, a detection unit that detects a subject corresponding to some of a plurality of pieces of dictionary data by using a part of the dictionary data from among the plurality of pieces of dictionary data for each frame of an image of a plurality of frames obtained by an imaging unit, and a switching unit that switches the dictionary data used by the detection unit in a plurality of frames according to the detection result of the subject by the detection unit.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to a subject detection technique in an image processing device and an imaging device. [Background technology]

[0002] Conventionally, digital cameras and other devices have been equipped with a tracking AF mode that detects human faces and eyes from images continuously output from an image sensor and continuously optimizes focus and exposure conditions for the detected human faces or eyes.

[0003] In recent years, it has become possible to use machine learning techniques to detect not only human faces and eyes, but also various types of subjects (Patent Document 1). In this technique, a specific subject is detected by inputting to a detector both an image and dictionary data that has been learned about the subject to be detected. By changing the dictionary data input to the detector, it is possible to detect different types of subjects from within an image.

[0004] As the number of types of subjects to be detected increases, the number of types of corresponding dictionary data also increases. In devices such as digital cameras, in which the number of logical operation circuits constituting a detector and processing power are limited, a control method for detecting multiple types of subjects by switching dictionary data for each frame is conceivable. Note that, here, one frame represents the time or cycle until the execution and completion of operation processing and camera control for an image output from an image sensor. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2010-154438 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, a control method that switches dictionary data for each frame in which an image is output alone cannot detect a detected subject in a frame in which dictionary data corresponding to that subject is not input, which causes a problem that stable tracking of the subject cannot be performed.

[0007] The present invention has been made in consideration of the above-mentioned problems, and has an object to provide an imaging device that is capable of detecting a plurality of types of subjects and stably tracking the subjects. [Means for solving the problem]

[0008] The image processing device according to the present invention includes a storage means for storing a plurality of dictionary data for detecting a plurality of different subjects from an image, a detection means for detecting a subject corresponding to a portion of the dictionary data from among the plurality of dictionary data for each frame of a plurality of frames of an image obtained by an imaging means, and a switching means for switching the dictionary data used by the detection means in the plurality of frames according to a result of the detection of the subject by the detection means. The switching means uses the dictionary data corresponding to the object detected by the detection means in a subsequent frame at a detection period shorter than that before the object was detected, and switches the dictionary data used by the detection means so that a combination of dictionary data used by the detection means is switched periodically for each frame. It is characterized by: Effect of the Invention

[0009] According to the present invention, it is possible to provide an imaging device that is capable of detecting a plurality of types of subjects and stably tracking the subjects. [Brief description of the drawings]

[0010] [Figure 1] 1 is a side sectional view showing the configuration of a digital single-lens camera that is a first embodiment of an imaging apparatus of the present invention. [Diagram 2] FIG. 2 is a block diagram of a control system of the imaging apparatus. [Diagram 3] 4 is a flowchart for explaining the operation of the imaging device for one frame. [Figure 4] 5 is a flowchart of a dictionary data priority calculation process according to the first embodiment. [Diagram 5] 5 is a flowchart for determining a dictionary data switching control method according to the first embodiment. [Figure 6] 6A and 6B are diagrams showing examples of detection periods of each dictionary data. [Figure 7] FIG. 13 is a diagram showing an example of dictionary data control assuming a specific case. [Figure 8] 11A and 11B are diagrams showing examples of detection periods of each dictionary data when local dictionary data is defined. [Figure 9] 13 is a diagram showing an example of dictionary data control assuming a specific case where local dictionary data is defined. FIG. [Figure 10] 13A and 13B are diagrams showing an example of a priority table and a detection period of dictionary data according to the second embodiment. [Figure 11] FIG. 11 is a diagram showing an example of dictionary data control assuming a specific case in the second embodiment. [Figure 12] FIG. 13 is a diagram showing data on subject detection frequency in the third embodiment. [Figure 13] FIG. 13 is a diagram showing an example of a detection period of dictionary data in the third embodiment. [Figure 14] 11 is a flowchart showing the contents of a continuity check process. [Figure 15] 13A and 13B are diagrams showing problems that arise when continuous shooting is started, etc. [Figure 16] FIG. 16 is a diagram showing a control concept for solving the problem shown in FIG. 15 . [Figure 17] 17 is a flowchart showing a control for realizing the control concept of FIG. 16. [Figure 18] 13A and 13B are diagrams showing the positional relationship of subjects when the subject is reselected as the main subject. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0012] (First embodiment) Fig. 1 is a side cross-sectional view showing the configuration of a digital single-lens camera (hereinafter also simply referred to as camera) 100 which is a first embodiment of an imaging device of the present invention. Fig. 2 is a block diagram showing the electrical configuration of the camera 100 of Fig. 1.

[0013] 1, in the camera 100 of this embodiment, a detachable and interchangeable lens unit 120 is attached to the front side (subject side) of the camera body 101. The lens unit 120 has a focus lens 121, an aperture 122, etc., and is electrically connected to the camera body 101 via a mount contact part 123. This makes it possible to adjust the amount of light taken into the camera body 101 and the focal position. The focus lens 121 can also be manually adjusted by the user.

[0014] The image sensor 104 is composed of a CCD sensor, a CMOS sensor, etc., and includes an infrared cut filter, a low-pass filter, etc. The image sensor 104 photoelectrically converts the subject image formed through the photographing optical system of the lens unit 120 during photographing, and transmits a signal for generating a photographed image to the calculation device 102. The calculation device 102 generates a photographed image from the received signal, stores it in the image storage unit 107, and displays it on the display unit 105 such as an LCD. The shutter 103 blocks light from the image sensor 104 when photographing is not being performed, and opens it when photographing to expose the image sensor 104 to light.

[0015] Next, the configuration related to control will be described with reference to Fig. 2. The arithmetic device 102 is equipped with a multi-core CPU capable of parallel processing of multiple tasks, RAM, and ROM, as well as dedicated circuits for executing specific arithmetic processing at high speed. With these hardware components, the arithmetic device 102 comprises a control unit 201, a main subject calculation unit 202, a tracking calculation unit 203, a focus calculation unit 204, and an exposure calculation unit 205. The control unit 201 controls each unit of the camera body 101 and the lens unit 120.

[0016] The main subject calculation unit 202 is configured to include a dictionary data priority calculation unit 211, a dictionary data switching control unit 212, a detector 213, and a main subject determination unit 214. The detector 213 performs a process of detecting a specific area (for example, a human face or eyes, or a dog's face or eyes) from an image. There are cases where no specific area is detected, and cases where multiple specific areas are detected. As a detection method, any known method such as AdaBoost or a convolutional neural network may be used. In addition, the implementation form may be a program running on a CPU, dedicated hardware, or a combination of these. In addition, the type of detected subject can be changed by switching the dictionary data input to the detector 213.

[0017] The dictionary data is data in which, for example, the characteristics of the corresponding object are registered, and control commands for the logic circuit are described for each object type. In this embodiment, the dictionary data for each object is stored in the ROM in the computing device 102 (multiple types of dictionary data are stored). Since the dictionary data exists for each object type and specific area, different types of objects can be detected by switching the dictionary data. For example, the dictionary data priority calculation unit 211 calculates the priority of the dictionary data for each predetermined period, such as for each frame, and the dictionary data switching control unit 212 determines the dictionary data to be input to the detector 213 based on the calculation result. The object detection result obtained from the detector 213 is sent to the main object determination unit 214, and the main object is determined from the detected objects and the main object area is set. The main object is determined using a known calculation method based on the size, position, reliability of the detection result, etc. If the detector 213 does not detect an object or a specific area, the main object area to be the main object is determined based on the past detection results, feature amounts such as edges of the target frame, object distance, etc.

[0018] In this embodiment, the detector 213 is a detector that performs subject detection using a CNN (Convolutional Neural Network), and the dictionary data that the detector uses to detect each subject is learned parameters that have been generated in advance by machine learning of the CNN in an external device (PC) or the imaging device 100.

[0019] The machine learning of the CNN may be performed by any method. For example, a predetermined computer such as a server may perform the machine learning of the CNN, and the imaging device 100 may acquire the trained CNN from the predetermined computer. For example, the CNN of the subject detection unit 204 may be trained by a predetermined computer performing supervised learning using the image data for learning as input and the position of the subject corresponding to the image data for learning as teacher data. In addition, the CNN of the dictionary estimation unit 205 may be trained by a predetermined computer performing supervised learning using the image data for learning as input and dictionary data corresponding to the subject of the image data for learning as teacher data. As a result, trained parameters of the CNN are generated. The CNN may be trained by the imaging device 100 or the image processing device described above.

[0020] The detector 213 used in this embodiment has a processing capability capable of detecting multiple types of subjects in one frame, but does not always detect all types of subjects that are targets of detection by the camera of this embodiment within one frame. In other words, this embodiment does not necessarily limit the processing capability of the detector 213, but in order to save processing speed and bus bandwidth, it applies detection of fewer than all types of subjects by switching dictionaries according to a predetermined setting.

[0021] The tracking calculation unit 203 tracks the main subject region based on the detection information of the main subject. The focus control unit 204 calculates the control value of the focus lens 121 for focusing on the main subject region. The exposure calculation unit 205 calculates the control values ​​of the aperture 122 and the image sensor 104 for achieving proper exposure of the main subject region.

[0022] The operation unit 106 includes a release switch, a mode dial, etc., and the control unit 201 can receive shooting instructions, mode change instructions, etc. from the user through the operation unit 106. The above is the configuration of the camera 100 according to the first embodiment of the present invention.

[0023] Next, the operation of one frame of the camera 100 in this embodiment will be described with reference to Fig. 3. The operation of the flowchart in Fig. 3 is repeatedly executed for each frame.

[0024] In step S301, the control unit 201 performs a process of reading out pixel signals from the image sensor 104. Image data is generated from the read out pixel signals. After storing the pixel signals and the generated image data in the RAM, the process proceeds to step S302.

[0025] In step S302, the process proceeds to step S306 only if it is the first frame, otherwise the process proceeds to step S303.

[0026] In step S303, the tracking calculation unit 203 performs a tracking process to calculate the position of the main subject region set in the previous frame in the current frame using the tracking reference information generated in step S310 described later for the previous frame. Any known method may be used as the algorithm used for the tracking process. For example, the tracking process can be performed by searching for an area in the current frame where the distance between the features is close, using the feature amount extracted from the main subject region set in the previous frame by a predetermined feature extraction means as the tracking reference information. In addition, the implementation form of the algorithm may be a program running on a CPU, dedicated hardware, or a combination of them. After the tracking process is completed, the tracking result (the position and size of the main subject region in the current frame) is stored in the RAM in the calculation device 102, and the process proceeds to S304.

[0027] In step S304, the focus calculation unit 204 calculates a control value for the focus lens 121 so as to focus on the tracking result, using the tracking result generated in step S303 and the signal information and image data generated in step S301. The focus lens control value may be calculated using any known method, such as a contrast method or a phase difference detection method. When the calculation of the focus lens control value is completed, the focus calculation unit 204 notifies the control unit 201 of the focus lens control value. The control unit 201 controls the focus lens 121 through the mount contact unit 123 based on the notified control value. After the above processing is completed, the process proceeds to step S305.

[0028] In step S305, the exposure calculation unit 205 calculates control values ​​for the image sensor 104 and the aperture 122 using the tracking result generated in step S303 and the signal information and image data generated in step S301 so that the exposure of the tracking result is appropriate. Any known method may be used to calculate the control values ​​for the image sensor 104 and the aperture 122. After the calculation of the control values ​​for the image sensor 104 and the aperture 122 is completed, the exposure calculation unit 205 notifies the control unit 201 of the control values ​​for the image sensor 104 and the aperture 122. The control unit 201 controls the image sensor 104 based on the notified control values ​​for the image sensor 104 and the aperture 122, and also controls the aperture 122 through the mount contact unit 123. After the above processing is completed, the process proceeds to step S306.

[0029] In step S306, the control unit 201 reads the state of the operation unit 106, and if the release switch is pressed, the process proceeds to step S307, otherwise the process proceeds to step S308.

[0030] In step S307, the control unit 201 performs imaging processing for a still image. Based on the control value calculated in step S305, the shutter 103 is driven to expose the image sensor 104. Image data is generated from pixel signals read from the exposed image sensor 104. The generated image data is stored in an external storage medium such as an SD card. After the above processing is completed, the process proceeds to step S308.

[0031] In step S308, the priority of each dictionary data is calculated. Details of this process will be described later with reference to Fig. 4. Next, the process proceeds to step S309.

[0032] In step S309, a dictionary data switching control method is determined based on the calculated priority of each dictionary data. Details of this process will be described later with reference to Fig. 5. Next, the process proceeds to step S310.

[0033] In step S310, the image data generated in step S301 and the dictionary data are input to detector 213, and a specific area of ​​the subject corresponding to the dictionary data is detected. The input dictionary data follows the dictionary data switching control method determined in step S309. Next, the process proceeds to step S311.

[0034] In step S311, the main subject region is determined from the detection result obtained in step S310, for example, by the method described above. The main subject region is determined using a known calculation method based on the part of the detection result (e.g., whole body, face, eyes, etc.), size, position, reliability of the detection result, etc. After information on the determined main subject region is stored in the RAM in the calculation device 102, the process proceeds to step S312.

[0035] In step S312, tracking calculation unit 203 generates tracking reference information to be used in the tracking process of step S303 of the next frame based on the information on the main subject region generated in step S311 and the image data generated in step S301. Then, the generated tracking reference information is stored in the RAM in calculation device 102.

[0036] When step S312 ends, the operation of one frame of the camera 100 ends. As long as the camera 100 continues to operate, the process returns to step S301 again and repeats the processes described up to this point. This concludes the description of the operation of one frame of the camera 100 in this embodiment.

[0037] Next, a method for calculating the priority of dictionary data in step S308 in FIG. 3 will be described with reference to FIG.

[0038] In step S400, the priority of each dictionary data set in the previous frame is initialized.

[0039] In step S401, it is determined whether a main subject region has been output in the previous frame by referring to main subject region information stored in the RAM in the arithmetic device 102. If a main subject region has been detected, the process proceeds to step S402, and if no subject has been detected, the process proceeds to step S405.

[0040] In step S402, the priority of the dictionary data corresponding to the main subject detected in the previous frame is set to "high." In Fig. 4, the priority of the dictionary data is set to two levels, "high" and "low." The method of setting the priority differs depending on the number of dictionary data and the processing power of the detector (types and number of times that can be detected per frame).

[0041] In step S403, it is determined whether local dictionary data is defined for the main subject. Depending on the subject, a whole region and a local region that is a part of the whole region are defined as a detection region. Specifically, for example, the relationship is as follows.

[0042] (For humans) Whole area: Face Local area: Eyes (For dogs) Whole area: Whole body / face Local area: Eyes In this embodiment, the detection of a local region (specific region) based on the dictionary data is not performed from the entire captured image, but by inputting an image obtained by cutting out a portion including the entire region of the detected subject and its vicinity from the original captured image to the detector 213. In other words, if the target subject is a person, the eyes, which are a local region, are detected only after the face of the person is detected. If there is corresponding local dictionary data, the process proceeds to step S404, and if there is no local dictionary data, the process proceeds to step S405.

[0043] In step S404, the priority of the local dictionary data corresponding to the main subject is set to "high", and the process proceeds to step S405.

[0044] In step S405, it is determined whether or not there is any dictionary data for which the priority has not been determined. If there is any dictionary data for which the priority has not been determined, the process proceeds to step S406, and if not, the dictionary data priority calculation process in step S308 ends.

[0045] In step S406, the priority of the dictionary data for which the priority has not been determined is set to "low."

[0046] Next, a method for determining the dictionary data switching control method in step S309 in FIG. 3 will be described with reference to FIG.

[0047] In step S500, the detection period of each dictionary data set determined in the previous frame is initialized. Here, the detection period is a parameter [frame / detection] indicating how many frames a single detection result is obtained. The unit of the detection period is not limited to this, and may be set, for example, by time [ms / detection]. The method of setting dictionary data used by the detector 213 to detect an object for a plurality of frames is not limited to the above. For example, a plurality of data tables indicating which combination of dictionary data the detector 213 uses in each frame may be stored in advance in the ROM in the computing device 102. The control unit 201 selects a table according to a user's setting of which object to detect in priority (e.g., person priority, animal (dog, cat, bird) priority, vehicle (2-wheel, 4-wheel) priority, etc.) and expands the table in the RAM. The detector 213 refers to the dictionary data in the order of the table stored in the RAM and detects the object corresponding to each dictionary data. The table indicating the input order of the dictionary data expanded in the RAM is rewritten at any time according to the detection situation of the object. For example, in response to detection of a specific subject such as a bird, in order to next detect the bird's eyes, the dictionary data corresponding to the bird's eyes is rewritten so that they are preferentially detected.

[0048] In step S501, it is determined whether there is any dictionary data with a "high" priority for which the detection period has not been set. If such dictionary data is present, the process proceeds to step S502, where the detection period of that dictionary is set to 1 [frame / detection] (the dictionary data is selected as the one for which detection results are obtained for each frame). Steps S501 and S502 are repeatedly executed until detection periods have been set for all dictionary data with a "high" priority. Next, the process proceeds to step S503.

[0049] In step S503, the detection period for the remaining dictionary data with "low" priority is set to a plurality of frames. Specific examples of detection periods for dictionary data with "low" priority will be described later with reference to Figures 6 and 7. Next, the process proceeds to step S504.

[0050] In step S504, a control schedule for the dictionary data is determined based on the detection cycle of each dictionary data and the number of times detection is possible per frame determined in the previous steps. A specific example of the control schedule for the dictionary data will be described later with reference to Figs. 6 and 7.

[0051] Next, a specific example of dictionary data control will be described with reference to Figs. 6 and 7. Fig. 6(a) shows the types of dictionary data, detection areas, and detector processing constraints in the control case assumed in this embodiment. In this embodiment, six types of dictionary data are prepared to detect subjects classified as people, dogs, cats, birds, four-wheeled vehicles, and two-wheeled vehicles. In this embodiment, the logic circuit block serving as the detector is one block, and detection can be performed three times in the time it takes to process one frame of image signals. In other words, it is assumed that three types of dictionary processing can be performed per frame. "Four-wheeled" refers to dictionary data for detecting four-wheeled vehicles, and examples of the vehicles include racing vehicles such as rally cars equipped with four wheels, and general passenger cars. "Two-wheeled" refers to dictionary data for detecting two-wheeled vehicles, and examples of the vehicles include motorbikes and bicycles.

[0052] Under the conditions of Fig. 6(a), the priority and detection cycle of each dictionary data are determined by steps S308 and S309 of Fig. 4 and Fig. 5. The results are shown in Fig. 6(b). Fig. 6(b) shows the detection cycle of each dictionary determined according to the detected main subject.

[0053] If no subject is detected, the detection cycle for all dictionary data is set to 2 [frames / detection]. On the other hand, if a person is detected as the subject, the detection cycle for only the dictionary data for people is set to 1 [frame / detection]. Similarly, if other subjects are detected, the detection cycle for only the dictionary data corresponding to the detected subject is set to 1 [frame / detection].

[0054] Next, an example of dictionary data control assuming a case where a dog 703 is detected in the middle of consecutive frames will be shown with reference to Fig. 7. In Fig. 7, 701 indicates dictionary data scheduled for each frame, and 702 indicates a captured image input to the detector 213.

[0055] In the example of FIG. 7, since no subject is detected in the first to fifth frames, the detection cycle of all dictionary data is set to 2 [frame / detection]. In the fourth frame, a dog 703 appears in the frame, and in the middle of the fifth frame, a dog's face 704 is output as a detection result, and the dog is determined as the main subject. From the sixth frame onwards, the detection cycle of the dog dictionary data is controlled to 1 [frame / detection], and the other dictionary data (people, cats, birds, four-wheeled vehicles, two-wheeled vehicles) is controlled to 2 [frame / detection]. From the sixth frame onwards, dictionary data corresponding to the detected main subject is input to the detector 213 in all frames (every frame from this frame onwards), so stable subject tracking is possible. In addition, since dictionary data other than the main subject is also input to the detector, even if a type of subject other than the main subject appears in the captured image, it is possible to detect it, and it is also possible to change the main subject to another type of subject.

[0056] Next, an example of dictionary data control when local dictionary data is present will be described. Fig. 8(a) shows the types of dictionary data and the processing constraints of the detector. Fig. 8(b) shows the detection cycles of each dictionary data and local dictionary data determined by steps S308 and S309 in Figs. 4 and 5. When a subject for which local dictionary data is defined is detected as the main subject, the local dictionary data is also set to a "high" priority and is controlled at a detection cycle of 1 [frame / detection]. Therefore, the detection cycle of dictionary data corresponding to subjects other than the main subject is relatively reduced.

[0057] Fig. 9 shows an example of dictionary data control based on the detection cycle of dictionary data determined by Fig. 8. A dog is detected in the fifth frame shown in Fig. 9, and the dictionary data control method is changed from the sixth frame onwards. From the sixth frame onwards, the corresponding local dictionary data is also input to the detector at a detection cycle of 1 [frame / detection], and eye 905, which is a local region of the dog, is detected.

[0058] As described above, in this embodiment, in a device in which the number of logical operation circuits constituting the detector is small compared to the number of dictionary data and processing power is poor, the priority of each dictionary data to be used for each frame is dynamically calculated according to the subject detection situation. Then, by changing the control method of the dictionary data, it is possible to detect multiple types of subjects, while enabling stable subject tracking after a specific subject is detected.

[0059] Second Embodiment Next, a second embodiment of the present invention will be described. In the first embodiment, the priority of each dictionary data other than the detected main subject is relatively lowered uniformly. In this embodiment, the priority is calculated for each dictionary data other than the main subject, and it is determined which dictionary data is to be preferentially input to detector 213. It is desirable to determine the relative priority of each dictionary data in accordance with an actual use case.

[0060] For example, if a dog is the main subject, it is thought that the subject that is most often photographed together with a dog is a person. Possible scenes include a walk, a person holding a dog, or a person playing with a dog, so the priority of the dictionary data for people should be set high. On the other hand, it is thought that there are few use cases in which a dog is photographed together with a four-wheeled or two-wheeled vehicle, so the priority of the dictionary data for four-wheeled or two-wheeled vehicles should be set low.

[0061] In actual control, based on this concept, the priority of each dictionary data is prepared in advance as a design table stored in ROM, and the priority of the dictionary data is determined by referring to this table according to the state of the detected subject.

[0062] Fig. 10(a) shows a specific example of a dictionary data priority table. Here, the priority of dictionary data is expressed in five stages from 5 (high) to 1 (low). Based on this table, the detection period of each dictionary data is calculated and a dictionary data switching control method is determined. Fig. 10(b) shows an example of the detection period of each dictionary data determined based on the priority of Fig. 10(a).

[0063] Next, an example of dictionary data control assuming a case where a dog 1103 is detected in the middle of consecutive frames will be shown with reference to Fig. 11. A dog 1103 is detected as a subject in the fifth frame, and dictionary data control is changed from the sixth frame onwards. Dictionary data corresponding to the dog's eyes, which are a local region of the dog, is input to the detector from the sixth frame onwards, so that the dog's eyes 1105 are detected. The detection cycle of dictionary data other than that of the dog is scheduled according to Fig. 10(b).

[0064] As described above, in this embodiment, the priority of each dictionary data is determined in accordance with an actual use case, which allows a device with a limited number of logic circuits constituting a detector and limited processing power to detect various types of subjects more quickly and to change the main subject more quickly.

[0065] (Third embodiment) Next, a third embodiment of the present invention will be described. In the third embodiment, the priority calculation step of each dictionary data shown in Fig. 3 is performed according to the past detection history. This embodiment differs from the first embodiment only in steps S308, S309, and S311 in Fig. 3, and the other processes are the same as those in the first embodiment.

[0066] In the process of determining a main subject from among the detected subjects in step S311, a process is added in which detection information of subjects detected in the same frame (simultaneously) as the main subject is accumulated for each subject type and stored as subject detection frequency data.

[0067] 12 is a bar graph showing an example of subject detection frequency data for each subject type, and indicates the detection frequency of other types of subjects in the same frame when a specific subject is detected as the main subject. As in the first embodiment, the subjects to be detected are people, dogs, cats, birds, four-wheeled vehicles, and two-wheeled vehicles, and the bar graph shows the detection frequency of other subjects when each of the subjects is detected as the main subject. Here, the detection frequency of the type of subject itself detected as the main subject is not included, and is therefore indicated as N / A (Not Applicable).

[0068] Next, in step S308, the subject detection frequency data accumulated in the previous step S311 is referenced, and the priority of each dictionary data for the current subject detection is determined according to the subject type detected as the main subject at that time. The dictionary data of a subject type detected frequently is set to a high priority, and the dictionary data of a subject type detected infrequently is set to a low priority.

[0069] Next, in step S309, the detection period for each dictionary data is determined based on the priority of the dictionary data that has already been determined. Fig. 13 shows an example of the determined detection period for each dictionary data.

[0070] As described above, in this embodiment, the priority of each dictionary data is determined based on the past subject detection frequency data, so that dictionary data control suited to the user can be performed. Therefore, even in a device with a limited number of logic operation circuits constituting a detector and a limited processing capacity, various types of subjects can be detected more quickly, and the main subject can be changed more quickly.

[0071] (Fourth embodiment) Next, a fourth embodiment of the present invention will be described. In subject detection, in order to prevent erroneous detection, an algorithm is used that detects a subject a certain number of times in succession before adopting the detection result (hereinafter, referred to as continuity check).

[0072] However, in shooting modes where immediacy is required in display and tracking control during continuous shooting, etc., continuity checks are not performed, and if a subject is detected even once, the result is immediately adopted as the main subject, and is reflected in tracking control, dictionary data control, and frame display.

[0073] However, there is a possibility of false positives in a single detection result. For example, there is a possibility of false positives such as detecting part of a two-wheeled vehicle as a dog. Even if there is a false positive, if the target subject continues to be detected continuously, there will be no major problems in tracking control. However, false positives often lead to an unstable detection state, which increases the possibility of causing problems in subject tracking.

[0074] In this embodiment, therefore, in a shooting mode that does not check continuity, such as a continuous shooting mode, all types of dictionary data are continued to operate even after a subject is detected once and adopted as the main subject, and if the subject is detected continuously a certain number of times or more, it is adopted again as the correct main subject. The tracking control, dictionary data control, and frame display control are also changed to match the newly adopted main subject.

[0075] The above continuity check is performed in the main subject determination process of S311 in Fig. 3. The continuity check process is described in Fig. 14. As described in Fig. 3, the flow chart in Fig. 3 is repeatedly performed for each frame, and therefore the operation of the flow chart in Fig. 14 is also performed for each frame.

[0076] In step S1401, it is checked whether or not there is a subject detection result in the input image. If there is a subject detection result, the process proceeds to step S1402. If there is no subject detection result, the number of consecutive detections is initialized to 0 in step S1403, and the detection result is not adopted (step S1407).

[0077] Next, in step S1402, it is checked whether the number of consecutive detections is 0 or whether the subject ID detected in the previous frame is the same as the subject ID detected in the current frame. Correlation processing between subjects between frames is performed before the continuity check processing, and subjects are linked by the subject ID. If the number of consecutive detections is 0, this is the first detection result, so the process proceeds to step S1404. Alternatively, if the previous subject ID and the current subject ID match, the process similarly proceeds to step S1404. For example, even if the number of consecutive detections is 1 or more, if the condition that the previous subject ID and the current subject ID match is met, the process proceeds to step S1404. If none of the above conditions are met, the process proceeds to step S1403.

[0078] In step S1403, the number of consecutive detections is initialized to 0, and in step S1407, the detection result in this case is not adopted.

[0079] In step S1404, the number of consecutive detections is incremented by 1, and the process proceeds to step S1405. The number of consecutive detections is stored in a memory or the like each time the operation in Fig. 14 is performed once. In step S1402 in Fig. 14, if the condition that the immediately preceding subject ID and the current subject ID match continues for multiple frames, the number of consecutive detections is counted up in the memory.

[0080] Next, in step S1405, it is checked whether the number of consecutive detections is equal to or greater than the threshold value of the number of consecutive detections. In a shooting mode that requires immediacy, such as continuous shooting, the threshold value of the number of consecutive detections is set to 1 (continuous shooting mode), and in other cases, the threshold value of the number of consecutive detections is set to a number greater than 1 (normal mode). If the number of consecutive detections exceeds the threshold value of the number of consecutive detections, the process proceeds to step S1406, where the detection result at that time is adopted and is adopted as the main subject after the continuity check process of FIG.

[0081] If the number of consecutive detections is less than the threshold value for the number of consecutive detections, the process proceeds to step S1407, and the detection result at that time is not adopted.

[0082] Next, a problem that is taken up in this embodiment at the start of continuous shooting etc. will be described with reference to FIG.

[0083] In FIG. 15, images 1501 and 1502 show input images of two wheels, and frame 1503 represents the AF frame. The detection state and tracking state for each image frame are displayed below the input image. In input image 1501, when continuous shooting is started without a main subject, subject detection is performed in the AF frame 1503, and thereafter the detected subject is tracked across the entire angle of view. Data 1504 represents dictionary data to be input in each image frame. In this example, it is assumed that three dictionary data can be set for one frame.

[0084] In Fig. 15, part of the two-wheeled vehicle is erroneously detected as a dog 1505 in the first frame, and the detection result of the dog 1505 is selected as the main subject and tracking is performed from the second frame onwards. The correct detection result in this image is two wheels, but because control is performed to increase the priority of the dictionary data for dogs from the second frame onwards, dictionary data for two-wheeled and four-wheeled vehicles is not set. Therefore, the target subject cannot be re-detected as two wheels. The control concept in this embodiment that solves this problem is shown in Fig. 16.

[0085] In FIG. 16, even if a part of two wheels is detected as a dog 1505 in the first frame, dictionary data control is not performed according to the dog selected as the main subject for a certain period from the second frame onwards, and dictionary data control is performed in the same way as when there is no main subject (dictionary data of all types is set). In FIG. 16, the detection result of two wheels 1506, which is a correct detection, is obtained multiple times from the second frame onwards, and the number of consecutive detections exceeds the threshold value for the number of consecutive detections in the eighth frame (set to four times in FIG. 16). Then, from the next ninth frame, the main subject is changed from the dog to two wheels, and dictionary data control is also changed according to the two wheels. By adopting the new subject detection results detected multiple times as the main subject, it is possible to correct erroneous detections while maintaining immediacy during continuous shooting, and to reduce the risk of erroneous tracking.

[0086] Next, a control flow for realizing the control concept of Fig. 16 is shown in Fig. 17. The control flow of Fig. 17 is performed in the main subject determination process of step S311 of Fig. 3. As described in Fig. 3, the flowchart of Fig. 3 is repeatedly performed for each frame, and therefore the operation of the flowchart of Fig. 17 is also performed for each frame.

[0087] 17, in step S1701, it is determined whether or not a main subject is already present. If a main subject is already present, the process ends. If no main subject is present, the process proceeds to step S1702, where it is determined whether or not the continuous shooting mode is selected.

[0088] If the continuous shooting mode is not selected, the continuity check can be performed in the normal mode, and the process proceeds to the continuity check in step S1706 (normal mode: number of continuous detections=4). If the continuous shooting mode is selected, a check is made in step S1703 to see if there is a tentative main subject. A tentative main subject here refers to a subject that is selected as the main subject without going through the continuity check (normal mode: number of continuous detections=4), such as the dog that was the main subject in frames 2 to 8 in FIG. 16. If this subject is the tentative main subject, it will be a tracking target, but the dictionary data control is the same as the dictionary data control when there is no main subject, in which all types of dictionary data are set uniformly.

[0089] If there is no tentative subject in step S1703, a continuity check is performed in step S1704 (continuous shooting mode: continuous detection count=1 time), and if there is a detection result in step S1705, it is adopted as the tentative main subject.

[0090] Next, in step S1709, the process ends by initializing the tentative main subject frame number to 0. The tentative main subject frame number is the number of image frames counted after the tentative main subject is set.

[0091] If a provisional subject is found in step S1703, the process proceeds to step S1708.

[0092] In step S1708, it is determined whether the number of tentative main subject frames exceeds the threshold value of the tentative main subject frames. If the number of tentative main subject frames exceeds the threshold value of the tentative main subject frames, it is determined that there is no subject more likely than the tentative main subject, and the tentative main subject is changed to the main subject in step S1710, and the process ends. Note that in this flowchart, the number of tentative main subject frames is stored in a memory or the like each time the operation in FIG. 17 is performed once, and the number of tentative main subject frames is counted up.

[0093] If the number of tentative main subject frames is equal to or less than the tentative main subject frame threshold, the process proceeds to step S1706. The tentative main subject frame threshold is set to limit the control state by the tentative main subject for a certain period of time, and the value is set to at least equal to or greater than the continuous detection count threshold.

[0094] Next, a continuity check (normal mode) is performed in step S1706, and if a detection result exceeding the threshold for the number of consecutive detections is obtained, in step S1707, that detection result is adopted as the main subject, and the process ends.

[0095] The above-described series of controls can also be applied to a case where the main subject that has been tracked during continuous shooting is lost and a new subject is detected again.

[0096] Also, only when the detection area of ​​the subject to be reselected and its neighboring area overlap with the detection area of ​​the provisional subject, the subject is reselected as the main subject. The positional relationship of the subjects is shown in Fig. 18. Reference numeral 1801 denotes the image area, 1802 denotes the detection area of ​​the provisional main subject, 1803 denotes the detection area of ​​the subject to be reselected, and 1804 denotes the neighboring area of ​​the detection area 1803.

[0097] In FIG. 18A, since there is no area that overlaps with detected area 1802, detected area 1803 is not selected as a new main subject.

[0098] 18(B) and 18(C), there is an area that overlaps with detection area 1802, so detection area 1803 is selected as the new main subject. Nearby area 1804 is set depending on the type of subject to be reselected and the amount of movement up to that point. For example, if the subject to be reselected is an animal, it is expected that it will move quickly and move a lot over several frames, so nearby area 1804 is set relatively wide. On the other hand, if the subject to be reselected is a person, it is unlikely that they will move a lot over several frames, so a setting method such as making the nearby area relatively narrow is conceivable.

[0099] By doing so, the risk of mistakenly selecting another surrounding subject can be reduced.

[0100] As described above, in this embodiment, in a shooting mode that does not perform a continuity check, such as a continuous shooting mode, even after a subject is detected once and adopted as a main subject, all types of dictionary data are continued to operate, and if the subject is detected continuously a certain number of times or more, it is adopted again as the correct main subject. The tracking control, dictionary data control, and frame display control are also changed according to the main subject that has been adopted again. In this way, it is possible to suppress a decrease in tracking performance due to erroneous detection while ensuring the immediacy of the tracking control and frame display control required in the continuous shooting mode.

[0101] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0102] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Industrial Applicability]

[0103] The present invention is not limited to imaging with digital cameras or digital video cameras, but can also be installed in imaging devices such as surveillance cameras, Web cameras, and mobile phones. [Explanation of symbols]

[0104] 102: Calculation device, 104: Image sensor, 202: Main subject calculation section, 203: Tracking calculation section, 211: Dictionary data priority calculation section, 212: Dictionary data switching control section, 213: Detector, 214: Main subject determination section

Claims

1. A storage means for storing a plurality of dictionary data for detecting a plurality of different subjects from an image; a detection means for detecting an object corresponding to a part of the dictionary data by using a part of the plurality of dictionary data for each frame of a plurality of frames of images obtained by an imaging means; a switching means for switching dictionary data used by the detection means in the plurality of frames in response to a result of the object detection by the detection means; Equipped with The image processing device is characterized in that the switching means uses dictionary data corresponding to a subject detected by the detection means in subsequent frames at a shorter detection period than before the subject was detected, and switches the dictionary data used by the detection means so that the combination of dictionary data used by the detection means is switched periodically for each frame.

2. 2. The image processing apparatus according to claim 1, wherein said detection means detects a main subject from said image.

3. 3. The image processing apparatus according to claim 2, wherein the main subject is a subject on which the image capturing means adjusts at least one of focus and exposure.

4. 4. The image processing device according to claim 1, wherein the storage means stores a plurality of types of tables for switching between the plurality of dictionary data for each frame, and the switching means switches the dictionary data to be used by the detection means using a table that has been set in advance by a user from among the plurality of types of tables.

5. The image processing device according to any one of claims 1 to 4, characterized in that the switching means switches the dictionary data used by the detection means to use second dictionary data corresponding to the detection of a local area of ​​the subject corresponding to the first dictionary data in response to the detection of a corresponding subject using the first dictionary data.

6. 6. The image processing apparatus according to claim 5, wherein the switching means switches the dictionary data used by the detection means so that, when the second dictionary data is present, the second dictionary data is preferentially used together with the first dictionary data.

7. 7. The image processing apparatus according to claim 1, wherein the switching means has a priority table for determining priorities of the plurality of dictionary data in accordance with the subject detected by the detection means.

8. 8. The image processing apparatus according to claim 7, wherein the switching means determines the priorities of the plurality of dictionary data based on the priority table.

9. 9. The image processing apparatus according to claim 1, wherein the switching means stores frequency data that is a cumulative count of the frequency of a type of object being detected simultaneously with a specific object.

10. 10. The image processing apparatus according to claim 9, wherein the switching means determines priorities of the plurality of dictionary data based on the frequency data.

11. 11. The image processing apparatus according to claim 1, wherein the plurality of dictionary data are convolutional neural networks trained by machine learning.

12. The image processing device according to any one of claims 1 to 11, characterized in that the switching means performs switching control of dictionary data so that, even if a first subject is detected and selected as a main subject in a specific shooting mode, if a second subject other than the first subject is detected consecutively a predetermined number of times or more, the second subject is re-selected as the main subject.

13. 13. The image processing apparatus according to claim 12, wherein the switching means performs the switching control even when the main subject being tracked is lost in the specific photographing mode and the subject is redetected.

14. 13 . The image processing device according to claim 12 , wherein the switching means reselects the main subject when the detection area of ​​the second subject and an area including the vicinity thereof overlap with the detection area of ​​the first subject.

15. An imaging means for capturing an image; An image processing device according to any one of claims 1 to 14, An imaging device comprising:

16. a detection step of detecting, for each frame of a plurality of frames of images obtained by the imaging means, a subject corresponding to a portion of a plurality of dictionary data for detecting a plurality of different subjects from the image, using the portion of dictionary data; a switching step of switching dictionary data used in the detection step in the plurality of frames according to a result of the detection of the object in the detection step; having A control method for an image processing device, characterized in that in the switching process, dictionary data corresponding to a subject detected in the detection process is used in subsequent frames at a shorter detection period than before the subject was detected, and the dictionary data used in the detection process is switched so that the combination of dictionary data used in the detection process is switched periodically for each frame.

17. A program for causing a computer to execute each step of the control method according to claim 16.

18. A computer-readable storage medium storing a program for causing a computer to execute each step of the control method according to claim 16.

Images