Image processing device and control method thereof

Abstract

To provide an image processing apparatus including a subject tracking function to realize favorable performance while suppressing power dissipation, and a control method thereof.SOLUTION: An image processing apparatus includes first tracking means and second tracking means having higher processing accuracy and higher arithmetic load than those of the first tracking means. The image processing apparatus calculates an evaluation value regarding a difficulty of tracking for a subject region to be tracked, which is determined on the basis of a frame to be processed to which tracking processing is to be applied. On the basis of the evaluation value, the image processing apparatus enables both the first tracking means and the second tracking means or disables at least one of the first tracking means and the second tracking means, regarding a frame after the frame to be processed.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present invention relates to an image processing device and a control method thereof, and in particular to a subject tracking technique. [Background technology]

[0002] Some image processing devices such as digital cameras have a function of tracking characteristic regions (subject tracking function) by detecting characteristic regions such as face regions over time. Also, devices that track subjects using trained neural networks are known (Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2017-156886 A Summary of the Invention [Problem to be solved by the invention]

[0004] By using a neural network, it may be possible to improve the accuracy of object tracking compared to using correlation or similarity between image regions. However, processing using a neural network requires a large amount of calculation, a high-speed processor, and a large-scale circuit, which results in a problem of high power consumption. For example, when object tracking using a neural network is applied to a moving image for live view display, battery consumption due to the live view display becomes a problem.

[0005] The present invention has been made in consideration of the above problems of the conventional technology, and has an object to provide an image processing device with a subject tracking function that realizes good performance while suppressing power consumption, and a control method thereof. [Means for solving the problem]

[0006] The above-mentioned purposes are1. A detection means for detecting a predetermined subject area from a frame as a candidate area, a determination means for determining a subject area to be tracked from the candidate area, and a calculation means for calculating an evaluation value relating to the difficulty of tracking the subject area determined by the determination means. No. 1 In the frame of No. 1 Before the frame The second image captured a tracking unit that performs a tracking process to estimate a position of an area corresponding to a subject area of ​​a tracking target in the frame, the tracking unit having a first tracking unit and a second tracking unit, and No. 1 After the frame The third image captured on and a control means for determining the operation of the first tracking means and the second tracking means for the frame, the second tracking means having a higher processing accuracy but a larger calculation load than the first tracking means, and the control means for determining whether to enable both the first tracking means and the second tracking means or disable at least one of the first tracking means and the second tracking means based on the evaluation value. When both the first tracking means and the second tracking means are enabled, the control means operates the second tracking means less frequently than the first tracking means. The object of the present invention is achieved by an image processing device comprising: Effect of the Invention

[0007] According to the present invention, it is possible to provide an image processing device having a subject tracking function that realizes good performance while suppressing power consumption, and a control method for the image processing device. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional arrangement of an imaging device according to a first embodiment; [Diagram 2] FIG. 13 is a diagram showing an example of display of a subject tracking result in the imaging device according to the first embodiment; [Diagram 3] Flowchart for difficulty score calculation processing in the first embodiment [Figure 4] Flowchart regarding control operation based on difficulty score in the first embodiment [Diagram 5] 1 is an exemplary timing chart relating to processing of an imaging device according to a first embodiment; [Figure 6]Timing chart regarding use of tracking processing results in the first embodiment [Figure 7] Flowchart regarding basic live view display operation in the first embodiment [Figure 8] Flowchart for live view display operation during subject tracking in the first embodiment [Figure 9] FIG. 13 is a block diagram showing an example of the functional arrangement of an imaging device according to a second embodiment. [Figure 10] FIG. 13 is a diagram showing an example of a live view display during subject tracking in the second embodiment. [Figure 11] Flowchart of difficulty score calculation process in the second embodiment [Figure 12] Flowchart of control operation based on difficulty score in the second embodiment [Figure 13] Flowchart regarding control operation without using difficulty score in the second embodiment [Figure 14] 13 is an exemplary timing chart relating to processing of an imaging device according to a second embodiment; [Figure 15] Flowchart showing a method for determining a subject region of a tracking target in the second embodiment [Figure 16] Flowchart for live view display operation during subject tracking in the second embodiment [Figure 17] FIG. 13 is a block diagram showing an example of the functional configuration of an imaging device according to the third to eighth embodiments. [Figure 18] FIG. 1 is a diagram showing a configuration of a line-of-sight information acquisition unit according to an embodiment; [Figure 19] FIG. 13 is a diagram showing functional blocks relating to enable / disable control of a subject detection unit in the third embodiment. [Figure 20] 11 is a flowchart showing the operation of the system control unit in the third embodiment. [Figure 21] 11 is a flowchart showing the operation of the system control unit in the third embodiment. [Figure 22] 11 is a flowchart showing the operation of the system control unit in the third embodiment. [Figure 23]13 is a flowchart showing the operation of the system control unit in the fourth embodiment. [Figure 24] Flowchart regarding the operation of the system control unit in the fifth embodiment [Diagram 25] Flowchart regarding the operation of the system control unit in the sixth embodiment [Figure 26] Flowchart regarding the operation of the system control unit in the seventh embodiment [Figure 27] Flowchart regarding the operation of the system control unit in the eighth embodiment [Figure 28] FIG. 13 is a block diagram showing an example of the functional configuration of an imaging device according to the 9th to 18th embodiments. [Figure 29] Flowchart regarding the operation of the system control unit in the ninth embodiment [Diagram 30] Flowchart regarding the operation of the system control unit in the tenth embodiment [Diagram 31] Flowchart regarding the operation of the system control unit in the eleventh embodiment [Diagram 32] Flowchart regarding the operation of the system control unit in the twelfth embodiment [Diagram 33] Flowchart regarding the operation of the system control unit in the thirteenth embodiment [Diagram 34] Flowchart regarding the operation of the system control unit in the fourteenth embodiment [Diagram 35] Flowchart regarding the operation of the system control unit in the fifteenth embodiment [Diagram 36] Flowchart regarding the operation of the system control unit in the sixteenth embodiment [Figure 37] Flowchart regarding the operation of the system control unit in the seventeenth embodiment [Figure 38] Flowchart regarding the operation of the system control unit in the 18th embodiment [Figure 39] FIG. 23 is a diagram showing an example of the rear appearance of an imaging device according to a twelfth embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in an imaging device such as a digital camera. However, an imaging device is merely one example of an image processing device to which the present invention can be applied, and the present invention can be implemented in any electronic device capable of image processing. Such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] <First embodiment> FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 100 as an example of an image processing device according to the first embodiment. The optical system 101 has a plurality of lenses including a movable lens such as a focus lens, and forms an optical image of the shooting range on the imaging plane of the image sensor 103 .

[0012] The control unit 102 has a CPU, and for example, loads a program stored in a ROM 123 into a RAM 122 and executes the program. The control unit 102 realizes the functions of the imaging device 100 by controlling the operation of each functional block. The ROM 123 is, for example, a rewritable non-volatile memory, and stores programs executable by the CPU of the control unit 102, setting values, GUI data, and the like. The RAM 122 is a system memory used to load a program executed by the CPU of the control unit 102 and to store values ​​required during the execution of a program. Although omitted in FIG. 1, the control unit 102 is connected to each functional block so as to be able to communicate with them.

[0013] The imaging element 103 may be, for example, a CMOS image sensor having a primary color Bayer array color filter. A plurality of pixels having photoelectric conversion regions are arranged two-dimensionally in the imaging element 103. The imaging element 103 converts an optical image formed by the optical system 101 into a group of electrical signals (analog image signals) by the plurality of pixels. The analog image signals are converted into digital image signals (image data) by an A / D converter included in the imaging element 103 and output. The A / D converter may be provided outside the imaging element 103.

[0014] The evaluation value generation unit 124 generates a signal and an evaluation value used for automatic focus detection (AF) from image data obtained from the image sensor 103, and calculates an evaluation value used for automatic exposure control (AE). The evaluation value generation unit 124 outputs the generated signal and evaluation value to the control unit 102. The control unit 102 controls the focus lens position of the optical system 101 and determines shooting conditions (exposure time, aperture value, ISO sensitivity, etc.) based on the signal and evaluation value obtained from the evaluation value generation unit 124. The evaluation value generation unit 124 may generate a signal and an evaluation value from display image data generated by a post-processing unit 114 described later.

[0015] The first pre-processing unit 104 applies color interpolation processing to the image data obtained from the imaging element 103. The color interpolation processing is also called demosaic processing, and is processing in which each of the pixel data constituting the image data has values ​​of R, G, and B components. The first pre-processing unit 104 may also apply reduction processing to reduce the number of pixels as necessary. The first pre-processing unit 104 stores the processed image data in the display memory 107.

[0016] The first image correction unit 109 applies correction processes such as white balance correction and shading correction, and conversion from RGB format to YUV format to the image data stored in the display memory 107. When applying the correction processes, the first image correction unit 109 may use image data of one or more frames different from the frame to be processed, among the image data stored in the display memory 107. The first image correction unit 109 may use, for example, image data of a frame chronologically before and / or after the frame to be processed, for the correction process. The first image correction unit 109 outputs the image data to which the process has been applied to the post-processing unit 114.

[0017] The post-processing unit 114 generates image data for recording or image data for display from the image data supplied from the first image correction unit 109. The post-processing unit 114 applies an encoding process to the image data, for example, and generates a data file for storing the encoded image data as image data for recording. The post-processing unit 114 supplies the image data for recording to the recording unit 118.

[0018] Furthermore, the post-processing unit 114 generates display image data to be displayed on the display unit 121, from the image data supplied from the first image correction unit 109. The display image data has a size according to the display size of the display unit 121. The post-processing unit 114 supplies the display image data to the information superimposition unit 120.

[0019] The recording unit 118 records the recording image data converted by the post-processing unit 114 on a recording medium 119. The recording medium 119 may be, for example, a semiconductor memory card, a built-in non-volatile memory, or the like.

[0020] The second pre-processing unit 105 applies color interpolation processing to the image data output by the imaging element 103. The second pre-processing unit 105 stores the processed image data in the tracking memory 108. The tracking memory 108 and the display memory 107 may be implemented as separate address spaces in the same memory space. The second pre-processing unit 105 may also apply a reduction process to reduce the number of pixels as necessary to reduce the processing load. Note that although the first pre-processing unit 104 and the second pre-processing unit 105 are described here as separate functional blocks, they may also be configured to use a common pre-processing unit.

[0021] The second image correction unit 106 applies correction processes such as white balance correction and shading correction, and conversion from RGB format to YUV format to the image data stored in the tracking memory 108. The second image correction unit 106 may also apply image processing suitable for subject detection processing to the image data. For example, if the representative luminance of the image data (for example, the average luminance of all pixels) is equal to or less than a predetermined threshold, the second image correction unit 106 may multiply the entire image data by a constant coefficient (gain) so that the representative luminance becomes equal to or greater than the threshold.

[0022] When applying the correction process, the second image correction unit 106 may use image data of one or more frames different from the frame to be processed, among the image data stored in the tracking memory 108. For example, the second image correction unit 106 may use image data of frames chronologically before and / or after the frame to be processed, for the correction process. The second image correction unit 106 stores the image data to which the process has been applied in the tracking memory 108.

[0023] In addition, functional blocks related to the subject tracking function, such as the second pre-processing unit 105 and the second image correction unit 106, do not need to operate when the subject tracking function is not performed. In addition, the image data to which the subject tracking function is applied is video data captured for live view display or recording. The video data has a predetermined frame rate, such as 30 fps, 60 fps, or 120 fps.

[0024] The detection unit 110 detects one or more regions (candidate regions) of predetermined candidate subjects from one frame of image data. The detection unit 110 also associates, for each detected region, the position and size within the frame, an object class indicating the type of candidate subject (car, airplane, bird, insect, human body, head, eye, cat, dog, etc.), and its reliability. The detection unit 110 also counts the number of detected regions for each object class.

[0025] The detection unit 110 can detect candidate regions using known techniques for detecting feature regions such as the face region of a person or an animal. For example, the detection unit 110 may be configured as a class classifier that has been trained using training data. There are no particular limitations on the classification algorithm. The detection unit 110 can be realized by training a classifier that implements multi-class logistic regression, a support vector machine, a random forest, a neural network, or the like. The detection unit 110 stores the detection results in the tracking memory 108.

[0026] The object determining unit 111 determines a subject area (main subject area) to be tracked from the candidate areas detected by the detection unit 110. The subject area to be tracked can be determined based on a priority given in advance to each item included in the detection result, such as object class and area size. Specifically, the total of the priority may be calculated for each candidate area, and the candidate area with the smallest total may be determined as the subject area to be tracked. Alternatively, among the candidate areas belonging to a specific object class, the candidate area closest to the center of the image or the focus detection area, or the largest candidate area may be determined as the subject area to be tracked. The object determining unit 111 stores information specifying the determined subject area in the tracking memory 108.

[0027] The difficulty level determination unit 112 calculates a difficulty score, which is an evaluation value indicating the difficulty level of tracking, for the subject area of ​​the tracking target determined by the target determination unit 111. For example, the difficulty level determination unit 112 can calculate the difficulty score by considering one or more factors that affect the difficulty level of tracking. Examples of factors that affect the difficulty level of tracking include, but are not limited to, the size of the subject area, the object class (type) of the subject, the total number of areas belonging to the same object class, and the position within the image. A specific example of a method of calculating the difficulty score will be described later. The difficulty level determination unit 112 outputs the calculated difficulty score to the tracking control unit 113.

[0028] The tracking control unit 113 determines whether to enable or disable each of the multiple tracking units included in the tracking unit 115 based on the difficulty score calculated by the difficulty determination unit 112. In this embodiment, the tracking unit 115 has multiple tracking units with different calculation loads and tracking accuracy. Specifically, the tracking unit 115 has a DL tracking unit 116 that tracks a subject using deep learning (DL) and a non-DL tracking unit 117 that tracks a subject without using DL. The DL tracking unit 116 has a higher processing accuracy than the non-DL tracking unit 117, but a larger calculation load than the non-DL tracking unit 117.

[0029] In this case, the tracking control unit 113 determines whether to enable or disable each of the DL tracking unit 116 and the non-DL tracking unit 117. The tracking control unit 113 also determines the operation frequency of the enabled tracking unit. The operation frequency is the frequency (fps) at which the tracking process is applied.

[0030] Tracking unit 115 estimates the subject area of ​​the tracking target from image data of a frame to be processed (current frame) stored in tracking memory 108, and obtains the position and size of the estimated subject area within the frame as a tracking result. Tracking unit 115 estimates the subject area of ​​the tracking target within the current frame, for example, using image data of the current frame and image data of a previous frame (e.g., the frame immediately preceding) captured before the current frame. Tracking unit 115 outputs the tracking result to information superimposition unit 120.

[0031] Here, tracking unit 115 estimates an area in the frame to be processed that corresponds to the subject area of ​​the tracking target in the past frame. In other words, the subject area of ​​the tracking target determined by target determination unit 111 for the frame to be processed is not the subject area of ​​the tracking target in the tracking process for the frame to be processed. The subject area of ​​the tracking target in the tracking process for the frame to be processed is the subject area of ​​the tracking target in the past frame. The subject area of ​​the tracking target determined by target determination unit 111 for the frame to be processed is used for the tracking process of the next frame when the subject of the tracking target is switched to another subject.

[0032] The tracking unit 115 has a DL tracking unit 116 that tracks a subject using deep learning (DL) and a non-DL tracking unit 117 that tracks a subject without using DL. Then, the tracking unit enabled by the tracking control unit 113 outputs a tracking result at an operation frequency set by the tracking control unit 113.

[0033] The DL tracking unit 116 estimates the position and size of the subject area of ​​the tracking target using a trained multi-layer neural network including a convolution layer. More specifically, the DL tracking unit 116 has a function of extracting feature points for the subject area of ​​each object class that may be a target and feature amounts contained in the feature points, and a function of associating the extracted feature points between frames. Therefore, the DL tracking unit 116 can estimate the position and size of the subject area of ​​the tracking target in the current frame from the feature points of the current frame that are associated with the feature points of the subject area of ​​the tracking target in the past frame.

[0034] The DL tracking unit 116 outputs the position, size, and reliability score of the subject region of the tracked target estimated for the current frame. The reliability score indicates the reliability of the correspondence of feature points between frames, i.e., the reliability of the estimation result of the subject region of the tracked target. If the reliability score indicates that the reliability of the correspondence of feature points between frames is low, it indicates that the subject region estimated in the current frame may be a region related to a different subject from the subject region of the tracked target in the past frame.

[0035] On the other hand, the non-DL tracking unit 117 estimates the subject area of ​​the tracked target in the current frame by a method that does not use deep learning. Here, the non-DL tracking unit 117 estimates the subject area of ​​the tracked target based on the similarity of color configuration. However, other methods such as pattern matching using the subject area of ​​the tracked target in a past frame as a template may be used. The non-DL tracking unit 117 outputs the position, size, and reliability score of the subject area of ​​the tracked target estimated for the current frame.

[0036] Here, the similarity of color configuration will be explained. For ease of explanation and understanding, it is assumed that the shape and size of the subject area to be tracked are the same in the past frame and the current frame. It is also assumed that the image data has a depth of 8 bits (values ​​0 to 255) for each RGB color component.

[0037] Non-DL tracking unit 117 divides the range of possible values ​​(0 to 255) for a certain color component (for example, the R component) into multiple regions. Then, non-DL tracking unit 117 classifies the pixels included in the subject region of the tracking target according to the region to which the R component value belongs (frequency of each value range), and sets the result as the color composition of the subject region of the tracking target.

[0038] As the simplest example, suppose that the range of possible values ​​for the R component (0 to 255) is divided into Red1, which is 0 to 127, and Red2, which is 128 to 255. Suppose that the color composition of the subject area to be tracked in the past frame was 50 pixels for Red1 and 70 pixels for Red2. Also suppose that the color composition of the subject area to be tracked in the current frame is 45 pixels for Red1 and 75 pixels for Red2.

[0039] In this case, the non-DL tracking unit 117 can calculate a score (similarity score) representing the similarity of the color configuration based on the difference in the number of pixels classified into the same value range as follows. Similarity score=|50-45|+|70-75|=10

[0040] If the color composition of the object area to be tracked in the current frame is Red1 with 10 pixels and Red2 with 110 pixels, the similarity score is Similarity score=|50-10|+|70-110|=80 In this way, the lower the similarity of the color configuration, the higher the similarity score. Alternatively, the smaller the similarity score, the higher the similarity of the color configuration.

[0041] The selection unit 125 adopts one of the tracking results of the DL tracking unit 116 and the non-DL tracking unit 117 based on the reliability score output by the DL tracking unit 116 and the similarity score output by the non-DL tracking unit 117. For example, when the reliability score is equal to or less than a predetermined reliability score threshold and the similarity score is equal to or less than a predetermined similarity score threshold, the selection unit 125 adopts the tracking result of the non-DL tracking unit 117, and adopts the tracking result of the DL tracking unit 116 in other cases. The selection unit 125 outputs the adopted tracking result to the information superimposition unit 120 and the control unit 102.

[0042] Here, whether to adopt the tracking result of DL tracking unit 116 or non-DL tracking unit 117 is determined based on the reliability score and the similarity score. However, other methods may be used to determine the result. For example, the tracking result of DL tracking unit 116 may be preferentially adopted by utilizing the fact that the accuracy of DL tracking unit 116 tends to be higher than the accuracy of non-DL tracking unit 117. Specifically, if the tracking result of DL tracking unit 116 is obtained, the tracking result of DL tracking unit 116 may be adopted, and if not, the tracking result of non-DL tracking unit 117 may be adopted.

[0043] The information superimposing unit 120 generates an image of a tracking frame based on the size of the subject area included in the tracking result output by the tracking unit 115. For example, the image of the tracking frame may be a frame-shaped image that represents a rectangular outline circumscribing the subject area. Then, the information superimposing unit 120 generates composite image data by superimposing the image of the tracking frame on the display image data output by the post-processing unit 114 so that the tracking frame is displayed at the position of the subject area included in the tracking result. The information superimposing unit 120 may also generate images that represent the current setting values ​​and states of the imaging device 100, and superimpose these images on the display image data output by the post-processing unit 114 so that they are displayed at predetermined positions. The information superimposing unit 120 outputs the composite image data to the display unit 121.

[0044] The display unit 121 may be, for example, a liquid crystal display or an organic EL display. The display unit 121 displays an image based on the composite image data output by the information superimposing unit 120. In this manner, a live view display for one frame is performed.

[0045] FIG. 2 is a diagram showing an example of a live view display. FIG. 2(a) shows an image 800 represented by the display image data output by the post-processing unit 114. FIG. 2(b) shows an image 802 represented by composite image data in which an image of a tracking frame 803 is superimposed on the display image data. Here, since only one candidate subject 801 exists in the shooting range, the candidate subject 801 is selected as a subject to be tracked. Then, the tracking frame 803 is superimposed so as to surround the candidate subject 801. Note that in the example of FIG. 2(b), the tracking frame 803 is composed of a combination of four hollow hook shapes, but the tracking frame 803 may have other shapes, such as a combination of solid hook shapes, a seamless frame, a combination of rectangles, or a combination of triangles. Also, the shape of the tracking frame 803 may be selectable by the user.

[0046] Next, the operation of calculating the difficulty score for the subject region of the tracking target, which is executed by the difficulty level determination unit 112, will be further described with reference to the flowchart in Fig. 3. Here, it is assumed that the object classes are classified into the following groups A to D. Groups A to C are defined such that object classes having similar characteristics regarding the difficulty of tracking belong to the same group. Also, object classes that are less important than the object classes belonging to groups A to C are classified into group D.

[0047] Group A mainly includes object classes of rigid subjects, such as a car object class, a train object class, a jet plane object class, a bicycle object class, a motorcycle object class, and a helicopter object class. Group B mainly includes object classes of subjects that move quickly and whose object region changes shape significantly, such as a bird object class and an insect object class. Group C (animals and humans) mainly includes object classes of subjects that include a face and body, such as a dog object class, a cat object class, and a human body object class. Group D includes other object classes, and object classes that belong to group D are considered to be less important. Note that the classification of groups A to D is merely an example, and the difficulty of tracking may be classified from other perspectives.

[0048] In S401, the difficulty level determination unit 112 initializes the difficulty level score to zero. In S402, the difficulty level determination unit 112 determines whether or not the object class of the subject region to be tracked belongs to group A. If it is determined that the object class belongs to group A, the difficulty level determination unit 112 executes S406, and if it is not determined that the object class belongs to group A, the difficulty level determination unit 112 executes S403.

[0049] In S406, the difficulty level determination unit 112 maintains the difficulty score. The object classes belonging to group A are object classes of subjects that are relatively easy to track. Therefore, the difficulty level determination unit 112 determines that the subject area of ​​the tracking target is easy to track, and does not add points to the difficulty score (does not increase the difficulty level).

[0050] In S403, the difficulty level determination unit 112 determines whether or not the object class of the subject region to be tracked belongs to group B. If it is determined that the object class belongs to group B, the difficulty level determination unit 112 executes S407, and if it is not determined that the object class belongs to group B, the difficulty level determination unit 112 executes S404.

[0051] In S407, the difficulty level determination unit 112 adds 1 point (+1) to the difficulty score. Object classes of subjects that move quickly and change shape significantly are classified into group B. Therefore, if the object class of the subject region of the tracking target belongs to group B, the difficulty level determination unit 112 determines that the subject region of the tracking target is difficult to track, and adds a point to the difficulty score (increases the difficulty).

[0052] In S404, the difficulty level determination unit 112 determines whether or not the object class of the subject region to be tracked belongs to group C. If it is determined that the object class belongs to group C, the difficulty level determination unit 112 executes S405. If it is determined that the object class belongs to group C, the difficulty level determination unit 112 executes S410.

[0053] In S405, the difficulty level determination unit 112 determines whether or not a subject region having a head object class and a subject region having an eye object class are detected within the subject region of the tracking target. If the difficulty level determination unit 112 determines that a subject region having a head object class and a subject region having an eye object class are detected within the subject region of the tracking target, the difficulty level determination unit 112 executes S408, and if not, executes S409.

[0054] In S408, the difficulty level determination unit 112 maintains the difficulty score. Group C is an object class of subjects having faces, such as dogs, cats, and human bodies. Since it is determined that both the head and the eyes, which are in a hierarchical relationship, have been detected, the difficulty level determination unit 112 determines that the subject region of the tracking target is easy to track, and does not add points to the difficulty score.

[0055] In S409, the difficulty level determination unit 112 adds 1 point to the difficulty score. Group C is an object class of subjects having faces, such as dogs, cats, and human bodies. Since it is determined that at least one of the head and the eyes, which are in a hierarchical relationship, has not been detected, the difficulty level determination unit 112 determines that the subject region of the tracking target is difficult to track, and adds a point to the difficulty score.

[0056] In S410, the difficulty level determination unit 112 maintains the difficulty score. Since the subject region of the tracking target belongs to group D and is not an object class of any of groups A, B, and C, the difficulty level determination unit 112 determines that the subject region of the tracking target is not a region related to the main subject, and does not add points to the difficulty score.

[0057] In S411, the difficulty level determination unit 112 determines whether the number of candidate regions having the same object class as the object class of the subject region of the tracking target is equal to or greater than a predetermined threshold N (N≧2). If the difficulty level determination unit 112 determines that the number of candidate regions having the same object class as the object class of the subject region of the tracking target is equal to or greater than the predetermined threshold N, it executes S412, and if not, it executes S413.

[0058] In S412, the difficulty determination unit 112 adds 1 to the difficulty score. If the number of candidate regions having the same object class as the subject region of the tracking target is equal to or greater than the threshold N, the difficulty determination unit 112 determines that the subject region of the tracking target is difficult to track, and adds 1 to the difficulty score.

[0059] In S413, the difficulty level determination unit 112 maintains the difficulty score. If the number of candidate regions having the same object class as the subject region of the tracking target is less than the threshold value N, the difficulty level determination unit 112 determines that the subject region of the tracking target is easy to track, and maintains the difficulty score.

[0060] In S414, the difficulty level determination unit 112 determines whether the number of types of object classes included in the candidate area detected by the detection unit 110 is equal to or greater than a predetermined threshold value O (O≧2). If the difficulty level determination unit 112 determines that the number of types of object classes included in the candidate area detected by the detection unit 110 is equal to or greater than the threshold value O, it executes S415, and if not, it executes S416.

[0061] In S415, the difficulty level determination unit 112 adds 1 point to the difficulty score. If the number of object class types in the candidate area is equal to or greater than the threshold value O, it is determined that the subject area of ​​the tracking target is difficult to track because candidate areas of various object classes are mixed in the shooting range, and the difficulty level determination unit 112 adds 1 point to the difficulty score.

[0062] In S416, difficulty determination unit 112 maintains the difficulty score. If the number of object class types in the subject region of the tracking target is less than threshold value O, difficulty determination unit 112 determines that the subject region of the tracking target is in a state where it is easy for tracking unit 115 to track, and maintains the difficulty score.

[0063] In S417 , the difficulty level determination unit 112 outputs the calculated difficulty level score to the tracking control unit 113 .

[0064] The difficulty score calculated by the operation described here takes any one of the values ​​0, 1, 2, and 3. However, in practice, it may have finer increments. In practice, it is possible to classify more object classes and more groups than those exemplified here. Also, it is not necessary to determine the difficulty score for all of S402 to S404, S411, and S414, and it may be calculated based on at least one of these elements. Also, the difficulty score may be a binary value (0 or 1), and control may be performed such that the DL tracking unit 116 is enabled at a difficulty score of 1 (the non-DL tracking unit 117 is enabled or disabled) and the DL tracking unit 116 is disabled at a difficulty score of 0 (the non-DL tracking unit 117 is enabled).

[0065] By grouping in advance and widening the range that the difficulty score can take, the difficulty determination unit 112 can calculate a more accurate difficulty score. Also, by associating speed information with the object class, in addition to the object class of the subject area to be tracked, the moving speed on the image may be estimated from the size and speed information, and reflected in the difficulty score. For example, if the estimated moving speed is equal to or greater than a threshold, a point is added to the difficulty score. Also, the reliability score and similarity score calculated in a past frame (for example, the frame immediately before) may be included in the difficulty score calculation process.

[0066] Next, the operation of the tracking control unit 113 will be further described with reference to the flowchart in Fig. 4. Here, it is assumed that the difficulty level determination unit 112 calculates the difficulty level score by the operation described with reference to Fig. 3. Therefore, the difficulty level score has a value between 0 and 3.

[0067] In S501, the tracking control unit 113 determines whether or not the difficulty score is 3. If it is determined that the difficulty score is 3, the tracking control unit 113 executes S504, and if it is not determined that the difficulty score is 3, the tracking control unit 113 executes S502. In S502, the tracking control unit 113 determines whether or not the difficulty score is 2. If it is determined that the difficulty score is 2, the tracking control unit 113 executes S505, and if it is not determined that the difficulty score is 2, the tracking control unit 113 executes S503. In S503, the tracking control unit 113 determines whether or not the difficulty score is 1. If it is determined that the difficulty score is 1, the tracking control unit 113 executes S506, and if it is not determined that the difficulty score is 1, the tracking control unit 113 executes S507.

[0068] In S504, the tracking control unit 113 determines to enable the DL tracking unit 116 and disable the non-DL tracking unit 117. The tracking control unit 113 also sets the operation frequency of the DL tracking unit 116 to 60 fps (every frame). Of the calculated difficulty scores, difficulty score 3 is the state in which tracking is most difficult. Therefore, the tracking control unit 113 determines to use the DL tracking unit 116, which has better performance than the non-DL tracking unit 117. On the other hand, the tracking control unit 113 determines that the possibility of obtaining good results even if the non-DL tracking unit 117 is used is low, and determines not to use the non-DL tracking unit 117.

[0069] In S505, the tracking control unit 113 determines to enable both the DL tracking unit 116 and the non-DL tracking unit 117. The tracking control unit 113 also sets the operation frequency of the DL tracking unit 116 to 30 fps (every other frame) and the operation frequency of the non-DL tracking unit 117 to 60 fps. The difficulty score 2 is a state in which the difficulty of tracking is somewhat high. Therefore, the tracking control unit 113 determines to use both the DL tracking unit 116 and the non-DL tracking unit 117. On the other hand, the tracking control unit 113 reduces the operation frequency of the DL tracking unit 116 more than that of the non-DL tracking unit 117, thereby reducing power consumption. The tracking control unit 113 also maintains the tracking performance of tracking by setting the operation frequency of the non-DL tracking unit 117 to every frame.

[0070] In S506, the tracking control unit 113 determines to enable both the DL tracking unit 116 and the non-DL tracking unit 117. The tracking control unit 113 also sets the operation frequency of the DL tracking unit 116 to 15 fps (every two frames) and the operation frequency of the non-DL tracking unit 117 to 60 fps. Difficulty score 1 is a state in which the difficulty of tracking is somewhat low. Therefore, the tracking control unit 113 determines to use both the DL tracking unit 116 and the non-DL tracking unit 117. On the other hand, the tracking control unit 113 further reduces the operation frequency of the DL tracking unit 116 compared to the case of difficulty score 2, thereby further reducing power consumption. On the other hand, the tracking control unit 113 maintains the tracking performance of tracking by setting the operation frequency of the non-DL tracking unit 117 to every frame.

[0071] In S507, the tracking control unit 113 determines to disable the DL tracking unit 116 and to enable the non-DL tracking unit 117. The tracking control unit 113 also sets the operation frequency of the non-DL tracking unit 117 to 30 fps (every other frame). S507 is executed when the difficulty score is 0, which is the lowest level of difficulty in tracking. Therefore, the tracking control unit 113 determines that there is a high possibility that the non-DL tracking unit 117 can obtain a result with sufficient accuracy, and determines to use only the non-DL tracking unit 117. Furthermore, the tracking control unit 113 further reduces power consumption by setting the operation frequency of the non-DL tracking unit 117 to every other frame.

[0072] The control operations described here are merely examples, and may vary depending on the range of values ​​that the difficulty score can take and the increments of the values. Also, the frame rate of the video is assumed to be 60 fps here, but it may be 120 fps. In this case, when the operation frequency of the non-DL tracking unit 117 is set to 120 fps, the operation frequency of the DL tracking unit 116 may be set to less than 15 fps, for example, 5 fps.

[0073] Also, it is not necessary to control differently for each difficulty score. For example, if the difficulty score is equal to or greater than a first threshold, DL tracking unit 116 may be enabled and non-DL tracking unit 117 may be disabled, or if the difficulty score is less than a second threshold, DL tracking unit 116 may be disabled and non-DL tracking unit 117 may be enabled.

[0074] 5 is a timing chart showing an example of the operation timing of each of the above-mentioned units when the image capture device 100 performs the subject tracking function. t600, t601, t602, t603, t604, t605, and t606 are timings at which the detection unit 110 starts detection processing in each frame. At t600, the detection unit 110 starts detection processing 610 for the first frame (0th frame).

[0075] When the detection unit 110 detects one or more candidate areas and their object classes for the 0th frame image data by the detection process 610, the target determination unit 111 starts a determination process 611 for determining a subject area to be tracked. When the subject area to be tracked for the 0th frame image data is determined by the determination process 611, the difficulty level determination unit 112 starts a difficulty level determination process 612 for the subject area to be tracked.

[0076] When the difficulty score is calculated by the difficulty determination process 612 and the subject area determined by the determination process 611 is an area related to a subject different from the most recently determined subject area, the tracking control unit 113 reviews the control of the DL tracking unit 116 and the non-DL tracking unit 117. Specifically, the tracking unit 113 updates the enable / disable and operation frequency of the DL tracking unit 116 and the non-DL tracking unit 117 according to the difficulty score of the subject area of ​​the tracking target determined for the 0th frame image data. In addition, the tracking unit 115 updates the subject area to be tracked with the subject area of ​​the tracking target determined for the 0th frame image data. Therefore, for the image data of the 1st frame, the updated subject area is tracked.

[0077] On the other hand, if the subject area determined in the determination process 611 and the most recently determined subject area are areas relating to the same subject, the tracking control unit 113 does not review the control of the DL tracking unit 116 and the non-DL tracking unit 117. Furthermore, the tracking unit 115 does not update the subject area to be tracked.

[0078] In FIG. 5, there is no change in the subject to be tracked in the 0th frame, the 2nd frame in which detection process 624 starts from t602, the 4th frame in which detection process 633 starts from t604, and the 6th frame in which detection process 643 starts from t606.

[0079] On the other hand, for the first frame of image data in which detection processing 616 is started from t601, an example is shown in which the subject region of the tracking target determined in determination processing 617 and the subject region determined most recently are regions relating to different subjects.

[0080] In this case, in order to change the subject to be tracked in the next frame, tracking control unit 113 uses the result of difficulty determination process 618 to review the enable / disable control of DL tracking unit 116 and non-DL tracking unit 117. Here, for the next frame, non-DL tracking unit 117 is disabled, DL tracking unit 116 is enabled, and the operation frequency of DL tracking unit 116 is switched to twice the operation frequency in the 0th and 1st frames.

[0081] Similarly, in the third frame in which detection processing 629 starts from t603, the subject to be tracked changes, and the tracking control unit 113 reviews the enable / disable control of the DL tracking unit 116 and the non-DL tracking unit 117. Here, for the fourth frame in which detection processing 633 starts from t604, the DL tracking unit 116 is disabled and the non-DL tracking unit 117 is enabled. Also, the operation frequency of the non-DL tracking unit 117 is set to half the operation frequency setting for the 0th and 1st frames.

[0082] At t600, the tracking unit 115 starts DL tracking processing 613 and non-DL tracking processing 614 for the 0th frame. Here, it is assumed that at the time of the 0th frame, both the DL tracking unit 116 and the non-DL tracking unit 117 are enabled by the tracking control unit 113, and the operation frequency of the non-DL tracking unit 117 is set to twice the operation frequency of the DL tracking unit 116.

[0083] In FIG. 5, the DL tracking process and the non-DL tracking process indicated by dashed rectangles indicate that they are not executed because the operation frequency of the DL tracking unit 116 and the non-DL tracking unit 117 is set lower than every frame.

[0084] In the 0th frame, the DL tracking unit 116 performs DL tracking processing 613, and the non-DL tracking unit 117 performs non-DL tracking processing 614, and outputs the tracking processing results. Here, the DL tracking processing 613 can output a tracking result with higher accuracy than the non-DL tracking processing 614. Therefore, the tracking unit 115 adopts the tracking result of the DL tracking processing 613 by the selection unit 125. Then, in the superimposition processing 615, the information superimposition unit 120 generates synthetic image data on which a tracking frame based on the tracking result of the DL tracking processing 613 is superimposed. At t601, the display unit 121 executes display processing 623 of the synthetic image data for the 0th frame.

[0085] At t601, the non-DL tracking unit 117 starts non-DL tracking processing 620 for the first frame. The DL tracking unit 116 does not perform tracking processing for the first frame according to the operation frequency setting. Since only the tracking result of the non-DL tracking unit 117 is obtained in the first frame, in the superimposition processing 622, the information superimposition unit 120 generates composite image data on which a tracking frame based on the tracking result of the non-DL tracking unit 117 is superimposed.

[0086] Since the non-DL tracking process has a lower processing load than the DL tracking process, it is completed earlier than the DL tracking process. In order to output the tracking result at the same timing as when the DL tracking process is performed, the tracking unit 115 performs a wait process 621 following the non-DL tracking unit process. At t602, the display unit 121 executes a display process 628 of the composite image data of the first frame.

[0087] At t602, the DL tracking unit 116 starts DL tracking processing 625 for the second frame. Since the non-DL tracking unit 117 is disabled for the second frame, the non-DL tracking processing is not executed. Since only the tracking result of the DL tracking unit 116 is obtained in the second frame, in the superimposition processing 627, the information superimposition unit 120 generates composite image data on which a tracking frame based on the tracking result of the DL tracking unit 116 is superimposed. At t603, the display unit 121 executes display processing 632 of the composite image data for the first frame.

[0088] At t603, the DL tracking unit 116 starts DL tracking processing 630 for the third frame. Before starting DL tracking processing 625 for the second frame, the operation frequency of the DL tracking unit 116 is changed to twice the operation frequency in the zeroth frame, so that DL tracking processing is also executed in the third frame following the second frame. The superimposition processing 631 and display processing 638 are the same as those for the second frame.

[0089] At t604, the non-DL tracking unit 117 starts non-DL tracking processing 635 for the fourth frame. Since the DL tracking unit 116 is disabled for the fourth frame, the DL tracking processing is not executed. Since the operation frequency of the non-DL tracking unit 117 is set to half the operation frequency for the 0th and 1st frames, the non-DL tracking processing 635 for the fourth frame is executed, but the non-DL tracking processing 640 for the fifth frame is not executed. The waiting processing 636, the superimposition processing 637, and the display processing 642 are the same as those for the first frame.

[0090] At t605, the tracking unit 115 does not execute either DL tracking processing 639 or non-DL tracking processing 640. Therefore, in superimposition processing 641 for the fifth frame, the information superimposition unit 120 generates synthetic image data on which a tracking frame based on the tracking result of non-DL tracking processing 635 obtained in the fourth frame is superimposed. Note that the information superimposition unit 120 may generate synthetic image data on which a tracking frame is not superimposed in the superimposition processing for frames on which neither DL tracking processing nor non-DL tracking processing is executed, such as the fifth frame.

[0091] At t606, the non-DL tracking unit 117 starts non-DL tracking processing 644 for the sixth frame. Since the DL tracking unit 116 is disabled, the DL tracking processing is not executed. The non-DL tracking processing 644, waiting processing 645, superimposition processing 646, and display processing 647 for the sixth frame are the same as those for the fourth frame.

[0092] In this way, when the subject to be tracked is switched to another subject, the presence or absence of execution of DL tracking processing and non-DL tracking processing and the frequency of operation are reviewed based on the difficulty score calculated for the new subject area of ​​the tracking target to be used in the next frame. Therefore, it is possible to dynamically execute an appropriate tracking processing according to the difficulty score, and it is possible to reduce power consumption.

[0093] The tracking process in this embodiment can use the tracking results for a frame prior to the frame to be processed. The tracking process using past tracking results will be described with reference to the timing chart shown in FIG.

[0094] t700 is the tracking process start timing for the 0th frame. Thereafter, t70n (n=1, 2, ...) indicates the tracking process start timing for the nth frame. At t702 and t704, the subject to be tracked is switched, and the control of the DL tracking unit 116 and the non-DL tracking unit 117 by the tracking control unit 113 is reviewed.

[0095] When enabled, the DL tracking unit 116 and the non-DL tracking unit 117 perform tracking processing on a frame according to the operation frequency. In addition, the DL tracking unit 116 and the non-DL tracking unit 117 use the tracking result obtained for the previous frame for tracking processing of the processing target frame.

[0096] The processing accuracy of DL tracking unit 116 tends to be higher than the processing accuracy of non-DL tracking unit 117 based on similarity, etc. Therefore, for the DL tracking process and / or non-DL tracking process of the processing target frame, the tracking result of the DL tracking process for the frame immediately preceding the processing target frame is used if available. Note that if the subject to be tracked is different between the processing target frame and the frame immediately preceding, the tracking result for the frame immediately preceding is not used.

[0097] Therefore, non-DL tracking processing 711 for the 1st frame uses the tracking result of DL tracking processing 710 for the 0th frame. Because the subjects to be tracked are different in the 1st and 2nd frames, DL tracking processing 712 for the 2nd frame does not use the tracking result of the 1st frame. DL tracking processing 713 for the 3rd frame uses the tracking result of DL tracking processing 712 for the 2nd frame. Because the subjects to be tracked are different in the 3rd and 4th frames, non-DL tracking processing 715 for the 4th frame does not use the tracking result of the 3rd frame. Because DL tracking processing 714 has not been executed in the 4th frame, non-DL tracking processing 716 for the 5th frame uses the tracking result of non-DL tracking processing 715 for the 4th frame.

[0098] FIG. 7 is a flowchart of a live view display operation of the imaging device 100 when the subject tracking function is enabled. In S200, the control unit 102 controls the image sensor 103 to capture one frame and read out the image data. In S201, the first pre-processing unit 104 applies pre-processing to image data read from the imaging element 103. In S202, the first pre-processing unit 104 stores the image data to which the pre-processing has been applied in the display memory 107.

[0099] In S203, first image correction unit 109 starts applying a predetermined image correction process to the image data read from display memory 107. In S204, first image correction unit 109 determines whether or not all image correction processes to be applied have been completed, and if it is determined that all have been completed, it outputs the image data to which the image correction process has been applied to post-processing unit 114. Furthermore, if it is not determined that all image correction processes have been completed, first image correction unit 109 continues the image correction process.

[0100] In S205, the post-processing unit 114 generates image data for display from the image data to which the image correction process has been applied by the first image correcting unit 109, and outputs the image data to the information superimposing unit 120.

[0101] In S206, the information superimposing unit 120 generates data of a composite image in which an image of the tracking frame and other information are superimposed on the captured image, using the display image data generated by the post-processing unit 114, the image data of the tracking frame, and the image data indicating other information. The information superimposing unit 120 outputs the composite image data to the display unit 121.

[0102] In S207, the display unit 121 displays the composite image data generated by the information superimposing unit 120. This completes the live view display for one frame.

[0103] FIG. 8 is a flowchart relating to the operation of the subject tracking function of the imaging device 100. The processing of S300 to S304 is the same as the processing of S200 to S204 in FIG. 7, except that the tracking memory 108 is used instead of the display memory 107, and the image correction processing is image correction processing for tracking.

[0104] In S305, the detection unit 110 executes a detection process of a subject region for the image data for which all image correction processes have been completed. The detection unit 110 stores the position, size, object class, number of subject regions belonging to the same object class, reliability score, and the like for each of the detected subject regions (candidate regions) in the tracking memory 108 as detection results.

[0105] In S306, the object determining unit 111 determines one object region to be tracked from the information on the candidate regions stored in the tracking memory 108. The object determining unit 111 stores information on the determined object region to be tracked in the tracking memory 108.

[0106] In S307, the difficulty determination unit 112 calculates a difficulty score based on the information on the subject area of ​​the tracking target and the information on the candidate area, and based on the tracking candidate subject information read from the tracking memory 108. In addition, the tracking control unit 113 determines whether the DL tracking unit 116 and the non-DL tracking unit 117 are enabled or disabled, and the operation frequency of the enabled tracking unit, based on the difficulty score.

[0107] In S308, the tracker 115 determines whether or not the DL tracker 116 has executed the tracking process for the processing target frame, and if it is determined that the process has been executed, the tracker 115 executes S309, and if it is determined that the process has not been executed, the tracker 115 executes S310.

[0108] In S309 , the tracker 115 selects the tracking result of the DL tracker 116 via the selector 125 and outputs the result to the information superimposing unit 120 . In S310, the tracking unit 115 uses the selection unit 125 to adopt the tracking result of the non-DL tracking unit 117 and outputs it to the information superimposition unit 120. Note that when the non-DL tracking unit 117 is not performing tracking processing either, it does not output the tracking result or outputs the tracking result of the previous frame.

[0109] In S311, the tracking control unit 113 determines whether or not the subject area used in the tracking process of the processing target frame is an area related to the same subject as the tracking target subject area determined in S306. This determination can also be said to be a determination of whether or not the tracking target subject has changed. For example, if the reliability score included in the tracking result in the processing target frame is equal to or less than a threshold, the tracking control unit 113 can determine that the tracking target subject has changed. Note that this is just one example, and the determination may be made by other methods.

[0110] Tracking control unit 113 executes S312 if it is determined that the subject to be tracked has been switched, and executes S313 if it is not determined that the subject to be tracked has been switched.

[0111] In S312, tracking control unit 113 determines whether DL tracking unit 116 and non-DL tracking unit 117 are enabled or disabled and their operation frequencies for the next frame, based on the difficulty score of the subject region of the tracking target determined for the processing target frame in S306.

[0112] In S313, the information superimposing unit 120 generates data of a composite image on which an image of the tracking frame is superimposed, based on the display image data and the tracking result adopted in S309 or S310. The processes in S314 and S315 are similar to those in S206 and S207.

[0113] As described above, according to this embodiment, in an imaging device using a first tracking unit and a second tracking unit that has higher processing accuracy but requires a larger amount of calculation than the first tracking unit, the first and / or second tracking unit is enabled or disabled based on the degree of difficulty of tracking. Therefore, a tracking unit that is unlikely to produce good tracking results is disabled, or one of the tracking units is disabled. Tracking By disabling the other tracking unit when there is a high possibility of obtaining good tracking results using only one tracking unit, it is possible to reduce power consumption while preventing a decrease in tracking accuracy.

[0114] Since tracking processing is generally performed on moving images, the effect of reducing power consumption is particularly large when a tracking unit that utilizes computationally intensive machine learning such as deep learning is used as the second tracking unit.

[0115] <Second embodiment> Next, a second embodiment of the present invention will be described. 9 is a block diagram showing an example of the functional configuration of an image capture device 100' which is an example of an image processing device according to the second embodiment, and the same components as those in the image capture device 100 described in the first embodiment are given the same reference numerals. In this embodiment, the operation of the target determination unit, rather than the tracking unit, is controlled by the difficulty score.

[0116] The following describes the differences from the first embodiment. The difficulty determination unit 112 of this embodiment calculates a difficulty score, which is an evaluation value indicating the difficulty of the process (target determination process) of determining the subject area of ​​the tracking target from the detected candidate areas, based on the detection results of the detection unit 110 stored in the tracking memory 108.

[0117] For example, the difficulty level determination unit 112 can calculate the difficulty score by considering one or more factors that affect the difficulty of the object determination process. Examples of factors that affect the difficulty of the object determination process include, but are not limited to, the size of the candidate area, the object class (type) of the candidate area, the number of candidate areas belonging to the same object class, and the position of the candidate area in the image. A specific example of a method for calculating the difficulty score will be described later. The difficulty level determination unit 112 outputs the calculated difficulty score to the object determination control unit 135.

[0118] The object determination control unit 135 determines whether to enable or disable each of the multiple object determination units included in the object determination unit 130 based on the difficulty score calculated by the difficulty level judgment unit 112. In this embodiment, the object determination unit 130 has multiple object determination units with different calculation loads and object determination accuracy. Specifically, the object determination unit 130 has a DL object determination unit 131 that performs object determination processing using deep learning (DL) and a non-DL object determination unit 132 that performs object determination processing without using DL. The DL object determination unit 131 has a higher processing accuracy than the non-DL object determination unit 132, but a larger calculation load than the non-DL object determination unit 132.

[0119] In this case, the target determination control unit 135 determines whether to enable or disable each of the DL target determination unit 131 and the non-DL target determination unit 132. Note that when one candidate area is detected or when no candidate area is detected, the target determination control unit 135 disables both the DL target determination unit 131 and the non-DL target determination unit 132.

[0120] The object determining unit 130 determines a subject region (main subject region) to be tracked from the candidate regions detected by the detection unit 110. The object determining unit 130 stores information specifying the determined subject region in the tracking memory 108.

[0121] The object determination unit 130 has a DL object determination unit 131 that performs object determination processing using deep learning (DL), and a non-DL object determination unit 132 that performs object determination processing without using DL. Of the DL object determination unit 131 and the non-DL object determination unit 132, the object determination unit enabled by the object determination control unit 135 outputs information that identifies the determined subject region.

[0122] The DL target determination unit 131 determines the subject region of the tracking target from a plurality of candidate regions using a trained multilayer neural network. The multilayer neural network may be, for example, a convolutional neural network (CNN). The training of the multilayer neural network can be performed using learning data in which size information, position information within the screen, object class information, and reliability for each of the candidate regions are used as input data, and a correct main subject region is used as training data. Note that the training data can be prepared, for example, by having a human view the frame to be processed and determining a correct main subject region.

[0123] The learning of the multilayer neural network may be performed by another device or by the imaging device 100'. When the learning is performed by the imaging device 100', the DL target determination unit 131 is set to a learning mode, and the multilayer neural network can be learned by supplying learning data (a set of input data and teacher data) to the DL target determination unit 131 from, for example, the recording medium 119. The learning may be performed using a known optimization algorithm until the value of the loss function becomes less than a predetermined value.

[0124] When training the multilayer neural network in another device, for example, parameters of the multilayer neural network that has completed training in the other device may be supplied to the DL target determination unit 131 via, for example, the recording medium 119. The multilayer neural network trained in the other device has the same configuration as the multilayer neural network implemented in the DL target determination unit 131. The training method may be the same as that in the case of training in the imaging device 100'.

[0125] DL target determination unit 131 provides the trained multi-layer neural network with size information, position information within the screen, object class information, and reliability as test data for each candidate region detected by detection unit 110. The trained multi-layer neural network outputs, for example, the probability that each candidate region is a main subject region.

[0126] The DL target determination unit 131 determines the candidate area with the highest output of the multilayer neural network as the subject area (main subject area) to be tracked, among the multiple candidate areas detected by the detection unit 110. Then, the DL target determination unit 131 outputs size information, position information within the screen, and object class information of the determined main subject area.

[0127] On the other hand, the non-DL target determination unit 132 determines the subject region of the tracking target from multiple candidate regions by a method that does not use deep learning. The non-DL target determination unit can determine the subject region of the tracking target in the same manner as the target determination unit 111 of the first embodiment. The non-DL target determination unit 132 stores information that identifies the determined subject region in the tracking memory 108.

[0128] The selection unit 133 adopts either the information on the main subject region output by the DL target determination unit 131 or the information on the main subject region output by the non-DL target determination unit 132. The selection unit 133 decides which information on the main subject region to adopt, for example, based on the position information of the main subject region output by the DL target determination unit 131 and the non-DL target determination unit 132. The selection unit 133 can adopt, for example, the information on the main subject region that is closer to the focus detection region currently set in the imaging device 100' (the one closer to the focus detection region). The selection unit 133 stores the adopted information on the main subject region in the tracking memory 108.

[0129] When both DL target determination unit 131 and non-DL target determination unit 132 are disabled by target determination control unit 135, target determination unit 130 determines that there is one candidate area or that no candidate area has been detected. If there is one candidate area, target determination unit 130 determines that candidate area is the main subject area. If no candidate area has been detected, target determination unit 130 notifies tracking unit 134 of the absence of a main subject and stops tracking processing.

[0130] The tracking unit 134 estimates the subject area of ​​the tracking target from the image data of the frame to be processed (current frame) stored in the tracking memory 108, and obtains the position and size of the estimated subject area in the frame as the tracking result. The tracking unit 134 can estimate the subject area of ​​the tracking target based on the similarity of color composition, for example, in the same manner as the non-DL tracking unit 117 in the first embodiment. Note that the color composition information of the subject area of ​​the tracking target may be updated by the subject area of ​​the tracking target in a previous frame, and the tracking process of the next frame may be executed.

[0131] The tracking unit 134 may estimate the subject area of ​​the tracking target in the current frame using pattern matching with the subject area of ​​the tracking target in a past frame as a template. In this case, the template used for pattern matching may be updated for each frame with the subject area of ​​the tracking target determined by the target determination unit 130. The tracking unit 134 may also estimate the position and size of the subject area of ​​the tracking target in the current frame using a trained multilayer neural network. The tracking unit 134 outputs the position, size, and reliability score for the subject area of ​​the tracking target estimated for the current frame.

[0132] The information superimposing unit 120 generates an image of the tracking frame based on the size of the main subject area included in the tracking result output by the tracking unit 134. Then, the information superimposing unit 120 outputs the composite image data generated in the same manner as in the first embodiment to the display unit 121.

[0133] Fig. 10 is a diagram showing an example of live view display on the display unit 121 using display image data output by the post-processing unit 114. In Fig. 10, a dashed line 1004 indicates a central region of the image, which is not included in the actual display image data. Images 1010 and 1020 show display examples of composite image data in which an image of a tracking frame 1015 is superimposed on the display image data. Image 1010 is an image one frame after image 1000, and image 1020 is an image several frames after image 1010.

[0134] When automobiles 1001, 1002, and 1003 are detected as candidate regions in image 1000, object determining unit 130 determines the region of automobile 1003 closest to the central region indicated by dashed line 1004 as the subject region to be tracked. As a result, the region of automobile 1003 is tracked in image 1010 one frame later, and tracking frame 1015 is displayed superimposed so as to surround the region of automobile 1003. Furthermore, in image 1020 several frames later, tracking frame 1015 is also displayed superimposed on the region of automobile 1003.

[0135] Images 1030 and 1040 show an example of live view display when there is only one candidate region. Image 1040 is an image one frame after image 1030. In image 1030, automobile 1031 is detected as the only candidate region. In this case, object determination control unit 135 disables both DL object determination unit 131 and non-DL object determination unit 132 of object determination unit 130. Then, object determination unit 130 determines the region of automobile 1031 as the subject region of the tracking target. As a result, in image 1040 one frame after, tracking frame 1015 is superimposed and displayed so as to surround the region of automobile 1031, which is the tracking target. In this embodiment, as in the first embodiment, the shape of tracking frame 1015 is not limited to a combination of four hollow hook shapes.

[0136] Next, the operation of calculating the difficulty score for the subject region of the tracking target, which is executed by the difficulty level determination unit 112 in this embodiment, will be further described with reference to the flowchart in FIG. In S1101, the difficulty level determination unit 112 initializes the difficulty level score to zero.

[0137] In S1102, the difficulty level determination unit 112 determines whether or not a candidate area exists in the center of the image. For example, if a candidate area including a part or all of the area in the center of the screen indicated by the dashed line 1004 in Fig. 10 exists, the difficulty level determination unit 112 determines that a candidate area exists in the center of the image. Also, for example, if a candidate area including a part or all of the area in the center of the screen indicated by the dashed line 1004 does not exist, the difficulty level determination unit 112 determines that a candidate area does not exist in the center of the image. Then, if it is determined that a candidate area does not exist in the center of the image, the difficulty level determination unit 112 executes S1103, and if it is determined that a candidate area exists in the center of the image, the difficulty level determination unit 112 executes S1104.

[0138] If a focus detection area is set in advance, the difficulty level determination unit 112 performs a similar determination on the focus detection area instead of the area at the center of the screen. If the difficulty level determination unit 112 determines that no candidate area exists in the focus detection area, it executes S1103, and if it determines that a candidate area exists in the focus detection area, it executes S1104.

[0139] In S1103, the difficulty determination unit 112 adds one point (+1) to the difficulty score. If the candidate area is not present in the focus detection area or the center of the image, the difficulty determination unit 112 determines that it is difficult to determine the subject area to be tracked, and adds a point to the difficulty score (increases the difficulty). Then, the difficulty determination unit 112 executes S1105.

[0140] In S1104, the difficulty determination unit 112 maintains the difficulty score. If the candidate area exists in the focus detection area or the center of the image, the difficulty determination unit 112 determines that it is easy to determine the subject area to be tracked, and does not add points to the difficulty score (does not increase the difficulty). Then, the difficulty determination unit 112 executes S1105.

[0141] In S1105, the difficulty level determination unit 112 determines whether the number of types of candidate areas (types of object classes) is equal to or greater than a preset threshold N (whether N or more types of candidate areas have been detected). For example, if candidate areas associated with a human object class, a dog object class, a cat object class, and a car object class have been detected, the number of types of candidate areas will be four. If the threshold N is three, the difficulty level determination unit 112 determines that the number of types of candidate areas is equal to or greater than the threshold N, and executes S1106. On the other hand, if candidate areas associated with a dog object class and a cat object class have been detected, the number of types of candidate areas will be two. If the threshold N is three, the difficulty level determination unit 112 determines that the number of types of candidate areas is not equal to or greater than the threshold N, and executes S1107.

[0142] In S1106, the difficulty level determination unit 112 adds (the number of types of candidate areas - N + 1) points to the difficulty score. If N or more types of candidate areas are detected, the difficulty level determination unit 112 determines that it is difficult to determine the subject area of ​​the tracking target, and adds more points to the difficulty score (increases the difficulty) the more types there are. Then, the difficulty level determination unit 112 executes S1108.

[0143] In S1107, the difficulty level determination unit 112 maintains the difficulty score. If the number of types of candidate regions is less than N, the difficulty level determination unit 112 determines that it is easy to determine the subject region of the tracking target, and does not add points to the difficulty score (does not increase the difficulty). Then, the difficulty level determination unit 112 executes S1108.

[0144] S1108 and S1115 indicate that the processes between them are repeated. Specifically, the processes from S1109 to S1114 are repeated a number of times equal to the number of types of detected candidate regions. For example, if four types of candidate regions are detected, the processes from S1109 to S1114 are repeated four times for each type of candidate region.

[0145] In S1109, the difficulty level determination unit 112 determines whether or not the number of candidate regions associated with the same type (object class) is equal to or greater than a preset threshold value M. For example, assume that the detection unit 110 has detected four candidate regions for the dog object class and two candidate regions for the cat object class, and the threshold value M is 3. In this case, the processes from S1109 to S1114 are executed once for the dog object class and once for the cat object class.

[0146] In the processing for the dog object class, in S1109 the difficulty level determination unit 112 determines that the number of candidate regions for the dog object class is 4, which is greater than or equal to the threshold value M, and executes S1110. On the other hand, in the processing for the cat object class, in S1109 the difficulty level determination unit 112 determines that the number of candidate regions for the cat object class is 2, which is less than or equal to the threshold value M, and executes S1111.

[0147] In S1110, the difficulty level determination unit 112 adds (the number of candidate areas of the object class to be processed - M + 1) to the difficulty score. If the number of candidate areas of the same object class is equal to or greater than the threshold M, it is determined that it is difficult to determine the subject area of ​​the tracking target, and adds points to the difficulty score (increases the difficulty). Then, the difficulty level determination unit 112 executes S1112.

[0148] In S1111, the difficulty level determination unit 112 maintains the difficulty score. If the number of candidate regions of the same object class is less than the threshold value M, it is determined that it is easy to determine the subject region of the tracking target, and does not add points to the difficulty score (does not increase the difficulty). Then, the difficulty level determination unit 112 executes S1112.

[0149] In S1112, the difficulty level determination unit 112 determines whether the variance of the sizes of candidate regions of the same object class is less than a preset threshold value L. Here, the size of a candidate region may be, for example, the product of the number of horizontal pixels and the number of vertical pixels of the candidate region. The size of a candidate region is normalized by the size of the largest candidate region among the candidate regions of the same object class (≦1). Using the normalized size, the variance can be calculated as the average of the squared values ​​of the differences between the size of each candidate region and the average size.

[0150] For example, assume that four candidate regions for the dog object class and two candidate regions for the cat object class are detected, and the preset threshold L is 0.1. Furthermore, when S1112 is executed for the dog object class, if the variance of the sizes of the candidate regions is 0.2, the difficulty level determination unit 112 determines that the variance of the sizes of the candidate regions of the same object class is not less than L, and executes S1114. On the other hand, when S1112 is executed for the cat object class, if the variance of the sizes of the candidate regions is 0.0025, the difficulty level determination unit 112 determines that the variance of the sizes of the candidate regions of the same object class is less than L, and executes S1113.

[0151] In S1113, the difficulty level determination unit 112 adds 1 point to the difficulty score. When the variance of the sizes of candidate areas of the same object class is less than L, it is considered that candidate areas related to the same type of subject have similar sizes. Therefore, the difficulty level determination unit 112 determines that it is difficult to determine the subject area of ​​the tracking target for the object class to be processed, and adds a point to the difficulty score (increases the difficulty). Then, the difficulty level determination unit 112 executes S1115.

[0152] In S1114, the difficulty level determination unit 112 maintains the difficulty score. If the variance of the sizes of the candidate regions of the same object class is not less than L, it is considered that there is variation in the sizes of the candidate regions for the same type of subject. Therefore, the difficulty level determination unit 112 determines that it is easy to determine the subject region of the tracking target for the object class to be processed, and does not add points to the difficulty score (does not increase the difficulty). Then, the difficulty level determination unit 112 executes S1115.

[0153] After executing the processes from S1109 to S1114 for each of all detected types (object classes), the difficulty level determination unit 112 executes S1116. In S1116, the difficulty level determination unit 112 outputs the calculated difficulty level score to the object determination control unit 135.

[0154] Note that the method of calculating the difficulty score described here is merely an example, and the difficulty score can be calculated by various other methods. For example, instead of adding 1 to the difficulty score in S1103, a value multiplied by a preset gain coefficient may be added. Also, the difficulty score may be calculated using only the number of types of candidate areas and the number of candidate areas related to the same type of subject (belonging to the same object class). Alternatively, the difficulty score may be calculated based on only one item. Also, the difficulty score may be a binary value (0 or 1), and the DL target determination unit 131 may be controlled so that a difficulty score of 1 enables the DL target determination unit 131 and a difficulty score of 0 disables the DL target determination unit 131.

[0155] Next, the operation of the object determination control unit 135 will be further described with reference to the flowchart of Fig. 12. Here, it is assumed that the difficulty level determination unit 112 has calculated the difficulty level score by the operation described with reference to Fig. 11.

[0156] In S1201, the object determination control unit 135 determines whether or not the difficulty score is greater than a preset threshold A, and if it is determined that it is greater, executes S1203, and if not, executes S1202.

[0157] In S1202, the target determination control unit 135 determines whether the difficulty score is greater than a threshold B (<A) set in advance. If it is determined to be greater, S1204 is executed; otherwise, S1205 is executed.

[0158] In S1203, the target determination control unit 135 determines to enable the DL target determination unit 131 and disable the non-DL target determination unit 132. When the difficulty score is greater than A, the difficulty of target determination is high. Therefore, the target determination control unit 135 determines to use the DL target determination unit 131 with better performance than the non-DL target determination unit 132. Also, the target determination control unit 135 determines that it is unlikely to obtain good results even if the non-DL target determination unit 132 is used, and decides not to use the non-DL target determination unit 132.

[0159] In S1204, the target determination control unit 135 determines to disable the DL target determination unit 131 and enable the non-DL target determination unit 132. When the difficulty score is less than or equal to A and greater than B, the difficulty of target determination is medium. Therefore, the target determination control unit 135 determines that it is highly likely to obtain a sufficiently accurate result with the non-DL target determination unit 132 alone, and decides to use only the non-DL target determination unit 132. By disabling the DL target determination unit 131, which consumes more power than the non-DL target determination unit 132, power consumption can be reduced.

[0160] In S1205, the target determination control unit 135 determines to disable both the DL target determination unit 131 and the non-DL target determination unit 132. When the difficulty score is less than or equal to B, the difficulty of target determination is low. Therefore, the target determination control unit 135 determines that target determination is possible without using the DL target determination unit 131 and the non-DL target determination unit 132, and disables both the DL target determination unit 131 and the non-DL target determination unit 132. Thereby, power consumption can be further reduced.

[0161] Note that, for example, when one or no candidate areas are detected, it is possible to determine the subject area of ​​the tracking target without using the DL target determination unit 131 and the non-DL target determination unit 132. This also applies when one of the detected candidate areas is very large and is located in the center of the screen or in the focus detection area.

[0162] The operation of the target determination control unit 135 described here is merely an example. For example, the control does not need to be different for each difficulty score. For example, the non-DL target determination unit may be enabled in S1205 as in S1204. Alternatively, if the difficulty score is not determined to be greater than A in S1201, S1204 may be executed immediately.

[0163] 13, an example of a method in which the target determination control unit 135 determines whether to enable or disable the DL target determination unit 131 and the non-DL target determination unit 132 without using a difficulty score will be described. Here, it is assumed that the target determination control unit 135 determines whether to enable or disable the DL target determination unit 131 and the non-DL target determination unit 132 using a detected object class and the number of candidate regions belonging to the same object class.

[0164] It is assumed here that the DL target determination unit 131 has a multi-layer neural network that has been trained using training data related to a specific scene, for example, a scene of a sport using a ball. The training data is a set of input data and corresponding training data. Here, the input data can include size information, position information within the screen, object class information, and reliability of each of a plurality of candidate areas extracted from image data of a sport using a ball. In addition, training data can be used that sets candidate areas belonging to a human object class as the correct answer among the candidate areas included in the image data.

[0165] The DL target determination unit 131 having such a learned multi-layer neural network can perform highly accurate target determination processing when a ball sports scene is captured by the imaging device 100'. Furthermore, the image data used to generate the input data can be limited to image data in which the number of candidate regions for the human object class is four or more. This allows the DL target determination unit 131 to perform more accurate target determination processing for images of ball sports scenes that include four or more "people."

[0166] The following describes the operation of the object determination control unit 135 when the DL object determination unit 131 can perform more accurate object determination processing on an image of a scene involving a ball sport that includes four or more “people.”

[0167] In S1301, the object determination control unit 135 determines whether or not four or more candidate regions for the human object class and one or more candidate regions for the ball object class have been detected by the detection unit 110. If it is determined that four or more candidate regions for the human object class and one or more candidate regions for the ball object class have been detected, the object determination control unit 135 executes S1304, and if not, executes S1302.

[0168] In S1302, the object determination control unit 135 determines whether or not three candidate regions for the human object class and one or more candidate regions for the ball object class have been detected by the detection unit 110. If it is determined that three or more candidate regions for the human object class and one or more candidate regions for the ball object class have been detected, the object determination control unit 135 executes S1305, and if not, executes S1303.

[0169] In S1301, the object determination control unit 135 determines whether or not two candidate regions for the human object class have been detected by the detection unit 110. If it is determined that two or more candidate regions for the human object class have been detected, the object determination control unit 135 executes S1306, and if not, the object determination control unit 135 executes S1307.

[0170] In S1304, the object determination control unit 135 determines to enable the DL object determination unit 131 and disable the non-DL object determination unit 132. When four or more candidate regions of the human object class and one or more candidate regions of the ball object class are detected, the scene is similar to the scene used for training the multilayer neural network of the DL object determination unit 131. Therefore, the object determination control unit 135 determines to use only the DL object determination unit 131, since the DL object determination unit 131 can optimally perform object determination.

[0171] In S1305, the object determination control unit 135 determines to enable the DL object determination unit 131 and to enable the non-DL object determination unit 132. When three candidate regions of the human object class and one or more candidate regions of the ball object class are detected, the scene is similar to the scene used for training the multilayer neural network of the DL object determination unit 131, but is not the same scene. On the other hand, since the object determination process needs to be performed for three candidate regions of the human object class, it is highly likely that the accuracy will be insufficient if only the non-DL object determination unit is enabled. Therefore, the object determination control unit 135 determines to use both the DL object determination unit 131 and the non-DL object determination unit 132 in order to achieve a highly accurate object determination process.

[0172] In S1306, the target determination control unit 135 determines to disable the DL target determination unit 131 and enable the non-DL target determination unit 132. When two candidate regions of the human object class are detected, one of them may be selected as the subject region to be tracked, and the difficulty of target determination is low. Therefore, the target determination control unit 135 determines that there is a high possibility that the non-DL target determination unit 132 can obtain a result with sufficient accuracy, and determines to use only the non-DL target determination unit 132. By disabling the DL target determination unit 131, which consumes more power than the non-DL target determination unit 132, it is possible to reduce power consumption.

[0173] In S1307, the target determination control unit 135 determines to disable both the DL target determination unit 131 and the non-DL target determination unit 132. When one or no candidate area of ​​the human object class is detected, the difficulty of target determination is the lowest. Therefore, the target determination control unit 135 determines that the subject area of ​​the tracking target can be determined without using the DL target determination unit 131 and the non-DL target determination unit 132, and disables both the DL target determination unit 131 and the non-DL target determination unit 132. This can further reduce power consumption. In this case, the target determination unit 130 determines the only detected candidate area as the subject area of ​​the tracking target without using the DL target determination unit 131 and the non-DL target determination unit 132, and outputs the information. When no candidate area is detected, the target determination unit 130 outputs the determination result of, for example, one frame before again.

[0174] The control operations described here are merely examples. The target determination control unit 135 can determine which target determination unit to enable, taking into consideration whether the target image is an image of a scene in which the DL target determination unit 131 and the non-DL target determination unit 132 can accurately determine the target.

[0175] In addition, a configuration may be adopted in which the DL target determination unit 131 can switch scenes for which targets can be determined with high accuracy by switching between parameters obtained by training a multilayer neural network using training data prepared for each type of scene. For example, trained parameters corresponding to the scene being photographed by the imaging device 100' can be set in the multilayer neural network and used.

[0176] In addition, when the imaging device 100' has a user's gaze detection function, the non-DL target determination unit 132 may determine a target based on the user's gaze. For example, the non-DL target determination unit 132 can determine a candidate area including image coordinates that the user focuses on as a subject area to be tracked.

[0177] When the non-DL target determining unit 132 is set to perform target determination based on the user's line of sight and the target determination control unit 135 decides to enable the DL target determining unit 131, the target determination control unit 135 can disable the non-DL target determining unit 132. This is because in a scene with a high level of target determination difficulty in which the DL target determining unit 131 is enabled, it is considered that the user's line of sight is unlikely to be fixed on the main subject region.

[0178] 14 is a timing chart showing an example of the operation timing of each of the above-mentioned units when the imaging device 100' performs a subject tracking function including an object determination process. t1400, t1401, t1402, t1403, and t1404 are timings at which the detection unit 110 starts detection processing in each frame. At t1400, the detection unit 110 starts detection processing 1410 for the first frame (0th frame).

[0179] The detection unit 110 detects one or more candidate areas and their object classes for the 0th frame of image data through detection process 1410. The difficulty level determination unit 112 executes the difficulty level determination process 1411 as described with reference to Fig. 11 based on the information of the candidate areas detected in the 0th frame of image data, and calculates a difficulty score.

[0180] Based on the difficulty score calculated by the difficulty level judgment process 1411, the object determination control unit 135 determines whether the DL object determination unit 131 and the non-DL object determination unit 132 are enabled or disabled for the 0th frame image data.

[0181] 14, a target determination process indicated by a dashed line (e.g., DL target determination process 1412) indicates that the target determination unit (DL target determination unit 131) that executes the target determination process has been disabled by the target determination control unit 135. On the other hand, a target determination process indicated by a solid line (e.g., non-DL target determination process 1413) indicates that the target determination unit (non-DL target determination unit 132) that executes the target determination process has been enabled by the target determination control unit 135.

[0182] The tracking unit 134 executes tracking processing 1414 targeting the subject region of the tracking target output from the target determination unit 130. The tracking unit 134 estimates the position of the subject region of the tracking target in the image data to be processed. Then, the tracking unit 134 outputs information (position, size, etc.) of the estimated subject region of the tracking target in the image data of the 0th frame as a tracking result to the information superimposition unit 120. The tracking processing 1414 is shown by a dashed line because the subject region of the tracking target has not been determined in the 0th frame and therefore no tracking processing is actually performed.

[0183] By the tracking frame superimposition process 1415, the information superimposition unit 120 generates composite image data in which a tracking frame based on the result of the tracking process 1414 is superimposed on the display image data of the 0th frame. Note that the information superimposition unit 120 is shown by a dashed line because the subject area of ​​the tracking target has not been determined in the 0th frame and therefore the tracking frame is not actually superimposed.

[0184] The display unit 121 displays the composite image data of the 0th frame by the display process 1416. As described above, since the tracking frame superimposition process 1415 is not actually performed, the display unit 121 displays the display image data without the tracking frame superimposed.

[0185] The detection unit 110 detects one or more candidate regions and their object classes in the image data of the first frame through detection process 1420. The difficulty level determination unit 112 executes difficulty level determination process 1421 based on the information of the candidate regions detected in the image data of the first frame, and calculates a difficulty score.

[0186] The object determination control unit 135 determines whether to enable or disable the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the first frame, based on the difficulty score calculated by the difficulty assessment process 1421. Here, it is assumed that the object determination control unit 135 determines to enable the DL object determination unit 131 and to disable the non-DL object determination unit 132.

[0187] The non-DL target determination process 1423 is not actually performed, and is therefore indicated by a dashed line. On the other hand, the DL target determination process 1422 is executed by the DL target determination unit 131. The target determination unit 130 outputs information on the subject region of the tracking target determined by the DL target determination unit 131.

[0188] In tracking process 1424, tracking unit 134 estimates the position in the first frame image data of the subject area of ​​the tracking target determined in non-DL target determination process 1413 for the 0th frame image. Then, tracking unit 134 outputs information (position, size, etc.) of the estimated subject area of ​​the tracking target in the first frame image data to information superimposition unit 120 as the tracking result.

[0189] In a tracking frame superimposition process 1425, the information superimposition unit 120 generates composite image data by superimposing a tracking frame based on the result of the tracking process 1424 onto the display image data of the first frame. A display process 1426 causes the display unit 121 to display the composite image data of the first frame.

[0190] The detection unit 110 detects one or more candidate regions and their object classes in the image data of the second frame through detection process 1430. The difficulty level determination unit 112 executes difficulty level determination process 1431 based on the information of the candidate regions detected in the image data of the second frame, and calculates a difficulty score.

[0191] The target determination control unit 135 determines whether to enable or disable the DL target determination unit 131 and the non-DL target determination unit 132 for the image data of the second frame, based on the difficulty score calculated by the difficulty assessment process 1431. Here, it is assumed that only one candidate region is detected in the detection process 1430, and the target determination control unit 135 determines to disable both the DL target determination unit 131 and the non-DL target determination unit 132.

[0192] Therefore, the DL object determination process 1432 and the non-DL object determination process 1433 are not performed, and are indicated by dashed lines. In this case, the object determination unit 130 determines the only detected candidate area as the subject area to be tracked.

[0193] In tracking process 1434, tracking unit 134 estimates the position in the second frame image data where the subject area of ​​the tracking target determined for the first frame image in DL target determination process 1422 exists. Then, tracking unit 134 outputs information (position, size, etc.) of the estimated subject area of ​​the tracking target in the second frame image data to information superimposition unit 120 as a tracking result.

[0194] In a tracking frame superimposition process 1435, the information superimposition unit 120 generates composite image data by superimposing a tracking frame based on the result of the tracking process 1434 onto the display image data of the second frame. Display processing 1436 causes the display unit 121 to display the composite image data of the second frame.

[0195] The detection unit 110 detects one or more candidate areas and their object classes for the image data of the third frame through detection process 1440. The difficulty level determination unit 112 executes difficulty level determination process 1441 based on the information of the candidate areas detected in the image data of the third frame, and calculates a difficulty score.

[0196] The object determination control unit 135 determines whether to enable or disable the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the third frame, based on the difficulty score calculated by the difficulty assessment process 1441. Here, it is assumed that the object determination control unit 135 determines to enable both the DL object determination unit 131 and the non-DL object determination unit 132.

[0197] Therefore, the DL target determination unit 131 executes DL target determination processing 1442, and the non-DL target determination unit 132 executes non-DL target determination processing 1443. Then, the selection unit 133 selects one of the results of the DL target determination processing 1442 and the non-DL target determination processing 1443, and outputs it as information on the subject area of ​​the tracking target determined for the third frame. For example, if the two subject areas determined by the respective target determination processing are the same, the selection unit 133 selects one of the predetermined results. On the other hand, if the two subject areas determined by the respective target determination processing are different, the selection unit 133 selects one of the determination results according to a predetermined condition. For example, the selection unit 133 can select a determination result indicating a larger subject area, a subject area whose center of gravity is closer to the center of the image, or a subject area whose center of gravity is closer to the focus detection area.

[0198] In tracking process 1444, tracking unit 134 estimates the position where the subject area of ​​the tracking target determined for the second frame image exists in the third frame image data. Then, tracking unit 134 outputs information (position, size, etc.) of the estimated subject area of ​​the tracking target in the third frame image data to information superimposition unit 120 as the tracking result.

[0199] In a tracking frame superimposition process 1445, the information superimposition unit 120 generates composite image data by superimposing a tracking frame based on the result of the tracking process 1444 onto the display image data of the third frame. Display processing 1446 causes the display unit 121 to display the composite image data of the third frame.

[0200] The detection unit 110 detects one or more candidate areas and their object classes for the fourth frame of image data through detection process 1450. The difficulty level determination unit 112 executes difficulty level determination process 1451 based on the information on the candidate areas detected in the fourth frame of image data, and calculates a difficulty score.

[0201] The object determination control unit 135 determines whether to enable or disable the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the fourth frame, based on the difficulty score calculated by the difficulty assessment process 1451. Here, it is assumed that the object determination control unit 135 determines to enable both the DL object determination unit 131 and the non-DL object determination unit 132.

[0202] Therefore, the DL target determination unit 131 executes DL target determination processing 1452, and the non-DL target determination unit 132 executes non-DL target determination processing 1453. The selection unit 133 selects and outputs one of the determination results, as described in relation to the processing of the third frame.

[0203] In tracking process 1454, tracking unit 134 estimates the position where the subject area of ​​the tracking target determined for the third frame image exists in the fourth frame image data. Then, tracking unit 134 outputs information (position, size, etc.) of the estimated subject area of ​​the tracking target in the fourth frame image data to information superimposition unit 120 as the tracking result.

[0204] In a tracking frame superimposition process 1455, the information superimposition unit 120 generates composite image data by superimposing a tracking frame based on the result of the tracking process 1454 onto the display image data of the fourth frame. A display process 1456 causes the display unit 121 to display the composite image data of the fourth frame.

[0205] The timing chart in FIG. 14 is merely an example, and various modifications are possible. For example, in FIG. 14, the detection process and the difficulty level determination process are performed for each frame, but the frequency of execution may be reduced. By executing once per multiple frames, power consumption can be reduced. In this case, the subject area of ​​the tracking target is updated with the result of the target determination process for the frame in which the detection process and the difficulty level determination process are performed, and the result of the target determination process for the frame in which the detection process and the difficulty level determination process are not performed may not be used for updating.

[0206] The live view display operation in this embodiment is the same as that described in the first embodiment with reference to FIG. 7, and therefore will not be described.

[0207] FIG. 15 is a flowchart relating to the operational control of the target determining unit used to determine the subject region of the tracking target. The processing in S300 to S305 is the same as the processing described in the first embodiment with reference to Fig. 8, and therefore the description will be omitted. Note that the image correction processing is a correction processing for tracking and detection.

[0208] In S1501, the difficulty level determination unit 112 calculates a difficulty score indicating the difficulty of object determination based on the candidate area detection result in S305, for example, as described with reference to Fig. 11. The difficulty level determination unit 112 outputs the calculated difficulty score to the object determination control unit 135.

[0209] In S1502, the object determination control unit 135 determines whether the difficulty score is equal to or greater than a first threshold value set in advance. If it is determined that the difficulty score is equal to or greater than the first threshold value, the object determination control unit 135 executes S1503, and if it is not determined that the difficulty score is equal to or greater than the first threshold value, the object determination control unit 135 executes S1504.

[0210] In S1503, the target determination control unit 135 determines to enable the DL target determination unit 131, and executes control to enable the DL target determination unit 131. In S1504, the object determination control unit 135 determines to disable the DL object determination unit 131, and executes control to disable the DL object determination unit 131. Furthermore, the object determination control unit 135 executes S1505.

[0211] In S1505, the object determination control unit 135 determines whether the difficulty score is less than a preset second threshold. Here, the second threshold is smaller than the first threshold (difficulty is low). If it is determined that the difficulty score is less than the second threshold, the object determination control unit 135 executes S1506, and if it is not determined that the difficulty score is less than the second threshold, the object determination control unit 135 executes S1507.

[0212] In S1506 , the target determination control unit 135 determines to disable the non-DL target determination unit 132 , and executes control to disable the non-DL target determination unit 132 . In S1507, the target determination control unit 135 determines to enable the non-DL target determination unit 132, and executes control to enable the non-DL target determination unit 132.

[0213] In S1508, if only one of the DL target determination unit 131 and the non-DL target determination unit 132 is enabled, the target determination unit 130 selects, in the selection unit 133, the determination result of the enabled target determination unit.

[0214] When both the DL target determination unit 131 and the non-DL target determination unit 132 are enabled, if the determination results are the same subject area, the target determination unit 130 selects any one of the determination results with the selection unit 133. When the determination results are different, the target determination unit 130 selects one of the determination results based on a predetermined condition as described above with the selection unit 133. For example, the target determination unit 130 selects with the selection unit 133 the determination result indicating the subject area having a central position closer to the center of the screen. The target determination unit 130 stores the determination result selected with the selection unit 133 in the tracking memory 108 as information on the subject area of ​​the tracking target.

[0215] When both DL target determination unit 131 and non-DL target determination unit 132 are disabled, target determination unit 130 checks the number of candidate regions detected in S305. When the number of detected candidate regions is one, target determination unit 130 determines the detected candidate region as a subject region to be tracked, and stores the information in tracking memory 108. When the number of candidate regions is zero, there is no subject candidate to be tracked. In this case, target determination unit 130 may store information on the subject region of the tracking target output immediately before in tracking memory 108, or may disable tracking processing for the next frame.

[0216] This completes the process of determining the subject area to be tracked for one frame. Thereafter, the same process is executed for each frame, or for each set of frames.

[0217] Fig. 16 is a flowchart relating to the operation of the subject tracking function of the image pickup device 100', and the same reference numerals are used for the same processes as in Fig. 15. The description of the processes in S300 to S304 will be omitted.

[0218] In S1601, the tracking unit 134 determines whether or not the subject area of ​​the tracking target has been determined in the process of S1508 for the frame immediately preceding the frame to be processed. If it is determined that the subject area of ​​the tracking target has been determined in the previous frame, the tracking unit 134 executes S1602, and if it is not determined that the subject area of ​​the tracking target has been determined, the tracking unit 134 executes S1603.

[0219] In S1602, the tracking unit 134 updates the information of the subject region of the tracking target with the information determined for the previous frame. Also, if the tracking process was disabled in the previous frame, the tracking unit 134 executes control to start the tracking process. This is because, as described above, if no candidate region is detected, the tracking process for the next frame may be disabled.

[0220] In S1603, the tracking unit 134 determines whether or not the tracking is continuing. The tracking is continuing when the tracking process is enabled and the subject area of ​​the tracking target is determined in the previous frame. If the tracking unit 134 determines that the tracking is continuing, it executes S1604, and if the tracking is not continuing, it executes S314.

[0221] In S1604, the tracking unit 134 executes a tracking process for the determined object region. The tracking unit 134 outputs the estimated size of the object region and its position within the image to the information superimposing unit 120 as a tracking result.

[0222] The processing of S313 to S315 is the same as that described in the first embodiment with reference to Fig. 8, and therefore the description will be omitted. However, if it is determined in S1603 that the tracking is not in a continuing state and S314 is executed, there is no tracking result, so the tracking frame is not superimposed.

[0223] In this embodiment, in an image processing device using a first object determining unit that determines a subject region of a tracking object and a second object determining unit that is more accurate than the first object determining unit but requires a larger amount of calculation, the first and / or second tracking units are enabled or disabled based on the difficulty of object determination. Therefore, by disabling an object determining unit that is unlikely to be able to determine an object with high accuracy, or disabling the other object determining unit when it is likely that only one object determining unit can determine an object with high accuracy, it is possible to suppress power consumption while suppressing a decrease in object determination accuracy.

[0224] Since the object determination process is generally performed for each successive frame, the effect of reducing power consumption is particularly large when an object determination unit that utilizes computationally intensive machine learning such as deep learning is used as the second object determination unit.

[0225] <Third embodiment> Next, a third embodiment of the present invention will be described below. Fig. 17 is a block diagram showing an example of the functional arrangement of an image capture device 1700 as an example of an image processing device according to the third embodiment. The optical system 1701 and the image sensor 1704 may have the same configuration as the optical system 101 and the image sensor 103 of the image pickup device 100, and therefore the description thereof will be omitted. In this embodiment, a mechanical shutter 1703 (hereinafter, simply referred to as a shutter) is provided between the optical system 1701 and the image sensor 1704. Although it is provided The shutter 1703 is not essential. The operation of the shutter 1703 is controlled by a system control unit 1740. The shutter speed corresponds to the exposure time of the image sensor 1704.

[0226] The optical control unit 1702 calculates the defocus amount for each of the multiple focus detection areas by capturing an optical image formed by the optical system 1701, for example, by a phase difference AF sensor. The focus detection area may be, for example, a rectangular area predetermined within an imaging surface. The optical control unit 1702 determines the focus detection area in which the optical system 1701 is to be focused, based on the calculated defocus amount and a tracking result by the tracking unit 1718. Then, the optical control unit 1702 drives the focus lens of the optical system 1701 based on the calculated defocus amount for the determined focus detection area. As a result, the optical system 1701 is brought into focus on the subject within the determined focus detection area.

[0227] The imaging element 1704 outputs image data obtained by imaging to a first pre-processing unit 1711 and a second pre-processing unit 1712 of the image processing unit 1710. The first pre-processing unit 1711 processes the image data in the same manner as the first pre-processing unit 104 , and stores the processed image data in a display memory 1714 .

[0228] The first image correction unit 1716 applies the same processing as that of the first image correction unit 109 to the image data, and outputs the processed image data to a first post-processing unit 1719 .

[0229] The first post-processing unit 1719 generates image data for recording and image data for display in the same manner as the post-processing unit 114. The first post-processing unit 1719 supplies the image data for recording to the recording unit 1720. In addition, the first post-processing unit 1719 supplies the image data for display to the information superimposition unit 1721.

[0230] The recording unit 1720 records the image data for recording generated by the first post-processing unit 1719 on a recording medium 1730. The recording medium 1730 may be, for example, a semiconductor memory card, an internal non-volatile memory, or the like.

[0231] The second pre-processing unit 1712 applies color interpolation processing to the image data, similar to the second pre-processing unit 105, and stores the image data in the tracking memory 1715. 15 The first pre-processing unit 1711 and the second pre-processing unit 1712 may be implemented as separate address spaces in the same memory space. The second pre-processing unit 1712 may apply a reduction process to reduce the number of pixels as necessary to reduce the processing load. Although the first pre-processing unit 1711 and the second pre-processing unit 1712 are described here as separate functional blocks, they may be configured to use a common pre-processing unit.

[0232] The second image correction unit 1713 stores the tracking memory 17 15 The second image correction unit 1713 applies processing to the image data stored in the tracking memory 17 in the same manner as the second image correction unit 106. 15 Store in.

[0233] In this embodiment, the functional blocks related to the subject tracking function, such as the second pre-processing unit 1712 and the second image correction unit 1713, do not need to operate when the subject tracking function is not performed. Moreover, the image data to which the subject tracking function is applied is video data captured for live view display or recording. The video data has a predetermined frame rate, such as 30 fps, 60 fps, or 120 fps.

[0234] The detection unit 1717 detects one or more regions (candidate regions) of predetermined candidate subjects from one frame of image data. The detection unit 1717 also associates, for each detected region, the position and size within the frame, an object class indicating the type of candidate subject (car, airplane, bird, insect, human body, head, eye, cat, dog, etc.), and its reliability. The detection unit 1717 also finds the number of detected candidate regions for each object class.

[0235] In this embodiment, the detection unit 1717 has a plurality of detection units with different computational loads and object determination accuracy. Specifically, the detection unit 1717 has a DL detection unit 1722 that performs object determination processing using deep learning (DL) and a non-DL detection unit 1723 that performs object determination processing without using DL. The DL detection unit 1722 has a higher processing accuracy than the non-DL detection unit 1723, but a larger computational load than the non-DL detection unit 1723.

[0236] The DL detection unit 1722 and the non-DL detection unit 1723 each detect candidate regions, associate object classes and their reliability with types of candidate regions, and count the number of candidate regions for each object class. In addition, the DL detection unit 1722 and the non-DL detection unit 1723 obtain size and position information for the detected candidate regions.

[0237] The DL detection unit 1722 can be realized by a multi-class classifier using machine learning. The DL detection unit 1722 can be implemented using multi-class logistic regression, a support vector machine, a random forest, a neural network, or the like.

[0238] The non-DL detection unit 1723 detects a candidate region by a known method that does not use machine learning. The non-DL detection unit 1723 can detect the candidate region by, for example, pattern matching using a template prepared in advance for each object class, feature detection using feature information prepared in advance for each object class, detection based on distance information, or the like.

[0239] Similar to the information superimposing unit 120, the information superimposing unit 1721 generates an image of the tracking frame based on the center position and size of the subject area of ​​the tracking target output by the detection unit 1717 or the tracking unit 1718. Then, the information superimposing unit 1721 generates composite image data in which the image of the tracking frame is superimposed on the display image data output by the first post-processing unit 1719. The information superimposing unit 1721 outputs the composite image data to the display unit 1731.

[0240] The non-volatile memory 1753 is a rewritable non-volatile memory such as an EEPROM, and stores programs executable by the CPU of the system control unit 1740, setting values, GUI data, and the like.

[0241] The system memory 1754 is, for example, a RAM, and is used to load the programs executed by the CPU of the system control unit 1740 and to store values ​​required during the execution of the programs.

[0242] The system control unit 1740 has a CPU, and executes, for example, a program stored in a nonvolatile memory 1753 by loading it into a system memory 1754. The system control unit 1740 realizes the functions of the imaging device 1700 by controlling the operation of each functional block.

[0243] A release button 1750 , a mode changeover switch 1751 , and an operation unit 1752 are input devices for inputting instructions to the system control unit 1740 . The release button 1750 has a switch SW1 that turns on when pressed halfway, and a switch SW2 that turns on when pressed all the way. The system control unit 1740 recognizes the ON of SW1 as an instruction to prepare for shooting a still image, and the ON of SW2 as an instruction to start shooting a still image, and executes an operation according to each instruction.

[0244] A system control unit 1740 executes AF (autofocus) processing, AE (auto exposure) processing, AWB (auto white balance) processing, EF (pre-flash) processing, and the like in response to a shooting preparation instruction.

[0245] In addition, the system control unit 1740 executes still image shooting processing using the results of the AF processing and AE processing executed in response to the shooting preparation instruction in response to the shooting instruction. The still image shooting processing includes opening and closing control of the shutter 1703, operation control of the image sensor 1704, generation of image data for recording in the image processing unit 1710, and recording to the recording medium 1730. The system control unit 1740 controls each functional block for this series of still image shooting processing.

[0246] The motion sensor 1760 outputs a signal according to the motion of the imaging device 1700, and outputs the signal to the system control unit 1740. The motion sensor 1760 may be, for example, an acceleration sensor or an angular velocity sensor. Furthermore, if the optical system 101 has a motion sensor for camera shake correction, the motion sensor 1760 may be omitted and the motion of the imaging device 1700 may be detected using the motion sensor for camera shake correction.

[0247] The mode changeover switch 1751 changes the operation mode of the system control unit 1740 to one of a still image shooting mode, a video shooting mode, a playback mode, etc. Modes included in the still image shooting mode include an auto shooting mode, an auto scene determination mode, a manual mode, an aperture priority mode (Av mode), and a shutter speed priority mode (Tv mode). In addition, there are various scene modes, which are shooting settings according to shooting scenes, a program AE mode, a custom mode, etc. The mode changeover switch 1751 can directly change to one of these modes included in the menu button. Alternatively, after the mode changeover switch 1751 is once switched to the menu button, another operating member can be used to change to one of these modes included in the menu button. Similarly, the video shooting mode may also include multiple modes.

[0248] Operation unit 1752 is a general term for input devices other than release button 1750 and mode changeover switch 1751. Functions are assigned to the input devices included in operation unit 1752 according to the operating state of imaging device 1700, for example, through a selection operation of a function icon displayed on display unit 1731. The input devices have names according to the assigned functions.

[0249] For example, direction buttons, a determination (SET) button, an end button, a back button, an image forward button, a jump button, a narrowing down button, an attribute change button, a menu button, and the like are included in the operation unit 1752. For example, when the menu button is pressed, a menu screen in which various settings can be made is displayed on the display unit 1731. The user can make various settings by operating the menu screen displayed on the display unit 1731 using the direction buttons and the SET button.

[0250] In this embodiment, a gaze information acquisition unit (gaze detection means) is provided as an input device included in the operation unit 1752. The gaze information acquisition unit detects the user's gaze or the user's gaze position within the display unit 1731. Fig. 18 is a diagram showing an example of the configuration of the gaze information acquisition unit 1800. Here, the display unit 1731 is provided within a peer-type finder, and the position within the display unit 1731 at which the user is gazing is detected by detecting the gaze direction from the rotation angle of the user's eyeball 1802 looking into the finder.

[0251] 18, line-of-sight information acquisition section 1800 has image sensor 1801a, light receiving lens 1801b, dichroic mirror 1801c, and illumination light source 1801e. Eyepiece lens 1801d is a finder optical system and is not directly related to acquisition of line-of-sight information.

[0252] The user can observe the display on the display unit 1731 through the eyepiece 1801d and the dichroic mirror 1801c. The illumination light source 1801e projects infrared light toward the outside of the viewfinder. When the user is looking through the viewfinder, the infrared light is reflected by the eyeball 1802, and the reflected light enters the viewfinder. The reflected light is reflected upward by the dichroic mirror 1801c, and an eyeball image formed by the light receiving lens 1801b is captured by the image sensor 1801a. Image data of the eyeball image is transmitted from the image sensor 1801a to the system control unit 1740.

[0253] The system control unit 1740 extracts, for example, the pupil area from the eyeball image, and detects the rotation angle of the eyeball from the position of the pupil area. The system control unit 1740 then obtains the user's gaze direction from the rotation angle of the eyeball. The gaze direction may be obtained by calculation, or may be obtained by referring to a table that stores the relationship between the rotation angle and the gaze direction. The system control unit 1740 detects the position (coordinates) within the display unit 1731 where the user is gazing based on the gaze direction. The gaze position may be obtained from the rotation angle of the eyeball.

[0254] In this example, it is assumed that one eye is looking through the viewfinder, and gaze information such as the gaze direction and gaze position of both eyes is obtained for one eye. However, if the environment allows acquisition of eyeball images of both eyes, gaze information may be obtained for each eye. When obtaining gaze information for both eyes, the accuracy of the gaze information can be improved by, for example, using the gaze information of one eye that is considered to have a higher detection accuracy.

[0255] The line-of-sight information acquisition unit 1800 may have any other configuration as long as it can identify the position in the display unit 1731 that the user is gazing at. The display unit 1731 does not necessarily have to be disposed inside the imaging device 1700. For example, the display unit 1731, the illumination light source 1801e, and the light receiving lens 1801b may be provided on the back surface of the imaging device 1700.

[0256] Next, based on the line of sight or gaze position detected by the line of sight information acquisition unit 1800, the DL detection unit 17 22 and non-DL detection section 17 The operation of system control unit 1740, which controls whether 23 is enabled or disabled, will now be described.

[0257] Fig. 19 is a diagram showing functional blocks related to the operation of controlling the enable / disable of the subject detection unit based on the line of sight or gaze position. Note that in Fig. 19, system control unit 1740 is shown as three functional blocks for the sake of convenience, but in reality, the functions of these functional blocks are realized by the CPU of system control unit 1740 executing a program.

[0258] The operation of system control unit 1740 for controlling the enable / disable of the subject detection unit based on the line of sight or gaze position will be described below with reference to the flowchart shown in FIG. 20 and FIGS. S2000 is image acquisition processing, which will be described in detail with reference to the flowchart of FIG. In S2100, the system control unit 1740 controls the image sensor 1704 to perform an image capturing operation for one frame, thereby converting the subject optical image formed by the optical system 1701 into RAW image data made up of data of a plurality of pixels.

[0259] In S2102, the first pre-processing unit 1711 of the image processing unit 1710 applies demosaic processing to the RAW image data to generate image data in which each pixel has RGB components. The first pre-processing unit 1711 stores the generated image data in the display memory 1714. Note that the first pre-processing unit 1711 may convert the image data from RGB format to YUV format before storing it in the display memory 1714.

[0260] In S2103, the first image correction unit 1716 applies a predetermined image correction process to the image data stored in the display memory 1714. The image correction process includes a process of calculating an evaluation value. The first image correction unit 1716 may vary the read range of the image data depending on the process to be applied.

[0261] For example, when image processing is applied to image data of one line (pixel row), there are cases where only the image data of the one line to which the processing is applied is read out, and cases where image data of multiple lines including the one line to which the processing is applied is read out. The image processing may also be applied to image data of a rectangular area of ​​a predetermined size. The first image correction unit 1716 executes the image correction processing while appropriately storing the image processing results in the display memory 1714.

[0262] In S2104, the first image correction unit 1716 determines whether or not all image correction processes to be applied to one frame's worth of image data have been completed. If it is determined that all image correction processes to be applied have been completed, the first image correction unit 1716 outputs the image data after the image correction processes to the first post-processing unit 1719, and outputs the evaluation value to the system control unit 1740. This completes the image acquisition process for one frame. On the other hand, if it is determined that there is still image correction process remaining to be applied, the first image correction unit 1716 repeatedly executes S2103.

[0263] Except for the fact that the image data is stored in tracking memory 1715 and that second image correction unit 1713 applies image processing for tracking, similar processing is performed in second pre-processing unit 1712, tracking memory 1715, and second image correction unit 1713.

[0264] 20, in S2001, the tracking unit 1718 applies tracking processing to the image data acquired in S2000 and stored in the tracking memory 1715. It is assumed that the tracking unit 1718 applies the non-DL tracking processing applied by the non-DL tracking unit 117 in the first embodiment. The tracking unit 1718 outputs the position and size of the subject area of ​​the tracking target as the tracking result.

[0265] In S2002, the system control unit 1740 executes a process of determining whether or not the subject has been lost sight of. The process of determining whether or not the subject has been lost sight of will be described in detail with reference to the flowchart of FIG. In S2200, system control unit 1740 acquires the user's gaze information through gaze information acquisition unit 1800. The gaze information acquired here may be the gaze direction or the gaze position.

[0266] In S2201, the gaze analysis unit 1900 calculates the degree of irregularity of the gaze information. The degree of irregularity of the gaze information is an evaluation value of the stability of the gaze position of the user. When the degree of irregularity is large, the gaze position of the user is not stable and moves irregularly within the screen of the display unit 1731. Therefore, when the degree of irregularity of the gaze information is large, it is considered that the user has lost sight of the main subject and is searching for it within the screen.

[0267] In S2003, the irregularity determining unit 1901 determines whether or not the user has lost sight of the subject based on the degree of irregularity calculated in S2201.

[0268] The degree of irregularity of the gaze information in S2201 may be calculated by any method. In addition, the determination in S2003 may be performed using a threshold value according to the degree of irregularity. An example of a method for calculating and determining the degree of irregularity is shown below. -Use frequency information on the time axis of gaze position or line of sight The frequency spectrum is calculated, and the magnitude of the spectrum above a certain frequency is calculated as the degree of irregularity. If the degree of irregularity exceeds a threshold, it is determined that the subject has been lost. Calculate the absolute value of the autocorrelation of gaze position or gaze path as the degree of irregularity. If the gaze movement is regular, the autocorrelation in adjacent time intervals will approach 1, and if the gaze movement is random, the autocorrelation will approach 0. If the degree of irregularity is below a threshold, it is determined that the subject has been lost. The absolute value of the cross-correlation between the average value of the magnitude of the velocity vector or acceleration vector of the gaze position or line of sight and the average value of the magnitude of the velocity vector or acceleration vector of the trajectory of each subject area during a certain time span T is calculated as the degree of irregularity. If the irregularity degree for all object regions is less than the threshold value, it is determined that the object has been lost.

[0269] If the irregularity determination unit 1901 determines in S2003 that the subject has been lost, the process determination unit 1902 executes S2004. If the irregularity determination unit 1901 does not determine in S2003 that the subject has been lost, the process determination unit 1902 executes S2005.

[0270] In S2004, the processing determination unit 1902 determines to enable the DL detection unit 1722. The non-DL detection unit 1723 may be disabled or enabled. The DL detection unit 1722 in the image processing unit 1710 executes a candidate area detection process using machine learning (here, DL) as described above. When only one candidate area is detected by the DL detection unit 1722, the system control unit 1740 determines the detected candidate area as the subject area to be tracked.

[0271] On the other hand, when a plurality of candidate areas are detected by the DL detection unit 1722, the system control unit 1740 determines the subject area of ​​the tracking target from the plurality of candidate areas. The system control unit 1740 can determine the subject area of ​​the tracking target based on the size and position of the candidate area as described in the previous embodiment, or by other methods. There is no particular limitation on other methods, but for example, the user may select the subject area of ​​the tracking target from the detected candidate areas. For example, an index similar to the tracking frame may be superimposed on each of the detected candidate areas, and the user may select the subject area of ​​the tracking target through the operation unit 1752. For example, the user can select the subject area of ​​the tracking target through the operation of the direction button and the SET button, or through the detection of the gaze position using the gaze information acquisition unit 1800.

[0272] In S2005, the processing determination unit 1902 determines to disable the DL detection unit 1722 and to enable the non-DL detection unit 1723. As described above, the non-DL detection unit 1723 in the image processing unit 1710 detects candidate areas by a method that does not use machine learning. In this case, since it is determined that the user has not lost sight of the subject, the system control unit 1740 can determine the subject area to be tracked from among the candidate areas based on the line of sight information.

[0273] As in the first embodiment, an object determining section may be provided in the image processing section 1710, and the object determining section may determine the subject area to be tracked from the candidate area.

[0274] In S2006, system control unit 1740 outputs information about the subject region of the tracking target to tracking unit 1718 and information superimposing unit 1721. As a result, the information about the subject region of the tracking target is updated.

[0275] In S2007, the system control unit 1740 disables both the DL detection unit 1722 and the non-DL detection unit 1723 to reduce power consumption.

[0276] In S2008, the information superimposing unit 1721 generates composite image data by superimposing the tracking frame indicating the subject area of ​​the tracking target updated in S2006 on the display image data from the first post-processing unit 1719, and outputs the composite image data to the display unit 1731.

[0277] In this embodiment, a first subject detection unit that detects a candidate area for the subject area of ​​the tracking target and a second subject detection unit that is more accurate than the first subject detection unit but requires a larger amount of calculation are used depending on whether the user has lost sight of the subject. Specifically, the second subject detection unit is used when the user has lost sight of the subject, and is not used when the user has not lost sight of the subject. When the user has not lost sight of the subject, the first subject detection unit can detect the candidate area with sufficient accuracy, so by disabling the second subject detection unit, it is possible to suppress the deterioration of detection accuracy while suppressing power consumption.

[0278] <Fourth embodiment> Next, a fourth embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are enabled or disabled depending on the autofocus (AF) mode of the imaging device. Since this embodiment can be implemented by the imaging device 1700 similar to the third embodiment, the embodiment will be described using the components shown in FIG.

[0279] Here, the imaging device 1700 has, as AF modes, a single AF mode in which the focus detection result is maintained, and a continuous AF mode in which focus detection is performed continuously. The single AF mode is likely to be used for shooting still or static subjects such as landscapes and portraits. On the other hand, the continuous AF mode is likely to be used for shooting moving subjects such as vehicles and sports scenes.

[0280] Therefore, the movement of the main subject that the user is about to capture can be inferred from the set AF mode. The AF mode may be explicitly specified by the user, or may be automatically set according to the capture mode of the image capture device 1700, such as a sports mode or a landscape mode. Alternatively, the AF mode may be set by the system control unit 1740 based on other conditions.

[0281] The AF mode is a setting related to focus detection operation when shooting still images, while subject tracking in the still image shooting mode is applied to moving images captured for live view display in shooting standby state.

[0282] In this embodiment, in single AF mode, the subject is likely to be stationary or the subject's movement is small, and subject detection is likely to be relatively easy, so the DL detection unit 1722 is disabled and the non-DL detection unit 1723 is enabled. On the other hand, in continuous AF mode, the subject is likely to be moving or the subject's movement is large, and subject detection is likely to be relatively difficult, so the DL detection unit 1722 is enabled and the non-DL detection unit 1723 is disabled. By such control, it is possible to suppress power consumption while suppressing a decrease in the accuracy of subject detection.

[0283] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 23. In Fig. 23, the steps performing the same processes as in the third embodiment are given the same reference numbers as in Fig. 20, and the description thereof will be omitted. The processes of S2000 and S2001 are as described in the third embodiment.

[0284] In S2302, the system control unit 1740 determines whether the current AF mode of the imaging device 1700 is the continuous AF mode. If the system control unit 1740 determines that the current AF mode of the imaging device 1700 is the continuous AF mode, it enables the DL detection unit 1722 in S2004. Power consumption can be reduced by disabling the non-DL detection unit 1723, but it may also be enabled. On the other hand, if the system control unit 1740 does not determine that the current AF mode of the imaging device 1700 is the continuous AF mode, it enables the non-DL detection unit 1723 and disables the DL detection unit 1722 in S2005.

[0285] Below The subsequent processing is similar to that in the third embodiment, and therefore a description thereof will be omitted.

[0286] In this embodiment, a first subject detection unit that detects a candidate area for the subject area of ​​the tracking target and a second subject detection unit that is more accurate than the first subject detection unit but requires a larger amount of calculation are used depending on the autofocus mode of the imaging device. Specifically, the second subject detection unit is used when an autofocus mode that is likely to be used for shooting a moving subject is set, and is not used when an autofocus mode that is likely to be used for shooting a still subject is set. When the subject does not move much, the first subject detection unit can detect the candidate area with high accuracy, so by disabling the second subject detection unit, it is possible to suppress the deterioration of detection accuracy while suppressing power consumption.

[0287] <Fifth embodiment> Next, a fifth embodiment of the present invention will be described. In this embodiment, the DL detection unit 1722 and the non-DL detection unit 1723 are enabled or disabled depending on the frame rate of the moving image to which the tracking process is applied. Since this embodiment can be implemented by the same imaging device 1700 as the third embodiment, the embodiment will be described using the components shown in FIG.

[0288] The higher the frame rate, the smaller the movement of the subject area between frames. Therefore, when the frame rate is high, the movement of the subject area to be tracked is small, and the possibility of subject blurring is low. Therefore, there is a high possibility that sufficient accuracy can be obtained by the non-DL detection unit 1723. Therefore, there is little need for detection by the DL detection unit 1722 for each frame.

[0289] Therefore, when the frame rate of the video to which the tracking process is applied is higher than a predetermined first threshold (fps), the frequency at which the DL detection unit 1722 is enabled is reduced compared to when the frame rate is equal to or lower than the first threshold. This makes it possible to reduce power consumption while suppressing deterioration in detection accuracy. Note that the first threshold can be determined experimentally, for example. Note that a plurality of thresholds may be provided, and the frequency at which the DL detection unit 1722 is enabled may be controlled in three or more stages.

[0290] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 24. Here, it is assumed that it has already been determined that the frame rate of the moving image to which the tracking process is applied is higher than a predetermined first threshold value (fps). In Fig. 24, the same reference numbers as in Fig. 20 are used for steps that perform the same processes as in the third embodiment, and the description thereof will be omitted. The processes of S2000 and S2001 are as described in the third embodiment.

[0291] In S2402, the system control unit 1740 determines whether the value of the variable COUNT used as a counter for the number of frames is a predetermined value N (an integer equal to or greater than 2). If the frame rate of the video to which the tracking process is applied is equal to or less than a first threshold value (fps), and the DL detection unit 1722 is enabled every M frames, the value of N is determined to satisfy the relationship N>M. Here, M=1 and N=10. If the system control unit 1740 determines that the value of the variable COUNT is N, it executes S2403, and if not, it executes S2404.

[0292] Since N=10, S2403 is executed once every 10 frames. In S2403, the system control unit 1740 initializes the value of the variable COUNT to 1. After that, in S2004, the system control unit 1740 enables the DL detection unit 1722. Although power consumption can be reduced by disabling the non-DL detection unit 1723, it may also be enabled.

[0293] Meanwhile, in S2404, the system control unit 1740 increments the value of the variable COUNT by 1. After that, in S2405, the system control unit 1740 enables the non-DL detection unit 1723 and disables the DL detection unit 1722. The subsequent processing is similar to that in the third embodiment, and therefore will not be described.

[0294] This completes the operation for one frame in the live view display. With the above operations, when the frame rate of the video to which the tracking process is applied is high, the DL detection unit 1722 is enabled for one frame out of 10 frames and disabled for the remaining nine frames. In frames where the DL detection unit 1722 is disabled, the non-DL detection unit 1723 is enabled.

[0295] In this embodiment, the frequency of using the second subject detection unit, which is a subject detection unit that detects candidate areas for the subject area of ​​the tracking target and has higher accuracy but requires more calculations than the first subject detection unit, is controlled according to the frame rate of the video to which the tracking process is applied. Specifically, when the frame rate of the video to which the tracking process is applied is high, the frequency of enabling the second subject detection unit is lower than when the frame rate is not high. When the subject moves little, the first subject detection unit can detect the candidate area with high accuracy, so by suppressing the frequency of using the second subject detection unit, it is possible to suppress the deterioration of detection accuracy and suppress power consumption.

[0296] <Sixth embodiment> Next, a sixth embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are enabled or disabled depending on the shutter speed of the imaging device. Since this embodiment can be implemented by the imaging device 1700 similar to the third embodiment, the components shown in FIG. 17 will be used for the description.

[0297] When the shutter speed is fast, the possibility of subject blurring is low. Therefore, there is a high possibility that sufficient accuracy can be obtained with the non-DL detection unit 1723. Therefore, when the shutter speed is fast, the DL detection unit 1722 is disabled.

[0298] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 25. In Fig. 25, the steps performing the same processes as in the third embodiment are given the same reference numbers as in Fig. 20, and the description thereof will be omitted. The processes of S2000 and S2001 are as described in the third embodiment.

[0299] In S2502, the system control unit 1740 determines whether or not the currently set shutter speed is faster than a threshold value. The threshold value used here can be determined, for example, experimentally depending on the performance of the non-DL detection unit 1723 and the like.

[0300] If the shutter speed is not determined to be faster than the threshold, the system control unit 1740 enables in S2004 the DL detection unit 1722. Although power consumption can be reduced by disabling the non-DL detection unit 1723, it may also be enabled. On the other hand, if it is not determined that the shutter speed is faster than the threshold value, in S2005 the system control unit 1740 enables the non-DL detection unit 1723 and disables the DL detection unit 1722. The subsequent processing is similar to that in the third embodiment, and therefore description thereof will be omitted.

[0301] In this embodiment, a first subject detection unit that detects a candidate area for the subject area of ​​the tracking target and a second subject detection unit that is more accurate than the first subject detection unit but requires a larger amount of calculation are used depending on the shutter speed of the imaging device. Specifically, the second subject detection unit is used when the shutter speed is not fast (slow) and is not used when the shutter speed is fast. When the shutter speed is fast, the possibility of subject blurring is low and the first subject detection unit can detect the candidate area with high accuracy, so by disabling the second subject detection unit, it is possible to suppress power consumption while suppressing a decrease in detection accuracy.

[0302] <Seventh embodiment> Next, a seventh embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are enabled or disabled in response to a user instruction. Since this embodiment can be implemented by the same imaging device 1700 as the third embodiment, the components shown in FIG. 17 will be used to describe it.

[0303] In this embodiment, the DL detection unit 1722 is enabled only when the user enables the DL subject detection process. Any user instruction may be given to enable the DL subject detection process. The user does not need to consciously give an instruction to enable the DL subject detection process, and any instruction that results in the DL subject detection process being enabled may be used.

[0304] As an example, it is assumed here that operation unit 1752 has an input device (DL subject detection button) to which a function for switching between enabling and disabling DL subject detection processing is assigned. System control unit 1740 switches between enabling and disabling DL subject detection processing each time the DL subject detection button is pressed. Alternatively, system control unit 1740 may enable DL subject detection processing only while the DL subject detection button is pressed. Furthermore, the DL subject detection button may be a software key using display unit 1731.

[0305] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 26. In Fig. 26, the steps that perform the same processes as in the third embodiment are given the same reference numbers as in Fig. 20, and the description thereof will be omitted. The processes of S2000 and S2001 are as described in the third embodiment.

[0306] In S2602, the system control unit 1740 determines whether the DL subject detection button is ON. Here, it is determined that the DL subject detection button is ON when it is pressed. If the system control unit 1740 determines that the DL subject detection button is ON, it enables the DL detection unit 1722 in S2004. Although power consumption can be reduced by disabling the non-DL detection unit 1723, it may also be enabled. On the other hand, if the system control unit 1740 does not determine that the DL subject detection button is ON, it enables the non-DL detection unit 1723 in S2005 and disables the DL detection unit 1722. The subsequent processing is similar to that in the third embodiment, and therefore description thereof will be omitted.

[0307] In this embodiment, among the subject detection units that detect candidate areas for the subject area of ​​the tracking target, the second subject detection unit, which is more accurate than the first subject detection unit but requires more calculation, is enabled or disabled according to a user instruction. Specifically, the second subject detection unit is enabled only when a user instruction to enable the second subject detection unit is detected. This makes it possible to reduce power consumption because the second subject detection unit is disabled when not required by the user.

[0308] <Eighth embodiment> Next, an eighth embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are enabled or disabled depending on the operation status of the device. Since this embodiment can be implemented by the same imaging device 1700 as the third embodiment, the components shown in FIG. 17 will be used to describe it.

[0309] Specifically, when it is determined that a pan, tilt, or zoom operation for changing the shooting range is being performed, the DL detection unit 1722 is disabled. This is because when these operations are being performed, the user may be adjusting the main subject so that it fits within the shooting range. Therefore, even if the DL detection unit 1722 is enabled during these operations, there is a possibility that a meaningful detection result will not be obtained. In this embodiment, the DL detection unit 1722 may be enabled only when it is determined that the image capture device 1700 is not being panned, tilted, or zoomed.

[0310] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 27. In Fig. 27, the steps that perform the same processes as in the third embodiment are given the same reference numbers as in Fig. 20, and the description thereof will be omitted. The processes of S2000 and S2001 are as described in the third embodiment.

[0311] In S2702, the system control unit 1740 determines whether a pan, tilt, or zoom operation is being performed on the imaging device 1700. The system control unit 1740 can detect a pan, tilt, or zoom operation by detecting, for example, a motion vector between frames of a moving image captured for live view display or a change in size of a feature area, or by the output of the motion sensor 1760. A zoom operation can also be detected by operating a zoom button on the operation unit 1752.

[0312] If it is determined that a panning, tilting, or zooming operation is being performed, the system control unit 1740 executes S2708. On the other hand, if it is not determined that a panning, tilting, or zooming operation is being performed, the system control unit 1740 executes S2702.

[0313] In S2702, the system control unit 1740 enables the DL detection unit 1722 or the non-DL detection unit 1723. When the DL detection unit 1722 is enabled, the non-DL detection unit 1723 can be disabled to reduce power consumption, but may be enabled. The condition for enabling the DL detection unit 1722 may be, for example, one or more of the conditions described in the third to seventh embodiments. If the condition for enabling the DL detection unit 1722 is not satisfied, the system control unit 1740 disables the DL detection unit 1722 and enables the non-DL detection unit 1723. The processing from S2006 onwards is similar to that in the third embodiment, and therefore will not be described.

[0314] In this embodiment, among the subject detection units that detect candidate areas for the subject area of ​​the tracking target, the second subject detection unit, which is more accurate than the first subject detection unit but requires a larger amount of calculation, can be enabled only when no operation is performed to change the shooting range. By not enabling the second subject detection unit when the shooting range is not determined, power consumption can be efficiently reduced. Furthermore, by not enabling the first subject detection unit when the shooting range is not determined, power consumption can be further reduced.

[0315] (Modification) Note that the enable / disable of DL subject detection and non-DL subject detection may be controlled based on conditions different from those described in the above-mentioned third to eighth embodiments. For example, the enable / disable of DL subject detection and non-DL subject detection may be controlled according to the shooting mode (still image shooting mode or video shooting mode), the pressing state of the release button (not pressed, half-pressed, fully pressed), the state of the tracking mode, or the type of tracking object.

[0316] <Ninth embodiment> Next, a ninth embodiment of the present invention will be described. Fig. 28 is a block diagram showing an example of the functional configuration of an imaging device 1700' as an example of an image processing device according to the ninth embodiment. The same reference numerals are used to designate the same components as those of the imaging device 1700 described using Fig. 17, and duplicated explanations will be omitted. Note that the detection unit 1717 of this embodiment may be either the DL detection unit 1722 or the non-DL detection unit 1723. Here, it is assumed to be the same as the non-DL detection unit 1723.

[0317] The imaging device 1700' of this embodiment has a new component, an object determining unit 2801, and the tracking unit 1718 has a DL tracking unit 2802 and a non-DL tracking unit 2803, similar to the first embodiment.

[0318] Similar to the object determining unit 111 of the first embodiment, the object determining unit 2801 determines a subject area (main subject area) to be tracked from the candidate area detected by the detection unit 1717. If there is one candidate area detected, the object determining unit 2801 determines the detected candidate area as the main subject area. If multiple candidate areas are detected, the object determining unit 2801 can determine the main subject area based on a predetermined priority order.

[0319] The object determining unit 2801 can determine the main subject region based on, for example, a priority order based on the size of the candidate region or a priority order based on the distance from the focus detection region. Specifically, the largest candidate region may be determined as the main subject region, or the candidate region closest to the focus detection region may be determined as the main subject region. In addition, the main subject region may be determined taking into consideration multiple priorities, such as determining the largest candidate region among the candidate regions whose distance from the focus detection region is less than a threshold value as the main subject region. Priorities other than those exemplified here may also be considered.

[0320] The tracking unit 1718 estimates the subject area of ​​the tracking target from the image data of the frame to be processed (current frame) stored in the tracking memory 1715, and obtains the position and size of the estimated subject area in the frame as the tracking result. For example, the tracking unit 115 estimates the position and size in the current frame of the main subject area determined by the target determination unit 2801 for a previous frame (for example, the previous frame) captured before the current frame. The tracking unit 115 outputs the tracking result to the information superimposition unit 1721.

[0321] The tracking unit 1718 has a DL tracking unit 2802 that tracks a subject using deep learning (DL) and a non-DL tracking unit 2803 that tracks a subject without using DL. The configurations of the DL tracking unit 2802 and the non-DL tracking unit 2803 may be the same as the DL tracking unit 116 and the non-DL tracking unit 117 described in the first embodiment, respectively, and detailed description will be omitted. In this embodiment, the system control unit 1740 controls the enable and disable of the DL tracking unit 2802 and the non-DL tracking unit 2803, and the operation frequency when enabled.

[0322] In this embodiment, the DL tracking unit 2802 also outputs the position and size of the estimated main subject region and the reliability score as tracking results. The non-DL tracking unit 2803 also outputs the position and size of the estimated main subject region and the similarity score as tracking results.

[0323] The tracking unit 1718 adopts one of the tracking results of the DL tracking unit 2802 and the non-DL tracking unit 2803 based on the reliability score output by the DL tracking unit 2802 and the similarity score output by the non-DL tracking unit 2803. For example, when the reliability score is equal to or less than a predetermined reliability score threshold and the similarity score is equal to or less than a predetermined similarity score threshold, the tracking unit 1718 adopts the tracking result of the non-DL tracking unit 2803. In other cases, the tracking unit 1718 adopts the tracking result of the DL tracking unit 2802. The tracking unit 1718 outputs the adopted tracking result to the information superimposing unit 1721 and the system control unit 1740.

[0324] Here, whether to adopt the tracking result of DL tracking unit 2802 or non-DL tracking unit 2803 is determined based on the reliability score and the similarity score. However, other methods may be used to determine the result. For example, the tracking result of DL tracking unit 2802 may be preferentially adopted by utilizing the fact that the accuracy of DL tracking unit 2802 tends to be higher than the accuracy of non-DL tracking unit 2803. Specifically, if the tracking result of DL tracking unit 2802 is obtained, the tracking result of DL tracking unit 2802 may be adopted, and if not, the tracking result of non-DL tracking unit 2803 may be adopted.

[0325] Next, a description will be given of the control operation of the DL tracking unit 2802 and the non-DL tracking unit 2803 performed by the system control unit 1740 in this embodiment. In this embodiment, the system control unit 1740 controls whether to enable the DL tracking unit 2802 or the non-DL tracking unit 2803 depending on the AF mode set in the imaging device 1700'.

[0326] Here, the image pickup device 1700' has a single AF mode and a continuous AF mode, similar to the fourth embodiment. The single AF mode is likely to be used for shooting still or static subjects such as landscapes and portraits. On the other hand, the continuous AF mode is likely to be used for shooting moving subjects such as vehicles and sports scenes.

[0327] In single AF mode, it is highly likely that the subject is still or that its movement is small. Therefore, the non-DL tracking section is enabled instead of the DL tracking section. On the other hand, in continuous AF mode, it is highly likely that the subject is moving or that its movement is large. Therefore, the DL tracking section is enabled instead of the non-DL tracking section. This type of control makes it possible to reduce power consumption while suppressing any deterioration in tracking accuracy.

[0328] FIG. 29 is a flowchart relating to the operation of the system control unit 1740 in this embodiment. S2900 is an image acquisition process similar to the process described with reference to FIG. 21 in the third embodiment.

[0329] In S2901, system control unit 1740 instructs detection unit 1717 to detect candidate areas. Detection unit 1717 applies subject area detection processing to one frame of image, and stores the detection results in tracking memory 1715. The detection results include the total number of detected candidate areas, and the position and size of each candidate area within the image.

[0330] In S2902, the system control unit 1740 instructs the object determining unit 2801 to determine a subject area (main subject area) to be tracked from among the candidate areas. The object determining unit 2801 reads out information on the candidate areas stored in the tracking memory 1715, and when multiple candidate areas are detected, determines the main subject area based on the priority order as described above. The object determining unit 2801 stores information identifying the main subject area in the tracking memory 1715.

[0331] In S2903, the system control unit 1740 determines whether the AF mode currently set in the imaging device 1700' is the continuous AF mode. If it is determined that the AF mode currently set is the continuous AF mode, the system control unit 1740 executes S2904, and if it is not determined that the AF mode currently set is the continuous AF mode, the system control unit 1740 executes S2905.

[0332] In S2904, the system control unit 1740 enables the DL tracking unit 2802 and disables the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 executes tracking processing for the current frame. Also, in S2905, the system control unit 1740 enables the non-DL tracking unit 2803 and disables the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 executes tracking processing for the current frame. The main subject region used in the tracking process in S2904 and S2905 is the main subject region most recently determined for a frame older than the current frame.

[0333] In S2906, system control unit 1740 updates the information on the main subject region stored in tracking memory 1715 with the result of the tracking process in S2904 or S2905. The result of the tracking process is also output to information superimposition unit 1721.

[0334] In S2907, the system control unit 1740 disables the DL tracking unit 2802 and the non-DL tracking unit 2803. In S2908, the system control unit 1740 instructs the information superimposing unit 1721 to superimpose a tracking frame based on the tracking result on the display image of the current frame. The information superimposing unit 1721 generates an image of the tracking frame having a position and size based on the tracking result, and generates composite image data by superimposing the image on the display image data of the current frame. The information superimposing unit 1721 outputs the composite image data to the display unit 1731.

[0335] This completes the operation for one frame in the live view display. In this embodiment, in an imaging device using a first tracking unit and a second tracking unit that has higher processing accuracy than the first tracking unit but requires a larger amount of calculation, the first and / or second tracking units are enabled or disabled depending on the AF mode. Therefore, by disabling a tracking unit that is unlikely to obtain a good tracking result, or by disabling the other tracking unit when there is a high possibility that a good tracking result can be obtained by only one processing unit, it is possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0336] <Tenth embodiment> Next, a tenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit 2802 and the non-DL tracking unit 2803 are enabled or disabled depending on the frame rate of the moving image to which the tracking process is applied. Since this embodiment can be implemented by the same imaging device 1700' as the ninth embodiment, it will be described using the components shown in FIG.

[0337] The higher the frame rate, the smaller the fluctuation in the position and size of the subject area between frames. Therefore, when the frame rate is high, the accuracy of the tracking process by the non-DL tracking unit 2803 is considered to be sufficiently high. In this case, the DL tracking unit 2802 is unlikely to be required. However, by using the DL tracking unit 2802 with a certain degree of frequency, the accuracy of the tracking process can be more reliably maintained.

[0338] Therefore, when the frame rate of the video to which the tracking process is applied is higher than a predetermined first threshold (fps), the frequency at which the DL tracking unit 2802 is enabled is reduced compared to when the frame rate is equal to or lower than the first threshold. This makes it possible to reduce power consumption while suppressing deterioration in detection accuracy. Note that the first threshold can be determined experimentally, for example. Note that a plurality of thresholds may be provided, and the frequency at which the DL tracking unit 2802 is enabled may be controlled in three or more stages.

[0339] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 30. Here, it is assumed that it has already been determined that the frame rate of the moving image to which the tracking process is applied is higher than a predetermined first threshold value (fps). In Fig. 30, the same reference numbers as in Fig. 29 are used for steps that perform the same processes as in the ninth embodiment, and the description thereof will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0340] In S3001, the system control unit 1740 determines whether the value of the variable COUNT used as a counter for the number of frames is a predetermined value N (an integer equal to or greater than 2). If the frame rate of the video to which the tracking process is applied is equal to or less than a first threshold value (fps), and the DL tracking unit 2802 is enabled every M frames, the value of N is determined to satisfy the relationship N>M. Here, M=1 and N=10. If the system control unit 1740 determines that the value of the variable COUNT is N, it executes S3002, and if not, it executes S3003.

[0341] Since N=10, S3002 is executed once every 10 frames. In S3002, the system control unit 1740 initializes the value of a variable COUNT to 1. After that, in S2904, the system control unit 1740 enables the DL tracking unit 2802. Although power consumption can be reduced by disabling the non-DL tracking unit 2803, it may also be enabled.

[0342] Meanwhile, in S3003, the system control unit 1740 increments the value of the variable COUNT by 1. After that, in S2905, the system control unit 1740 enables the non-DL tracking unit 2803 and disables the DL tracking unit 2802. The subsequent processing is similar to that in the ninth embodiment, and therefore will not be described.

[0343] This completes the operation for one frame in the live view display. With the above operations, if the frame rate of the video to which the tracking process is applied is high, the DL tracking unit 2802 is enabled for one frame out of 10 frames and disabled for the remaining nine frames. In frames where the DL tracking unit 2802 is disabled, the non-DL tracking unit 2803 is enabled.

[0344] In this embodiment, the frequency of using the second tracking unit, which is more accurate than the first tracking unit but requires more calculation, is controlled according to the frame rate of the video to which the tracking process is applied. Specifically, when the frame rate of the video to which the tracking process is applied is high, the frequency of enabling the second tracking unit is lower than when the frame rate is not high. When the subject moves little, the first tracking unit can perform tracking with high accuracy, so by suppressing the frequency of using the second tracking unit, it is possible to suppress the deterioration of detection accuracy and suppress power consumption. <Eleventh embodiment> Next, an eleventh embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on the shutter speed of the imaging device. Since this embodiment can be implemented by an imaging device 1700' similar to the ninth embodiment, it will be described using the components shown in FIG.

[0345] When the shutter speed is fast during video shooting, the fluctuation in the position and size of the subject area between frames is small, just as when the frame rate is high. Therefore, there is a high possibility that sufficient accuracy can be obtained by the tracking process of the non-DL tracking unit 2803. Therefore, when the shutter speed is fast, the DL tracking unit 2802 is disabled.

[0346] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 31. In Fig. 31, the steps that perform the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 19, and the description will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0347] In S3101, the system control unit 1740 determines whether or not the currently set shutter speed is faster than a threshold value. The threshold value used here can be determined, for example, experimentally depending on the performance of the non-DL tracking unit 2803 and the like.

[0348] If it is not determined that the shutter speed is faster than the threshold, in S2904 the system control unit 1740 enables the DL tracking unit 2802. Although power consumption can be reduced by disabling the non-DL tracking unit 2803, it may also be enabled. On the other hand, if it is not determined that the shutter speed is faster than the threshold value, in S2905 the system control unit 1740 enables the non-DL tracking unit 2803 and disables the DL tracking unit 2802. The subsequent processing is similar to that in the third embodiment, and therefore description thereof will be omitted.

[0349] In this embodiment, the first tracking unit and the second tracking unit, which has higher processing accuracy than the first tracking unit but requires a larger amount of calculation, are used depending on the shutter speed of the imaging device. Specifically, the second tracking unit is used when the shutter speed is not fast (slow), and is not used when the shutter speed is fast. When the shutter speed is fast, the change in position and size of the subject area between frames is small, and the first tracking unit can perform tracking processing with high accuracy, so the second tracking unit is disabled. This makes it possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0350] <Twelfth embodiment> Next, a twelfth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled in response to a user instruction. Since this embodiment can be implemented by an imaging device 1700' similar to the ninth embodiment, the components shown in FIG. 28 will be used for the description.

[0351] In this embodiment, the DL tracking unit 2802 is enabled only when the user enables the DL tracking process. Any user instruction may be used to enable the DL tracking process. The user does not need to consciously instruct the DL tracking process, and any instruction that results in the DL tracking process being enabled may be used.

[0352] As an example, it is assumed here that the operation unit 1752 has an input device (DL tracking button) assigned with a function of switching between enable and disable of DL tracking processing. FIG. 39 is a diagram showing an example of the rear appearance of the imaging device 1700′. Here, it is assumed that the display unit 1731 is provided on the rear of the imaging device 1700′. Some input devices 3901 to 3903 constituting the operation unit 1752 are provided on the rear of the imaging device 1700′. The input device 3903 capable of direction input and decision input functions as a direction key and a decision button. The input device 3901 or 3902 can be used as the DL tracking button.

[0353] System control unit 1740 switches DL tracking between enabled and disabled each time the DL tracking button is pressed. Alternatively, system control unit 1740 may enable DL tracking only while the DL tracking button is pressed. In addition, the DL tracking button may be a software key using display unit 1731.

[0354] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 32. In Fig. 32, the steps that perform the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 29, and the description will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0355] In S3201, the system control unit 1740 determines whether the DL tracking button is ON. Here, it is determined that the DL tracking button is ON when pressed. If the system control unit 1740 determines that the DL tracking button is ON, it enables the DL tracking unit 2802 in S2904. Although power consumption can be reduced by disabling the non-DL tracking unit 2803, it may be enabled. On the other hand, if the system control unit 1740 does not determine that the DL tracking button is ON, it enables the non-DL tracking unit 2803 in S2905 and disables the DL tracking unit 2802. The subsequent processing is similar to that in the ninth embodiment, and therefore will not be described.

[0356] In this embodiment, among the tracking units that track the subject area of ​​the tracking target, the second subject tracking unit, which is more accurate than the first tracking unit but requires a larger amount of calculation, is enabled or disabled according to a user's instruction. Specifically, the second tracking unit is enabled only when a user instruction to enable the second tracking unit is detected. This makes it possible to reduce power consumption because the second tracking unit is disabled when the user does not need it.

[0357] <Thirteenth embodiment> Next, a thirteenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on the operation status of the device. Since this embodiment can be implemented by an imaging device 1700' similar to the ninth embodiment, the components shown in FIG. 28 will be used for the description.

[0358] Specifically, when it is determined that a pan, tilt, or zoom operation for changing the shooting range is being performed, the DL tracking unit 2802 is disabled. This is because when these operations are being performed, the user may be adjusting the main subject so that it fits within the shooting range. Therefore, even if the DL tracking unit 2802 is enabled during these operations, there is a possibility that a meaningful detection result will not be obtained. In this embodiment, the DL tracking unit 2802 may be enabled only when it is determined that the image capture device 1700' is not being panned, tilted, or zoomed.

[0359] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 33. In Fig. 33, the steps performing the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 29, and the description thereof will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0360] In S3301, the system control unit 1740 determines whether a pan, tilt, or zoom operation is being performed on the imaging device 1700'. The system control unit 1740 can detect the pan, tilt, or zoom operation by detecting, for example, a motion vector between frames of a moving image captured for live view display or a change in size of a feature area, or by the output of the motion sensor 1760. A zoom operation can also be detected by operating a zoom button on the operation unit 1752.

[0361] If it is determined that a panning, tilting, or zooming operation is being performed, the system control unit 1740 executes S2908. On the other hand, if it is not determined that a panning, tilting, or zooming operation is being performed, the system control unit 1740 executes S3302.

[0362] In S3302, the system control unit 1740 enables the DL tracking unit 2802 or the non-DL tracking unit 2803. When the DL tracking unit 2802 is enabled, the non-DL detection unit 1723 can be disabled to reduce power consumption, but may be enabled. The condition for enabling the DL tracking unit 2802 may be, for example, one or more of the conditions described in the ninth to twelfth embodiments. If the condition for enabling the DL tracking unit 2802 is not satisfied, the system control unit 1740 disables the DL tracking unit 2802 and enables the non-DL tracking unit 2803. The processing from S2906 onwards is similar to that of the ninth embodiment, and therefore will not be described.

[0363] In this embodiment, among the tracking units that track the subject area of ​​the tracking target, the second tracking unit, which is more accurate than the first tracking unit but requires more calculations, can be enabled only when no operation is performed to change the shooting range. By not enabling the second tracking unit when the shooting range is not determined, power consumption can be efficiently reduced.

[0364] <Fourteenth embodiment> Next, a fourteenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on the shooting mode of the imaging device 1700'. Since this embodiment can be implemented by the imaging device 1700' similar to the ninth embodiment, it will be described using the components shown in FIG.

[0365] The imaging device 1700' has a plurality of imaging modes including a still image imaging mode and a video imaging mode. The imaging modes may be provided according to the imaging scene or the special effects to be applied, such as a night view mode, a sports mode, a filter mode, etc. Here, an imaging mode for ultimately capturing still images is treated as a still image imaging mode, and an imaging mode for capturing videos is treated as a video imaging mode.

[0366] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 34. In Fig. 34, the steps performing the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 29, and the description thereof will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0367] In S3401, the system control unit 1740 determines whether the shooting mode of the imaging device 1700' is a still image shooting mode, for example, based on the state of the mode changeover switch 1751. If the system control unit 1740 determines that the shooting mode of the imaging device 1700' is a still image shooting mode, it executes S3402. On the other hand, if the system control unit 1740 does not determine that the shooting mode of the imaging device 1700' is a still image shooting mode, it executes S3403.

[0368] In S3402, system control unit 1740 sets the count setting value to a first value (N1), and executes S3405. N1 is an integer equal to or greater than 1. In S3403, system control unit 1740 sets the count setting value to a second value (N2), and executes S3405. N2 is an integer of 1 or more.

[0369] In S3405, the system control unit 1740 determines whether the value of the variable COUNT used as a counter for the number of frames is the count setting value (N1 or N2). If it is determined that the value of the variable COUNT is the count setting value, the system control unit 1740 executes S3406, and if not, it executes S3407.

[0370] In S3406, the system control unit 1740 initializes the value of the variable COUNT to 1. After that, in S2904, the system control unit 1740 enables the DL tracking unit 2802. Although power consumption can be reduced by disabling the non-DL tracking unit 2803, it may also be enabled.

[0371] On the other hand, in S3407, system control unit 1740 increments the value of variable COUNT by 1. Thereafter, system control unit 1740 enables non-DL tracking unit 2803 and disables DL tracking unit 2802 in S2905.

[0372] S2904 is executed each time the variable COUNT becomes N1 or N2. That is, the DL tracker 2802 is enabled once every N1 frames in still image shooting mode and once every N2 frames in video shooting mode, and the DL tracker 2802 is disabled for the remaining frames.

[0373] The first value and the second value control the frequency at which the DL tracking unit 2802 is enabled according to the shooting mode. The first value and the second value are stored in advance in, for example, the non-volatile memory 1753. Note that the first value and the second value can be determined in consideration of the shutter speed, the frame rate, and the like.

[0374] For example, the frame rate of the video to which the tracking process is applied is assumed to be higher in video shooting mode than in still image shooting mode. In this case, by lowering the frequency at which the DL tracking unit 2802 is enabled in video shooting mode compared to still image shooting mode, it is possible to suppress the deterioration of tracking accuracy while suppressing power consumption. Note that this is just one example, and by independently controlling the frequency at which the DL tracking unit 2802 is enabled for each shooting mode, it is possible to enable the DL tracking unit 2802 at a frequency suitable for the shooting mode.

[0375] In this embodiment, the frequency of activating the second tracking unit, which is more accurate but requires more calculations than the first tracking unit, among the tracking units that track the subject area of ​​the tracking target, is controlled according to the shooting mode. By determining an appropriate count setting value according to the shooting mode, it is possible to reduce power consumption while suppressing deterioration of tracking performance.

[0376] <Fifteenth embodiment> Next, a fifteenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on the state of the release button being pressed. Since this embodiment can be implemented by an imaging device 1700' similar to the ninth embodiment, the embodiment will be described using the components shown in FIG.

[0377] The release button 1750 has a switch SW1 that turns ON when pressed halfway and a switch SW2 that turns ON when pressed all the way, and has three states: OFF for both switches, ON for only switch SW1, and ON for switches SW1 and SW2. The system control unit 1740 recognizes the ON of SW1 as an instruction to prepare for shooting a still image, and the ON of SW2 as an instruction to start shooting a still image.

[0378] In this embodiment, unless the release button 1750 is pressed halfway or fully, the image capture standby state is entered, and power consumption is reduced by disabling the DL tracking unit 2802. In this manner, in this embodiment, the DL tracking unit 2802 can be enabled only when the image capture device 1700' has received an instruction to prepare for image capture or an instruction to start image capture.

[0379] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 35. In Fig. 35, the steps performing the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 29, and the description thereof will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0380] In S3501, the system control unit 1740 determines whether the release button 1750 is half-pressed or fully pressed. For example, if the switch SW1 is ON, the system control unit 1740 can determine that the release button 1750 is half-pressed or fully pressed. When the switch SW2 is ON, the switch SW1 is ON, so there is no need to detect the state of the switch SW2.

[0381] If it is determined that the release button 1750 is half-pressed or fully pressed, the system control unit 1740 executes S3502. On the other hand, if it is not determined that the release button 1750 is half-pressed or fully pressed, the system control unit 1740 executes S2908. In this case, since the camera is in a standby state for shooting, tracking processing is not performed on the frame to be processed (this may be performed by the non-DL tracking unit 2803).

[0382] In S3502, the system control unit 1740 enables the DL tracking unit 2802 or the non-DL tracking unit 2803. When the DL tracking unit 2802 is enabled, the non-DL tracking unit 2803 can be disabled to reduce power consumption, but may be enabled. The condition for enabling the DL tracking unit 2802 may be one or more of the conditions described in the ninth to fourteenth embodiments. If the condition for enabling the DL tracking unit 2802 is not met, the system control unit 1740 disables the DL tracking unit 2802 and enables the non-DL tracking unit 2803. The condition for enabling the DL tracking unit 2802 may be different depending on whether the release button is half-pressed or fully pressed. The process from S2906 onwards is the same as in the ninth embodiment, and therefore will not be described.

[0383] In this embodiment, among the tracking units that track the subject area of ​​the tracking target, the second tracking unit, which is more accurate than the first tracking unit but requires a larger amount of calculation, can be enabled only when a shooting preparation instruction or a shooting start instruction is input. This makes it possible to efficiently reduce power consumption by not enabling the second tracking unit in a shooting standby state in which a shooting preparation instruction and a shooting start instruction are not input. Furthermore, if the first tracking unit is not enabled in a shooting standby state, power consumption can be further reduced.

[0384] <Sixteenth embodiment> Next, a sixteenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on the tracking mode of the imaging device 1700'. Since this embodiment can be implemented by the imaging device 1700' similar to the ninth embodiment, it will be described using the components shown in FIG.

[0385] The tracking mode is a mode in which focus control is continuously performed so that the main subject is focused even if the main subject moves from the center of the shooting range, and is suitable for shooting a subject that is constantly moving within the shooting range or a subject whose movement is difficult to predict, such as a small animal. The tracking mode may be set by the mode changeover switch 1751, or may be set by operating one of the input devices included in the operation unit 1752. Note that the subject to be focused in the tracking mode may be a subject corresponding to the main subject area determined by the object determination unit 2801. Note that the tracking mode may be set explicitly or indirectly in conjunction with the setting of a specific function.

[0386] When the tracking mode is set, it is considered that the subject to be focused on is likely to move, so the DL tracking unit 2802 is enabled. On the other hand, when the tracking mode is not set, it is considered that the subject to be focused on is unlikely to move, so the DL tracking unit 2802 is disabled and the non-DL tracking unit 2803 is enabled.

[0387] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 36. In Fig. 36, the steps performing the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 29, and the description thereof will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0388] In S3601, the system control unit 1740 determines whether or not the tracking mode is set in the imaging device 1700′. The system control unit 1740 can determine whether or not the tracking mode is set from the state of the mode changeover switch 1751, for example.

[0389] System control unit 1740 executes S2904 if it is determined that the tracking mode is set, and executes S2905 if it is not determined that the tracking mode is set. In S2904, the system control unit 1740 enables the DL tracking unit 2802 and disables the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 executes tracking processing for the current frame.

[0390] Also, in S2905, the system control unit 1740 enables the non-DL tracking unit 2803 and disables the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 executes tracking processing for the current frame. The processing from S2906 onwards is similar to that in the ninth embodiment, and therefore the description will be omitted.

[0391] In this embodiment, among the tracking units that track the subject area of ​​the tracking target, the second tracking unit, which has higher accuracy but requires more calculations than the first tracking unit, is enabled when the tracking mode is set, and disabled when the tracking mode is not set. By disabling the second tracking unit when it is considered that the second tracking unit is less necessary, it is possible to suppress the decrease in tracking accuracy and reduce power consumption.

[0392] <17th embodiment> Next, a seventeenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on whether the main subject to be tracked is an easy-to-track subject or not. Since this embodiment can be implemented by an imaging device 1700' similar to the ninth embodiment, it will be described using the components shown in FIG.

[0393] The detection unit 1717 of this embodiment associates an object class indicating the type of subject and its reliability with the detected candidate area, similar to the detection unit 110 of the first embodiment. If the main subject area to be tracked is an easy-to-track subject, there is a high possibility that sufficient accuracy can be obtained with the non-DL tracking unit 2803, and therefore the DL tracking unit 2802 is disabled.

[0394] Hereinafter, the operation of the system control unit 1740 in this embodiment will be described with reference to the flowchart in Fig. 37. In Fig. 37, the same reference numbers as in Fig. 29 are used for steps that perform the same processes as in the ninth embodiment, and descriptions thereof will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0395] In S3701, the system control unit 1740 determines whether the type of the subject related to the main subject region is an easy subject to track, based on the object class associated with the main subject region determined by the object determination unit 2801. The system control unit 1740 can determine whether the type of the subject related to the main subject region is an easy subject to track, for example, by referring to the object class and the difficulty of tracking stored in advance in the non-volatile memory 1753. The difficulty of tracking here is the difficulty of non-DL tracking.

[0396] For example, non-DL tracking is difficult for subjects such as birds and insects whose shape and / or color change significantly depending on the angle and state (for example, whether or not they are flying). Note that subjects that are difficult to track in a non-DL manner are not limited to subjects whose shape and / or color change significantly, as they also depend on the non-DL tracking algorithm. Note that the object classes and the tracking difficulty levels can be associated in advance according to the object classes that the detection unit 1717 can detect and the performance of the non-DL tracking unit.

[0397] If the system control unit 1740 determines that the type of the subject related to the main subject region is an easy-to-track subject, it executes S2905, and if not, it executes S2904. In S2904, the system control unit 1740 enables the DL tracking unit 2802 and disables the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 executes tracking processing for the current frame.

[0398] Also, in S2905, the system control unit 1740 enables the non-DL tracking unit 2803 and disables the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 executes tracking processing for the current frame. The processing from S2906 onwards is similar to that in the ninth embodiment, and therefore the description will be omitted.

[0399] In this embodiment, among the tracking units that track the subject area of ​​the tracking target, the second tracking unit, which has higher accuracy but requires more calculations than the first tracking unit, is disabled if the subject related to the subject area of ​​the tracking target is one that can be easily tracked by the first tracking unit. By disabling the second tracking unit when it is considered that the second tracking unit is less necessary, it is possible to suppress the decrease in tracking accuracy and reduce power consumption.

[0400] <Eighteenth embodiment> Next, an 18th embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are enabled or disabled depending on whether the user has lost sight of the subject. Since this embodiment can be implemented by the same imaging device 1700' as the 9th embodiment, the components shown in FIG. 28 will be used for the description. However, the imaging device 1700' is provided with the line-of-sight information acquisition unit 1800 shown in FIG. 18 as an input device included in the operation unit 1752, similar to the imaging device 1700 according to the third embodiment. The system control unit 1740 also has a functional block as shown in FIG. 19.

[0401] The operation of the system control unit 1740 in this embodiment will be described below with reference to the flowchart in Fig. 38. In Fig. 38, the steps that perform the same processes as in the ninth embodiment are given the same reference numbers as in Fig. 29, and the steps that perform the same processes as in the third embodiment are given the same reference numbers as in Fig. 20, and their descriptions will be omitted. The processes from S2900 to S2902 are as described in the ninth embodiment.

[0402] In S2002, the system control unit 1740 executes the subject loss determination process described with reference to FIGS. In S2003, the system control unit 1740 (irregularity determination unit 1901) determines whether or not the user has lost sight of the subject based on the degree of irregularity calculated in the loss-of-sight determination process.

[0403] If the irregularity determination unit 1901 determines in S2003 that the subject has been lost, the processing determination unit 1902 executes S2904. If the irregularity determination unit 1901 does not determine in S2003 that the subject has been lost, the processing determination unit 1902 executes S2905.

[0404] In S2904, the system control unit 1740 (processing determination unit 1902) enables the DL tracking unit 2802 and disables the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 executes tracking processing for the current frame. Also, in S2905, the system control unit 1740 (processing determination unit 1902) enables the non-DL tracking unit 2803 and disables the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 executes tracking processing for the current frame. The processing from S2906 onwards is similar to that in the ninth embodiment, and therefore the description will be omitted.

[0405] In this embodiment, among the tracking units that track the subject area of ​​the tracking target, a first tracking unit and a second tracking unit that is more accurate than the first tracking unit but requires a larger amount of calculation are used depending on whether the user has lost sight of the subject. Specifically, the second tracking unit is used when the user has lost sight of the subject, and is not used when the user has not lost sight of the subject. When the user has not lost sight of the subject, the first tracking unit can perform tracking processing with sufficient accuracy, so by disabling the second tracking unit, it is possible to suppress the deterioration of tracking accuracy while suppressing power consumption.

[0406] (Other embodiments) The above-described embodiments may be combined and implemented. For example, two or more of the detection unit, the target determination unit, and the tracking unit may have two components with different calculation amounts and accuracy that realize the same function, and their enable / disable may be dynamically controlled according to the embodiment.

[0407] In the above-described embodiment, the frame periods during which DL tracking and non-DL tracking are enabled may be made different, and both DL tracking and non-DL tracking may be enabled in a period that is the least common multiple of the respective periods.

[0408] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0409] 102: control unit, 110: detection unit, 111: object determination unit, 112: difficulty determination unit, 113: tracking control unit, 115: tracking unit, 116: DL tracking unit, 117: non-DL tracking unit, 120: information superimposition unit

Claims

1. A detection means for detecting a predetermined subject area as a candidate area from a first frame to be processed; A determining means for determining a subject area to be tracked from the candidate area; A calculation means for calculating an evaluation value relating to a degree of difficulty of tracking the subject area determined by the determination means; a tracking unit that performs a tracking process to estimate a position of an area in the first frame that corresponds to a subject area of ​​a tracking target in a second frame captured before the first frame, the tracking unit having a first tracking unit and a second tracking unit; a control means for determining operations of the first tracking means and the second tracking means for a third frame captured after the first frame based on the evaluation value; The second tracking means has a higher processing accuracy but a larger calculation load than the first tracking means, the control means enables both the first tracking means and the second tracking means, or disables at least one of the first tracking means and the second tracking means, based on the evaluation value; When both the first tracking means and the second tracking means are enabled, the control means reduces the operation frequency of the second tracking means more than that of the first tracking means.

13. An image processing device comprising:

2. The image processing device according to claim 1 , wherein the calculation means calculates the evaluation value based on one or more of a size of the subject region, a position of the subject region, a type of the subject, and a number of subject regions relating to the same type of subject.

3. The larger the evaluation value, the higher the difficulty of the tracking process.

3. The image processing apparatus according to claim 1, wherein the control means disables the first tracking means and enables the second tracking means when the evaluation value is equal to or greater than a first threshold value.

4. The larger the evaluation value, the higher the difficulty of the tracking process.

4. The image processing device according to claim 1, wherein the control means disables the second tracking means and enables the first tracking means when the evaluation value is less than a second threshold value.

5. The image processing device according to any one of claims 1 to 4, characterized in that when the subject area determined by the determination means and the subject area whose position is estimated by the tracking means relate to the same subject, the control means does not change the operation of the first tracking means and the second tracking means for the third frame from the operation of the first tracking means and the second tracking means for the first frame.

6. The image processing device according to any one of claims 1 to 5, characterized in that when the subject area determined by the determination means and the subject area whose position is estimated by the tracking means relate to different subjects, the control means determines the operations of the first tracking means and the second tracking means for the third frame based on the evaluation value.

7. 7. The image processing apparatus according to claim 1, wherein the second tracking means executes the tracking process by using a neural network.

8. A detection means for detecting a predetermined subject area as a candidate area from a first frame to be processed; A determining means for determining a subject area to be tracked from the candidate area; a tracking unit that performs a tracking process to estimate a position of an area in the first frame that corresponds to a subject area of ​​a tracking target in a second frame captured before the first frame, the tracking unit having a first tracking unit and a second tracking unit; a control means for determining operations of the first tracking means and the second tracking means for a third frame captured after the first frame based on the type of the subject detected by the detection means, The second tracking means has a higher processing accuracy but a larger calculation load than the first tracking means, the control means enables both the first tracking means and the second tracking means, or disables at least one of the first tracking means and the second tracking means, based on the type of the subject detected by the detection means.

13. An image processing device comprising:

9. A control method for an image processing device, comprising: the image processing device has a tracking means for performing a tracking process to estimate a position of an area in a first frame to be processed that corresponds to a subject area of ​​a tracking target in a second frame captured prior to the first frame; The tracking means includes a first tracking means and a second tracking means having a higher processing accuracy but a larger calculation load than the first tracking means, The control method comprises: A detection step of detecting a predetermined subject region as a candidate region from a first frame; a determining step of determining a subject region to be tracked from the candidate region; a calculation step of calculating an evaluation value relating to a degree of difficulty of tracking the subject region determined in the determination step; a control step of controlling operations of the first tracking means and the second tracking means for a third frame captured after the first frame based on the evaluation value, The control step includes enabling both the first tracking means and the second tracking means, or disabling at least one of the first tracking means and the second tracking means, based on the evaluation value; When both the first tracking means and the second tracking means are enabled, the control step reduces an operation frequency of the second tracking means compared to the first tracking means.

23. A method for controlling an image processing apparatus comprising the steps of:

10. A control method for an image processing device, comprising: the image processing device has a tracking means for performing a tracking process to estimate a position of an area in a first frame to be processed that corresponds to a subject area of ​​a tracking target in a second frame captured prior to the first frame; The tracking means includes a first tracking means and a second tracking means having a higher processing accuracy but a larger calculation load than the first tracking means, The control method comprises: A detection step of detecting a predetermined subject region as a candidate region from a first frame; a determining step of determining a subject region to be tracked from the candidate region; a control step of controlling operations of the first tracking means and the second tracking means for a third frame captured after the first frame, based on the type of the subject detected in the detection step; the control step enables both the first tracking means and the second tracking means, or disables at least one of the first tracking means and the second tracking means, based on the type of the subject detected in the detection step.

23. A method for controlling an image processing apparatus comprising the steps of:

11. A program for causing a computer included in an image processing apparatus to function as each of the means included in the image processing apparatus according to any one of claims 1 to 8.

12. A detection means for detecting a predetermined subject area as a candidate area from a first frame to be processed; a calculation means for calculating an evaluation value relating to a degree of difficulty of a determination process for determining an area to be a target of a tracking process from the candidate area based on a detection result of the candidate area; a target determining means having a first determining means and a second determining means, the target determining means performing the determining process on the candidate area detected by the detecting means; a control means for controlling the operation of the first determination means and the second determination means based on the evaluation value; The second determination means has a higher processing accuracy but a larger calculation load than the first determination means, the control means enables both the first determination means and the second determination means, disables one of the first determination means and the second determination means, or disables both of the first determination means and the second determination means, based on the evaluation value; The larger the evaluation value, the higher the difficulty of the determination process; the control means disables both the first determination means and the second determination means when the evaluation value is equal to or less than a first threshold value.

13. An image processing device comprising:

13. 13 . The image processing apparatus according to claim 12 , wherein the calculation means calculates the evaluation value based on at least one of the size, position, and type of the candidate region, and the number of candidate regions relating to the same type of subject.

14. The image processing device described in claim 12 or 13, characterized in that the control means enables the second determination means if the evaluation value is greater than a second threshold value that is greater than the first threshold value.

15. The image processing device described in Claim 14, characterized in that the control means disables the second determination means and enables the first determination means if the evaluation value is less than the second threshold and greater than the first threshold.

16. A detection means for detecting a predetermined subject area as a candidate area from a first frame to be processed; A target determination means having a first determination means and a second determination means having higher accuracy but a larger calculation load than the first determination means, and performing a determination process to determine a target area for tracking processing from the candidate area detected by the detection means; a control means for controlling operations of the first determination means and the second determination means based on a result of the detection of the candidate area; the control means enables both the first determination means and the second determination means, disables one of the first determination means and the second determination means, or disables both of the first determination means and the second determination means, based on the type and number of the detected candidate regions.

13. An image processing device comprising:

17. 17. The image processing device according to claim 16, wherein the control means disables the second determination means and enables the first determination means if the number of candidate regions of a predetermined type among the detected candidate regions is less than a first threshold value.

18. The image processing device according to claim 16 or 17, characterized in that the control means enables the second determination means if, among the detected candidate areas, the number of first type candidate areas is equal to or greater than a first threshold and the number of second type candidate areas is equal to or greater than a second threshold.

19. 19. The image processing device according to claim 16, wherein the control means disables both the first determination means and the second determination means if the number of candidate regions for any type of the detected candidate regions is less than a third threshold value.

20. a gaze detection unit for detecting a gaze of a user of the image processing device; The first determination means executes the determination process based on a line of sight detected by the line of sight detection means, 20. The image processing apparatus according to claim 12, wherein the control means disables the first determination means when the second determination means is enabled.

21. 21. The image processing apparatus according to claim 12, wherein the second determination means executes the determination process by using a trained neural network.

22. A control method for an image processing device, comprising: A detection step in which a detection means detects a predetermined subject area as a candidate area from a first frame to be processed; a calculation step of calculating an evaluation value relating to a degree of difficulty of a determination process for determining an area to be a target of a tracking process from the candidate area based on a detection result of the candidate area by a calculation means; a control step of controlling an operation of a target determining means for performing the determination process, the target determining means being included in the image processing device, based on the evaluation value; The object determination means includes a first determination means and a second determination means having higher accuracy but a larger calculation load than the first determination means, In the control step, both of the first determination means and the second determination means are enabled, one of the first determination means and the second determination means is disabled, or both of the first determination means and the second determination means are disabled based on the evaluation value. The larger the evaluation value, the higher the difficulty of the determination process; In the control step, when the evaluation value is equal to or less than a first threshold value, both of the first determination means and the second determination means are disabled.

23. A method for controlling an image processing apparatus comprising the steps of:

23. A control method for an image processing device, comprising: A detection step in which a detection means detects a predetermined subject area as a candidate area from a first frame to be processed; A control step in which a control unit controls an operation of a target determination unit that performs a determination process of determining a target area to be tracked from the candidate area, based on a detection result of the candidate area; The object determination means includes a first determination means and a second determination means having higher accuracy but a larger calculation load than the first determination means, the control step includes enabling both the first determination means and the second determination means, disabling one of the first determination means and the second determination means, or disabling both of the first determination means and the second determination means, based on the type and number of the detected candidate regions.

23. A method for controlling an image processing apparatus comprising the steps of:

24. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 12 to 21.

Images