Image processing device and control method thereof

Abstract

Disclosed are an image processing device and a control method thereof that achieves good performance while reducing power consumption. The image processing device includes a first tracking component and a second tracking component, the second tracking component having higher processing accuracy and a greater computational load than the first tracking component. The image processing device calculates an evaluation value related to the difficulty of tracking a subject region, which is determined as a tracking target in a target frame to which tracking processing is applied. Based on the evaluation value, the image processing device activates both the first tracking component and the second tracking component, or deactivates at least one of the first tracking component and the second tracking component, for frames subsequent to the target frame.

Description

Technical Field

[0001] The present invention relates to an image processing device and a control method thereof, and in particular to a subject tracking technology. Background Art

[0002] Some image processing devices such as digital cameras have a function (subject tracking function) of tracking a feature region such as a face region by detecting the feature region over time. Devices that track subjects using a pre-trained neural network are also known (Patent Document 1).

[0003] Prior art literature

[0004] Patent Literature

[0005] Patent Document 1: Japanese Patent Application Laid-Open No. 2017-156886 Summary of the Invention

[0006] Problems to be solved by the invention

[0007] Using neural networks can improve subject tracking accuracy compared to using correlations or similarities between image regions. However, processing using neural networks involves a high computational load and requires a high-speed processor and large-scale circuitry, resulting in high power consumption. For example, if subject tracking using neural networks is applied to moving images for live view display, battery drain caused by the live view display can become a problem.

[0008] The present invention has been conceived in view of these problems of the conventional technology. An object of the present invention is to provide an image processing apparatus having a subject tracking function that achieves good performance while suppressing power consumption, and a control method thereof.

[0009] Solutions for solving problems

[0010] According to one aspect of the present invention, an image processing device is provided. The image processing device is characterized by comprising: a detection unit for detecting one or more regions of a predetermined subject in a target frame as candidate regions; a determination unit for determining a target region to be tracked from the candidate regions; a calculation unit for calculating an evaluation value related to tracking difficulty based on the target region determined by the determination unit; a tracking unit for performing tracking processing for estimating a position in the target frame of a region corresponding to the target region to be tracked in a frame preceding the target frame, the tracking unit comprising a first tracking unit and a second tracking unit; and a control unit for determining, based on the evaluation value, an operation of the first tracking unit on a subsequent frame of the target frame and an operation of the second tracking unit on a subsequent frame of the target frame, wherein the second tracking unit has higher processing accuracy and a larger computational load than the first tracking unit, and the control unit for enabling both the first tracking unit and the second tracking unit or disabling at least one of the first tracking unit and the second tracking unit based on the evaluation value.

[0011] According to another aspect of the present invention, an image processing device is provided. The image processing device is characterized in that it includes: a detection unit for detecting one or more regions of a predetermined subject in a target frame as candidate regions; a determination unit for determining a target region to be tracked from the candidate regions; a tracking unit for performing tracking processing for estimating a position in the target frame of a region corresponding to a target region to be tracked in a frame preceding the target frame, the tracking unit including a first tracking unit and a second tracking unit; and a control unit for determining, based on the type of the subject detected by the detection unit, an operation of the first tracking unit on a subsequent frame of the target frame and an operation of the second tracking unit on a subsequent frame of the target frame, wherein the second tracking unit has higher processing accuracy and a larger computational load than the first tracking unit, and the control unit enables both the first tracking unit and the second tracking unit, or disables at least one of the first tracking unit and the second tracking unit, based on the type of the subject detected by the detection unit.

[0012] According to another aspect of the present invention, there is provided an image processing device. The image processing device is characterized in that it includes: a detection unit for detecting one or more regions of a predetermined subject in an object frame as candidate regions; a calculation unit for calculating an evaluation value related to the difficulty of a determination process for determining a region to be subjected to tracking processing from the candidate regions based on a detection result of the candidate regions; an object determination unit including a first determination unit and a second determination unit, the object determination unit performing the determination process on the candidate regions detected by the detection unit; and a control unit for controlling the operation of the first determination unit and the operation of the second determination unit based on the evaluation value, wherein the second determination unit has higher processing accuracy and a larger computational load than the first determination unit, and the control unit enables both the first determination unit and the second determination unit, disables one of the first determination unit and the second determination unit, or disables both the first determination unit and the second determination unit based on the evaluation value.

[0013] According to another aspect of the present invention, an image processing device is provided. The image processing device is characterized in that it includes: a detection unit for detecting one or more regions of a predetermined subject in an object frame as candidate regions; an object determination unit for performing determination processing, the determination processing being used to determine a region to be subjected to tracking processing from the candidate regions detected by the detection unit, the object determination unit including a first determination unit and a second determination unit, the second determination unit having higher processing accuracy and a larger computational load than the first determination unit; and a control unit for controlling the operation of the first determination unit and the operation of the second determination unit based on the detection result of the candidate regions, wherein the control unit enables both the first determination unit and the second determination unit, or disables one of the first determination unit and the second determination unit, or disables both the first determination unit and the second determination unit, based on the type and number of the detected candidate regions.

[0014] According to still another aspect of the present invention, there is provided a non-transitory machine-readable medium storing a program for causing a computer to function as various components of the electronic device according to the present invention.

[0015] Effects of the Invention

[0016] According to the present invention, it is possible to provide an image processing apparatus having a subject tracking function that achieves good performance while suppressing power consumption, and a control method thereof.

[0017] Other features and advantages of the present invention will become apparent from the following description taken in conjunction with the accompanying drawings. Note that in the accompanying drawings, the same reference numerals represent the same or similar components. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments of the invention and, together with the description, serve to explain the principles of the invention.

[0019] Figure 1 is a block diagram illustrating an example of a functional configuration of an image pickup apparatus according to the first embodiment.

[0020] Figure 2A : is a diagram showing an example of display of subject tracking results in the image pickup apparatus according to the first embodiment.

[0021] Figure 2B : is a diagram showing an example of display of subject tracking results in the image pickup apparatus according to the first embodiment.

[0022] Figure 3 This is a flowchart related to the difficulty score calculation process in the first embodiment.

[0023] Figure 4 This is a flowchart related to the control operation based on the difficulty score in the first embodiment.

[0024] Figure 5 is an example timing chart related to the processing of the image pickup apparatus according to the first embodiment.

[0025] Figure 6 This is a sequence diagram related to the use of the tracking processing results in the first embodiment.

[0026] Figure 7 is a flowchart related to the basic live view display operation in the first embodiment.

[0027] Figure 8A This is a flowchart related to the live view display operation during subject tracking in the first embodiment.

[0028] Figure 8B This is a flowchart related to the live view display operation during subject tracking in the first embodiment.

[0029] Figure 9 is a block diagram illustrating an example of a functional configuration of an image pickup apparatus according to the second embodiment.

[0030] Figure 10 is a diagram showing an example of live view display during subject tracking in the second embodiment.

[0031] Figure 11AThis is a flowchart related to the difficulty score calculation process in the second embodiment.

[0032] Figure 11B This is a flowchart related to the difficulty score calculation process in the second embodiment.

[0033] Figure 12 This is a flowchart related to the control operation based on the difficulty score in the second embodiment.

[0034] Figure 13 This is a flowchart related to control operations that do not use difficulty scores in the second embodiment.

[0035] Figure 14 is an example timing chart related to the processing of the image pickup apparatus according to the second embodiment.

[0036] Figure 15A This is a flowchart related to the method for determining a subject region to be tracked in the second embodiment.

[0037] Figure 15B This is a flowchart related to the method for determining a subject region to be tracked in the second embodiment.

[0038] Figure 16 This is a flowchart related to the live view display operation during subject tracking in the second embodiment.

[0039] Figure 17 is a block diagram illustrating an example of the functional configuration of an image pickup apparatus according to the third to eighth embodiments.

[0040] Figure 18 is a diagram showing a configuration of a visual line information acquisition device according to an embodiment.

[0041] Figure 19 is a diagram showing functional blocks related to activation / deactivation control of a subject detection unit in the third embodiment.

[0042] Figure 20 is a flowchart related to the operation of the system control unit according to the third embodiment.

[0043] Figure 21 is a flowchart related to the operation of the system control unit according to the third embodiment.

[0044] Figure 22 is a flowchart related to the operation of the system control unit according to the third embodiment.

[0045] Figure 23 is a flowchart related to the operation of the system control unit according to the fourth embodiment.

[0046] Figure 24 is a flowchart related to the operation of the system control unit according to the fifth embodiment.

[0047] Figure 25 is a flowchart related to the operation of the system control unit according to the sixth embodiment.

[0048] Figure 26 is a flowchart related to the operation of the system control unit according to the seventh embodiment.

[0049] Figure 27 is a flowchart related to the operation of the system control unit according to the eighth embodiment.

[0050] Figure 28 is a block diagram illustrating an example of the functional configuration of an image pickup apparatus according to the ninth to eighteenth embodiments.

[0051] Figure 29 is a flowchart related to the operation of the system control unit according to the ninth embodiment.

[0052] Figure 30 is a flowchart related to the operation of the system control unit according to the tenth embodiment.

[0053] Figure 31 is a flowchart related to the operation of the system control unit according to the eleventh embodiment.

[0054] Figure 32 is a flowchart related to the operation of the system control unit according to the twelfth embodiment.

[0055] Figure 33 is a flowchart related to the operation of the system control unit according to the thirteenth embodiment.

[0056] Figure 34 is a flowchart related to the operation of the system control unit according to the fourteenth embodiment.

[0057] Figure 35 is a flowchart related to the operation of the system control unit according to the fifteenth embodiment.

[0058] Figure 36 is a flowchart related to the operation of the system control unit according to the sixteenth embodiment.

[0059] Figure 37 is a flowchart related to the operation of the system control unit according to the seventeenth embodiment.

[0060] Figure 38 is a flowchart related to the operation of the system control unit according to the eighteenth embodiment.

[0061] Figure 39is a diagram showing an example of the outer appearance of the rear side of the image pickup apparatus in the twelfth embodiment. DETAILED DESCRIPTION

[0062] Hereinafter, the present invention will be described in detail based on exemplary embodiments of the present invention with reference to the accompanying drawings. Note that the following embodiments do not limit the present invention as set forth in the claims. In addition, although several features are described in the embodiments, the present invention does not necessarily require all of these features, and multiple features can be combined as needed. In addition, in the accompanying drawings, the same or similar configurations are given the same reference numerals, and their redundant descriptions will be omitted.

[0063] Note that the following embodiments will describe the case where the present invention is applied to an imaging device such as a digital camera. However, an imaging device is only one example of an image processing device to which the present invention is applicable, and the present invention can be implemented in any electronic device capable of image processing. Examples of such electronic devices include computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and dashboard cameras. However, these are merely examples, and the present invention can also be applied to other electronic devices.

[0064] First embodiment

[0065] Figure 1 1 is a block diagram illustrating an example of a functional configuration of an image pickup apparatus 100 serving as an example of an image processing apparatus according to the first embodiment.

[0066] The optical system 101 has a plurality of lenses including a movable lens such as a focus lens, and forms an optical image of an imaging range on an imaging surface of the image sensor 103 .

[0067] The control unit 102 includes a CPU, and loads a program stored in the ROM 123 into the RAM 122, for example, and executes the program. The control unit 102 realizes the functions of the imaging apparatus 100 by controlling the operations of the respective functional blocks. The ROM 123 is, for example, a rewritable nonvolatile memory, and stores programs, setting values, GUI data, and the like that can be executed by the CPU of the control unit 102. The RAM 122 is a system memory that is used to load programs executed by the CPU of the control unit 102, store values ​​required when executing the programs, and the like. Although in Figure 1 Although not shown in the figure, the control unit 102 is communicably connected to the respective functional blocks.

[0068] Image sensor 103 may be a CMOS image sensor having, for example, a primary color Bayer array filter. Multiple pixels, each having a photoelectric conversion region, are arranged two-dimensionally in image sensor 103. Image sensor 103 uses these pixels to convert the optical image formed by optical system 101 into a set of electrical signals (analog image signals). An A / D converter within image sensor 103 converts the analog image signals into digital image signals (image data). The A / D converter may be provided external to image sensor 103.

[0069] The evaluation value generation unit 124 generates signals and evaluation values ​​used in autofocus detection (AF) based on image data obtained from the image sensor 103, and calculates evaluation values ​​used in automatic exposure control (AE). The evaluation value generation unit 124 outputs the generated signals and evaluation values ​​to the control unit 102. Based on the signals and evaluation values ​​obtained from the evaluation value generation unit 124, the control unit 102 controls the focus lens position of the optical system 101 and determines shooting conditions (such as exposure time, aperture value, and ISO sensitivity). The evaluation value generation unit 124 can generate signals and evaluation values ​​based on display image data generated by the post-processing unit 114, which will be described later.

[0070] The first pre-processing unit 104 applies color interpolation processing to the image data obtained from the image sensor 103. Color interpolation processing is also called "demosaicing" and is a process of assigning R, G, and B component values ​​to each instance of pixel data constituting the image data. The first pre-processing unit 104 may also apply reduction processing for reducing the number of pixels as needed. The first pre-processing unit 104 stores the image data to which the processing has been applied in the video memory 107.

[0071] The first image correction unit 109 applies correction processing, such as white balance correction processing and shading correction processing, as well as conversion processing from RGB format to YUV format, to the image data stored in the video memory 107. Note that when applying the correction processing, the first image correction unit 109 may use image data from at least one frame different from the frame to be processed, among the image data stored in the video memory 107. For example, the first image correction unit 109 may use image data from frames chronologically preceding and / or following the target frame in the correction processing. The first image correction unit 109 outputs the image data to which the processing has been applied to the post-processing unit 114.

[0072] The post-processing unit 114 generates recording image data, display image data, and the like based on the image data supplied from the first image correction unit 109. For example, the post-processing unit 114 applies encoding processing to the image data and generates a data file storing the encoded image data as the recording image data. The post-processing unit 114 supplies the recording image data to the recording unit 118.

[0073] The post-processing unit 114 also generates display image data for display in the display unit 121 based on the image data supplied from the first image correction unit 109. The display image data has a size based on the display size of the display unit 121. The post-processing unit 114 supplies the display image data to the information superimposing unit 120.

[0074] The recording unit 118 records the recorded image data converted by the post-processing unit 114 in the recording medium 119. The recording medium 119 may be, for example, a semiconductor memory card or a built-in nonvolatile memory.

[0075] The second pre-processing unit 105 applies color interpolation processing to the image data output from the image sensor 103. The second pre-processing unit 105 stores the processed image data in the tracking memory 108. The tracking memory 108 and the video memory 107 can be implemented as separate address spaces within the same memory space. The second pre-processing unit 105 can also apply a reduction process to reduce the number of pixels as needed to reduce the processing load. Although the first pre-processing unit 104 and the second pre-processing unit 105 are described here as separate functional blocks, these units can be configured using a common pre-processing unit.

[0076] The second image correction unit 106 applies correction processing such as white balance correction processing and shading correction processing, as well as conversion processing for converting from RGB format to YUV format, to the image data stored in the tracking memory 108. The second image correction unit 106 may also apply image processing suitable for subject detection processing to the image data. For example, if the representative brightness of the image data (e.g., the average brightness of all pixels) is less than a predetermined threshold, the second image correction unit 106 may multiply the entire image data by a set coefficient (gain) so that the representative brightness becomes at least the threshold.

[0077] Note that when applying the correction processing, the second image correction unit 106 may use image data from at least one frame different from the frame to be processed, among the image data stored in the tracking memory 108. For example, the second image correction unit 106 may use image data of frames chronologically preceding and / or following the target frame in the correction processing. The second image correction unit 106 stores the image data to which the processing has been applied in the tracking memory 108.

[0078] Note that when the subject tracking function is not used, functional blocks related to the subject tracking function, such as the second pre-processing unit 105 and the second image correction unit 106, do not need to be operated. Furthermore, the image data to which the subject tracking function is applied is moving image data captured for live view display or recording. The moving image data has a predetermined frame rate, such as 30 fps, 60 fps, or 120 fps.

[0079] The detection unit 110 detects at least one region (candidate region) of a predetermined candidate subject from image data of a frame. Furthermore, for each detected region, the detection unit 110 associates an object class indicating the position, size, and type of the candidate subject (such as car, airplane, bird, insect, human body, head, pupil, cat, or dog) within the frame with its confidence level. The number of detected regions is also counted for each object class.

[0080] The detection unit 110 can detect candidate regions using well-known techniques for detecting feature regions, such as the facial regions of people or animals. For example, the detection unit 110 can be configured as a class discriminator pre-trained using training data. The discrimination (classification) algorithm is not particularly limited. The detection unit 110 can be implemented by training a discriminator for implementing multi-class logistic regression, support vector machines, random forests, neural networks, and the like. The detection unit 110 stores the detection results in the tracking memory 108.

[0081] The object determination unit 111 determines the subject area to be tracked (main subject area) from the candidate areas detected by the detection unit 110. For example, the subject area to be tracked can be determined based on a priority order assigned in advance to each item included in the detection result (such as the object class and the size of the area). Specifically, the total of the priority order can be calculated for each candidate area, and the candidate area with the lowest total can be determined as the subject area to be tracked. Alternatively, the candidate area closest to the center of the image or the focus detection area or the largest candidate area among the candidate areas belonging to a specific object class can be determined as the subject area to be tracked. The object determination unit 111 stores information for identifying the determined subject area in the tracking memory 108.

[0082] The difficulty determination unit 112 calculates a difficulty score, which is an evaluation value indicating the difficulty of tracking, for the subject region to be tracked, determined by the object determination unit 111. For example, the difficulty determination unit 112 may calculate the difficulty score by taking into account at least one factor that affects the difficulty of tracking. Examples of factors that affect the difficulty of tracking include the size of the subject region, the object class (type) of the subject, the total number of regions belonging to the same object class, and the position within the image. A specific example of the method for calculating the difficulty score will be given later. The difficulty determination unit 112 outputs the calculated difficulty score to the tracking control unit 113.

[0083] The tracking control unit 113 determines whether to activate or deactivate each of the plurality of tracking units included in the tracking unit 115 based on the difficulty score calculated by the difficulty determination unit 112. In this embodiment, the tracking unit 115 includes a plurality of tracking units having different computational loads and tracking accuracies. Specifically, the tracking unit 115 includes a DL tracking unit 116 that uses deep learning (DL) for subject tracking and a non-DL tracking unit 117 that does not use DL for subject tracking. It is assumed that the DL tracking unit 116 has higher processing accuracy than the non-DL tracking unit 117, but also has a greater computational load than the non-DL tracking unit 117.

[0084] In this case, the tracking control unit 113 determines whether to activate or deactivate each of the DL tracking unit 116 and the non-DL tracking unit 117. The tracking control unit 113 also determines the operating frequency of the tracking unit to be activated. The "operating frequency" is the frequency (in fps) at which the tracking process is applied.

[0085] The tracking unit 115 estimates the subject region to be tracked based on the image data of the target frame (current frame) stored in the tracking memory 108, and obtains the position and size of the estimated subject region within the frame as a tracking result. For example, the tracking unit 115 uses the image data of the current frame and the image data of a previous frame (e.g., the previous frame) captured before the current frame to estimate the subject region to be tracked within the current frame. The tracking unit 115 outputs the tracking result to the information superimposition unit 120.

[0086] Here, tracking unit 115 estimates an area within the target frame that corresponds to the subject area to be tracked in the previous frame. In other words, the subject area to be tracked determined by target determination unit 111 for the target frame is not the subject area to be tracked in the tracking process performed on the target frame. The subject area to be tracked in the tracking process performed on the target frame is the subject area to be tracked in the previous frame. When the subject to be tracked switches to another subject, the subject area to be tracked determined by target determination unit 111 for the target frame is used for tracking processing in the next frame.

[0087] The tracking unit 115 includes a DL tracking unit 116 that uses deep learning (DL) to track an object and a non-DL tracking unit 117 that does not use DL to track an object. The tracking unit 115, activated by the tracking control unit 113, outputs tracking results at the operating frequency set by the tracking control unit 113.

[0088] The DL tracking unit 116 uses a pre-trained multi-layer neural network including convolutional layers to estimate the position and size of the subject area to be tracked. More specifically, the DL tracking unit 116 has a function for extracting feature points of the subject area for each object class that may be an object and the feature quantities contained in these feature points, as well as a function for associating the extracted feature points between frames. Therefore, the DL tracking unit 116 can estimate the position and size of the subject area to be tracked in the current frame based on the feature points in the current frame that are associated with the feature points of the subject area to be tracked in the previous frame.

[0089] The DL tracking unit 116 outputs the position, size, and confidence score of the object region to be tracked estimated for the current frame. The confidence score indicates the confidence level of the feature point association between frames (i.e., the confidence level of the estimated result of the object region to be tracked). A low confidence score indicating the confidence level of the feature point association between frames means that the object region estimated in the current frame may be a different object-related region than the object region to be tracked in the previous frame.

[0090] On the other hand, the non-DL tracking unit 117 estimates the subject region to be tracked in the current frame using a method that does not use deep learning. Here, it is assumed that the non-DL tracking unit 117 estimates the subject region to be tracked based on the similarity of color composition. However, other methods may be used, such as pattern matching using the subject region to be tracked in past frames as a template. The non-DL tracking unit 117 outputs the position, size, and confidence level score of the subject region to be tracked estimated for the current frame.

[0091] Here, we will describe the similarity of color composition. For ease of description and understanding, it is assumed that the shape and size of the subject area to be tracked are the same between the past frame and the current frame. It is also assumed that the image data has a depth of 8 bits (values ​​from 0 to 255) for each RGB color component.

[0092] The non-DL tracking unit 117 divides the range of possible values ​​(0 to 255) of a given color component (e.g., R component) into a plurality of regions. The non-DL tracking unit 117 then uses the result of classifying the pixels included in the subject region to be tracked by the region to which the R component value belongs (the frequency of each value range) as the color composition of the subject region to be tracked.

[0093] As the simplest example, assume that the range of possible values ​​of the R component (0 to 255) is divided into Red1 corresponding to 0 to 127 and Red2 corresponding to 128 to 255. Furthermore, assume that the color composition of the subject area to be tracked in the past frame is 50 pixels of Red1 and 70 pixels of Red2. Furthermore, assume that the color composition of the subject area to be tracked in the current frame is 45 pixels of Red1 and 75 pixels of Red2.

[0094] In this case, the non-DL tracking unit 117 can calculate a score indicating the similarity of color compositions (similarity score) as follows based on the difference between the numbers of pixels classified into the same value range.

[0095] Similarity score = |50-45| + |70-75| = 10

[0096] Assuming that the color composition of the subject area to be tracked in the current frame is 10 pixels of Red1 and 110 pixels of Red2, the similarity score will be:

[0097] Similarity score = |50-10| + |70-110| = 80

[0098] In this way, the similarity score increases as the similarity of the color composition decreases. Alternatively, a lower similarity score indicates a higher similarity of the color composition.

[0099] The selection unit 125 adopts the tracking result from the DL tracking unit 116 or the non-DL tracking unit 117 based on the confidence level score output by the DL tracking unit 116 and the similarity score output by the non-DL tracking unit 117. For example, if the confidence level score is not greater than a predetermined confidence level score threshold and the similarity score is not greater than a predetermined similarity score threshold, the selection unit 125 adopts the tracking result from the non-DL tracking unit 117, and otherwise adopts the tracking result from the DL tracking unit 116. The selection unit 125 outputs the adopted tracking result to the information superposition unit 120 and the control unit 102.

[0100] Here, the determination of which of the tracking results from the DL tracking unit 116 and the non-DL tracking unit 117 to adopt is made based on the confidence level score and the similarity score. However, this determination can be made by other methods. For example, the tracking result from the DL tracking unit 116 can be preferentially adopted by utilizing the fact that the accuracy of the DL tracking unit 116 tends to be higher than the accuracy of the non-DL tracking unit 117. Specifically, if a tracking result from the DL tracking unit 116 is available, the tracking result from the DL tracking unit 116 can be adopted, and if not, the tracking result from the non-DL tracking unit 117 can be adopted.

[0101] The information overlay unit 120 generates a tracking frame image based on the size of the subject area included in the tracking results output by the tracking unit 115. For example, the tracking frame image can be a frame-shaped image representing a rectangular outline that delimits the subject area. The information overlay unit 120 then generates composite image data by overlaying the tracking frame image on the display image data output by the post-processing unit 114, so that the tracking frame is displayed at the location of the subject area included in the tracking results. The information overlay unit 120 can also generate images representing the current settings or status of the imaging device 100, and overlay these images on the display image data output by the post-processing unit 114 so that these images are displayed at a predetermined location. The information overlay unit 120 outputs the composite image data to the display unit 121.

[0102] The display unit 121 may be, for example, a liquid crystal display or an organic EL display. The display unit 121 displays an image based on the synthesized image data output by the information superimposition unit 120. In this manner, a live view display of one frame is performed.

[0103] Figure 2A and Figure 2B is a diagram showing an example of a live view display. Figure 2A An image 800 represented by display image data output by the post-processing unit 114 is shown. Figure 2B An image 802 represented by the synthesized image data is shown, in which the image of the tracking frame 803 is superimposed on the display image data. Here, there is only one candidate subject 801 in the imaging range, so the candidate subject 801 is selected as the subject to be tracked. The tracking frame 803 is superimposed so as to surround the candidate subject 801. Figure 2B In the example shown in FIG, tracking frame 803 is composed of a combination of four hollow bracket shapes, but tracking frame 803 can be a combination of non-hollow bracket shapes, a complete frame, a combination of rectangular shapes, a combination of triangular shapes, or other forms. The form of tracking frame 803 can also be selected by the user.

[0104] Next, we will refer to Figure 3 The flowchart in further describes the operation performed by the difficulty determination unit 112 to calculate the difficulty score associated with the subject region to be tracked. Here, it is assumed that the object classes are classified into the following groups A to D. Groups A to C are defined so that object classes with similar characteristics regarding tracking difficulty belong to the same group. Object classes that are less important than those belonging to groups A to C are classified into group D.

[0105] Group A mainly includes object classes of rigid subjects, such as a car object class, a train object class, a jet object class, a bicycle object class, a motorcycle object class, and a helicopter object class. Group B mainly includes object classes of subjects that move quickly and whose shape of the subject area changes significantly, such as a bird object class and an insect object class. Group C (animal / human) mainly includes object classes of subjects that include faces and bodies, such as a dog object class, a cat object class, and a human object class. Group D includes other object classes, and it is assumed that the object classes belonging to Group D have lower importance. Note that the classification of Group A to Group D is merely an example, and classification with respect to tracking difficulty can also be performed from other perspectives.

[0106] In S401 , the difficulty level determination unit 112 initializes the difficulty level score to zero.

[0107] In S402 , the difficulty determination unit 112 determines whether the object class of the subject region to be tracked belongs to group A, executes S406 if determined to belong to group A, and executes S403 if not determined to belong to group A.

[0108] In S406, the difficulty determination unit 112 maintains the difficulty score. The object class belonging to group A is a class of objects that are relatively easy to track. Therefore, the difficulty determination unit 112 determines that the object region to be tracked is easy to track and does not add points to the difficulty score (does not increase the difficulty).

[0109] In S403 , the difficulty determination unit 112 determines whether the object class of the subject region to be tracked belongs to group B, executes S407 if determined to belong to group B, and executes S404 if not determined to belong to group B.

[0110] In S407, the difficulty judgment unit 112 adds 1 point (+1) to the difficulty score. The object class of a subject that moves quickly and changes shape significantly is classified as group B. Therefore, if the object class of the subject area to be tracked belongs to group B, the difficulty judgment unit 112 determines that the subject area to be tracked is difficult to track and adds points to the difficulty score (increasing the difficulty).

[0111] In S404 , the difficulty determination unit 112 determines whether the object class of the subject region to be tracked belongs to group C, executes S405 if determined to belong to group C, and executes S410 if not determined to belong to group C.

[0112] In S405, the difficulty determination unit 112 determines whether a subject region with a head object class and a subject region with a pupil object class are detected within the subject region to be tracked. If it is determined that a subject region with a head object class and a subject region with a pupil object class are detected within the subject region to be tracked, the difficulty determination unit 112 executes S408, and if not, executes S409.

[0113] In S408, the difficulty determination unit 112 maintains the difficulty score. Group C is primarily an object class of subjects with faces (such as dogs, cats, and humans). Because both the head and the pupil are detected in a hierarchical relationship, the difficulty determination unit 112 determines that the subject region to be tracked is easy to track and does not add points to the difficulty score.

[0114] In S409, the difficulty determination unit 112 adds 1 point to the difficulty score. Group C is an object class primarily consisting of subjects with faces (such as dogs, cats, and humans). Because it was determined that at least one of the head and pupils in a hierarchical relationship was not detected, the difficulty determination unit 112 determines that the subject region to be tracked is difficult to track and adds a point to the difficulty score.

[0115] In S410, the difficulty determination unit 112 maintains the difficulty score. Because the subject region to be tracked belongs to group D and is not an object class in group A, group B, or group C, the difficulty determination unit 112 determines that the subject region to be tracked is not an area related to the main subject and does not add points to the difficulty score.

[0116] In S411, the difficulty judgment unit 112 judges whether the number of candidate regions having the same object class as the object class of the subject region to be tracked is at least a predetermined threshold value N (N ≥ 2). If it is determined that the number of candidate regions having the same object class as the object class of the subject region to be tracked is at least the predetermined threshold value N, the difficulty judgment unit 112 executes S412, and if not, executes S413.

[0117] In S412, the difficulty judgment unit 112 adds 1 point to the difficulty score. If the number of candidate regions having the same object class as the subject region to be tracked is at least the threshold value N, the difficulty judgment unit 112 determines that the subject region to be tracked is difficult to track and adds a point to the difficulty score.

[0118] In S413, the difficulty judgment unit 112 maintains the difficulty score. If the number of candidate regions having the same object class as the subject region to be tracked is less than the threshold N, the difficulty judgment unit 112 determines that the subject region to be tracked is easy to track and maintains the difficulty score.

[0119] In S414, the difficulty judgment unit 112 judges whether the number of types of object classes included in the candidate area detected by the detection unit 110 is at least a predetermined threshold value O (O ≥ 2). If it is judged that the number of types of object classes included in the candidate area detected by the detection unit 110 is at least the threshold value O, the difficulty judgment unit 112 executes S415, and if not, executes S416.

[0120] In step S415, the difficulty judgment unit 112 adds 1 point to the difficulty score. If the number of types of object classes in the candidate region is at least the threshold value 0, candidate regions of various object classes are mixed within the imaging range, and thus the subject region to be tracked is determined to be difficult to track, and the difficulty score is added.

[0121] In S416, the difficulty judgment unit 112 maintains the difficulty score. If the number of types of object classes in the subject region to be tracked is less than the threshold value 0, the difficulty judgment unit 112 determines that the tracking unit 115 can easily track the subject region to be tracked and maintains the difficulty score.

[0122] In S417 , the difficulty determination unit 112 outputs the calculated difficulty score to the tracking control unit 113 .

[0123] The difficulty score calculated by the operations described here can take values ​​of 0, 1, 2, or 3. However, these values ​​can actually have finer step sizes. In fact, more object classes than those described here as examples can be classified and classified into more groups. In addition, it is not necessary to judge all factors in S402 to S404, S411, and S414, and the difficulty score can be calculated based on at least one of these factors. In addition, the difficulty score can be a binary value (0 or 1), wherein the DL tracking unit 116 is controlled to be enabled when the difficulty score is 1 (the non-DL tracking unit 117 is enabled or disabled), and the DL tracking unit 116 is disabled when the difficulty score is 0 (the non-DL tracking unit 117 is enabled).

[0124] The difficulty determination unit 112 can calculate a more accurate difficulty score by pre-grouping and expanding the range of possible difficulty scores. Additionally, by associating speed information with an object class, in addition to the object class of the subject area to be tracked, the speed of movement in the image can be estimated based on size and speed information and reflected in the difficulty score. For example, if the estimated speed of movement is at least a threshold, points are added to the difficulty score. Furthermore, the confidence level score and similarity score calculated in a past frame (e.g., the previous frame) can be included in the difficulty score calculation process.

[0125] Next, we will refer to Figure 4 The flowchart in further describes the operation of the tracking control unit 113. Here, it is assumed that the difficulty judgment unit 112 Figure 3 The difficulty score is calculated by the described operation. Therefore, the difficulty score has a value from 0 to 3.

[0126] In S501 , the tracking control unit 113 determines whether the difficulty score is 3, executes S504 if determined to be 3, and executes S502 if not.

[0127] In S502 , the tracking control unit 113 determines whether the difficulty score is 2, executes S505 if determined to be 2, and executes S503 if not.

[0128] In S503 , the tracking control unit 113 determines whether the difficulty score is 1, executes S506 if it is determined that the difficulty score is 1, and executes S507 if not.

[0129] In S504, the tracking control unit 113 determines to activate the DL tracking unit 116 and deactivate the non-DL tracking unit 117. The tracking control unit 113 also sets the operating frequency of the DL tracking unit 116 to 60 fps (frames per second). Difficulty score 3 is the state with the highest tracking difficulty among the calculated difficulty scores. Therefore, the tracking control unit 113 determines to use the DL tracking unit 116, which has better performance than the non-DL tracking unit 117. On the other hand, the tracking control unit 113 determines that it is unlikely to obtain good results even when using the non-DL tracking unit 117, and determines not to use the non-DL tracking unit 117.

[0130] In S505, the tracking control unit 113 determines to enable both the DL tracking unit 116 and the non-DL tracking unit 117. The tracking control unit 113 also sets the operating frequency of the DL tracking unit 116 to 30 fps (every other frame) and the operating frequency of the non-DL tracking unit 117 to 60 fps. Difficulty score 2 is a state in which the difficulty of tracking is slightly higher. Therefore, the tracking control unit 113 determines to use both the DL tracking unit 116 and the non-DL tracking unit 117. However, the tracking control unit 113 reduces power consumption by setting the operating frequency of the DL tracking unit 116 to be lower than the operating frequency of the non-DL tracking unit 117. The tracking control unit 113 also maintains the tracking performance of tracking by setting the operating frequency of the non-DL tracking unit 117 to every frame.

[0131] In S506, the tracking control unit 113 determines to enable both the DL tracking unit 116 and the non-DL tracking unit 117. The tracking control unit 113 also sets the operating frequency of the DL tracking unit 116 to 15fps (every three frames) and the operating frequency of the non-DL tracking unit 117 to 60fps. Difficulty score 1 is a state in which the difficulty of tracking is slightly lower. Therefore, the tracking control unit 113 determines to use both the DL tracking unit 116 and the non-DL tracking unit 117. However, the tracking control unit 113 further reduces power consumption by setting the operating frequency of the DL tracking unit 116 to be even lower than when the difficulty score is 2. On the other hand, the tracking control unit 113 also maintains the tracking performance of tracking by setting the operating frequency of the non-DL tracking unit 117 to every frame.

[0132] In S507, the tracking control unit 113 determines to deactivate the DL tracking unit 116 and activate the non-DL tracking unit 117. The tracking control unit 113 also sets the operating frequency of the non-DL tracking unit 117 to 30 fps (every other frame). S507 is executed when the difficulty score is 0, which is the state with the lowest tracking difficulty. Therefore, the tracking control unit 113 determines that sufficiently accurate results can be obtained using the non-DL tracking unit 117, and determines to use only the non-DL tracking unit 117. In addition, the tracking control unit 113 further reduces power consumption by setting the operating frequency of the non-DL tracking unit 117 to every other frame.

[0133] The control operation described here is merely an example and may vary depending on the range of possible values ​​of the difficulty score and the step size of the value. Furthermore, although the frame rate of the moving image is described as 60 fps here, the frame rate may be 120 fps. In this case, when the operating frequency of the non-DL tracking unit 117 is set to 120 fps, the operating frequency of the DL tracking unit 116 may be set to less than 15 fps (e.g., 5 fps).

[0134] Furthermore, it is not absolutely necessary to change the control for each difficulty score. For example, when the difficulty score is at least a first threshold, the DL tracking unit 116 may be enabled and the non-DL tracking unit 117 may be disabled; when the difficulty score is less than a second threshold, the DL tracking unit 116 may be disabled and the non-DL tracking unit 117 may be enabled, and so on.

[0135] Figure 5 1 is a timing chart showing an example of the operation timing of each of the above-mentioned units when the subject tracking function is implemented in the imaging apparatus 100. t600, t601, t602, t603, t604, t605, and t606 are times when the detection unit 110 starts detection processing in each frame. At t600, the detection unit 110 starts detection processing 610 for the first frame (frame 0).

[0136] When the detection unit 110 detects at least one candidate region and its object class in the image data of the 0th frame through the detection process 610, the object determination unit 111 starts the determination process 611 for determining the subject region to be tracked. When the subject region to be tracked is determined in the image data of the 0th frame through the determination process 611, the difficulty determination unit 112 starts the difficulty determination process 612 for the subject region to be tracked.

[0137] When the difficulty score is calculated in the difficulty determination process 612 and the subject region determined in the determination process 611 is a different subject-related region than the most recently determined subject region, the tracking control unit 113 modifies the control of the DL tracking unit 116 and the non-DL tracking unit 117. Specifically, the activation or deactivation and operating frequency of the DL tracking unit 116 and the non-DL tracking unit 117 are updated based on the difficulty score of the subject region to be tracked determined for the image data in the 0th frame. Furthermore, the tracking unit 115 updates the subject region to be tracked to the subject region to be tracked determined for the image data in the 0th frame. Therefore, the updated subject region is tracked for the image data of the first frame.

[0138] On the other hand, when the subject region determined by the determination process 611 and the most recently determined subject region are regions related to the same subject, the tracking control unit 113 does not modify the control of the DL tracking unit 116 and the non-DL tracking unit 117. The tracking unit 115 also does not update the subject region to be tracked.

[0139] exist Figure 5 , the 0th frame, the second frame when the detection process 624 starts at t602, the fourth frame when the detection process 633 starts at t604, and the sixth frame when the detection process 643 starts at t606 are cases where the subject to be tracked does not change.

[0140] On the other hand, the image data of the first frame in which the detection process 616 starts at t601 indicates that the subject region to be tracked determined by the determination process 617 and the most recently determined subject region are examples of regions related to different subjects.

[0141] In this case, the subject to be tracked changes in the next frame, so the tracking control unit 113 uses the result of the difficulty determination process 618 to modify the activation / deactivation control of the DL tracking unit 116 and the non-DL tracking unit 117. Here, for the next frame, the non-DL tracking unit 117 is deactivated, the DL tracking unit 116 is activated, and the operating frequency of the DL tracking unit 116 is switched to twice the operating frequency in the 0th frame and the first frame.

[0142] The subject to be tracked similarly changes in the third frame in which the detection process 629 starts at t603, and the tracking control unit 113 modifies the activation / deactivation control of the DL tracking unit 116 and the non-DL tracking unit 117. Here, for the fourth frame in which the detection process 633 starts at t604, the DL tracking unit 116 is deactivated, and the non-DL tracking unit 117 is activated. In addition, the operating frequency of the non-DL tracking unit 117 is set to a frequency half that of the operating frequency settings for the 0th and first frames.

[0143] At t600, the tracking unit 115 starts the DL tracking process 613 and the non-DL tracking process 614 of frame 0. Here, it is assumed that at the time point of frame 0, the tracking control unit 113 has enabled both the DL tracking unit 116 and the non-DL tracking unit 117, and has set the operating frequency of the non-DL tracking unit 117 to twice the operating frequency of the DL tracking unit 116.

[0144] exist Figure 5 , since the operation frequency of the DL tracking unit 116 and the non-DL tracking unit 117 is set to be lower than each frame, the DL tracking process and the non-DL tracking process indicated by the dotted rectangle are not performed.

[0145] In frame 0, the DL tracking unit 116 performs DL tracking processing 613, and the non-DL tracking unit 117 performs non-DL tracking processing 614, and both output tracking processing results. Here, the DL tracking processing 613 can output a tracking result with higher accuracy than the tracking result of the non-DL tracking processing 614. Therefore, the tracking unit 115 adopts the tracking result from the DL tracking processing 613 selected by the selection unit 125. Then, in the superimposition processing 615, the information superimposition unit 120 generates composite image data superimposed with a tracking frame based on the tracking result from the DL tracking processing 613. At t601, the display unit 121 performs display processing 623 for the composite image data of frame 0.

[0146] At t601, the non-DL tracking unit 117 starts the non-DL tracking process 620 for the first frame. Due to the operating frequency setting, the DL tracking unit 116 does not perform tracking processing on the first frame. Only the tracking result from the non-DL tracking unit 117 is obtained for the first frame, and therefore, in the superimposition process 622, the information superimposition unit 120 generates synthesized image data on which a tracking frame based on the tracking result from the non-DL tracking unit 117 is superimposed.

[0147] Note that the non-DL tracking process has a lower processing load than the DL tracking process and therefore ends earlier than the DL tracking process. The tracking results are output at a similar timing to when performing the DL tracking process, and therefore the tracking unit 115 performs the waiting process 621 after the non-DL tracking process. At t602, the display unit 121 performs the display process 628 of the synthesized image data of the first frame.

[0148] At t602, the DL tracking unit 116 starts DL tracking processing 625 for the second frame. For the second frame, the non-DL tracking unit 117 is deactivated, and therefore, the non-DL tracking processing is not performed. Only the tracking results from the DL tracking unit 116 are obtained for the second frame, and therefore, in superimposition processing 627, the information superimposition unit 120 generates composite image data superimposed with a tracking frame based on the tracking results from the DL tracking unit 116. At t603, the display unit 121 performs display processing 632 for the composite image data of the first frame.

[0149] At t603, the DL tracking unit 116 starts the DL tracking process 630 for the third frame. Before starting the DL tracking process 625 for the second frame, the operating frequency of the DL tracking unit 116 has been changed to twice the operating frequency in the 0th frame, and therefore the DL tracking process is also performed in the third frame after the second frame. The superposition process 631 and the display process 638 are similar to those performed in the second frame.

[0150] At t604, the non-DL tracking unit 117 starts the non-DL tracking process 635 for the fourth frame. For the fourth frame, the DL tracking unit 116 is deactivated, and thus, the DL tracking process is not performed. The operating frequency of the non-DL tracking unit 117 is set to half the operating frequency of the 0th and first frames, and thus, the non-DL tracking process 635 for the fourth frame is performed, while the non-DL tracking process 640 for the fifth frame is not performed. The waiting process 636, the superimposition process 637, and the display process 642 are similar to those performed for the first frame.

[0151] At t605, the tracking unit 115 performs neither the DL tracking process 639 nor the non-DL tracking process 640. Therefore, in the superimposition process 641 performed on the fifth frame, the information superimposition unit 120 generates synthetic image data on which a tracking frame based on the tracking result obtained in the non-DL tracking process 635 in the fourth frame is superimposed. Note that the information superimposition unit 120 may generate synthetic image data on which no tracking frame is superimposed in the superimposition process for a frame (such as the fifth frame) on which neither the DL tracking process nor the non-DL tracking process is performed.

[0152] At t606, the non-DL tracking unit 117 starts the non-DL tracking process 644 for the sixth frame. The DL tracking unit 116 is deactivated, and thus no DL tracking process is performed. The non-DL tracking process 644, the waiting process 645, the superimposition process 646, and the display process 647 for the sixth frame are similar to those performed in the fourth frame.

[0153] In this way, when the subject to be tracked is switched to another subject, the decision on whether to perform DL tracking or non-DL tracking, as well as the frequency of such execution, is modified based on the difficulty score calculated for the new subject area to be tracked in the next frame. This allows for dynamic execution of appropriate tracking processing based on the difficulty score, which in turn allows for suppression of power consumption.

[0154] The tracking process in this embodiment can use the tracking results of the frame before the target frame. Figure 6 The timing diagram shown describes the trace processing using past trace results.

[0155] t700 indicates the start timing of the tracking process for frame 0. Thereafter, t70n (where n=1, 2, etc.) indicates the start timing of the tracking process for frame n. At t702 and t704, the subject to be tracked is switched, and the control of the DL tracking unit 116 and the non-DL tracking unit 117 by the tracking control unit 113 is modified.

[0156] The DL tracking unit 116 and the non-DL tracking unit 117 perform tracking processing on the frame according to the operating frequency when set to enabled. Additionally, the DL tracking unit 116 and the non-DL tracking unit 117 use the tracking result obtained one frame ago for the tracking processing of the frame to be processed.

[0157] The processing accuracy of the DL tracking unit 116 tends to be higher than the processing accuracy of the non-DL tracking unit 117 based on similarity or the like. Therefore, (if available) the tracking result from the DL tracking process of the frame preceding the frame to be processed is used for the DL tracking process and / or the non-DL tracking process of the frame to be processed. Note that when the subject to be tracked differs between the frame to be processed and the previous frame, the tracking result of the previous frame is not used.

[0158] Therefore, the non-DL tracking process 711 of the first frame uses the tracking result from the DL tracking process 710 of the 0th frame. The subject to be tracked differs between the first and second frames, and therefore the tracking result from the first frame is not used for the DL tracking process 712 of the second frame. The DL tracking process 713 of the third frame uses the tracking result from the DL tracking process 712 of the second frame. The subject to be tracked differs between the third and fourth frames, and therefore the tracking result from the third frame is not used for the non-DL tracking process 715 of the fourth frame. The DL tracking process 714 is not performed in the fourth frame, and therefore the non-DL tracking process 716 of the fifth frame uses the tracking result from the non-DL tracking process 715 of the fourth frame.

[0159] Figure 7 1 is a flowchart related to a live view display operation performed when the subject tracking function of the imaging apparatus 100 is enabled.

[0160] In S200 , the control unit 102 controls the image sensor 103 to capture one frame and read out image data.

[0161] In S201 , the first pre-processing unit 104 applies pre-processing to the image data read out from the image sensor 103 .

[0162] In S202 , the first pre-processing unit 104 stores the image data to which the pre-processing is applied in the video memory 107 .

[0163] In S203, the first image correction unit 109 starts applying a predetermined image correction process to the image data read from the video memory 107. In S204, the first image correction unit 109 determines whether all the image correction processes to be applied are completed, and if it is determined that all the image correction processes to be applied are completed, it outputs the image data to which the image correction processes have been applied to the post-processing unit 114. If it is determined that all the image correction processes are not completed, the first image correction unit 109 continues the image correction process.

[0164] In S205 , the post-processing unit 114 generates image data for display based on the image data to which the image correction process is applied by the first image correction unit 109 , and outputs the image data for display to the information superimposing unit 120 .

[0165] At S206, information superimposition unit 120 generates composite image data in which the tracking frame and other information are superimposed on the captured image, using the display image data, the tracking frame image data, and the image data indicating other information generated by post-processing unit 114. Information superimposition unit 120 outputs this composite image data to display unit 121.

[0166] In S207, the display unit 121 displays the synthesized image data generated by the information superimposing unit 120. This completes the live view display of one frame.

[0167] Figure 8A and Figure 8B This is a flowchart related to the operation of the subject tracking function of the imaging apparatus 100 .

[0168] The processing from S300 to S304 is the same as that of the video memory 107 except that the tracking memory 108 is used instead of the video memory 107 and the image correction processing is an image correction processing for tracking. Figure 7 The processing from S200 to S204 in is similar.

[0169] In step S305, the detection unit 110 performs object region detection on the image data that has undergone all image correction processing. For each detected object region (candidate region), the detection unit 110 stores the position and size, object class, number of object regions belonging to the same object class, and confidence level score as a detection result in the tracking memory 108.

[0170] In S306, the object determination unit 111 determines one subject region to be tracked based on the information on the candidate region stored in the tracking memory 108. The object determination unit 111 stores the information on the determined subject region to be tracked in the tracking memory 108.

[0171] In S307, the difficulty determination unit 112 calculates a difficulty score based on the information about the subject region to be tracked and the information about the candidate region, and based on the tracking candidate subject information read out from the tracking memory 108. The tracking control unit 113 also determines whether to enable or disable the DL tracking unit 116 and the non-DL tracking unit 117 based on the difficulty score, and determines the operation frequency of the enabled tracking units.

[0172] In S308 , the tracking unit 115 determines whether the DL tracking unit 116 has performed tracking processing on the frame to be processed, and if so, executes S309 ; otherwise, executes S310 .

[0173] In S309 , the tracking unit 115 adopts the tracking result from the DL tracking unit 116 selected by the selection unit 125 , and outputs the tracking result to the information superposition unit 120 .

[0174] In S310, the tracking unit 115 adopts the tracking result from the non-DL tracking unit 117 selected by the selection unit 125 and outputs the tracking result to the information superposition unit 120. Note that when the non-DL tracking unit 117 has not yet performed the tracking process, the tracking result is not output, or the tracking result of the previous frame is output.

[0175] In S311, the tracking control unit 113 determines whether the subject region used in the tracking process for the frame to be processed is the same region associated with the subject to be tracked as the subject region determined in S306. This determination can also be considered as determining whether the subject to be tracked has switched. For example, if the confidence level score included in the tracking result for the target frame is not greater than a threshold, the tracking control unit 113 may determine that the subject to be tracked has switched. Note that this is merely an example, and other methods may be used for this determination.

[0176] If it is determined that the subject to be tracked has been switched, the tracking control unit 113 executes S312 , and if it is determined that the subject to be tracked has not been switched, the tracking control unit 113 executes S313 .

[0177] In S312 , the tracking control unit 113 determines whether to activate or deactivate the DL tracking unit 116 and the non-DL tracking unit 117 and determines the operation frequency based on the difficulty score of the subject region to be tracked determined in S306 for the target frame.

[0178] In S313 , the information superimposition unit 120 generates data of a synthesized image in which the image of the tracking frame is superimposed, based on the display image data and the tracking result adopted in S309 or S310 .

[0179] The processing of S314 and S315 is similar to the processing of S206 and S207.

[0180] As described above, according to this embodiment, in an imaging apparatus using a first tracking unit and a second tracking unit having higher processing accuracy but a greater computational load than the first tracking unit, activation or deactivation of the first and / or second tracking units is controlled based on the difficulty of tracking. Therefore, by deactivating a tracking unit that is unlikely to achieve good tracking results, deactivating the other tracking units when good tracking results are likely to be achieved from only one of the tracking units, and so on, it is possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0181] Since tracking processing is generally performed on moving images, the effect of suppressing power consumption is particularly large when a tracking unit utilizing machine learning with a high calculation load, such as deep learning, is used as the second tracking unit.

[0182] Second embodiment

[0183] Next, a second embodiment of the present invention will be described.

[0184] Figure 9 1 is a block diagram showing an example of the functional configuration of an image pickup apparatus 100′ serving as an example of an image processing apparatus according to the second embodiment, and the same reference numerals are given to the same configurations as those of the image pickup apparatus 100 described in the first embodiment. This embodiment uses a difficulty score to control the operation of an object determination unit (instead of controlling the operation of a tracking unit).

[0185] Areas different from the first embodiment will be described below.

[0186] The difficulty judgment unit 112 of this embodiment calculates a difficulty score that is an evaluation value indicating the difficulty of processing (object determination processing) for determining a subject region to be tracked from detected candidate regions based on the detection results from the detection unit 110 stored in the tracking memory 108.

[0187] For example, the difficulty determination unit 112 may calculate a difficulty score by taking into account at least one factor that affects the difficulty of the object determination process. Examples of factors that affect the difficulty of the object determination process include the size of the candidate region, the object class (type) of the candidate region, the number of candidate regions belonging to the same object class, and the location of the candidate region within the image. A specific example of a method for calculating the difficulty score will be provided later. The difficulty determination unit 112 outputs the calculated difficulty score to the object determination control unit 135.

[0188] The object determination control unit 135 determines whether to enable or disable each of the multiple object determination units included in the object determination unit 130 based on the difficulty score calculated by the difficulty judgment unit 112. In the present embodiment, the object determination unit 130 includes a plurality of object determination units having different computational loads and object determination accuracy. Specifically, the object determination unit 130 includes a DL object determination unit 131 that performs object determination processing using deep learning (DL) and a non-DL object determination unit 132 that performs object determination processing without using DL. It is assumed that the DL object determination unit 131 has a higher processing accuracy than the non-DL object determination unit 132, but also has a larger computational load than the non-DL object determination unit 132.

[0189] In this case, the object determination control unit 135 determines whether to enable or disable each of the DL object determination unit 131 and the non-DL object determination unit 132. Note that the object determination control unit 135 disables both the DL object determination unit 131 and the non-DL object determination unit 132 when only one candidate area is detected and when no candidate area is detected.

[0190] The object determination unit 130 determines a subject region (main subject region) to be tracked from among the candidate regions detected by the detection unit 110. The object determination unit 130 stores information for identifying the determined subject region in the tracking memory 108.

[0191] The object determination unit 130 includes a DL object determination unit 131 that performs object determination processing using deep learning (DL) and a non-DL object determination unit 132 that performs object determination processing without using DL. In the DL object determination unit 131 and the non-DL object determination unit 132, the object determination unit activated by the object determination control unit 135 outputs information for identifying the determined subject area.

[0192] The DL object determination unit 131 uses a pre-trained multi-layer neural network to determine the subject area to be tracked from multiple candidate areas. For example, the multi-layer neural network can be a convolutional neural network (CNN). For example, the multi-layer neural network can be trained by using training data that uses the size information, position information within the picture, object class information, and confidence level of each candidate area as input data, and uses the main subject area as the correct answer as supervision data. Note that the supervision data can be prepared by, for example, a person who observes the object frame and determines the main subject area as the correct answer.

[0193] The multi-layer neural network can be trained by a separate device or by the imaging device 100'. When the imaging device 100' is used, the multi-layer neural network can be trained by, for example, placing the DL object determination unit 131 in a learning mode and supplying training data (a set of input data and supervisory data) from the recording medium 119 to the DL object determination unit 131. Note that learning can be performed using a known optimization algorithm until the value of the loss function becomes less than a predetermined value.

[0194] When the multi-layer neural network is trained by a separate device, for example, the parameters of the multi-layer neural network trained by the separate device can be supplied to the DL target determination unit 131 via, for example, the recording medium 119. The multi-layer neural network trained by the separate device has a configuration similar to that of the multi-layer neural network implemented in the DL target determination unit 131. The training method can also be the same as the method used when training is performed using the imaging device 100'.

[0195] The DL object determination unit 131 provides the size information, position information within the screen, object class information, and confidence level of each candidate region detected by the detection unit 110 as test data to the trained multi-layer neural network. The trained multi-layer neural network outputs the probability that each candidate region is, for example, the main subject region.

[0196] The DL object determination unit 131 determines the candidate region with the highest output from the multi-layer neural network among the multiple candidate regions detected by the detection unit 110 as the subject region to be tracked (the main subject region). The DL object determination unit 131 then outputs the size information, position information within the screen, and object class information of the determined main subject region.

[0197] On the other hand, the non-DL object determination unit 132 determines a subject region to be tracked from among a plurality of candidate regions using a method that does not use deep learning. The non-DL object determination unit can determine a subject region to be tracked in a manner similar to the object determination unit 111 of the first embodiment. The non-DL object determination unit 132 stores information identifying the determined subject region in the tracking memory 108.

[0198] The selection unit 133 uses either the information on the main subject area output by the DL target determination unit 131 or the information on the main subject area output by the non-DL target determination unit 132. The selection unit 133 determines which main subject area information to use based on, for example, the positional information of the main subject area output by the DL target determination unit 131 and the non-DL target determination unit 132. For example, the selection unit 133 may use information on a main subject area that is closer in distance to the focus detection area currently set in the imaging apparatus 100' (closer to the focus detection area). The information on the main subject area used by the selection unit 133 is stored in the tracking memory 108.

[0199] When the object determination control unit 135 deactivates both the DL object determination unit 131 and the non-DL object determination unit 132, the object determination unit 130 determines that only one candidate region has been detected or that no candidate region has been detected. If there is a candidate region, the object determination unit 130 determines that candidate region as the main subject region. If even one candidate region has not been detected, the object determination unit 130 notifies the tracking unit 134 that there is no main subject and stops tracking processing.

[0200] The tracking unit 134 estimates the subject region to be tracked based on the image data of the target frame (current frame) stored in the tracking memory 108, and obtains the position and size of the estimated subject region within the frame as a tracking result. Similar to the non-DL tracking unit 117 in the first embodiment, for example, the tracking unit 134 can estimate the subject region to be tracked based on the similarity of color composition. Note that the color composition information of the subject region to be tracked can be updated based on the subject region to be tracked in the previous frame, and tracking processing can be performed for the next frame.

[0201] Note that the tracking unit 134 can estimate the subject region to be tracked in the current frame using pattern matching using the subject region to be tracked in the previous frame as a template. In this case, the template used for pattern matching can be updated for each frame using the subject region to be tracked determined by the object determination unit 130. The tracking unit 134 can also use a pre-trained multi-layer neural network to estimate the position and size of the subject region to be tracked in the current frame. The tracking unit 134 outputs the estimated position, size, and confidence level score of the subject region to be tracked for the current frame.

[0202] The information superposition unit 120 generates an image of a tracking frame based on the size of the main subject area included in the tracking result output by the tracking unit 134. The information superposition unit 120 outputs the synthesized image data generated in a similar manner to the first embodiment to the display unit 121.

[0203] Figure 10 is a diagram showing an example of live view display in the display unit 121 using the display image data output by the post-processing unit 114. Figure 10 In FIG, dotted line 1004 indicates the center area of ​​the image and is not included in the actual display image data. Images 1010 and 1020 show examples of display of composite image data in which an image of tracking frame 1015 is superimposed on the display image data. Image 1010 is an image one frame after image 1000, and image 1020 is an image several frames after image 1010.

[0204] In image 1000, when cars 1001, 1002, and 1003 are detected as candidate regions, object determination unit 130 determines the region of car 1003 closest to the center region indicated by dotted line 1004 as the subject region to be tracked. As a result, in image 1010 one frame later, the region of car 1003 undergoes tracking processing, and a tracking frame 1015 is superimposed and displayed to surround the region of car 1003. In image 1020 several frames later, tracking frame 1015 is also superimposed and displayed on the region of car 1003.

[0205] Images 1030 and 1040 indicate an example of a live view display when there is only one candidate area. Image 1040 is an image one frame after image 1030. In image 1030, a car 1031 is detected as the only candidate area. In this case, the object determination control unit 135 deactivates both the DL object determination unit 131 and the non-DL object determination unit 132 in the object determination unit 130. Then, the object determination unit 130 determines the area of ​​the car 1031 as the subject area to be tracked. As a result, in image 1040 one frame later, the tracking frame 1015 is superimposed and displayed to surround the area of ​​the car 1031 to be tracked. Similar to the first embodiment, the shape of the tracking frame 1015 is not limited to the combination of the four hollow bracket shapes in this embodiment.

[0206] Next, we will refer to Figure 11A and Figure 11B The flowchart in further describes the operation performed by the difficulty judgment unit 112 in this embodiment for calculating the difficulty score related to the object area to be tracked.

[0207] In S1101 , the difficulty level determination unit 112 initializes the difficulty level score to zero.

[0208] In S1102, the difficulty judgment unit 112 judges whether the candidate region exists in the center of the image. For example, if there is a candidate region including Figure 10If some or all of the candidate regions are within the area at the center of the image indicated by the dotted line 1004, difficulty determination unit 112 determines that the candidate region exists at the center of the image. Alternatively, for example, if some or all of the candidate regions are not within the area including the center of the image indicated by the dotted line 1004, difficulty determination unit 112 determines that the candidate region does not exist at the center of the image. Difficulty determination unit 112 then executes S1103 if it is not determined that the candidate region exists at the center of the image, and executes S1104 if it is determined that the candidate region exists at the center of the image.

[0209] Note that if a focus detection area is set in advance, the difficulty judgment unit 112 performs a similar judgment on the focus detection area (instead of the area in the center of the screen). Then, the difficulty judgment unit 112 executes S1103 if it determines that the candidate area is not present in the focus detection area, and executes S1104 if it determines that the candidate area is present in the focus detection area.

[0210] In step S1103, the difficulty determination unit 112 adds one point (+1) to the difficulty score. If no candidate region exists in the focus detection area or the image center, the difficulty determination unit 112 determines that it is difficult to determine the subject region to be tracked and adds points to the difficulty score (increasing the difficulty). The difficulty determination unit 112 then executes step S1105.

[0211] In step S1104, the difficulty determination unit 112 maintains the difficulty score. If the candidate area is located in the focus detection area or the image center, the difficulty determination unit 112 determines that it is easy to determine the subject area to be tracked and does not add points to the difficulty score (does not increase the difficulty). The difficulty determination unit 112 then executes step S1105.

[0212] In S1105, the difficulty judgment unit 112 determines whether the number of types of candidate regions (types of object classes) is at least a preset threshold value N (whether at least N types of candidate regions have been detected). For example, if candidate regions associated with the human object class, the dog object class, the cat object class, and the car object class are detected, there are four types of candidate regions. If the threshold value N is 3, the difficulty judgment unit 112 determines that the types of candidate regions exceed the threshold value N and executes S1106. On the other hand, if candidate regions associated with the dog object class and the cat object class are detected, there are two types of candidate regions. If the threshold value N is 3, the difficulty judgment unit 112 determines that the types of candidate regions do not exceed the threshold value N and executes S1107.

[0213] In S1106, the difficulty determination unit 112 adds (the number of candidate region types - N + 1) points to the difficulty score. When at least N types of candidate regions are detected, the difficulty determination unit 112 determines that it is difficult to determine the subject region to be tracked, and adds more points to the difficulty score (increasing the difficulty) as the number of types increases. The difficulty determination unit 112 then executes S1108.

[0214] In S1107, the difficulty determination unit 112 maintains the difficulty score. If the number of candidate region types is less than N, the difficulty determination unit 112 determines that it is easy to determine the subject region to be tracked and does not add points to the difficulty score (does not increase the difficulty). The difficulty determination unit 112 then executes S1108.

[0215] S1108 and S1115 indicate that the processing therebetween is repeated. Specifically, the processing from S1109 to S1114 is repeated a number of times equal to the number of types of candidate regions detected. For example, if four types of candidate regions are detected, the processing from S1109 to S1114 is repeated four times, once for each type of candidate region.

[0216] In S1109, the difficulty determination unit 112 determines whether the number of candidate regions associated with the same type (object class) is at least a preset threshold value M. For example, assume that the detection unit 110 detects four candidate regions associated with the dog object class and two candidate regions associated with the cat object class, and the threshold value M is 3. In this case, the processing from S1109 to S1114 is performed once for the dog object class and once for the cat object class.

[0217] In the processing for the dog object class, in S1109, the number of candidate regions for the dog object class is 4, so the difficulty determination unit 112 determines that it exceeds the threshold value M and executes S1110. On the other hand, in the processing for the cat object class, in S1109, the number of candidate regions for the cat object class is 2, so the difficulty determination unit 112 determines that it does not exceed the threshold value M and executes S1111.

[0218] In S1110, the difficulty determination unit 112 adds (the number of candidate regions of the object class being processed - M + 1) points to the difficulty score. When the number of candidate regions of the same object class is at least the threshold value M, it is determined that it is difficult to determine the subject region to be tracked, and points are added to the difficulty score (increasing the difficulty). The difficulty determination unit 112 then executes S1112.

[0219] In S1111, the difficulty determination unit 112 maintains the difficulty score. If the number of candidate regions of the same object class is less than the threshold value M, it is determined that it is easy to determine the subject region to be tracked, and no points are added to the difficulty score (the difficulty is not increased). The difficulty determination unit 112 then executes S1112.

[0220] In step S1112, the difficulty determination unit 112 determines whether the size variance of candidate regions associated with the same object class is less than a preset threshold value L. Here, the size of a candidate region may be, for example, the product of the number of horizontal pixels and the number of vertical pixels in the candidate region. The size of a candidate region is a value normalized by the size of the largest candidate region among the candidate regions of the same object class (≤ 1). Using the normalized size, the variance may be calculated as the average of the squares of the differences between the sizes of the individual candidate regions and the average of the sizes.

[0221] For example, assume that four candidate regions of the dog object class and two candidate regions of the cat object class are detected, and the preset threshold L is 0.1. If, when executing S1112 for the dog object class, the size variance of the candidate regions is 0.2, the difficulty determination unit 112 determines that the size variance of the candidate regions of the same object class is not less than L, and executes S1114. On the other hand, if, when executing S1112 for the cat object class, the size variance of the candidate regions is 0.0025, the difficulty determination unit 112 determines that the size variance of the candidate regions of the same object class is less than L, and executes S1113.

[0222] In S1113, the difficulty determination unit 112 adds 1 point to the difficulty score. When the size variance of candidate regions of the same object class is less than L, it is assumed that candidate regions associated with the same type of subject have similar sizes. Therefore, the difficulty determination unit 112 determines that it is difficult to determine the subject region to be tracked for the object class being processed, and adds a point to the difficulty score (increasing the difficulty). The difficulty determination unit 112 then executes S1115.

[0223] In S1114, the difficulty determination unit 112 maintains the difficulty score. If the size variance of candidate regions of the same object class is not less than L, it is assumed that candidate regions associated with the same type of subject have varying sizes. Therefore, the difficulty determination unit 112 determines that it is easy to determine the subject region to be tracked for the object class being processed and does not add points to the difficulty score (does not increase the difficulty). The difficulty determination unit 112 then executes S1115.

[0224] Once the processing from S1109 to S1114 is performed on all the detected types (object classes), the difficulty judgment unit 112 executes S1116.

[0225] In S1116 , the difficulty judgment unit 112 outputs the calculated difficulty score to the object determination control unit 135 .

[0226] Note that the method of calculating the difficulty score described here is merely an example, and the difficulty score can be calculated by various other methods. For example, instead of adding 1 point to the difficulty score in S1103, a value multiplied by a preset gain coefficient can be added. Additionally, the difficulty score can be calculated using only the number of types of candidate areas and the number of candidate areas associated with the same type of subject (belonging to the same object class). Alternatively, the difficulty score can be calculated based on only one item. Additionally, the difficulty score can be a binary value (0 or 1), where control is performed to enable the DL object determination unit 131 when the difficulty score is 1, and to disable the DL object determination unit 131 when the difficulty score is 0.

[0227] Next, we will refer to Figure 12 The flowchart in FIG further describes the operation of the object determination control unit 135. Here, it is assumed that the difficulty judgment unit 112 Figure 11A and Figure 11B The difficulty score is calculated by the operations described.

[0228] In S1201 , the object determination control unit 135 determines whether the difficulty score is greater than a preset threshold value A, executes S1203 if it is determined that the difficulty score is greater than the preset threshold value A, and executes S1202 if not.

[0229] In S1202 , the object determination control unit 135 determines whether the difficulty score is greater than a preset threshold value B (<A), executes S1204 if determined to be greater than the preset threshold value B, and executes S1205 if not.

[0230] In step S1203, the object determination control unit 135 determines to enable the DL object determination unit 131 and disable the non-DL object determination unit 132. When the difficulty score is greater than A, the difficulty in determining the object is high. As such, the object determination control unit 135 determines to use the DL object determination unit 131, which has better performance than the non-DL object determination unit 132. Alternatively, the object determination control unit 135 determines that a good result is unlikely to be obtained even when the non-DL object determination unit 132 is used, and determines not to use the non-DL object determination unit 132.

[0231] In step S1204, the object determination control unit 135 determines to deactivate the DL object determination unit 131 and activate the non-DL object determination unit 132. When the difficulty score is not greater than A but greater than B, the difficulty of determining the object is moderately difficult. Therefore, the object determination control unit 135 determines that sufficiently accurate results can be obtained using the non-DL object determination unit 132, and determines to use only the non-DL object determination unit 132. By deactivating the DL object determination unit 131, which consumes more power than the non-DL object determination unit 132, power consumption can be reduced.

[0232] In S1205, the object determination control unit 135 determines to deactivate the DL object determination unit 131 and deactivate the non-DL object determination unit 132. When the difficulty score is not greater than B, the difficulty of determining the object is low. Therefore, the object determination control unit 135 determines that the object can be determined without using the DL object determination unit 131 and the non-DL object determination unit 132, and deactivates both the DL object determination unit 131 and the non-DL object determination unit 132. This makes it possible to further reduce power consumption.

[0233] Note that, for example, when only one candidate area is detected or no candidate area is detected, the subject area to be tracked can be determined without using the DL object determination unit 131 and the non-DL object determination unit 132. This also applies when one of the multiple detected candidate areas is extremely large and exists in the center of the screen or in the focus detection area.

[0234] The operation of the object determination control unit 135 described here is merely an example. For example, it is not necessary to change the control for each difficulty score. For example, similar to S1204, the non-DL object determination unit can be enabled in S1205. Alternatively, if it is determined in S1201 that the difficulty score is greater than A, S1204 can be immediately executed.

[0235] Next, we will refer to Figure 13 The flowchart in describes an example of a method in which the object determination control unit 135 determines activation and deactivation of the DL object determination unit 131 and the non-DL object determination unit 132 without using the difficulty score. It is assumed here that the object determination control unit 135 determines activation and deactivation of the DL object determination unit 131 and the non-DL object determination unit 132 using the detected object class and the number of candidate regions belonging to the same object class.

[0236] It is assumed here that the DL object determination unit 131 has a multi-layer neural network pre-trained using training data related to a specific scene (e.g., a sports scene using a ball). The training data is a collection of input data and corresponding supervisory data. Here, the size information, position information within the picture, object class information, and confidence level of each candidate area among multiple candidate areas extracted from the image data of sports using a ball can be used as input data. Supervisory data for the correct answer can be used, which includes candidate areas belonging to the human object class among the candidate areas included in the image data.

[0237] When the camera 100' captures a sports scene involving a ball, the DL object determination unit 131, equipped with a multi-layer neural network trained in this manner, can perform highly accurate object determination processing. Furthermore, the image data used to generate input data can be limited to image data containing at least four candidate regions for the human object class. This allows the DL object determination unit 131 to perform more accurate object determination processing on images of sports scenes involving a ball, including at least four "people."

[0238] The operation of the object determination control unit 135 in a case where the DL object determination unit 131 can perform more accurate object determination processing on an image of a scene including at least four “people” in a sport using a ball will be described below.

[0239] In step S1301, the object determination control unit 135 determines whether the detection unit 110 has detected at least four candidate regions of the human object class and at least one candidate region of the ball object class. If it is determined that at least four candidate regions of the human object class and at least one candidate region of the ball object class have been detected, the object determination control unit 135 executes step S1304, and if not, executes step S1302.

[0240] In step S1302, the object determination control unit 135 determines whether the detection unit 110 has detected three candidate regions of the human object class and at least one candidate region of the ball object class. If it is determined that at least three candidate regions of the human object class and at least one candidate region of the ball object class have been detected, the object determination control unit 135 executes step S1305, and if not, executes step S1303.

[0241] In S1301, the object determination control unit 135 determines whether the detection unit 110 has detected two candidate regions of the human object class. If it is determined that at least two candidate regions of the human object class are detected, the object determination control unit 135 executes S1306, and if not, executes S1307.

[0242] In step S1304, the object determination control unit 135 determines to activate the DL object determination unit 131 and deactivate the non-DL object determination unit 132. When at least four candidate regions of the human object class and at least one candidate region of the ball object class are detected, the scene is the same as the scene used to train the multi-layer neural network of the DL object determination unit 131. As such, under the assumption that the DL object determination unit 131 can optimally determine the object, the object determination control unit 135 determines to use only the DL object determination unit 131.

[0243] In step S1305, the object determination control unit 135 determines to enable the DL object determination unit 131 and to enable the non-DL object determination unit 132. When at least three candidate regions of the human object class and at least one candidate region of the ball object class are detected, the scene is similar to, but not identical to, the scene used to train the multi-layer neural network of the DL object determination unit 131. On the other hand, object determination processing is necessary for the three candidate regions of the human object class, and therefore, if only the non-DL object determination unit is enabled, accuracy may be insufficient. Therefore, in order to achieve highly accurate object determination processing, the object determination control unit 135 determines to use both the DL object determination unit 131 and the non-DL object determination unit 132.

[0244] In S1306, the object determination control unit 135 determines to deactivate the DL object determination unit 131 and activate the non-DL object determination unit 132. When two candidate regions of the human object class are detected, one candidate region can be selected as the subject region to be tracked, and the difficulty of determining the object is low. Therefore, the object determination control unit 135 determines that sufficiently accurate results can be obtained using the non-DL object determination unit 132 and determines to use only the non-DL object determination unit 132. By deactivating the DL object determination unit 131, which consumes more power than the non-DL object determination unit 132, power consumption can be reduced.

[0245] In S1307, the object determination control unit 135 determines to deactivate both the DL object determination unit 131 and the non-DL object determination unit 132. When one candidate area of ​​the human object class is detected or no candidate area of ​​the human object class is detected, the difficulty of determining the object is the lowest. Therefore, the object determination control unit 135 determines that the subject area to be tracked can be determined without using the DL object determination unit 131 and the non-DL object determination unit 132, and deactivates both the DL object determination unit 131 and the non-DL object determination unit 132. This makes it possible to further reduce power consumption. In this case, the object determination unit 130 determines the detected only candidate area as the subject area to be tracked without using the DL object determination unit 131 and the non-DL object determination unit 132, and outputs this information. When no candidate area is detected, the object determination unit 130 outputs the determination result from one frame ago again, for example.

[0246] The control operation described here is merely an example. The object determination control unit 135 may determine the object determination unit to be activated, considering whether the object image is an image of a scene in which the DL object determination unit 131 and the non-DL object determination unit 132 can accurately determine the object.

[0247] Note that this configuration can make it possible to switch the scene in which the DL object determination unit 131 can accurately determine the object by switching the parameters obtained when training the multi-layer neural network using training data prepared for each type of scene. For example, pre-trained parameters corresponding to the scene captured by the camera 100' can be set and used in the multi-layer neural network.

[0248] Additionally, if the camera 100' has a user sight line detection function, the non-DL object determination unit 132 can determine the object based on the user's sight line. For example, the non-DL object determination unit 132 can determine the candidate area containing the image coordinates of the user's attention as the subject area to be tracked.

[0249] When the object determination control unit 135 determines to activate the DL object determination unit 131 in a state where the non-DL object determination unit 132 is set to determine an object based on the user's line of sight, the object determination control unit 135 may deactivate the non-DL object determination unit 132. This is because, in a scene where it is very difficult to determine an object with the DL object determination unit 131 activated, it is considered unlikely that the user's line of sight will be fixed on the main subject area.

[0250] Figure 14This is a timing diagram illustrating an example of the operation timing of each of the above-described units when implementing a subject tracking function including object identification processing in the imaging apparatus 100'. t1400, t1401, t1402, t1403, and t1404 are times when the detection unit 110 starts detection processing in each frame. At t1400, the detection unit 110 starts detection processing 1410 for the first frame (frame 0).

[0251] Through the detection process 1410, the detection unit 110 detects at least one candidate region and its object class in the image data of the 0th frame. Based on the information about the candidate region detected in the image data of the 0th frame, the difficulty judgment unit 112 performs the following steps as shown in FIG. Figure 11A and Figure 11B The difficulty judgment process 1411 is performed and a difficulty score is calculated.

[0252] Based on the difficulty score calculated by the difficulty judgment process 1411 , the object determination control unit 135 determines whether to activate or deactivate the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the 0th frame.

[0253] exist Figure 14 , the object determination process shown by the dotted line (e.g., the DL object determination process 1412) indicates that the object determination control unit 135 has deactivated the object determination unit (the DL object determination unit 131) for executing the object determination process. On the other hand, the object determination process shown by the solid line (e.g., the non-DL object determination process 1413) indicates that the object determination control unit 135 has activated the object determination unit (the non-DL object determination unit 132) for executing the object determination process.

[0254] The tracking unit 134 performs tracking processing 1414 for the subject area to be tracked output from the object determination unit 130. The tracking unit 134 estimates the position of the subject area to be tracked in the image data to be processed. The tracking unit 134 then outputs the estimated information (position, size, etc.) of the subject area to be tracked in the image data of the 0th frame to the information superposition unit 120 as a tracking result. The tracking processing 1414 is indicated by a dotted line because, in the 0th frame, the subject area to be tracked has not yet been determined, and tracking processing will not actually be performed.

[0255] Through the tracking frame superimposition processing 1415, the information superimposition unit 120 generates composite image data in which a tracking frame based on the result of the tracking processing 1414 is superimposed on the display image data of the 0th frame. Note that the information superimposition unit 120 is indicated by a dotted line because, in the 0th frame, the subject region to be tracked has not yet been determined and, in reality, a tracking frame will not be superimposed.

[0256] Through the display process 1416, the display unit 121 displays the synthesized image data of the frame 0. As described above, the tracking frame superimposing process 1415 has not actually been performed yet, and therefore the display unit 121 displays the display image data on which no tracking frame is superimposed.

[0257] Through detection processing 1420, detection unit 110 detects at least one candidate region and its object class in the image data of the first frame. Based on information related to the candidate region detected in the image data of the first frame, difficulty determination unit 112 performs difficulty determination processing 1421 and calculates a difficulty score.

[0258] The object determination control unit 135 determines whether to activate or deactivate the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the first frame based on the difficulty score calculated by the difficulty judgment process 1421. Here, it is assumed that the object determination control unit 135 determines to activate the DL object determination unit 131 and deactivate the non-DL object determination unit 132.

[0259] The non-DL object determination process 1423 is not actually performed and is therefore indicated by a dotted line. On the other hand, the DL object determination process 1422 is performed by the DL object determination unit 131. The object determination unit 130 outputs information on the subject region to be tracked determined by the DL object determination unit 131.

[0260] Through the tracking process 1424, the tracking unit 134 estimates the position of the subject region to be tracked, which is determined for the image of frame 0 by the non-DL object determination process 1413, in the image data of the first frame. Then, the tracking unit 134 outputs information (such as the position and size) of the estimated subject region to be tracked in the image data of the first frame to the information superimposing unit 120 as a tracking result.

[0261] Through the tracking frame superimposing process 1425 , the information superimposing unit 120 generates synthesized image data in which the tracking frame based on the result of the tracking process 1424 is superimposed on the display image data of the first frame.

[0262] Through the display process 1426 , the display unit 121 displays the synthesized image data of the first frame.

[0263] Through detection processing 1430, detection unit 110 detects at least one candidate region and its object class in the image data of the second frame. Based on information related to the candidate region detected in the image data of the second frame, difficulty determination unit 112 performs difficulty determination processing 1431 and calculates a difficulty score.

[0264] The object determination control unit 135 determines whether to activate or deactivate the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the second frame based on the difficulty score calculated by the difficulty judgment process 1431. Here, it is assumed that only one candidate area is detected in the detection process 1430, and the object determination control unit 135 determines to deactivate both the DL object determination unit 131 and the non-DL object determination unit 132.

[0265] Therefore, neither the DL object determination process 1432 nor the non-DL object determination process 1433 is performed, and is indicated by a dotted line. In this case, the object determination unit 130 determines the detected only candidate region as the subject region to be tracked.

[0266] The tracking unit 134 estimates the position of the subject region to be tracked, determined for the image of the first frame by the non-DL object determination process 1422, in the image data of the second frame through the tracking process 1434. The tracking unit 134 then outputs information (such as the position and size) of the subject region to be tracked in the image data of the second frame as a tracking result to the information superimposing unit 120.

[0267] Through the tracking frame superimposition process 1435 , the information superimposition unit 120 generates synthesized image data in which the tracking frame based on the result of the tracking process 1434 is superimposed on the display image data of the second frame.

[0268] Through the display process 1436 , the display unit 121 displays the synthesized image data of the second frame.

[0269] Detection unit 110 detects at least one candidate region and its object class in the image data of the third frame through detection processing 1440. Based on information related to the candidate region detected in the image data of the third frame, difficulty determination unit 112 performs difficulty determination processing 1441 and calculates a difficulty score.

[0270] The object determination control unit 135 determines whether to activate or deactivate the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the third frame based on the difficulty score calculated by the difficulty judgment process 1441. Here, it is assumed that the object determination control unit 135 determines to activate both the DL object determination unit 131 and the non-DL object determination unit 132.

[0271] Therefore, the DL object determination unit 131 performs DL object determination processing 1442, and the non-DL object determination unit 132 performs non-DL object determination processing 1443. Then, the selection unit 133 selects one of the results from the DL object determination processing 1442 and the non-DL object determination processing 1443 and outputs it as information regarding the subject region to be tracked determined for the third frame. For example, if the two subject regions determined by the respective instances of the subject determination processing are the same, the selection unit 133 selects a predetermined one of them. On the other hand, if the two subject regions determined by the respective instances of the subject determination processing are different, the selection unit 133 selects one of the determination results based on predetermined conditions. For example, the selection unit 133 may select a determination result indicating a larger subject region, a subject region with a center of gravity closer to the center of the image, or a subject region with a center of gravity closer to the focus detection area.

[0272] Through the tracking process 1444, the tracking unit 134 estimates the position of the subject region to be tracked, determined for the image of the second frame, in the image data of the third frame. The tracking unit 134 then outputs information (such as the position and size) of the estimated subject region to be tracked in the image data of the third frame to the information superimposing unit 120 as a tracking result.

[0273] Through the tracking frame superimposition process 1445 , the information superimposition unit 120 generates synthesized image data in which the tracking frame based on the result of the tracking process 1444 is superimposed on the display image data of the third frame.

[0274] Through the display process 1446 , the display unit 121 displays the synthesized image data of the third frame.

[0275] Through detection processing 1450, detection unit 110 detects at least one candidate region and its object class in the image data of the fourth frame. Based on information related to the candidate region detected in the image data of the fourth frame, difficulty determination unit 112 performs difficulty determination processing 1451 and calculates a difficulty score.

[0276] The object determination control unit 135 determines whether to activate or deactivate the DL object determination unit 131 and the non-DL object determination unit 132 for the image data of the fourth frame based on the difficulty score calculated by the difficulty judgment process 1451. Here, it is assumed that the object determination control unit 135 determines to activate both the DL object determination unit 131 and the non-DL object determination unit 132.

[0277] Therefore, the DL object determination unit 131 performs a DL object determination process 1452, and the non-DL object determination unit 132 performs a non-DL object determination process 1453. As described with respect to the process of the third frame, the selection unit 133 selects and outputs one determination result.

[0278] Through the tracking process 1454, the tracking unit 134 estimates the position of the subject region to be tracked, determined for the image of the third frame, in the image data of the fourth frame. The tracking unit 134 then outputs the estimated information (such as the position and size) of the subject region to be tracked in the image data of the fourth frame to the information superimposing unit 120 as a tracking result.

[0279] Through the tracking frame superimposition process 1455 , the information superimposition unit 120 generates synthesized image data in which the tracking frame based on the result of the tracking process 1454 is superimposed on the display image data of the fourth frame.

[0280] Through the display process 1456 , the display unit 121 displays the synthesized image data of the fourth frame.

[0281] Figure 14 The timing diagrams in are merely examples and can be changed in various ways. For example, although Figure 14 Detection processing and difficulty determination processing are performed for each frame, but the execution frequency can be reduced. Power consumption can be reduced by performing the processing every few frames. In this case, the results of the object determination processing for the frames in which detection processing and difficulty determination processing are performed can be used to update the subject area to be tracked, and the results of the object determination processing for the frames in which detection processing and difficulty determination processing are not performed do not need to be used for updating.

[0282] The live view display operation in this embodiment is similar to the reference Figure 7 The operation is the same as that described in the first embodiment and therefore will not be described here.

[0283] Figure 15A and Figure 15B This is a flowchart related to control of the operation of a subject determination unit for determining a subject region to be tracked.

[0284] Processing and Reference from S300 to S305 Figure 8A The processing described in the first embodiment is the same and will therefore not be described here. Note that the image correction processing is correction processing for tracking and for detection.

[0285] In S1501, the difficulty judgment unit 112 determines the candidate region based on the candidate region detection result from S305. Figure 11A and Figure 11BThe difficulty score indicating the difficulty of determining the object is calculated as described. The difficulty judgment unit 112 outputs the calculated difficulty score to the object determination control unit 135.

[0286] In step S1502, the object determination control unit 135 determines whether the difficulty score is at least a preset first threshold. If it is determined that the difficulty score is at least the first threshold, the object determination control unit 135 executes step S1503. If it is not determined that the difficulty score is at least the first threshold, the object determination control unit 135 executes step S1504.

[0287] In S1503 , the object determination control unit 135 determines to activate the DL object determination unit 131 , and performs control for activating the DL object determination unit 131 .

[0288] In S1504, the object determination control unit 135 determines to deactivate the DL object determination unit 131, and performs control for deactivating the DL object determination unit 131. Furthermore, the object determination control unit 135 performs S1505.

[0289] In step S1505, the object determination control unit 135 determines whether the difficulty score is less than a preset second threshold. Here, the second threshold is lower than the first threshold (lower difficulty). If the difficulty score is determined to be less than the second threshold, the object determination control unit 135 executes step S1506. If the difficulty score is not determined to be less than the second threshold, the object determination control unit 135 executes step S1507.

[0290] In S1506 , the object determination control unit 135 determines to deactivate the non-DL object determination unit 132 , and performs control for deactivating the non-DL object determination unit 132 .

[0291] In S1507 , the object determination control unit 135 determines to activate the non-DL object determination unit 132 , and performs control for activating the non-DL object determination unit 132 .

[0292] In S1508 , if only one of the DL object determination unit 131 and the non-DL object determination unit 132 is enabled, the object determination unit 130 selects the determination result from the enabled object determination unit using the selection unit 133 .

[0293] If both the DL object determination unit 131 and the non-DL object determination unit 132 are enabled and the respective determination results indicate the same subject region, the object determination unit 130 selects one of the determination results using the selection unit 133. If the determination results differ, the object determination unit 130 selects one of the determination results based on a predetermined condition using the selection unit 133 as described above. For example, the object determination unit 130 selects the determination result indicating the subject region having a center position closer to the center of the screen using the selection unit 133. The object determination unit 130 stores the determination result selected by the selection unit 133 as information regarding the subject region to be tracked in the tracking memory 108.

[0294] If both the DL object determination unit 131 and the non-DL object determination unit 132 are inactive, the object determination unit 130 confirms the number of candidate regions detected in step S305. If the number of detected candidate regions is one, the object determination unit 130 determines the detected candidate region as a subject region to be tracked and stores information related thereto in the tracking memory 108. If the number of candidate regions is zero, there is no subject to be tracked. In this case, the object determination unit 130 may store the information related to the subject region to be tracked, which was output immediately before, in the tracking memory 108, or may deactivate tracking processing for the next frame.

[0295] This completes the operation for determining the subject area to be tracked for one frame. Thereafter, similar processing is repeated for each frame or every predetermined number of frames.

[0296] Figure 16 is a flowchart related to the operation of the subject tracking function in the imaging apparatus 100, and is Figure 15A and Figure 15B The same processing will be given the same reference numerals in the figure. The processing of S300 to S304 will not be described.

[0297] In S1601, the tracking unit 134 determines whether the subject region to be tracked has been determined in the process of S1508 of the frame before the target frame. If it is determined that the subject region to be tracked has been determined in the previous frame, the tracking unit 134 executes S1602. If it is determined that the subject region to be tracked has not been determined, the tracking unit 134 executes S1603.

[0298] In step S1602, the tracking unit 134 updates the information related to the subject region to be tracked using the information determined for the previous frame. If the tracking process was disabled in the previous frame, the tracking unit 134 controls the start of the tracking process. This is because, as described above, if even one candidate region has not been detected, the tracking process can be disabled for the next frame.

[0299] In step S1603, the tracking unit 134 determines whether the tracking continuation state is valid. The "tracking continuation state" is a state in which tracking processing is enabled and the subject area to be tracked has been determined in the previous frame. If the state is determined to be the tracking continuation state, the tracking unit 134 executes step S1604. If the state is not determined to be the tracking continuation state, the tracking unit 134 executes step S314.

[0300] In S1604, the tracking unit 134 performs tracking processing of the determined subject region. The tracking unit 134 outputs the estimated size and position within the image of the subject region to the information superposition unit 120 as a tracking result.

[0301] Processing and reference from S313 to S315 Figure 8B The processing described in the first embodiment is the same and will therefore not be described here. However, if it is determined in S1603 that the state is not the tracking continuation state and S314 is executed, this means that there is no tracking result and therefore the tracking frame is not superimposed.

[0302] In this embodiment, in an image processing device that uses a first object determination unit for determining a subject region to be tracked, and a second object determination unit that has higher accuracy but a greater computational load than the first object determination unit, whether to activate or deactivate the first and / or second tracking units is controlled based on the difficulty of determining the subject. Therefore, by deactivating an object determination unit that is unlikely to be able to make an accurate determination, and by deactivating the other object determination unit when only one of the object determination units is likely to be able to make an accurate determination, it is possible to suppress power consumption while suppressing a decrease in object determination accuracy.

[0303] Since the object determination process is generally performed on consecutive frames at a time, the effect of suppressing power consumption is particularly large when an object determination unit utilizing machine learning with a high computational load, such as deep learning, is used as the second object determination unit.

[0304] Third embodiment

[0305] Next, a third embodiment of the present invention will be described. Figure 17 17 is a block diagram illustrating an example of a functional configuration of an image pickup apparatus 1700 serving as an example of an image processing apparatus according to the third embodiment.

[0306] The optical system 1701 and the image sensor 1704 can have a configuration similar to that of the optical system 101 and the image sensor 103 in the imaging apparatus 100, and therefore will not be described again. In this embodiment, a mechanical shutter 1703 (hereinafter simply referred to as "shutter") is provided between the optical system 1701 and the image sensor 1704, but the shutter 1703 is not essential. The operation of the shutter 1703 is controlled by the system control unit 1740. The shutter speed corresponds to the exposure time of the image sensor 1704.

[0307] The optical control unit 1702 calculates the defocus amount for each of a plurality of focus detection areas by capturing an optical image formed by the optical system 1701 using, for example, a phase difference AF sensor. The focus detection area may be, for example, a predetermined rectangular area within the imaging plane. Based on the calculated defocus amount and the tracking results from the tracking unit 1718, the optical control unit 1702 determines the focus detection area on which the optical system 1701 is to focus. The optical control unit 1702 then drives the focus lens of the optical system 1701 based on the calculated defocus amount for the determined focus detection area. This causes the optical system 1701 to focus on the subject within the determined focus detection area.

[0308] The image sensor 1704 outputs image data obtained by imaging to a first pre-processing unit 1711 and a second pre-processing unit 1712 in the image processing unit 1710 .

[0309] The first pre-processing unit 1711 performs processing similar to that of the first pre-processing unit 104 on the image data and stores the result in the video memory 1714 .

[0310] The first image correction unit 1716 applies processing similar to that of the first image correction unit 109 to the image data, and outputs the image data to which the processing has been applied to the first post-processing unit 1719 .

[0311] The first post-processing unit 1719 generates recording image data, display image data, and the like in a manner similar to the post-processing unit 114. The first post-processing unit 1719 supplies the recording image data to the recording unit 1720. The first post-processing unit 1719 supplies the display image data to the information superimposing unit 1721.

[0312] The recording unit 1720 records the recorded image data generated by the first post-processing unit 1719 in the recording medium 1730. The recording medium 1730 may be, for example, a semiconductor memory card or a built-in nonvolatile memory.

[0313] Similar to the second pre-processing unit 105, the second pre-processing unit 1712 applies color interpolation processing to the image data and stores the result in the tracking memory 1715. Also in this embodiment, the tracking memory 1715 and the video memory 107 can be implemented as separate address spaces within the same memory space. The second pre-processing unit 1712 can also apply a reduction process to reduce the number of pixels as needed to reduce the processing load. Although the first pre-processing unit 1711 and the second pre-processing unit 1712 are described here as separate functional blocks, these units can be configured using a common pre-processing unit.

[0314] The second image correction unit 1713 applies a process similar to that of the second image correction unit 106 to the image data stored in the tracing memory 1715. The second image correction unit 1713 stores, in the tracing memory 1715, the image data to which the process has been applied.

[0315] Also in this embodiment, when the subject tracking function is not used, functional blocks related to the subject tracking function (such as the second pre-processing unit 1712 and the second image correction unit 1713) do not need to operate. Additionally, image data to which the subject tracking function is applied is moving image data captured for the purpose of live view display or recording. The moving image data has a predetermined frame rate, such as 30 fps, 60 fps, or 120 fps.

[0316] The detection unit 1717 detects at least one region (candidate region) of a predetermined candidate subject from the image data of a frame. Additionally, for each detected region, the detection unit 1717 associates an object class indicating the position, size, and type of the candidate subject (such as car, airplane, bird, insect, human body, head, pupil, cat, or dog) within the frame with its confidence level. The number of detected candidate regions is also obtained for each object class.

[0317] In this embodiment, the detection unit 1717 includes multiple detection units with different computational loads and object determination accuracy. Specifically, the detection unit 1717 includes a DL detection unit 1722 that performs object determination processing using deep learning (DL) and a non-DL detection unit 1723 that performs object determination processing without using DL. It is assumed that the DL detection unit 1722 has higher processing accuracy than the non-DL detection unit 1723, but also has a greater computational load than the non-DL detection unit 1723.

[0318] The DL detection unit 1722 and the non-DL detection unit 1723 each detect candidate regions, associate object classes and their confidence levels with the types of the candidate regions, and count the number of candidate regions for each object class. The DL detection unit 1722 and the non-DL detection unit 1723 also obtain information about the sizes and locations of the detected candidate regions.

[0319] The DL detection unit 1722 can be implemented by a multi-class discriminator using machine learning. The DL detection unit 1722 can be implemented using multi-class logistic regression, support vector machine, random forest, neural network, etc.

[0320] The non-DL detection unit 1723 detects candidate areas using a known method that does not use machine learning. For example, the non-DL detection unit 1723 can detect candidate areas through pattern matching using templates prepared in advance for each object class, feature detection using feature information prepared in advance for each object class, or detection based on distance information.

[0321] Similar to the information superposition unit 120, the information superposition unit 1721 generates an image of a tracking frame based on the center position and size of the subject area to be tracked output by the detection unit 1717 or the tracking unit 1718. The information superposition unit 1721 then generates composite image data in which the image of the tracking frame is superimposed on the display image data output from the first post-processing unit 1719. The information superposition unit 1721 outputs the composite image data to the display unit 1731.

[0322] The nonvolatile memory 1753 is, for example, a rewritable nonvolatile memory such as an EEPROM, and stores programs executable by the CPU of the system control unit 1740 , setting values, GUI data, or the like.

[0323] The system memory 1754 is, for example, a RAM, and is used to load programs executed by the CPU of the system control unit 1740 and to store values ​​and the like required while the programs are being executed.

[0324] The system control unit 1740 includes a CPU, and, for example, loads programs stored in the nonvolatile memory 1753 into the system memory 1754 and executes the programs. The system control unit 1740 realizes the functions of the imaging apparatus 1700 by controlling the operations of the respective functional blocks.

[0325] The release button 1750 , the mode change switch 1751 , and the operation unit 1752 are input devices for inputting instructions to the system control unit 1740 .

[0326] The release button 1750 includes a switch SW1 that turns on when pressed halfway and a switch SW2 that turns on when pressed fully. The system control unit 1740 recognizes SW1 turning on as a still image shooting preparation instruction and SW2 turning on as a still image shooting start instruction, and performs operations according to the respective instructions.

[0327] In response to the shooting preparation instruction, the system control unit 1740 performs auto focus (AF) processing, auto exposure (AE) processing, auto white balance (AWB) processing, flash pre-emission (EF) processing, and the like.

[0328] In addition, in response to a shooting instruction, the system control unit 1740 uses the results of the AF processing and AE processing executed in response to the shooting preparation instruction to execute a still image shooting process. The still image shooting process includes control for opening / closing the shutter 1703, controlling the operation of the image sensor 1704, and generating recorded image data in the image processing unit 1710 and recording the recorded image data in the recording medium 1730. The system control unit 1740 controls various functional blocks related to this series of still image shooting processes.

[0329] The motion sensor 1760 outputs a signal according to the motion of the imaging apparatus 1700 and outputs the signal to the system control unit 1740. For example, the motion sensor 1760 may be an accelerometer or an angular velocity sensor. When the optical system 101 includes a motion sensor for image stabilization, the motion of the imaging apparatus 1700 may be detected using the motion sensor for image stabilization without providing the motion sensor 1760.

[0330] The mode change switch 1751 switches the operating mode of the system control unit 1740 between a still image shooting mode, a moving image shooting mode, and a playback mode, etc. The still image shooting mode includes an automatic shooting mode, an automatic scene determination mode, a manual mode, an aperture priority mode (Av mode), and a shutter speed priority mode (Tv mode) as modes. There are also various types of scene modes, program AE mode, and custom mode, etc., which are shooting settings for different shooting scenes. The mode change switch 1751 can be used to directly switch to any of these modes included in the menu button. Alternatively, mode switching can be performed by first switching to the menu button using the mode change switch 1751 and then switching to one of the modes included in the menu button using other operating components. Similarly, the moving image shooting mode can include multiple modes.

[0331] The "operation unit 1752" is a general term for input devices other than the release button 1750 and the mode change switch 1751. Functions are assigned to the input devices included in the operation unit 1752 according to the operating state of the imaging apparatus 1700, for example, by selecting a function icon displayed on the display unit 1731. The input devices are named according to the functions assigned to them.

[0332] For example, the operation unit 1752 includes a direction button, a setting button, an end button, a return button, a next image button, a jump button, a sort button, a property change button, and a menu button. For example, when the menu button is pressed, a menu screen in which various types of settings can be made is displayed on the display unit 1731. The user can make various types of settings by operating the menu screen displayed on the display unit 1731 using the direction buttons and the setting button.

[0333] In this embodiment, a visual line information obtaining unit (visual line detecting means) is provided as an input device included in the operation unit 1752. The visual line information obtaining unit detects the visual line of the user or the gaze position of the user within the display unit 1731. Figure 18 is a diagram illustrating an example of the configuration of the line of sight information obtaining unit 1800. Here, it is assumed that the display unit 1731 is provided in a peek-type viewfinder, and the position in the display unit 1731 that the user is looking at is detected by detecting the direction of the line of sight based on the rotation angle of the user's eyeball 1802 as it looks toward the viewfinder.

[0334] exist Figure 18 In FIG, the visual line information obtaining unit 1800 includes an image sensor 1801a, a light receiving lens 1801b, a dichroic mirror 1801c, and an illumination light source 1801e. The eyepiece 1801d is a viewfinder optical system and is not directly related to obtaining visual line information.

[0335] The user can observe the display on the display unit 1731 through the eyepiece 1801d and the dichroic mirror 1801c. The illumination light source 1801e projects infrared light toward the exterior of the viewfinder. When the user is looking directly into the viewfinder, the infrared light is reflected by the eyeball 1802, and the reflected light enters the viewfinder. The reflected light is reflected upward by the dichroic mirror 1801c, and the image sensor 1801a captures an image of the eyeball formed by the light-receiving lens 1801b. Image data for the eyeball image is transmitted from the image sensor 1801a to the system control unit 1740.

[0336] For example, the system control unit 1740 extracts a pupil area from the eye image and detects the eye's rotation angle from the position of the pupil area. The system control unit 1740 then obtains the user's gaze direction based on the eye's rotation angle. The gaze direction can be obtained by calculation or by referencing a table storing the relationship between rotation angle and gaze direction. Based on the gaze direction, the system control unit 1740 detects the position (coordinates) within the display unit 1731 that the user is gazing at. Note that the gaze position can be obtained based on the eye's rotation angle.

[0337] Here, it is assumed that a single eye is viewing the viewfinder, and line of sight information, representing both eyes' line of sight directions and gaze positions, is obtained for that single eye. However, if the environment allows for obtaining eyeball images of both eyes, line of sight information can be obtained for each eye. When obtaining line of sight information for both eyes, its accuracy can be improved by, for example, using a single instance of line of sight information that is considered to have higher detection accuracy.

[0338] Note that any other configuration may be used as long as the line of sight information obtaining unit 1800 can identify the position in the display unit 1731 that the user is looking at. Note also that the display unit 1731 does not absolutely need to be provided within the imaging device 1700. For example, the display unit 1731, the illumination light source 1801e, and the light-receiving lens 1801b may be provided on the back surface of the imaging device 1700.

[0339] Next, operations of the system control unit 1740 for controlling activation and deactivation of the DL detection unit 1722 and the non-DL detection unit 1723 based on the line of sight or gaze position detected by the line of sight information obtaining unit 1800 will be described.

[0340] Figure 19 : is a diagram showing functional blocks related to an operation for controlling activation / deactivation of a subject detection unit based on a line of sight or gaze position. Figure 19 The system control unit 1740 is illustrated as three functional blocks for simplicity, but actually the CPU of the system control unit 1740 realizes the functions of these functional blocks by executing programs.

[0341] The following will refer to Figure 20 The flowchart and Figure 17 and 19 The operation of the system control unit 1740 controlling activation / deactivation of the subject detection unit based on the line of sight or gaze position will be described.

[0342] S2000 indicates image acquisition processing. Figure 21 The flowchart in describes the image acquisition process in detail.

[0343] In S2100, the system control unit 1740 controls the image sensor 1704 and performs an image capturing operation for one frame. As a result, the subject optical image formed by the optical system 1701 is converted into RAW image data composed of a plurality of instances of pixel data.

[0344] In S2102, the first pre-processing unit 1711 of the image processing unit 1710 applies demosaicing to the RAW image data and generates image data in which each pixel has RGB components. The first pre-processing unit 1711 stores the generated image data in the video memory 1714. Note that the first pre-processing unit 1711 may convert the image data from RGB format to YUV format before storing the image data in the video memory 1714.

[0345] In S2103, the first image correction unit 1716 applies a predetermined image correction process to the image data stored in the video memory 1714. Note that the image correction process also includes an evaluation value calculation process. The first image correction unit 1716 can make the readout range of the image data different depending on the process to be applied.

[0346] For example, for image processing applied to image data of a row (pixel row), only the image data of the row to which the processing is applied may be read out, or image data of multiple rows including the row to which the processing is applied may be read out. Additionally, image processing may be applied to image data within a rectangular area of ​​a predetermined size. First image correction unit 1716 performs image correction processing while appropriately storing the image processing results in video memory 1714.

[0347] In step S2104, first image correction unit 1716 determines whether all image correction processing to be applied to the image data for one frame has been completed. If it is determined that all image correction processing to be applied has been completed, first image correction unit 1716 outputs the image data after image correction processing to first post-processing unit 1719 and outputs the evaluation value to system control unit 1740. This completes the image acquisition processing for one frame. On the other hand, if it is determined that image correction processing to be applied still remains, first image correction unit 1716 repeats step S2103.

[0348] Note that similar processing is performed by the second pre-processing unit 1712 , the tracking memory 1715 , and the second image correction unit 1713 , except that the image data is stored in the tracking memory 1715 and the second image correction unit 1713 applies image processing for tracking.

[0349] Back to Figure 20In S2001, the tracking unit 1718 applies tracking processing to the image data obtained in S2000 and stored in the tracking memory 1715. Note that it is assumed that the tracking unit 1718 applies the non-DL tracking processing applied by the non-DL tracking unit 117 in the first embodiment. The tracking unit 1718 outputs the position and size of the subject area to be tracked as a tracking result.

[0350] In S2002, the system control unit 1740 executes a subject loss determination process. Figure 22 The flowchart in describes the subject loss judgment process in detail.

[0351] In S2200, the system control unit 1740 obtains the user's line of sight information through the line of sight information obtaining unit 1800. The obtained line of sight information may be a line of sight direction or a gaze position.

[0352] In step S2201, the sight line analysis unit 1900 calculates the irregularity level of the sight line information. The irregularity level of the sight line information is an evaluation value for the stability of the user's gaze position. A high irregularity level indicates that the user's gaze position is unstable and is moving irregularly within the display unit 1731. Therefore, when the irregularity level of the sight line information is high, it is assumed that the user is unable to see the main subject and is searching on the screen.

[0353] Based on the irregularity level calculated in S2201 , the irregularity determination unit 1901 determines in S2003 whether the user cannot see the subject.

[0354] The irregularity level of the sight line information in S2201 can be calculated by any method. Additionally, a threshold value based on the irregularity level can be used to make the determination in S2003. The following is an example of a method for calculating and determining the irregularity level.

[0355] Use the frequency information of the gaze position or line of sight in the time axis direction

[0356] The frequency spectrum is calculated, and the amplitude of the spectrum of at least a given frequency is obtained as the irregularity level. When the irregularity level exceeds a threshold value, it is determined that the subject is not visible.

[0357] Calculate the absolute value of the autocorrelation of the gaze position or line of sight as the irregularity level

[0358] If the movement of the line of sight is regular, the autocorrelation is close to 1 in adjacent time spans, and if the movement of the line of sight is random, the autocorrelation is close to 0. When the irregularity level is less than the threshold, it is determined that the subject is not visible.

[0359] The absolute value of the cross-correlation between the average value of the magnitude of the velocity vector or acceleration vector of the gaze position or line of sight at a given time span T and the average value of the magnitude of the velocity vector or acceleration vector of the trajectory of each object area is calculated as the irregularity level.

[0360] When the irregularity level is smaller than the threshold value for all the subject regions, it is determined that the subject has been lost.

[0361] If the irregularity determination unit 1901 determines that the subject is lost in S2003, the process determination unit 1902 executes S2004. If the irregularity determination unit 1901 does not determine that the subject is lost in S2003, the process determination unit 1902 executes S2005.

[0362] In S2004, the processing determination unit 1902 determines whether to enable the DL detection unit 1722. The non-DL detection unit 1723 can be disabled or enabled. As described above, the DL detection unit 1722 in the image processing unit 1710 performs candidate region detection processing using machine learning (here, "DL"). If the DL detection unit 1722 detects only one candidate region, the system control unit 1740 determines the detected candidate region as the subject region to be tracked.

[0363] On the other hand, if the DL detection unit 1722 has detected multiple candidate areas, the system control unit 1740 determines the subject area to be tracked from the multiple candidate areas. The system control unit 1740 can determine the subject area to be tracked based on the size and position of the candidate area, etc., as described in the previous embodiment, or by other methods. Other methods are not particularly limited, but, for example, can enable the user to select the subject area to be tracked from the detected candidate areas. For example, an indicator similar to a tracking frame can be superimposed on each of the detected candidate areas, and the user can select the subject area to be tracked through the operation unit 1752. For example, the user can select the subject area to be tracked by operating the direction button and the setting button or by detecting the gaze position using the line of sight information acquisition unit 1800.

[0364] In S2005, the processing determination unit 1902 determines to deactivate the DL detection unit 1722 and activate the non-DL detection unit 1723. As described above, the non-DL detection unit 1723 in the image processing unit 1710 detects candidate areas using a method that does not use machine learning. In this case, it is determined that the user will not lose sight of the subject, and therefore the system control unit 1740 can determine the subject area to be tracked from among the candidate areas based on the line of sight information.

[0365] Note that, as in the first embodiment, the image processing unit 1710 may be provided with an object determination unit, and the object determination unit may determine the subject region to be tracked from among the candidate regions.

[0366] In S2006, the system control unit 1740 outputs the information about the subject region to be tracked to the tracking unit 1718 and the information superimposition unit 1721. As a result, the information about the subject region to be tracked is updated.

[0367] In S2007 , the system control unit 1740 deactivates both the DL detection unit 1722 and the non-DL detection unit 1723 to suppress power consumption.

[0368] In S2008 , the information superimposition unit 1721 generates composite image data in which the tracking frame indicating the subject region to be tracked updated in S2006 is superimposed on the display image data from the first post-processing unit 1719 , and outputs the composite image data to the display unit 1731 .

[0369] In this embodiment, whether the user cannot see the subject determines whether to use the first subject detection unit for detecting candidate areas for the subject area to be tracked, or the second subject detection unit, which has higher accuracy but a greater computational load than the first subject detection unit. Specifically, the second subject detection unit is used when the user cannot see the subject, and is not used when the user cannot lose sight of the subject. When the user cannot lose sight of the subject, the first subject detection unit can detect the candidate area with sufficient accuracy, and the second subject detection unit is therefore deactivated. This makes it possible to suppress power consumption while suppressing a decrease in detection accuracy.

[0370] Fourth embodiment

[0371] Next, a fourth embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are controlled to be enabled or disabled according to the auto focus (AF) mode of the imaging device. This embodiment can be implemented using an imaging device 1700 similar to the imaging device in the third embodiment, and thus will be used. Figure 17 This embodiment will be described with reference to the constituent elements shown.

[0372] Here, it is assumed that the imaging device 1700 has a single AF mode for retaining the focus detection result and a continuous AF mode for continuously performing the focus detection operation as AF modes. The single AF mode may be used to capture still or slightly moving subjects such as landscapes or portraits. On the other hand, the continuous AF mode may be used to capture moving subjects such as vehicles or sports scenes.

[0373] Therefore, the motion of the main subject that the user is trying to capture can be estimated from the set AF mode. The AF mode can be explicitly specified by the user, or can be automatically set according to the shooting mode of the camera 1700 (such as sports mode or landscape mode). The system control unit 1740 can also set the mode based on other conditions.

[0374] Note that the AF mode is a setting related to the focus detection operation when shooting a still image. However, subject tracking in the still image shooting mode is applied to moving images shot for live view display in a shooting standby state.

[0375] In this embodiment, when the single AF mode is used, the subject is likely to be stationary or moving slightly, which makes it relatively easy to detect the subject, and thus the DL detection unit 1722 is deactivated and the non-DL detection unit 1723 is activated. On the other hand, when the continuous AF mode is used, the subject is likely to be in motion or moving significantly, which makes it relatively difficult to detect the subject, and thus the DL detection unit 1722 is activated and the non-DL detection unit 1723 is deactivated. Such control makes it possible to suppress power consumption while suppressing a decrease in subject detection accuracy.

[0376] The following will refer to Figure 23 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 23 In the embodiment, steps for performing the same processing as that in the third embodiment are assigned to Figure 20 The same reference numerals are used in the same manner as in the third embodiment and will not be described again. The processing of S2000 and S2001 is the same as that described in the third embodiment.

[0377] In S2302, the system control unit 1740 determines whether the current AF mode of the imaging apparatus 1700 is the continuous AF mode. If the current AF mode of the imaging apparatus 1700 is determined to be the continuous AF mode, then in S2004, the system control unit 1740 activates the DL detection unit 1722. The non-DL detection unit 1723 may be deactivated to suppress power consumption, but may also be activated. On the other hand, if the current AF mode of the imaging apparatus 1700 is not determined to be the continuous AF mode, then in S2005, the system control unit 1740 activates the non-DL detection unit 1723 and deactivates the DL detection unit 1722.

[0378] The subsequent processing is the same as that in the third embodiment and will therefore not be described here.

[0379] In this embodiment, whether to use a first subject detection unit for detecting candidate areas for a subject area to be tracked, or a second subject detection unit, which has higher accuracy but a greater computational load than the first subject detection unit, is determined based on the autofocus mode of the imaging device. Specifically, the second subject detection unit is used when the autofocus mode is set, which is likely to be used for capturing moving subjects, and is not used when the autofocus mode is set, which is likely to be used for capturing stationary subjects. When the subject moves minimally, the first subject detection unit can accurately detect the candidate area, and therefore the second subject detection unit is deactivated. This makes it possible to suppress power consumption while suppressing a decrease in detection accuracy.

[0380] Fifth embodiment

[0381] Next, a fifth embodiment of the present invention will be described. In this embodiment, the DL detection unit 1722 and the non-DL detection unit 1723 are controlled to be enabled and disabled according to the frame rate of the moving image to which the tracking process is applied. This embodiment can be implemented using the camera apparatus 1700 similar to that in the third embodiment, and thus Figure 17 This embodiment will be described with reference to the constituent elements shown.

[0382] The amount of movement of the subject area between frames decreases as the frame rate increases. Consequently, when the frame rate is high, the amount of movement of the subject area to be tracked is low, making subject blur less likely. Therefore, sufficient accuracy may be achieved using the non-DL detection unit 1723. Therefore, it is less necessary to use the DL detection unit 1722 for detection of each frame.

[0383] Therefore, when the frame rate of the moving image to which tracking processing is applied is higher than the first threshold, the frequency of activating the DL detection unit 1722 is reduced compared to when the frame rate is not greater than the predetermined first threshold (fps). This makes it possible to reduce power consumption while suppressing a decrease in detection accuracy. Note that the first threshold can be determined experimentally, for example. Additionally, multiple thresholds can be set, and the frequency of activating the DL detection unit 1722 can be controlled in three or more stages.

[0384] The following will refer to Figure 24 The flowchart in FIG. 17 describes the operation of the system control unit 1740 in this embodiment. It is assumed here that the frame rate of the moving image to which the tracking process has been applied is higher than a predetermined first threshold value (fps). Additionally, in Figure 24 In the embodiment, steps for performing the same processing as that in the third embodiment are assigned to Figure 20 The same reference numerals are used in the same manner as in the third embodiment and will not be described again. The processing of S2000 and S2001 is the same as that described in the third embodiment.

[0385] In S2402, the system control unit 1740 determines whether the value of the variable COUNT, which serves as a counter for the number of frames, is a predetermined value N (an integer of at least 2). Assuming that the DL detection unit 1722 is enabled every M frames when the frame rate of the moving image to which tracking processing is applied is not greater than a first threshold (fps), the value of N can be determined to satisfy the relationship N > M. Here, it is assumed that M = 1 and N = 10. If the value of the variable COUNT is determined to be N, the system control unit 1740 executes S2403, and if not, executes S2404.

[0386] N=10, and therefore S2403 is executed once every 10 frames. In S2403, the system control unit 1740 resets the value of the variable COUNT to 1. Then, in S2004, the system control unit 1740 enables the DL detection unit 1722. The non-DL detection unit 1723 can be deactivated to suppress power consumption, but can also be enabled.

[0387] On the other hand, in S2404, the system control unit 1740 increases the value of the variable COUNT by 1. Then, in S2005, the system control unit 1740 enables the non-DL detection unit 1723 and disables the DL detection unit 1722. The subsequent processing is the same as that in the third embodiment and will not be described here.

[0388] This completes the operation for one frame in the live view display. Through the above processing, when the frame rate of the moving image to which the tracking process is applied is high, the DL detection unit 1722 is enabled for one frame out of every ten frames and is disabled for the remaining nine frames. The non-DL detection unit 1723 is enabled in the frames in which the DL detection unit 1722 is disabled.

[0389] In this embodiment, the subject detection unit used to detect candidate regions for a subject area to be tracked controls the frequency of using a second subject detection unit, which has higher accuracy but a greater computational load than the first subject detection unit, based on the frame rate of the moving image to which tracking processing is applied. Specifically, when the frame rate of the moving image to which tracking processing is applied is high, the frequency of activating the second subject detection unit is reduced compared to when the frame rate is low. When the subject moves minimally, the first subject detection unit can accurately detect the candidate region, and therefore the frequency of using the second subject detection unit is reduced. This allows for suppressing power consumption while minimizing a decrease in detection accuracy.

[0390] Sixth embodiment

[0391] Next, a sixth embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are controlled to be enabled or disabled according to the shutter speed of the imaging device. This embodiment can be implemented using an imaging device 1700 similar to the imaging device in the third embodiment, and thus Figure 17 This embodiment will be described with reference to the constituent elements shown.

[0392] When the shutter speed is fast, subject blur is less likely to occur. Therefore, sufficient accuracy may be achieved using the non-DL detection unit 1723. Therefore, when the shutter speed is fast, the DL detection unit 1722 is deactivated.

[0393] The following will refer to Figure 25 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 25 In the embodiment, steps for performing the same processing as that in the third embodiment are assigned to Figure 20 The same reference numerals are used in the same manner as in the third embodiment and will not be described again. The processing of S2000 and S2001 is the same as that described in the third embodiment.

[0394] In S2502, the system control unit 1740 determines whether the currently set shutter speed is faster than a threshold value. The threshold value used here can be determined experimentally based on the performance of the non-DL detection unit 1723, etc.

[0395] If it is determined that the shutter speed is faster than the threshold, in S2004 the system control unit 1740 enables the DL detection unit 1722. The non-DL detection unit 1723 may be deactivated to suppress power consumption, but may also be enabled.

[0396] On the other hand, if it is not determined that the shutter speed is faster than the threshold, in S2005 the system control unit 1740 enables the non-DL detection unit 1723 and deactivates the DL detection unit 1722. The subsequent processing is the same as that in the third embodiment and will therefore not be described here.

[0397] In this embodiment, whether to use the first subject detection unit for detecting candidate areas for the subject area to be tracked, or to use the second subject detection unit, which has higher accuracy but a greater computational load than the first subject detection unit, is determined based on the shutter speed of the imaging device. Specifically, the second subject detection unit is used when the shutter speed is not fast (slow), and is not used when the shutter speed is fast. When the shutter speed is fast, subject blur is less likely to occur, and the candidate area can be accurately detected by the first subject detection unit, so the second subject detection unit is deactivated. This makes it possible to suppress power consumption while suppressing a decrease in detection accuracy.

[0398] Seventh embodiment

[0399] Next, a seventh embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are controlled to be enabled or disabled in response to a user instruction. This embodiment can be implemented using a camera apparatus 1700 similar to the camera apparatus in the third embodiment, and thus will be used. Figure 17 This embodiment will be described with reference to the constituent elements shown.

[0400] In this embodiment, the DL detection unit 1722 is enabled only when the user enables the DL subject detection process. Any user instruction can be used to enable the DL subject detection process. The user does not need to intentionally instruct the DL subject detection process, and any instruction that effectively enables the DL subject detection process can be used.

[0401] Here, as an example, it is assumed that an input device (DL subject detection button) assigned with a function for switching DL subject detection processing between activation and deactivation is provided in the operation unit 1752. Each time the DL subject detection button is pressed, the system control unit 1740 switches the DL subject detection processing between activation and deactivation. Alternatively, the system control unit 1740 may activate the DL subject detection processing only while the DL subject detection button is being pressed. The DL subject detection button may also be a software key using the display unit 1731.

[0402] The following will refer to Figure 26 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 26 In the embodiment, steps for performing the same processing as that in the third embodiment are assigned to Figure 20 The same reference numerals are used in the same manner as in the third embodiment and will not be described again. The processing of S2000 and S2001 is the same as that described in the third embodiment.

[0403] In S2602, the system control unit 1740 determines whether the DL subject detection button is turned on (on). Here, when the button is being pressed, the DL subject detection button is determined to be on. If the DL subject detection button is determined to be on, the system control unit 1740 enables the DL detection unit 1722 in S2004. The non-DL detection unit 1723 can be deactivated to suppress power consumption, but the non-DL detection unit 1723 can also be enabled. On the other hand, if the DL subject detection button is not determined to be on, the system control unit 1740 enables the non-DL detection unit 1723 and deactivates the DL detection unit 1722 in S2005. The subsequent processing is the same as that in the third embodiment, and therefore will not be described here.

[0404] In this embodiment, in a subject detection unit for detecting candidate regions for subject areas to be tracked, a second subject detection unit, which has higher accuracy but a greater computational load than the first subject detection unit, is controlled to be activated or deactivated in response to a user instruction. Specifically, the second subject detection unit is activated only when a user instruction to activate it is detected. As a result, the second subject detection unit is deactivated when not required by the user, which reduces power consumption.

[0405] Eighth embodiment

[0406] Next, an eighth embodiment of the present invention will be described. In this embodiment, the DL detection unit and the non-DL detection unit are controlled to be enabled or disabled according to the operating situation of the device. This embodiment can be implemented using a camera device 1700 similar to the camera device in the third embodiment, and thus will use Figure 17 This embodiment will be described with reference to the constituent elements shown.

[0407] Specifically, when it is determined that a pan, tilt, or zoom operation, which is used to change the imaging range, is in progress, DL detection unit 1722 is deactivated. This is because while these operations are in progress, the user can make adjustments to bring the main subject into the imaging range. Consequently, even if DL detection unit 1722 is enabled during these operations, meaningful detection results may not be obtained. In this embodiment, DL detection unit 1722 may be enabled only when it is determined that no pan, tilt, or zoom operation is in progress in imaging device 1700.

[0408] The following will refer to Figure 27 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 27 In the embodiment, steps for performing the same processing as that in the third embodiment are assigned to Figure 20 The same reference numerals are used in the same manner as in the third embodiment and will not be described again. The processing of S2000 and S2001 is the same as that described in the third embodiment.

[0409] In S2702, the system control unit 1740 determines whether a pan, tilt, or zoom operation is being performed in the imaging apparatus 1700. For example, the system control unit 1740 can detect a pan, tilt, or zoom operation by detecting a change in a motion vector or a change in the size of a feature region between frames of a moving image being captured for live view display, or based on the output of the motion sensor 1760. A zoom operation can also be detected from a zoom button of the operating unit 1752 being operated.

[0410] If it is determined that a pan, tilt, or zoom operation is being performed, the system control unit 1740 executes S2008. On the other hand, if it is not determined that a pan, tilt, or zoom operation is being performed, the system control unit 1740 executes S2702.

[0411] In S2702, the system control unit 1740 activates the DL detection unit 1722 or the non-DL detection unit 1723. When the DL detection unit 1722 is activated, the non-DL detection unit 1723 may be deactivated to suppress power consumption, but the non-DL detection unit 1723 may also be activated. For example, at least one of the conditions described in the third to seventh embodiments may be used as the condition for activating the DL detection unit 1722. If the condition for activating the DL detection unit 1722 is not met, the system control unit 1740 deactivates the DL detection unit 1722 and activates the non-DL detection unit 1723. The processing from S2006 onwards is the same as that in the third embodiment and will not be described again here.

[0412] In this embodiment, the subject detection unit used to detect candidate regions for subject areas to be tracked can activate a second subject detection unit, which has higher accuracy but a greater computational load than the first subject detection unit, only when an operation to change the imaging range is not in progress. Ensuring that the second subject detection unit is not activated when the imaging range has not been finalized allows for efficient power consumption reduction. Furthermore, ensuring that the first subject detection unit is not activated when the imaging range has not been finalized allows for even greater power consumption reduction.

[0413] Variations

[0414] Note that the activation and deactivation of DL subject detection and non-DL subject detection may be controlled based on conditions different from those described in the third to eighth embodiments above. For example, the activation and deactivation of DL subject detection and non-DL subject detection may be controlled based on the shooting mode (still image shooting mode or moving image shooting mode), the pressed state of the release button (not pressed, half pressed, or fully pressed), the state of the tracking mode, or the type of the tracked object.

[0415] Ninth embodiment

[0416] Next, a ninth embodiment of the present invention will be described. Figure 28 1700′ is a block diagram showing an example of a functional configuration of an image pickup apparatus 1700′ serving as an example of an image processing apparatus according to the ninth embodiment. Figure 17The same configurations as in the described imaging apparatus 1700 are given the same reference numerals, and redundant description thereof will not be given. Note that the detection unit 1717 of this embodiment may be either the DL detection unit 1722 or the non-DL detection unit 1723. Here, it is assumed that it is the same as the non-DL detection unit 1723.

[0417] The imaging apparatus 1700 ′ of the present embodiment includes an object determination unit 2801 having a new configuration, and similarly to the first embodiment, the tracking unit 1718 includes a DL tracking unit 2802 and a non-DL tracking unit 2803 .

[0418] Similar to the object determination unit 111 of the first embodiment, the object determination unit 2801 determines a subject region to be tracked (primary subject region) from the candidate regions detected by the detection unit 1717. If one candidate region is detected, the object determination unit 2801 determines the detected candidate region as the primary subject region. If multiple candidate regions are detected, the object determination unit 2801 may determine the primary subject region based on a predetermined priority order.

[0419] The object determination unit 2801 can determine the primary subject area based on, for example, a priority order based on the size of the candidate areas or a priority order based on the distance from the focus detection area. Specifically, the largest candidate area can be considered the primary subject area, or the candidate area closest to the focus detection area can be considered the primary subject area. Additionally, the primary subject area can be determined based on multiple priority orders, for example, by considering the largest candidate area among the candidate areas whose distance from the focus detection area is less than a threshold as the primary subject area. Priority orders other than the ones given as examples here can also be considered.

[0420] Tracking unit 1718 estimates the subject region to be tracked based on the image data of the target frame (current frame) stored in tracking memory 1715, and obtains the estimated position and size of the subject region within the frame as a tracking result. For example, tracking unit 115 may estimate the position and size of the main subject region in the current frame, as determined by object determination unit 2801 for a past frame captured before the current frame (e.g., the previous frame). Tracking unit 115 outputs the tracking result to information superposition unit 1721.

[0421] The tracking unit 1718 includes a DL tracking unit 2802 for performing subject tracking using deep learning (DL) and a non-DL tracking unit 2803 for performing subject tracking without using DL. The configurations of the DL tracking unit 2802 and the non-DL tracking unit 2803 can be the same as the configurations of the DL tracking unit 116 and the non-DL tracking unit 117 described in the first embodiment, respectively, and therefore will not be described in detail. In this embodiment, the system control unit 1740 controls whether to enable or disable the DL tracking unit 2802 and the non-DL tracking unit 2803, and the operating frequency when enabled.

[0422] In this embodiment, the DL tracking unit 2802 outputs the estimated position and size of the main subject area and the confidence level score as the tracking result. The non-DL tracking unit 2803 outputs the estimated position and size of the main subject area and the similarity score as the tracking result.

[0423] The tracking unit 1718 adopts the tracking result from the DL tracking unit 2802 or the non-DL tracking unit 2803 based on the confidence level score output by the DL tracking unit 2802 and the similarity score output by the non-DL tracking unit 2803. For example, if the confidence level score is not greater than a predetermined confidence level score threshold and the similarity score is not greater than a predetermined similarity score threshold, the tracking unit 1718 adopts the tracking result from the non-DL tracking unit 2803. In other cases, the tracking unit 1718 adopts the tracking result from the DL tracking unit 2802. The tracking unit 1718 outputs the adopted tracking result to the information superposition unit 1721 and the system control unit 1740.

[0424] Here, the determination of which of the tracking results from the DL tracking unit 2802 and the tracking results from the non-DL tracking unit 2803 to adopt is made based on the confidence level score and the similarity score. However, this determination can be made by other methods. For example, the tracking results from the DL tracking unit 2802 can be preferentially adopted by using the fact that the accuracy of the DL tracking unit 2802 tends to be higher than the accuracy of the non-DL tracking unit 2803. Specifically, if the tracking results from the DL tracking unit 2802 are obtained, the tracking results from the DL tracking unit 2802 can be adopted, and if not, the tracking results from the non-DL tracking unit 2803 can be adopted.

[0425] Next, the control operation of the DL tracking unit 2802 and the non-DL tracking unit 2803 performed by the system control unit 1740 in this embodiment will be described. In this embodiment, the system control unit 1740 controls whether the DL tracking unit 2802 or the non-DL tracking unit 2803 is enabled according to the AF mode set in the imaging apparatus 1700'.

[0426] Here, similar to the fourth embodiment, it is assumed that the camera 1700' has a single AF mode and a continuous AF mode. The single AF mode may be used to capture still or slightly moving subjects such as landscapes or portraits. On the other hand, the continuous AF mode may be used to capture moving subjects such as vehicles or sports scenes.

[0427] In single AF mode, the subject may be stationary or moving very little. Therefore, the non-DL tracking unit is activated instead of the DL tracking unit. On the other hand, in continuous AF mode, the subject may be in motion or moving significantly. Therefore, the DL tracking unit is activated instead of the non-DL tracking unit. This control makes it possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0428] Figure 29 is a flowchart related to the operation of the system control unit 1740 according to the present embodiment.

[0429] S2900 is the same as that in the third embodiment. Figure 21 The processing described is similar to the image acquisition processing.

[0430] In step S2901, the system control unit 1740 instructs the detection unit 1717 to detect candidate regions. The detection unit 1717 applies subject region detection processing to the image of a single frame and stores the detection results in the tracking memory 1715. The detection results include the total number of detected candidate regions and the position and size of each candidate region within the image.

[0431] In step S2902, the system control unit 1740 instructs the object determination unit 2801 to determine a subject region (primary subject region) to be tracked from the candidate regions. The object determination unit 2801 reads information about the candidate regions stored in the tracking memory 1715, and when multiple candidate regions have been detected, determines the primary subject region based on the priority order as described above. The object determination unit 2801 stores the information used to identify the primary subject region in the tracking memory 1715.

[0432] In S2903, the system control unit 1740 determines whether the AF mode currently set in the imaging apparatus 1700' is the continuous AF mode. If the currently set AF mode is determined to be the continuous AF mode, the system control unit 1740 executes S2904, and if the currently set AF mode is not determined to be the continuous AF mode, the system control unit 1740 executes S2905.

[0433] In S2904, the system control unit 1740 activates the DL tracking unit 2802 and deactivates the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 performs tracking processing on the current frame.

[0434] On the other hand, in S2905, the system control unit 1740 activates the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 performs tracking processing on the current frame.

[0435] Note that the main subject region used in the tracking processing of S2904 and S2905 is the main subject region most recently determined for the past frame relative to the current frame.

[0436] In S2906 , the system control unit 1740 uses the result of the tracking process performed in S2904 or S2905 to update the information about the main subject area stored in the tracking memory 1715 . The result of the tracking process is also output to the information superposition unit 1721 .

[0437] In S2907 , the system control unit 1740 deactivates the DL tracking unit 2802 and the non-DL tracking unit 2803 .

[0438] At S2908, system control unit 1740 instructs information overlay unit 1721 to overlay and display the tracking frame based on the tracking results on the display image of the current frame. Information overlay unit 1721 generates an image with the tracking frame at a position and size based on the tracking results, and generates composite image data to combine with the display image data of the current frame. Information overlay unit 1721 outputs the composite image data to display unit 1731.

[0439] This completes the operation for one frame in the live view display. In this embodiment, in an imaging apparatus using a first tracking unit and a second tracking unit that has higher processing accuracy but a greater computational load than the first tracking unit, whether the first tracking unit and / or the second tracking unit are activated or deactivated is controlled according to the AF mode. Therefore, by deactivating a tracking unit that is unlikely to achieve good tracking results, deactivating the other tracking unit when good tracking results are likely to be achieved only from one of the processing units, and so on, it is possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0440] Tenth embodiment

[0441] The following will describe a tenth embodiment of the present invention. In this embodiment, the DL tracking unit 2802 and the non-DL tracking unit 2803 are controlled to be enabled and disabled according to the frame rate of the moving image to which the tracking process is applied. This embodiment can be implemented using a camera apparatus 1700' similar to the camera apparatus in the ninth embodiment, and thus Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0442] The amount of change in the position and size of the subject region between frames decreases as the frame rate increases. As such, when the frame rate is high, the tracking process of the non-DL tracking unit 2803 is considered sufficiently accurate. In this case, the DL tracking unit 2802 is unlikely to be necessary. However, using the DL tracking unit 2802 at a certain frequency allows the accuracy of the tracking process to be more reliably maintained.

[0443] Therefore, when the frame rate of the moving image to which tracking processing is applied is higher than the first threshold, the frequency of activating the DL tracking unit 2802 is reduced compared to when the frame rate is no greater than a predetermined first threshold (fps). This makes it possible to reduce power consumption while suppressing a decrease in detection accuracy. Note that the first threshold can be determined experimentally, for example. Additionally, multiple thresholds can be set, and the frequency of activating the DL tracking unit 2802 can be controlled in three or more stages.

[0444] The following will refer to Figure 30 The flowchart in FIG. 17 describes the operation of the system control unit 1740 in this embodiment. It is assumed here that the frame rate of the moving image to which the tracking process has been applied is higher than a predetermined first threshold value (fps). Additionally, in Figure 30 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0445] In S3001, the system control unit 1740 determines whether the value of a variable COUNT, which serves as a counter for the number of frames, is a predetermined value N (an integer of at least 2). Assuming that the DL tracking unit 2802 is enabled every M frames when the frame rate of the moving image to which tracking processing is applied is not greater than a first threshold (fps), the value of N can be determined to satisfy the relationship N > M. Here, it is assumed that M = 1 and N = 10. If the value of the variable COUNT is determined to be N, the system control unit 1740 executes S3002, and if not, executes S3003.

[0446] N=10, and therefore S3002 is performed once every 10 frames. In S3002, the system control unit 1740 resets the value of the variable COUNT to 1. Then, in S2904, the system control unit 1740 enables the DL tracking unit 2802. The non-DL tracking unit 2803 can be deactivated to suppress power consumption, but can also be enabled.

[0447] On the other hand, in S3003, the system control unit 1740 increases the value of the variable COUNT by 1. Then, in S2905, the system control unit 1740 enables the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. The subsequent processing is the same as that in the ninth embodiment and will not be described here.

[0448] This completes the operation for one frame in the live view display. Through the above processing, when the frame rate of the moving image to which the tracking process is applied is high, the DL tracking unit 2802 is enabled for one frame in every ten frames and is disabled for the remaining nine frames. The non-DL tracking unit 2803 is enabled in the frames in which the DL tracking unit 2802 is disabled.

[0449] In this embodiment, the frequency of using the second tracking unit, which has higher accuracy but a greater computational load than the first tracking unit, is controlled based on the frame rate of the moving image to which tracking processing is applied. Specifically, when the frame rate of the moving image to which tracking processing is applied is high, the frequency of activating the second tracking unit is reduced compared to when the frame rate is low. When the subject moves minimally, the first tracking unit can accurately track the subject, and therefore the frequency of using the second tracking unit is reduced. This allows for suppressing a decrease in detection accuracy while also reducing power consumption.

[0450] Eleventh embodiment

[0451] Next, an eleventh embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled according to the shutter speed of the imaging device. This embodiment can be implemented using an imaging device 1700' similar to the imaging device in the ninth embodiment, and thus Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0452] Similar to when the frame rate is high, as the shutter speed increases when capturing moving images, the amount of change in the position and size of the subject area between frames decreases. Therefore, it may be possible to achieve sufficiently accurate tracking processing using the non-DL tracking unit 2803. Therefore, when the shutter speed is fast, the DL tracking unit 2802 is deactivated.

[0453] The following will refer to Figure 31 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 31 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 19 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0454] In S3101, the system control unit 1740 determines whether the currently set shutter speed is faster than a threshold value. The threshold value used here can be determined experimentally based on the performance of the non-DL tracking unit 2803, etc.

[0455] If it is determined that the shutter speed is faster than the threshold, in S2904 the system control unit 1740 enables the DL tracking unit 2802. The non-DL tracking unit 2803 may be deactivated to suppress power consumption, but may also be enabled.

[0456] On the other hand, if it is not determined that the shutter speed is faster than the threshold, in S2905 the system control unit 1740 enables the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. The subsequent processing is the same as that in the third embodiment and is therefore not described here.

[0457] In this embodiment, the shutter speed of the imaging device determines whether to use the first tracking unit or the second tracking unit, which has higher processing accuracy but a greater computational load than the first tracking unit. Specifically, the second tracking unit is used when the shutter speed is not fast (slow), and is not used when the shutter speed is fast. When the shutter speed is fast, the position and size of the subject area hardly change between frames. Therefore, accurate tracking processing can be achieved by the first tracking unit, and the second tracking unit can be deactivated. This makes it possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0458] Twelfth embodiment

[0459] Next, a twelfth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled in response to a user instruction. This embodiment can be implemented using a camera apparatus 1700' similar to the camera apparatus in the ninth embodiment, and thus will be used. Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0460] In this embodiment, the DL tracking unit 2802 is enabled only when the user enables the DL tracking process. Any user instruction can be used to enable the DL tracking process. The user does not need to intentionally instruct the DL tracking process, and any instruction that effectively enables the DL tracking process can be used.

[0461] Here, as one example, it is assumed that an input device (DL tracking button) to which a function for switching the DL tracking process between activation and non-activation is assigned is provided in the operation unit 1752 . Figure 391700' is a diagram illustrating an example of the appearance of the back side of the imaging device 1700'. Here, it is assumed that the display unit 1731 is provided on the back side of the imaging device 1700'. Several input devices 3901 to 3903 constituting the operation unit 1752 are provided on the back side of the imaging device 1700'. The input device 3903, which enables direction input and confirmation input, functions as a directional key and an OK button. Input devices 3901 and 3902 can also function as a DL tracking button.

[0462] Whenever the DL tracking button is pressed, the system control unit 1740 switches DL tracking between enabled and disabled. Alternatively, the system control unit 1740 may enable DL tracking only while the DL tracking button is being pressed. The DL tracking button may also be a software key using the display unit 1731.

[0463] The following will refer to Figure 32 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 32 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0464] In S3201, the system control unit 1740 determines whether the DL tracking button is on. Here, when the DL tracking button is being pressed, the DL tracking button is determined to be on. If the DL tracking button is determined to be on, then in S2904, the system control unit 1740 enables the DL tracking unit 2802. The non-DL tracking unit 2803 can be deactivated to suppress power consumption, but it can also be enabled. On the other hand, if the DL tracking button is not determined to be on, then in S2905, the system control unit 1740 enables the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. The subsequent processing is the same as that in the ninth embodiment and will not be described here.

[0465] In this embodiment, in a tracking unit for tracking a target subject area, whether to activate or deactivate a second subject tracking unit, which has higher accuracy but a greater computational load than the first tracking unit, is controlled in response to a user instruction. Specifically, the second tracking unit is activated only when a user instruction to activate it is detected. As a result, the second tracking unit is deactivated when not required by the user, which makes it possible to suppress power consumption.

[0466] Thirteenth embodiment

[0467] The following will describe a thirteenth embodiment of the present invention. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled according to the operating situation of the device. This embodiment can be implemented using a camera device 1700' similar to the camera device in the ninth embodiment, and thus will use Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0468] Specifically, when it is determined that a pan, tilt, or zoom operation is being performed to change the imaging range, DL tracking unit 2802 is deactivated. This is because while such an operation is in progress, the user can make adjustments to bring the primary subject within the imaging range. Consequently, even if DL tracking unit 2802 is enabled during such an operation, meaningful detection results may not be obtained. In this embodiment, DL tracking unit 2802 may be enabled only when it is determined that a pan, tilt, or zoom operation is not being performed in imaging device 1700'.

[0469] The following will refer to Figure 33 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 33 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0470] At S3301, the system control unit 1740 determines whether a pan, tilt, or zoom operation is being performed on the imaging apparatus 1700'. For example, the system control unit 1740 can detect a pan, tilt, or zoom operation by detecting changes in motion vectors or the size of feature regions between frames of a moving image captured for live view display, or based on the output of the motion sensor 1760. A zoom operation can also be detected by operating a zoom button on the operating unit 1752.

[0471] If it is determined that a pan, tilt, or zoom operation is being performed, the system control unit 1740 executes S2908. On the other hand, if it is not determined that a pan, tilt, or zoom operation is being performed, the system control unit 1740 executes S3302.

[0472] In S3302, the system control unit 1740 activates the DL tracking unit 2802 or the non-DL tracking unit 2803. When the DL tracking unit 2802 is activated, the non-DL detection unit 1723 may be deactivated to suppress power consumption, but the non-DL detection unit 1723 may also be activated. For example, at least one of the conditions described in the ninth to twelfth embodiments may be used as a condition for activating the DL tracking unit 2802. If the condition for activating the DL tracking unit 2802 is not satisfied, the system control unit 1740 deactivates the DL tracking unit 2802 and activates the non-DL tracking unit 2803. The processing from S2906 onwards is the same as that in the ninth embodiment and is therefore not described here.

[0473] In this embodiment, in the tracking unit for tracking the subject area to be tracked, a second tracking unit, which has higher accuracy but a greater computational load than the first tracking unit, can be activated only when an operation to change the imaging range is not in progress. Ensuring that the second tracking unit is not activated when the imaging range has not been finalized effectively reduces power consumption.

[0474] Fourteenth embodiment

[0475] Next, a fourteenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled according to the shooting mode of the camera 1700'. This embodiment can be implemented using a camera 1700' similar to the camera in the ninth embodiment, and thus will be used. Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0476] The camera 1700′ has multiple shooting modes, including a still image shooting mode and a moving image shooting mode. Note that the shooting mode can be set according to the shooting scene and the special effects to be applied, such as night scene mode, sports mode, and filter mode. Here, the shooting mode that ultimately captures a still image will be considered a still image shooting mode, and the shooting mode that captures a moving image will be considered a moving image shooting mode.

[0477] The following will refer to Figure 34 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 34 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0478] In S3401, the system control unit 1740 determines whether the shooting mode of the imaging apparatus 1700' is the still image shooting mode, for example, based on the state of the mode change switch 1751. If the shooting mode of the imaging apparatus 1700' is determined to be the still image shooting mode, the system control unit 1740 executes S3402. On the other hand, if the shooting mode of the imaging apparatus 1700' is not determined to be the still image shooting mode, the system control unit 1740 executes S3403.

[0479] In S3402, the system control unit 1740 executes S3405 and sets the count value as the first value (N1). N1 is an integer of 1 or greater.

[0480] In S3403, the system control unit 1740 executes S3405 and sets the count value as the second value (N2). N2 is an integer of 1 or greater.

[0481] In S3405, the system control unit 1740 determines whether the value of the variable COUNT used as a counter of the frame number is the count setting value (N1 or N2). If the value of the variable COUNT is determined to be the count setting value, the system control unit 1740 executes S3406, and if not, the system control unit 1740 executes S3407.

[0482] In S3406, the system control unit 1740 resets the value of the variable COUNT to 1. Then, in S2904, the system control unit 1740 enables the DL tracking unit 2802. The non-DL tracking unit 2803 may be deactivated to suppress power consumption, but may also be enabled.

[0483] On the other hand, in S3407 , the system control unit 1740 increments the value of the variable COUNT by 1. Then, in S2905 , the system control unit 1740 activates the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802 .

[0484] Whenever the variable COUNT becomes N1 or N2, S2904 is executed. In other words, the DL tracking unit 2802 is enabled once every N1 frames in the still image shooting mode and once every N2 frames in the moving image shooting mode, and is deactivated for the remaining frames.

[0485] According to the shooting mode, the first value and the second value control the frequency for activating the DL tracking unit 2802. For example, the first value and the second value are pre-stored in the nonvolatile memory 1753. Note that the first value and the second value may also be determined in consideration of shutter speed or frame rate, etc.

[0486] For example, assume that the frame rate of a moving image to which tracking processing is applied is higher in moving image shooting mode than in still image shooting mode. In this case, the frequency of activating the DL tracking unit 2802 in moving image shooting mode is reduced compared to still image shooting mode, which makes it possible to suppress power consumption while suppressing a decrease in tracking accuracy. Note that this is merely an example, and by independently controlling the frequency of activating the DL tracking unit 2802 for each shooting mode, the DL tracking unit 2802 can be activated at a frequency suitable for the shooting mode.

[0487] In this embodiment, in a tracking unit for tracking a subject area to be tracked, the frequency of activating a second tracking unit, which has higher accuracy but a greater computational load than the first tracking unit, is controlled according to the shooting mode. Setting an appropriate count setting value according to the shooting mode makes it possible to reduce power consumption while suppressing a decrease in tracking performance.

[0488] Fifteenth embodiment

[0489] The fifteenth embodiment of the present invention will be described below. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled according to the state of the pressing state of the release button. This embodiment can be implemented using a camera apparatus 1700' similar to the camera apparatus in the ninth embodiment, and thus will be used. Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0490] Release button 1750 includes a switch SW1 that turns on when halfway pressed and a switch SW2 that turns on when fully pressed, and thus has a state in which both switches are off, a state in which only switch SW1 is on, and a state in which both switches SW1 and SW2 are on. System control unit 1740 recognizes that SW1 is turned on as a still image shooting preparation instruction, and recognizes that SW2 is turned on as a still image shooting start instruction.

[0491] In this embodiment, if the release button 1750 is not in a half-pressed or fully pressed state, the state is a shooting standby state, and therefore the DL tracking unit 2802 is deactivated to suppress power consumption. In this way, in this embodiment, the DL tracking unit 2802 can be activated only when a shooting preparation instruction or a shooting start instruction is being input to the imaging device 1700'.

[0492] The following will refer to Figure 35 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 35 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0493] In S3501, the system control unit 1740 determines whether the release button 1750 is in a half-pressed or fully pressed state. For example, if the switch SW1 is on, the system control unit 1740 can determine that the release button 1750 is in a half-pressed or fully pressed state. When the switch SW2 is on, the switch SW1 is also on, and therefore it is not necessary to detect the state of the switch SW2.

[0494] If it is determined that the release button 1750 is in the half-pressed or fully-pressed state, the system control unit 1740 executes S3502.

[0495] On the other hand, if it is determined that the release button 1750 is not in the half-pressed or fully-pressed state, the system control unit 1740 executes S2908. In this case, the state is the shooting standby state, and therefore the target frame is not tracked (or it can be tracked by the non-DL tracking unit 2803).

[0496] In S3502, the system control unit 1740 activates the DL tracking unit 2802 or the non-DL tracking unit 2803. When the DL tracking unit 2802 is activated, the non-DL tracking unit 2803 may be deactivated to suppress power consumption, but the non-DL tracking unit 2803 may also be activated. For example, at least one of the conditions described in the ninth to fourteenth embodiments may be used as a condition for activating the DL tracking unit 2802. If the condition for activating the DL tracking unit 2802 is not satisfied, the system control unit 1740 deactivates the DL tracking unit 2802 and activates the non-DL tracking unit 2803. The condition for activating the DL tracking unit 2802 may differ depending on whether the release button is in a half-pressed or fully pressed state. The processing from S2906 onwards is the same as that in the ninth embodiment and will therefore not be described again here.

[0497] In this embodiment, in the tracking unit used to track the subject area to be tracked, a second tracking unit, which has higher accuracy but a greater computational load than the first tracking unit, can be activated only when a shooting preparation command or a shooting start command is being input. As a result, in a shooting standby state where neither a shooting preparation command nor a shooting start command is being input, the second tracking unit is not activated, effectively suppressing power consumption. Furthermore, ensuring that the first tracking unit is not activated in the shooting standby state also allows for even greater power consumption suppression.

[0498] Sixteenth embodiment

[0499] The following will describe a sixteenth embodiment of the present invention. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled according to the tracking mode of the camera 1700'. This embodiment can be implemented using a camera 1700' similar to the camera in the ninth embodiment, and thus will be used. Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0500] Tracking mode is a mode in which focus control is continuously performed so that the main subject remains in focus even if it moves from the center of the imaging range. This mode is applied when capturing subjects that continuously move within the imaging range or subjects whose movements are difficult to predict (such as small animals). Tracking mode can be set using the mode change switch 1751 or by operating one of the input devices included in the operation unit 1752. Note that the subject focused on in tracking mode may be the subject corresponding to the main subject area determined by the object determination unit 2801. In addition to explicitly setting tracking mode, tracking mode can also be set indirectly in conjunction with the specific function being set.

[0501] When the tracking mode is set, the focused subject is considered likely to move, and thus the DL tracking unit 2802 is enabled. On the other hand, when the tracking mode is not set, the focused subject is considered unlikely to move, and thus the DL tracking unit 2802 is deactivated and the non-DL tracking unit 2803 is enabled.

[0502] The following will refer to Figure 36 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 36 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0503] In S3601, the system control unit 1740 determines whether the tracking mode is set in the imaging apparatus 1700'. For example, the system control unit 1740 may determine whether the tracking mode is set based on the state of the mode change switch 1751.

[0504] If it is determined that the tracking mode is set, the system control unit 1740 executes S2904 , and if it is determined that the tracking mode is not set, the system control unit 1740 executes S2905 .

[0505] In S2904, the system control unit 1740 activates the DL tracking unit 2802 and deactivates the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 performs tracking processing on the current frame.

[0506] On the other hand, in S2905, the system control unit 1740 activates the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 performs tracking processing on the current frame.

[0507] The processing from S2906 onwards is the same as that in the ninth embodiment and therefore will not be described here.

[0508] In this embodiment, in a tracking unit for tracking a subject area to be tracked, when a tracking mode is set, a second tracking unit having higher accuracy but a greater computational load than the first tracking unit is enabled, and when the tracking mode is not set, the second tracking unit is disabled. Deactivating the second tracking unit when the second tracking unit is deemed less necessary makes it possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0509] Seventeenth embodiment

[0510] Next, a seventeenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled and disabled depending on whether the main subject to be tracked is an easy-to-track subject. This embodiment can be implemented using a camera apparatus 1700' similar to the camera apparatus in the ninth embodiment, and thus will be used. Figure 28 This embodiment will be described with reference to the constituent elements shown.

[0511] Similar to the detection unit 110 in the first embodiment, it is assumed that the detection unit 1717 of this embodiment associates an object class indicating the subject type and its confidence level with each detected candidate region. If the main subject region to be tracked is an easy-to-track subject, the non-DL tracking unit 2803 may be able to achieve sufficient accuracy, and therefore the DL tracking unit 2802 is deactivated.

[0512] The following will refer to Figure 37 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 37 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals are used and will not be described again. The processing from S2900 to S2902 is the same as that described in the ninth embodiment.

[0513] In S3701, the system control unit 1740 determines whether the type of subject belonging to the main subject area is an easy-to-track subject based on the object class associated with the main subject area determined by the object determination unit 2801. The system control unit 1740 can determine whether the type of subject belonging to the main subject area is an easy-to-track subject by referring to the object class and the tracking difficulty stored in, for example, the non-volatile memory 1753. Here, it is assumed that the tracking difficulty is the difficulty of non-DL tracking.

[0514] Non-DL tracking is difficult for subjects with shapes and / or colors that vary significantly depending on the angle or state (e.g., whether they are flying), such as birds and insects. Note that subjects that are difficult to track with non-DL tracking also depend on the algorithm used for non-DL tracking, and therefore, subjects are not limited to subjects with shapes and / or colors that vary significantly. Note also that the association between object classes and tracking difficulty can be made in advance based on the object classes that can be detected by the detection unit 1717 and the performance of the non-DL tracking unit.

[0515] If the type of the subject belonging to the main subject area is a subject that is easy to track, the system control unit 1740 executes S2905 , and if not, executes S2904 .

[0516] In S2904, the system control unit 1740 activates the DL tracking unit 2802 and deactivates the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 performs tracking processing on the current frame.

[0517] On the other hand, in S2905, the system control unit 1740 activates the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 performs tracking processing on the current frame.

[0518] The processing from S2906 onwards is the same as that in the ninth embodiment and therefore will not be described here.

[0519] In this embodiment, in a tracking unit for tracking a subject area to be tracked, when a subject belonging to the subject area to be tracked is a subject that can be easily tracked by the first tracking unit, a second tracking unit, which has higher accuracy but a greater computational load than the first tracking unit, is deactivated. Deactivating the second tracking unit when the second tracking unit is deemed less necessary makes it possible to suppress power consumption while suppressing a decrease in tracking accuracy.

[0520] Eighteenth embodiment

[0521] Next, an eighteenth embodiment of the present invention will be described. In this embodiment, the DL tracking unit and the non-DL tracking unit are controlled to be enabled or disabled according to whether the user cannot see the subject. This embodiment can be implemented using a camera 1700' similar to the camera in the ninth embodiment, and thus will be used. Figure 28 However, as in the case of the imaging apparatus 1700 according to the third embodiment, it is assumed that the imaging apparatus 1700′ is provided with Figure 18 The visual line information obtaining unit 1800 is shown as an input device included in the operation unit 1752. It is also assumed that the system control unit 1740 has Figure 19 Function blocks shown.

[0522] The following will refer to Figure 38 The flowchart in describes the operation of the system control unit 1740 in this embodiment. Figure 38 In the embodiment, steps for performing the same processing as that in the ninth embodiment are assigned to Figure 29 The same reference numerals as in the third embodiment are given to steps for performing the same processing as that in the third embodiment. Figure 20 The same reference numerals are used in the figure, and these steps will not be described. The processing from S2900 to S2902 is the same as the processing described in the ninth embodiment.

[0523] In S2002, the system control unit 1740 executes a reference Figure 19 and Figure 22 The subject loss judgment process described.

[0524] In S2003 , the system control unit 1740 (irregularity determination unit 1901 ) determines whether the user cannot see the subject based on the irregularity level calculated in the loss determination process.

[0525] If the irregularity determination unit 1901 determines that the subject is lost in S2003, the process determination unit 1902 executes S2904. If the irregularity determination unit 1901 does not determine that the subject is lost in S2003, the process determination unit 1902 executes S2905.

[0526] In S2904, the system control unit 1740 (processing determination unit 1902) activates the DL tracking unit 2802 and deactivates the non-DL tracking unit 2803. As a result, the DL tracking unit 2802 performs tracking processing on the current frame.

[0527] Additionally, in S2905, the system control unit 1740 (processing determination unit 1902) activates the non-DL tracking unit 2803 and deactivates the DL tracking unit 2802. As a result, the non-DL tracking unit 2803 performs tracking processing on the current frame.

[0528] The processing from S2906 onwards is the same as that in the ninth embodiment and therefore will not be described here.

[0529] In this embodiment, the tracking unit used to track the area of ​​a subject to be tracked determines whether to use the first tracking unit or the second tracking unit, which has higher processing accuracy but a greater computational load than the first tracking unit, depending on whether the user cannot see the subject. Specifically, the second tracking unit is used when the user cannot see the subject, and is not used when the user cannot lose sight of the subject. When the user cannot lose sight of the subject, the first tracking unit can perform tracking processing with sufficient accuracy, and the second tracking unit is therefore deactivated. This makes it possible to suppress power consumption while preventing a decrease in tracking accuracy.

[0530] Other embodiments

[0531] The above embodiments may also be implemented in combination with each other. For example, at least two of the detection unit, the object determination unit, and the tracking unit may have two components that implement the same function but have different computational loads and accuracies, and whether these components are enabled or disabled may be dynamically controlled according to the embodiment.

[0532] In the above embodiment, the frame periods in which DL tracking and non-DL tracking are enabled may be different, and both DL tracking and non-DL tracking are enabled in a period that is the least common multiple of the respective periods.

[0533] The present invention can also be implemented as a process performed by supplying a program for implementing one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and then causing one or more processors of a computer of the system or device to read and execute the program. The present invention can also be implemented as a circuit (e.g., an ASIC) for implementing one or more functions.

[0534] The present invention is not limited to the above embodiments, and various changes and modifications can be made within the spirit and scope of the present invention. Therefore, in order to inform the public of the scope of the present invention, the following claims are made.

Claims

1. An image processing device, comprising: a detecting component for detecting one or more regions of a predetermined subject as candidate regions in a target frame of a moving image; A determining component, configured to determine a subject region to be tracked from the candidate regions; a calculation component for calculating an evaluation value related to the difficulty of tracking based on the object area determined by the determination component; a tracking component configured to perform a tracking process for estimating a position in the target frame of an area corresponding to an object area to be tracked in a frame preceding the target frame, the tracking component comprising a first tracking component and a second tracking component; as well as a control unit configured to determine, based on the evaluation value, an operation of the first tracking unit on a frame subsequent to the target frame and an operation of the second tracking unit on a frame subsequent to the target frame, wherein the second tracking component has higher processing accuracy and greater computational load than the first tracking component, and The control unit enables both the first tracking unit and the second tracking unit, or disables at least one of the first tracking unit and the second tracking unit, based on at least one of a frame rate of the moving image, a shutter speed used to capture the moving image, and whether a pan, tilt, or zoom operation is being performed to capture the moving image, and the evaluation value.

2. The image processing apparatus according to claim 1, wherein The calculation section calculates the evaluation value based on at least one of a size of the object region, a position of the object region, a type of the object, and a total number of object regions related to the same type of object.

3. The image processing apparatus according to claim 1 or 2, wherein: The higher the evaluation value, the more difficult the tracking process is, and When the evaluation value is at least a first threshold value, the control component deactivates the first tracking component and activates the second tracking component.

4. The image processing apparatus according to claim 1 or 2, wherein: The higher the evaluation value, the more difficult the tracking process is, and When the evaluation value is less than a second threshold, the control component deactivates the second tracking component and activates the first tracking component.

5. The image processing apparatus according to claim 1 or 2, wherein: When both the first tracking component and the second tracking component are enabled, the control component reduces the operating frequency of the second tracking component as compared to the operating frequency of the first tracking component.

6. The image processing apparatus according to claim 1 or 2, wherein: When the subject area determined by the determination component and the subject area whose position is estimated by the tracking component involve the same subject, the control component does not change the operation of the first tracking component on the subsequent frames and the operation of the second tracking component on the subsequent frames relative to the operation of the first tracking component on the object frame and the operation of the second tracking component on the object frame.

7. The image processing apparatus according to claim 1 or 2, wherein: When the subject region determined by the determination section and the subject region whose position is estimated by the tracking section relate to different subjects, the control section determines operations of the first tracking section and the second tracking section on subsequent frames based on the evaluation value.

8. The image processing apparatus according to claim 1 or 2, wherein: The second tracking section performs the tracking process using a neural network.

9. An image processing device comprising: a detecting component for detecting one or more regions of a predetermined subject as candidate regions in a target frame of a moving image; A determining component, configured to determine a subject region to be tracked from the candidate regions; a tracking component configured to perform a tracking process for estimating a position in the target frame of an area corresponding to an object area to be tracked in a frame preceding the target frame, the tracking component comprising a first tracking component and a second tracking component; as well as a control component for determining, based on the type of the subject detected by the detection component, an operation of the first tracking component on a subsequent frame of the object frame and an operation of the second tracking component on a subsequent frame of the object frame, wherein the second tracking component has higher processing accuracy and greater computational load than the first tracking component, and The control component enables both the first tracking component and the second tracking component, or disables at least one of the first tracking component and the second tracking component based on the frame rate of the moving image, the shutter speed used to capture the moving image, whether a pan, tilt or zoom operation is being performed to capture at least one of the moving images, and the type of the subject detected by the detection component.

10. A method for controlling an image processing device, wherein: The image processing apparatus includes a tracking unit configured to perform tracking processing for estimating a position in a target frame of a moving image of an area corresponding to an area of ​​a subject to be tracked in a frame preceding the target frame of the moving image, the tracking unit including a first tracking unit and a second tracking unit, the second tracking unit having higher processing accuracy and a larger computational load than the first tracking unit, and the control method including: a detecting step for detecting one or more regions of a predetermined subject as candidate regions in the target frame of the motion image; A determination step for determining a subject area to be tracked from the candidate areas; a calculating step for calculating an evaluation value related to difficulty of tracking based on the object area determined in the determining step; and a control step for controlling an operation of the first tracking means on a frame subsequent to the target frame and an operation of the second tracking means on a frame subsequent to the target frame based on the evaluation value, The control step enables both the first tracking component and the second tracking component, or disables at least one of the first tracking component and the second tracking component, based on the frame rate of the moving image, the shutter speed used to capture the moving image, and whether a pan, tilt, or zoom operation is being performed to capture the moving image, and the evaluation value.

11. A method for controlling an image processing device, wherein: The image processing apparatus includes a tracking unit configured to perform tracking processing for estimating a position in a target frame of a moving image of an area corresponding to an area of ​​a subject to be tracked in a frame preceding the target frame of the moving image, the tracking unit including a first tracking unit and a second tracking unit, the second tracking unit having higher processing accuracy and a larger computational load than the first tracking unit, and the control method including: a detecting step for detecting one or more regions of a predetermined subject as candidate regions in the target frame of the motion image; a determining step for determining a subject region to be tracked from the candidate regions; and a control step for controlling the operation of the first tracking component on subsequent frames of the object frame and the operation of the second tracking component on subsequent frames of the object frame based on the type of the object detected in the detection step, In which, the control step enables both the first tracking component and the second tracking component, or disables at least one of the first tracking component and the second tracking component based on the frame rate of the motion image, the shutter speed used to capture the motion image, whether a pan, tilt or zoom operation is being performed to capture at least one of the motion images, and the type of the subject detected in the detection step. 12 . A computer-readable storage medium storing a program for causing a computer included in an image processing apparatus to function as each component included in the image processing apparatus according to claim 1 .

13. An image processing device comprising: a detecting component for detecting one or more regions of a predetermined subject as candidate regions in a target frame of a moving image; a calculation section for calculating, based on the detection result of the candidate region, an evaluation value related to the difficulty of a determination process for determining a region to be subjected to tracking processing from among the candidate regions; an object determination component, comprising a first determination component and a second determination component, wherein the object determination component is configured to perform the determination process on the candidate area detected by the detection component; as well as a control section for controlling an operation of the first determination section and an operation of the second determination section based on the evaluation value, wherein the second determining component has higher processing accuracy and greater computational load than the first determining component, and The control section enables both the first determination section and the second determination section, or disables one of the first determination section and the second determination section, or disables both the first determination section and the second determination section, based on at least one of a frame rate of the moving image, a shutter speed for capturing the moving image, and whether a pan, tilt, or zoom operation is being performed to capture the moving image, and the evaluation value.

14. The image processing apparatus according to claim 13, wherein: The calculation section calculates the evaluation value based on at least one of a size, a position, and a type of each of the candidate regions and a total number of candidate regions associated with the same type of subject.

15. The image processing apparatus according to claim 13 or 14, wherein: The higher the evaluation value, the higher the difficulty of the determination process, and When the evaluation value is at least a first threshold value, the control section enables the second determination section.

16. The image processing apparatus according to claim 13 or 14, wherein: The higher the evaluation value, the higher the difficulty of the determination process, and When the evaluation value is smaller than a second threshold value, the control section deactivates the second determination section and activates the first determination section.

17. The image processing apparatus according to claim 13 or 14, wherein: The higher the evaluation value, the higher the difficulty of the determination process, and When the evaluation value is smaller than a third threshold value, the control section deactivates both the first determination section and the second determination section.

18. The image processing apparatus according to claim 13 or 14, further comprising: a sight line detection component for detecting the sight line of a user of the image processing device, wherein the first determining means performs the determining process based on the line of sight detected by the line of sight detecting means, and When the second determining means is activated, the control means deactivates the first determining means.

19. The image processing apparatus according to claim 13 or 14, wherein: The second determination section performs the determination process using a pre-trained neural network.

20. An image processing device comprising: a detecting component for detecting one or more regions of a predetermined subject as candidate regions in a target frame of a moving image; an object determining unit configured to perform a determination process for determining an area to be subjected to tracking processing from among the candidate areas detected by the detecting unit, the object determining unit comprising a first determining unit and a second determining unit, the second determining unit having higher accuracy and a larger computational load than the first determining unit; as well as a control component for controlling the operation of the first determining component and the operation of the second determining component based on the detection result of the candidate area, In which, the control component enables both the first determination component and the second determination component, or disables one of the first determination component and the second determination component, or disables both the first determination component and the second determination component, based on the frame rate of the motion image, the shutter speed used to capture the motion image, whether a pan, tilt or zoom operation is being performed to capture at least one of the motion images, and the type and number of detected candidate areas.

21. The image processing apparatus according to claim 20, wherein When the total number of candidate areas of a predetermined type among the detected candidate areas is less than a first threshold, the control section deactivates the second determination section and activates the first determination section.

22. The image processing apparatus according to claim 20 or 21, wherein: The control section enables the second determination section when the total number of first-type candidate areas among the detected candidate areas is at least a first threshold and the total number of second-type candidate areas is at least a second threshold.

23. The image processing apparatus according to claim 20 or 21, wherein: When the total number of any type of candidate areas among the detected candidate areas is smaller than a third threshold value, the control section deactivates both the first determination section and the second determination section.

24. The image processing apparatus according to claim 20 or 21, wherein: Also includes: a sight line detection component for detecting the sight line of a user of the image processing device, wherein the first determining means performs the determining process based on the line of sight detected by the line of sight detecting means, and When the second determining means is activated, the control means deactivates the first determining means.

25. The image processing apparatus according to claim 20 or 21, wherein: The second determination section performs the determination process using a pre-trained neural network.

26. A method for controlling an image processing device, comprising: a detecting step, performed by a detecting means, for detecting one or more regions of a predetermined subject as candidate regions in a target frame of a moving image; a calculation step, performed by a calculation means, for calculating an evaluation value related to difficulty of a determination process for determining a region to be subjected to tracking processing from among the candidate regions, based on a detection result of the candidate regions; as well as a control step, performed by a control means, for controlling an operation of an object determination means for performing the determination processing included in the image processing apparatus based on the evaluation value, wherein the object determining component includes a first determining component and a second determining component, the second determining component having higher accuracy and greater computational load than the first determining component, and The controlling step enables both the first determining means and the second determining means, or disables one of the first determining means and the second determining means, or disables both the first determining means and the second determining means, based on at least one of a frame rate of the moving image, a shutter speed for capturing the moving image, and whether a panning, tilting, or zooming operation is being performed to capture the moving image, and the evaluation value.

27. A method for controlling an image processing device, comprising: a detecting step, performed by a detecting means, for detecting one or more regions of a predetermined subject as candidate regions in a target frame of a moving image; as well as a control step, performed by a control unit, for controlling the operation of an object determination unit included in the image processing apparatus based on a detection result of the candidate area, wherein the object determination unit is configured to perform determination processing for determining an area to be subjected to tracking processing from among the candidate areas, wherein the object determining component includes a first determining component and a second determining component, the second determining component having higher accuracy and greater computational load than the first determining component, and The control step enables both the first determining component and the second determining component, or disables one of the first determining component and the second determining component, or disables both the first determining component and the second determining component, based on the frame rate of the moving image, the shutter speed used to capture the moving image, whether a pan, tilt or zoom operation is being performed to capture at least one of the moving images, and the type and number of detected candidate areas.

28. A computer-readable storage medium storing a program for causing a computer to function as each component included in the image processing apparatus according to any one of claims 13 to 25.

29. A computer program product comprising a program, wherein when the program is executed by a processor, the program implements the steps of the method for controlling an image processing apparatus according to any one of claims 10, 11, 26, and 27.

Images