Image processing device, imaging device, and image processing method

Abstract

To provide a technology for detecting a tracking target again from a captured image in consideration of the intention of a user.SOLUTION: An image processing apparatus registers a degree of occupancy which indicates a ratio of an image region of a tracking target in an image region of the tracking target in a captured image or in an image region of an object to which the tracking target belongs, and feature quantities of the tracking target. When it is determined that tracking of the tracking target in the captured image is unsuccessful, the image processing apparatus detects the image region of the tracking target again from the captured image.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to a technology for re-detecting a detection area of ​​a tracking target from a captured image. [Background technology]

[0002] Object detection processing, which detects objects from images, is being applied to the functions of imaging devices such as digital cameras. Until now, object detection processing has mostly been performed on objects of specific categories, such as human faces and facial organs (eyes, nose, mouth), or the entire human body. In recent years, with the development of deep learning, technology has been realized to detect objects of unspecified categories (hereinafter referred to as unspecified objects), such as animals and vehicles, by learning object-likeness using information on objects of various categories.

[0003] In digital cameras, object detection processing is applied to autofocus (AF) technology, which automatically focuses on a detected object as the subject. One AF technology is a tracking function that continuously focuses on the same subject. The tracking function is a function that identifies the tracking target in successive images, but if the tracking target is obscured by another object and becomes invisible, the tracking target will disappear. If the tracking target disappears, it must be redetected. Patent Document 1 discloses a technology for redetecting the tracking target when it disappears, in which the size of the search range is expanded and a search is performed again based on the tracking feature quantities of the tracking target. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Laid-Open No. 2009-17271 Summary of the Invention [Problem to be solved by the invention]

[0005] However, when identification is performed using only tracking features, if the tracking target is an animal or other object with a similar texture throughout, and the object size changes before and after redetection, the tracking target (tracked portion) may change. Furthermore, if redetection is performed when the tracking target is lost due to an unspecified object, many unspecified objects will be detected, making it difficult to detect the original tracking target from among them as intended by the user. The present invention provides a technology for redetecting a tracking target from a captured image while taking into account the user's intention. [Means for solving the problem]

[0006] One aspect of the present invention is characterized by comprising a registration means for registering an occupancy indicating the proportion of the image area of ​​the tracking target in the image area of ​​the tracking target in the captured image or the image area of ​​the object to which the tracking target belongs, and a feature amount of the tracking target, and a redetection means for redetecting the image area of ​​the tracking target from the captured image based on the occupancy amount and the feature amount when it is determined that tracking of the tracking target in the captured image has not been successful. [Effects of the Invention]

[0007] According to the present invention, it is possible to redetect a tracking target from a captured image while taking into account the user's intention. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an imaging device 100. [Figure 2] 1A is a block diagram showing an example of the hardware configuration of an imaging device 100, and FIG. 1B is a block diagram showing an example of the hardware configuration of an image analyzing device 200 and a learning device 700. FIG. [Figure 3] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 100. [Figure 4] 4 is a flowchart of processing performed by the imaging device 100. [Figure 5] 10 is a flowchart showing details of the process in step S401. [Figure 6]FIG. 1 is a diagram showing an example of the configuration of a neural network. [Figure 7] FIG. 10 is a diagram illustrating the occupancy rate of a detection region. [Figure 8] FIG. 10 is a diagram showing an example of a captured image. [Figure 9] FIG. 4 is a diagram showing an example of occupancy rates and tracking features stored in a storage unit 218. [Figure 10] FIG. 10 is a diagram showing an example of a tracking feature. [Figure 11] FIG. 7 is a block diagram showing an example of the functional configuration of a learning device 700. [Figure 12] 10 is a flowchart of a neural network learning process performed by the learning device 700. [Figure 13] FIG. 1 is a diagram illustrating a method for creating learning data. [Figure 14] FIG. 1A is a diagram showing an example of a first image, and FIG. 1B is a diagram showing an example of a second image. [Figure 15] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging device 100. [Figure 16] FIG. 10 is a diagram showing an example of a captured image. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0010] [First embodiment] In this embodiment, an imaging device will be described that acquires images of each frame in a moving image or still images captured periodically or irregularly as captured images, and performs a tracking process to track a tracking target (part or all of an object) detected / redetected from the captured images, and an AF process to automatically focus on the tracking target as a subject.

[0011] 1, the imaging device 100 according to this embodiment includes an image analysis device 200 that performs various analysis processes on the captured image and detects / redetects the "image region of the tracking target." Also, a learning device 700 that performs learning processes on the neural network used by the image analysis device 200 to perform the above-mentioned operations is connected to the imaging device 100.

[0012] First, an example of the hardware configuration of the image capture device 100 will be described using the block diagram in Fig. 2(a). Note that Fig. 2(a) shows the main configuration related to the following description, and does not exclude the image capture device 100 from including devices other than those shown in Fig. 2(a).

[0013] The arithmetic processing device 101 includes a processor such as a CPU (Central Processing Unit) and / or a GPU (Graphics Processing Unit), and a memory having a work area for the processor. The arithmetic processing device 101 executes various processes using computer programs and data stored in the storage device 102, thereby controlling the operation of the imaging device 100 (including the image analysis device 200) and the learning device 700.

[0014] The storage device 102 is a storage device such as a magnetic storage device, a semiconductor memory, etc. The storage device 102 stores computer programs and data for causing the arithmetic processing device 101 to control the operation of the imaging device 100 (including the image analysis device 200) and the learning device 700. The storage device 102 can also store captured images as files.

[0015] The imaging unit 105 includes a lens, an aperture, an imaging element such as a CCD or CMOS that converts light from the outside world into an analog signal, an A / D converter that converts the analog signal into a digital signal, and a generation circuit that generates a captured image based on the digital signal. The imaging unit 105 also includes an aperture control device, a focus control device, and the like. In the imaging unit 105, the imaging element photoelectrically converts light incident through the lens into an analog signal, the A / D converter converts the analog signal into a digital signal, and the generation circuit generates and outputs a captured image based on the digital signal. The imaging unit 105 also performs functions such as AF, AE, and AWB in response to instructions from the arithmetic processing unit 101. Still images captured periodically or irregularly by the imaging unit 105, or images of each frame in a moving image captured by the imaging unit 105, are stored in the storage device 102 as captured images.

[0016] The image analysis device 200 detects / redetects the tracking target from the captured image generated by the imaging unit 105. The arithmetic processing device 101 controls the imaging unit 105 to perform a tracking process for tracking the tracking target detected / redetected by the image analysis device 200 and an AF process for automatically focusing on the tracking target as a subject.

[0017] The input device 103 is a user interface such as a mouse, keyboard, touch panel device, or button, and can be operated by a user to input various instructions to the arithmetic processing device 101 .

[0018] The output device 104 is a device having a display screen such as a liquid crystal panel, and displays the processing results of the arithmetic processing device 101 on the display screen in the form of images, characters, etc. In this embodiment, a touch panel screen is configured by overlaying the input device 103 as a touch panel device on the display screen of the output device 104 as a liquid crystal panel. The touch panel screen displays the processing results of the arithmetic processing device 101 in the form of images, characters, etc., and also accepts operational inputs from the user.

[0019] 2(a), the arithmetic processing unit 101, the storage device 102, the imaging unit 105, the image analysis device 200, the input device 103, and the output device 104 are all connected to a system bus 107. The imaging device 100 may have, for example, an I / O unit for communicating between various devices. The I / O unit is, for example, an input / output unit such as a memory card or a USB cable, or a wired or wireless transmission / reception unit.

[0020] Next, an example of the hardware configuration of the image analyzing device 200 and the learning device 700 will be described using the block diagram in FIG. 2(b). For simplicity of explanation, this embodiment will be described assuming that the image analyzing device 200 and the learning device 700 have the same hardware configuration (FIG. 2(b)). However, the hardware configurations of the image analyzing device 200 and the learning device 700 may be different.

[0021] The arithmetic processing device 130 includes a processor such as a CPU and / or a GPU, and a memory having a work area for the processor. The arithmetic processing device 130 controls the overall operation of the image analyzing device 200 / learning device 700 by executing various processes using computer programs and data stored in the storage device 131.

[0022] The storage device 131 is a storage device such as a magnetic storage device, a semiconductor memory, etc. The storage device 131 stores computer programs and data for causing the arithmetic processing device 131 to control the operation of the image analysis device 200 / learning device 700.

[0023] The I / F 132 is a communication interface for performing data communication with an external device via a wired and / or wireless network. The I / F 132 of the image analyzing device 200 is a communication interface for performing data communication with the learning device 700. The I / F 132 of the learning device 700 is a communication interface for performing data communication with the image analyzing device 200. The arithmetic processing device 130, the storage device 131, and the I / F 132 are all connected to a system bus 133.

[0024] Next, an example of the functional configuration of the imaging device 100 will be described using the block diagram of FIG. 3. In the following, each functional unit shown in FIG. 3 may be described as the subject of processing. However, in reality, the functions of each functional unit shown in FIG. 3 except for the tracking unit 219, the AF processing unit 220, and the storage unit 218 are realized by the arithmetic processing device 130 executing a computer program that causes the arithmetic processing device 130 to realize the functions of the functional units. Similarly, the functions of the tracking unit 219 and the AF processing unit 220 are realized by the arithmetic processing device 101 executing a computer program that causes the arithmetic processing device 101 to realize the functions of the tracking unit 219 and the AF processing unit 220 among the functional units shown in FIG. 3.

[0025] The acquisition unit 210 acquires the captured image generated by the imaging unit 105. For example, the acquisition unit 210 acquires the full HD (1920 pixels×1280 pixels) captured image generated by the imaging unit 105 in real time (60 frames per second).

[0026] The acquisition unit 230 acquires information relating to an object or a part of the object (head, arm, leg, etc.) in the captured image acquired by the acquisition unit 210. An example of the functional configuration of the acquisition unit 230 is shown in the block diagram of FIG.

[0027] The extraction unit 211a extracts feature amounts (object feature amounts) from the captured image acquired by the acquisition unit 210. The estimation unit 212a uses the object feature amounts extracted from the captured image by the extraction unit 211a to estimate (detect) the entire image area of ​​the object in the captured image or a partial image area of ​​the object as a detection area. As a result, for each detection area estimated from the captured image, the estimation unit 212a acquires the position of the detection area in the captured image (center position, position of the upper left corner, etc.), the size of the detection area (vertical size and horizontal size), and a likelihood that the target included in the detection area is an object. Hereinafter, "feature amount" is synonymous with "feature vector" or "image feature."

[0028] For each detection area estimated from the captured image by the estimation unit 212a, the estimation unit 213a estimates the proportion of the detection area in the image area of ​​the object in the captured image as the occupancy rate, using the object feature amount extracted from the captured image by the extraction unit 211a.

[0029] The extraction unit 214a extracts a feature amount (tracking feature amount) corresponding to each detection area estimated from the captured image by the estimation unit 212a. The selection unit 240 selects a detection area of ​​a tracking target from the detection areas estimated from the captured image by the estimation unit 212a. The selection unit 240 then stores (registers) in the storage unit 218 the occupancy rate estimated by the estimation unit 213a for the selected detection area and the tracking feature amount extracted by the extraction unit 214a for the selected detection area. The selection unit 240 includes a selection unit 215, an input unit 216, and an input unit 217.

[0030] The input unit 216 displays the captured image acquired by the acquisition unit 210 on the display screen of the output device 104, and accepts a user operation to specify the position of the tracking target in the captured image. When the user operates the input device 103 to specify the position of the tracking target in the captured image, the input unit 216 acquires the image coordinates of the position.

[0031] The input unit 217 acquires the range of occupancy of the tracking target (occupancy range) input by the user operating the input device 103. The selection unit 215 selects the detection area of ​​the tracking target from the detection areas estimated from the captured image by the estimation unit 212a, based on the image coordinates acquired by the input unit 216 and the occupancy range acquired by the input unit 217. Then, the selection unit 215 stores (registers) in the storage unit 218 the occupancy estimated by the estimation unit 213a for the detection area of ​​the tracking target and the tracking feature amount extracted by the extraction unit 214a for the detection area of ​​the tracking target.

[0032] The tracking unit 219 performs tracking processing to track the tracking target in the captured image acquired by the acquisition unit 210, using the tracking feature amount of the detection area of ​​the tracking target stored in the storage unit 218 and the detection area of ​​the tracking target selected by the selection unit 215 or the image area of ​​the tracking target re-detected by the re-detection unit 250. Since the tracking processing is well known, a detailed description of the tracking processing will be omitted.

[0033] The AF processing unit 220 performs AF processing on the image area of ​​the tracking target that is being tracked by the tracking unit 219 in the captured image acquired by the acquisition unit 210. Since the AF processing is well known, a detailed description of the AF processing will be omitted.

[0034] The determination unit 221 determines whether or not the tracking process of the tracking target in the tracking unit 219 has been successful. The redetection unit 250 redetects the tracking target from the captured image when the determination unit 221 determines that "the tracking process of the tracking target in the tracking unit 219 has not been successful (has failed)." An example of the functional configuration of the redetection unit 250 is shown in the block diagram of Fig. 3(c).

[0035] 3(c), the extraction unit 211b, the estimation unit 212b, the estimation unit 213b, and the extraction unit 214b operate in the same manner as the extraction unit 211a, the estimation unit 212a, the estimation unit 213a, and the extraction unit 214a, respectively. That is, like the acquisition unit 230, the re-detection unit 250 acquires the occupancy rate and the tracking feature amount for each detection area from the captured image.

[0036] The processing unit 222 compares the occupancy and tracking feature acquired for each detection area with the occupancy and tracking feature of the detection area of ​​the tracking target stored in the memory unit 218, and identifies (redetects) the detection area of ​​the tracking target among the respective detection areas.

[0037] Next, the processing performed by the image capturing device 100 to perform tracking processing will be described with reference to the flowchart in Fig. 4. In step S401, processing is performed to store the occupancy and tracking feature amount for a detection region of the tracking target, which is part of the whole or part of an object included in the captured image, in the storage unit 218. Details of the processing in step S401 will be described with reference to the flowchart in Fig. 5.

[0038] In step S501, the acquisition unit 210 acquires a captured image generated by the imaging unit 105. This captured image is, for example, bitmap data of an RGB color image in which the pixel values ​​of R (red), G (green), and B (blue) at each pixel are all expressed in 8 bits.

[0039] In step S502, the extraction unit 211a extracts object features from the captured image acquired by the acquisition unit 210 in step S501. Various methods can be applied to extract object features from the captured image, but in this embodiment, the extraction unit 211a extracts object features from the captured image using a neural network. An example configuration of a neural network used to extract object features from the captured image is shown in FIG.

[0040] The extraction unit 211a is a neural network that performs a recognition task by repeating convolutional layers and pooling layers. The extraction unit 211a has multiple convolutional layers 511, 513, and 515 and multiple pooling layers 512 and 514, and extracts object features from an input image (captured image) 530 using these layers.

[0041] In the convolution layer, for example, a 3x3 size filter is set for multiple channels for the input image or feature map, a convolution operation is performed centered on the pixel of interest, and multiple feature maps 551, 553, 555 corresponding to the multiple channels are output.

[0042] The pooling layer generates reduced feature maps 552 and 554 by reducing the feature maps output from the convolution layer. When pooling is performed within a 2x2 range, the feature maps are reduced to 1 / 4 of their original size. Methods such as maximum value pooling and average value pooling can be used for pooling.

[0043] The configuration of the neural network applicable to the extraction unit 211a is not limited to the configuration shown in FIG. 6, and for example, the neural network may have more layers than the neural network shown in FIG. 6, or the number of channels may be changed.

[0044] In step S503, the estimation unit 212a estimates, as a detection area, the entire image area of ​​the object in the captured image or a partial image area of ​​the object, using the object feature amount extracted from the captured image by the extraction unit 211a in step S502. In this estimation, the estimation unit 212a estimates, for each detection area estimated from the captured image, the position of the detection area, the size of the detection area, and a likelihood representing the object-likeness of the detection area.

[0045] In this embodiment, the estimation unit 212a also estimates a detection area from a captured image using a neural network. Taking Fig. 6 as an example, by inputting a feature map 555 to a fully connected layer 556, the position, size, and likelihood of the detection area (detection frame) are output, thereby realizing estimation of the detection area.

[0046] In step S504, the estimation unit 213a estimates the occupancy of each detection region estimated from the captured image by the estimation unit 212a in step S503. In this embodiment, the estimation unit 213a also estimates the occupancy of each detection region using a neural network. Taking FIG. 6 as an example, the feature map 555 is input to the fully connected layer 556, and the occupancy of the detection region is output.

[0047] Here, the occupancy will be explained in more detail. The occupancy of a detection area is a degree that indicates to what extent the detection area captures the image area of ​​an object in a captured image. The occupancy of a detection area will be explained using a specific example shown in FIG. 7. In FIG. 7, a dog is used as an example of an object.

[0048] In FIG. 7, the captured image 600 includes an image area 601 of a dog, and the estimation unit 212a estimates a detection area 602 that includes the entire body of the dog and a detection area 603 that includes the head, which is part of the dog.

[0049] The detection area 602 captures the entire image area 601, and the proportion of the detection area 602 in the image area 601 is 100%, so the estimation unit 213a estimates that the occupancy rate of the detection area 602 is "1.0".

[0050] The detection area 603 captures a part of the image area 601, and if the detection area 603 occupies 20% of the image area 601, the estimation unit 213a estimates that the occupancy rate of the detection area 603 is "0.2".

[0051] In step S505, the extraction unit 214a extracts tracking features corresponding to each detection region estimated from the captured image by the estimation unit 212a in step S503. In this embodiment, the extraction unit 214a also extracts tracking features for each detection region from the captured image using a neural network. Taking FIG. 6 as an example, the extraction unit 214a acquires a feature map in map format, such as feature map 555, as tracking features. Here, the map size of the tracking features is set to 1×1×C (C is an arbitrary natural number) in terms of width, height, and channels. In this embodiment, tracking features extracted by a neural network will be described; however, this is not limiting; luminance values, RGB values, their histograms, SIFT features, SURF features, etc. may also be used as tracking features.

[0052] In step S506, the selection unit 215 selects the detection area of ​​the tracking target from the detection areas estimated from the captured image by the estimation unit 212a in step S503, based on the image coordinates acquired by the input unit 216 and the occupancy range acquired by the input unit 217.

[0053] The process of selecting detection areas for tracking by the selection unit 215 will be described below using Fig. 8(a) as an example. In a captured image 610, detection areas 611 to 616 are each detection areas estimated from the captured image 610 by the estimation unit 212a.

[0054] Detection area 611 is the detection area for the dog, and detection area 612 is the detection area for the head, which is part of the dog. Detection area 613 is the detection area for the tree, detection area 614 is the detection area for the group of flowers, which is part of the tree, and detection areas 615 and 616 are the detection areas for the flowers in the group of flowers. Point 617 indicates the designated position designated by the user using input device 103 as the position of the tracking target.

[0055] The input unit 216 acquires image coordinates corresponding to the point 617. Furthermore, the input unit 217 acquires, as an occupancy range, the "range that is acceptable as the occupancy of the tracking target" that the user inputs by operating the input device 103.

[0056] The selection unit 215 selects, from among the detection areas 611 to 616, a detection area whose occupancy is included in the occupancy range and which includes the image coordinates of point 617, as the detection area of ​​the tracking target. If there are multiple "detection areas whose occupancy is included in the occupancy range and which include the image coordinates of point 617," the selection unit 215 selects, from among the multiple detection areas, the detection area that is closest to the image coordinates of point 617, as the detection area of ​​the tracking target. Furthermore, if there is no detection area that includes the image coordinates of point 617, the selection unit 215 selects, from among the detection areas whose occupancy is included in the occupancy range, the detection area that is closest to the image coordinates of point 617, as the detection area of ​​the tracking target.

[0057] In the example of FIG. 8(a), the only detection area that includes point 617 is detection area 611. Here, if the occupancy range is "0.01 to 0.6," detection area 611 includes point 617, but its occupancy is 1.0, so it is not included in the occupancy range, and therefore detection area 611 is not selected as the detection area of ​​the tracking target. In such a case, of the detection areas with occupancies included in the occupancy range, the detection area that is closest to point 617 is selected as the detection area of ​​the tracking target. In the example of FIG. 8(a), the occupancy of detection area 612 (0.2) is included in the occupancy range, and of detection areas 612 to 616 excluding detection area 611, detection area 612 is the detection area that is closest to point 617. Therefore, detection area 612 is selected as the detection area of ​​the tracking target.

[0058] In step S507, the selection unit 215 stores (registers) the occupancy of the detection area of ​​the tracking target estimated by the estimation unit 213a in step S504 and the tracking feature of the detection area of ​​the tracking target extracted by the extraction unit 214a in step S505 in the storage unit 218. An example of the occupancy and the tracking feature stored in the storage unit 218 is shown in FIG.

[0059] Returning to FIG. 4, next, in step S402, the tracking unit 219 performs a tracking process to track the tracking target in the captured image acquired by the acquisition unit 210, using the tracking feature of the detection area of ​​the tracking target stored in the memory unit 218 and the detection area of ​​the tracking target selected by the selection unit 215 or the image area of ​​the tracking target re-detected by the re-detection unit 250.

[0060] In step S403, the determination unit 221 determines whether the tracking process in the tracking unit 219 has been successful. Various determination criteria are possible for determining whether the tracking process in the tracking unit 219 has been successful, and the determination unit 221 is not limited to a specific determination criterion. In this embodiment, the determination unit 221 calculates the similarity between the tracking feature of each detection area acquired from the captured image by the acquisition unit 230 and the tracking feature of the tracking target stored in the storage unit 218. Then, the determination unit 221 determines that "the tracking process in the tracking unit 219 has been successful" when there is one or more tracking feature of the tracking target stored in the storage unit 218, among the tracking feature of the tracking feature acquired from the captured image by the acquisition unit 230, whose similarity with the tracking feature of the tracking target stored in the storage unit 218 is equal to or greater than a threshold. On the other hand, if there is no tracking feature acquired by the acquisition unit 230 from the captured image whose similarity with the tracking feature of the tracking target stored in the memory unit 218 is equal to or greater than a threshold, the determination unit 221 determines that "the tracking process in the tracking unit 219 is not successful (failed)."

[0061] An example of a situation in which it is determined that "the tracking process in the tracking unit 219 has not been successful (failed)" is shown in Fig. 8(b). As shown in Fig. 8(b), when the dog being tracked is hidden behind a tree, which is another object, there is no detection area in the captured image where the tracking feature amount has a similarity to the tracking feature amount of the dog that is equal to or greater than the threshold. Therefore, in this case, it is determined that "the tracking process in the tracking unit 219 has not been successful (failed)".

[0062] If it is determined as a result of such determination that the tracking process in tracking unit 219 has been successful, the process proceeds to step S404. On the other hand, if it is determined that the tracking process in tracking unit 219 has not been successful, the process proceeds to step S406.

[0063] In step S404, the arithmetic processing unit 101 determines whether or not a condition for terminating the operation of the imaging device 100 is satisfied. For example, when the user operates the input device 103 to input an instruction to stop the operation of the imaging device 100 or to turn off the power of the imaging device 100, the arithmetic processing unit 101 determines that the condition for terminating the operation of the imaging device 100 is satisfied.

[0064] If it is determined that the conditions for ending the operation of the imaging device 100 are satisfied, the processing according to the flowchart in Fig. 4 ends. On the other hand, if it is determined that the conditions for ending the operation of the imaging device 100 are not satisfied, the processing proceeds to step S405.

[0065] In step S405, the acquisition unit 210 acquires the captured image generated by the imaging unit 105. Then, the process proceeds to step S402, and the tracking unit 219 performs tracking processing on the captured image acquired by the acquisition unit 210 in step S405.

[0066] In step S406, the extraction unit 211b extracts object features from the captured image acquired by the acquisition unit 210, similar to step S502. In step S407, the estimation unit 212b estimates, as a detection area, the entire image area of ​​an object in the captured image or a partial image area of ​​the object, similar to step S503, using the object features extracted from the captured image by the extraction unit 211b. As a result, the estimation unit 212b acquires, for each detection area estimated from the captured image, the position of the detection area, the size of the detection area, and a likelihood representing the object-likeliness of the detection area.

[0067] In step S408, the estimation unit 213b estimates the occupancy rate for each detection area estimated from the captured image by the estimation unit 212b, similar to step S504. In step S409, the extraction unit 214b extracts tracking features corresponding to each detection area estimated from the captured image by the estimation unit 212b, similar to step S505.

[0068] Here, at the end of the processing of step S409, it is assumed that detection areas 621 to 626 have been estimated from the captured image by the estimation unit 212b, as shown in FIG. 8(c). Detection area 625 is the detection area of ​​the dog, and detection area 626 is the detection area of ​​the head, which is part of the dog. Detection area 621 is the detection area of ​​a tree, detection area 622 is the detection area of ​​a group of flowers, which is part of the tree, and detection areas 623 and 624 are the detection areas of flowers in the group of flowers. An example of the occupancy and tracking feature amounts of each of the detection areas 621 to 626 at this time is shown in FIG. 10. "NO." is the reference number of each detection area, and "ID" is an identification number unique to each detection area. The tracking feature amounts of each detection area take the same map format as the tracking feature amounts of the tracking target, and the map size here is 1 x 1 x C in width, height, and channel.

[0069] In step S410, the processing unit 222 re-detects the detection area of ​​the tracking target from the captured image. First, the processing unit 222 retrieves the occupancy OCC T Then, the processing unit 222 obtains the occupancy OCC of the detection area in the captured image as shown in the following formula. T The detection area for which the occupancy rate is found to be included in the range based on the above is set as a candidate detection area.

[0070] OCC T -α <OCC ID <OCC T +α OCC ID is the occupancy of each detection area estimated from the captured image. α is a value related to the allowable range of fluctuation in the occupancy of the detection area of ​​the tracking target, and is set to 0.05 here, for example. OCC T = 0.2, then 0.15 <OCC ID OCC meets <0.25 ID 10, two candidate detection regions are selected: a detection region 622 with an occupancy rate of 0.20 and a detection region 626 with an occupancy rate of 0.18.

[0071] Next, the processing unit 222 determines, from among the candidate detection areas, the candidate detection area having the highest correlation value with the tracking feature of the tracking target acquired from the storage unit 218 and equal to or greater than a threshold value (≧0), as the detection area of ​​the tracking target. In the example of FIG. 10, the processing unit 222 determines, from among the candidate detection areas, the candidate detection area having the highest correlation value with the tracking feature of the tracking target acquired from the storage unit 218 and equal to or greater than a threshold value (≧0), as the detection area of ​​the tracking target. T The processing unit 222 calculates a correlation value X1 between the tracking feature F2(1, 1, C) of the tracking target and the tracking feature F2(1, 1, C) of the detection area 622. T The processing unit 222 calculates a correlation value X2 between the correlation value X1 (1, 1, C) and the tracking feature F6 (1, 1, C) of the detection area 626. If the correlation value X1 is higher than the correlation value X2 and the correlation value X1 is equal to or greater than the threshold, the processing unit 222 determines the detection area 622 as the detection area of ​​the tracking target. On the other hand, if the correlation value X2 is higher than the correlation value X1 and the correlation value X2 is equal to or greater than the threshold, the processing unit 222 determines the detection area 626 as the detection area of ​​the tracking target. Note that if both the correlation value X1 and the correlation value X2 are less than the threshold, the re-detection unit 250 does not determine the detection area of ​​the tracking target, but instead performs the same process on the next captured image input to redetect the detection area of ​​the tracking target. In this embodiment, the re-detection unit 250 redetects the detection area of ​​the tracking target for each captured image input within a predetermined period after starting re-detection of the detection area of ​​the tracking target. If the re-detection unit 250 is unable to determine the detection area of ​​the tracking target even after re-detecting the detection area of ​​the tracking target for each captured image input within a predetermined period after starting re-detection of the detection area of ​​the tracking target, it terminates operation as a re-detection failure.

[0072] Therefore, if the detection area of ​​the tracking target can be determined as a result of the redetection process in step S410, the process proceeds to step S402, and if the detection area of ​​the tracking target cannot be determined, the process proceeds to step S411.

[0073] In step S411, redetection unit 250 determines whether a preset period (predetermined time) has elapsed since redetection of the detection area of ​​the tracking target began. If the result of this determination is that the preset period (predetermined time) has elapsed since redetection of the detection area of ​​the tracking target began, the processing according to the flowchart in Fig. 4 ends. On the other hand, if the preset period (predetermined time) has not yet elapsed since redetection of the detection area of ​​the tracking target began, the processing proceeds to step S410.

[0074] In this manner, in this embodiment, an occupancy indicating the proportion of the image area of ​​the tracking target in the image area of ​​the tracking target in the captured image or in the image area of ​​the object to which the tracking target belongs, and a feature amount of the tracking target are registered. Then, if it is determined that tracking of the tracking target in the captured image has not been successful, the image area of ​​the tracking target is re-detected from the captured image based on the registered occupancy amount and feature amount.

[0075] As shown in FIG. 8(c), if the dog has moved further into the screen than the dog in the captured image of FIG. 8(a) (the captured image from which the occupancy and tracking feature amounts stored in the storage unit 218 were obtained), the size of the dog in FIG. 8(c) will be relatively smaller than the size of the dog in FIG. 8(a). Here, assume that the size of the entire dog in FIG. 8(c) is approximately the same as the size of the dog's head in FIG. 8(a). In this case, the tracking feature amounts of the dog's head and the entire dog will be relatively similar. Therefore, if the tracking target is detected using only the tracking feature amount, the entire dog in FIG. 8(c) may be mistakenly redetected as the dog's head. In this case, the tracking target will change from the dog's head to the entire dog before and after redetection. In this embodiment, the tracking target is redetected using the occupancy amount in addition to the tracking feature amount. Therefore, even if the size of the tracking target has changed from the captured image from which the occupancy and tracking feature amounts stored in the storage unit 218 were obtained, the tracking target can be redetected stably.

[0076] Furthermore, when detecting an unspecified object in Figure 8(c), many detection areas are estimated, such as dogs, trees, and parts of them, so it is difficult to identify the detection area of ​​the tracking target from among those detection areas using only the tracking feature amount.However, even if the tracking target is an unspecified object, it is possible to stably redetect the tracking target by redetecting the tracking target using the occupancy in addition to the tracking feature amount.

[0077] Next, a learning device 700 that performs learning processing of the neural networks used in the extraction units 211a / 211b, estimation units 212a / 212b, estimation units 213a / 213b, and extraction units 214a / 214b will be described.

[0078] In the following, when a common description is given for the extraction unit 211a and the extraction unit 211b, the extraction unit 211a and the extraction unit 211b will be collectively referred to as the extraction unit 211. Similarly, when a common description is given for the estimation unit 212a and the estimation unit 212b, the estimation unit 212a and the estimation unit 212b will be collectively referred to as the estimation unit 212. Similarly, when a common description is given for the estimation unit 213a and the estimation unit 213b, the estimation unit 213a and the estimation unit 213b will be collectively referred to as the estimation unit 213. Similarly, when a common description is given for the extraction unit 214a and the extraction unit 214b, the extraction unit 214a and the extraction unit 214b will be collectively referred to as the extraction unit 214.

[0079] An example of the functional configuration of the learning device 700 according to this embodiment will be described using the block diagram in Fig. 11. Note that the extraction unit 211, estimation unit 212, and estimation unit 213 in Fig. 11 are all shown as learning targets of the learning device 700, and do not indicate that the learning device 700 has these functional units.

[0080] 11 (excluding the extraction unit 211, the estimation unit 212, the estimation unit 213, and the storage unit 701) may be described below as the subject of processing. However, in reality, the functions of these functional units are realized by the arithmetic processing unit 130 executing a computer program that causes the arithmetic processing unit 130 to execute the functions of the functional units.

[0081] The storage unit 701 stores learning data used for training the neural networks used in the extraction unit 211, the estimation unit 212, and the estimation unit 213. The learning data includes a plurality of sets of learning images, area information indicating the position (center position, upper left corner position, etc.) and size (vertical size and horizontal size) of a reference area, which is an image area of ​​the entire or part of an object in the learning image, and the occupancy rate of the reference area.

[0082] The acquisition unit 702 acquires training data from the storage unit 701. The acquisition unit 703 acquires training images included in the training data acquired by the acquisition unit 702. The extraction unit 211 extracts object features from the training images acquired by the acquisition unit 703.

[0083] The estimation unit 212 estimates, as a detection area, the entire image area of ​​the object in the training image or a partial image area of ​​the object, using the object feature amount extracted from the training image by the extraction unit 211. As a result, for each detection area estimated from the training image, the estimation unit 212 calculates the position of the detection area in the training image (center position, position of the upper left corner, etc.), the size of the detection area (vertical size and horizontal size), and the likelihood that the target included in the detection area is the object.

[0084] The estimation unit 213 estimates, for each detection region estimated from the training image by the extraction unit 212, the proportion that the detection region occupies in the image region that includes the entire image region of the object in the training image as the occupancy.

[0085] The calculation unit 707 calculates, as the region error, the error (error based on the position error and size error) between the position and size of the detection region estimated by the estimation unit 212 from the training image and the position and size of the reference region indicated by the region information paired with the training image. The calculation unit 707 calculates, for example, the distance between the position of the detection region and the position of the reference region as the "position error." The calculation unit 707 also calculates, for example, the sum of the difference between the vertical size of the detection region and the vertical size of the reference region and the difference between the horizontal size of the detection region and the horizontal size of the reference region as the "size error." Then, for example, the calculation unit 707 calculates the sum of the "position error" and the "size error" for all detection regions, and calculates the sum of the sums calculated for all detection regions as the region error.

[0086] The calculation unit 708 calculates the difference between the occupancy of each detection region estimated from the training image by the estimation unit 213 and the occupancy of each reference region as an occupancy error. The learning unit 709 updates the parameters of the neural networks used in the extraction unit 211, estimation unit 212, and estimation unit 213 so as to reduce the region error calculated by the calculation unit 707 and the occupancy error calculated by the calculation unit 708. The neural network parameters are, for example, weight coefficients of the convolutional layer and fully connected layer in the neural network. This updating process realizes the learning process of the neural network.

[0087] The neural network learning process performed by the learning device 700 will be described with reference to the flowchart in Fig. 12. In step S801, the acquiring unit 702 acquires learning data from the storage unit 701. The acquiring unit 703 acquires learning images included in the learning data acquired by the acquiring unit 702. The learning data is created in advance and stored in the storage unit 701. A method for creating the learning data will now be described with reference to Fig. 13. The learning data may be created by the learning device 700 or by another device.

[0088] In Figure 13(a), a training image 1300 including a person 1350 has an image area 1310 of the entire body of the person 1350, an image area 1320 of the head of the person 1350, an image area 1330 of the torso of the person 1350, and an image area 1340 of the lower body of the person 1350.

[0089] In the training image 1400, which is a close-up image of the person 1450 in Figure 13(b), an image area 1402 of the head of the person 1450, an image area 1403 of the torso of the person 1450, and an image area 1404 including the entire person 1450 in the training image 1400 are set.

[0090] In a training image 1500 including an automobile 1550 in FIG. 13(c), an image region 1501 including the entire automobile 1550, image regions 1502 and 1503 of the headlights, image regions 1504 and 1505 of the tires, and an image region 1506 of the windshield are set.

[0091] In Figure 13(d), a training image 1600 including a cat 1650 has an image area 1601 of the cat's 1650's entire body, an image area 1602 of the head, an image area 1603 of the right eye, an image area 1604 of the left eye, and an image area 1605 of the torso set.

[0092] The image area of ​​an object or a part of an object on a training image may be set manually by a user using a user interface such as the input device 103, or may be set by setting an image area detected by a detector. Also, the image area detected by the detector may be manually corrected by a user.

[0093] In this way, area information indicating the position and size of the image area of ​​the object or a part of the image area of ​​the object set in the learning image is registered together with the learning image in the learning data. Note that, depending on the object, rotation information indicating the rotation direction and rotation angle of the image area may also be included in the learning data.

[0094] In addition, the ratio of the area of ​​the "entire image area of ​​the object or a part of the image area of ​​the object" to the area (number of pixels) of the entire image area of ​​the object in the training image is calculated as the occupancy of the image area, and the occupancy is registered in the training data together with the training image.

[0095] In the example of FIG. 13(a), the ratio of the area of ​​image region 1310 of person 1350 to the area of ​​image region 1310 of the entire body is calculated as the occupancy of image region 1310 (the occupancy in this case is 1.0). The ratio of the area of ​​image region 1320 of the head to the area of ​​image region 1310 of the entire body of person 1350 is calculated as the occupancy of image region 1320. The ratio of the area of ​​image region 1330 of the torso to the area of ​​image region 1310 is calculated as the occupancy of image region 1330. The ratio of the area of ​​image region 1340 of the lower body to the area of ​​image region 1310 is calculated as the occupancy of image region 1340. In the example of FIG. 13(a), the area of ​​image region 1340 of the lower body is about half the area of ​​image region 1310, so the occupancy of image region 1340 is calculated as 0.5.

[0096] As shown in the example of FIG. 13(b), in a learning image 1400 captured by capturing a close-up of a person 1450, an image region 1404 of a portion of the person 1450 is present, but the image region of the entire body of the person 1450 is not present. In such a case, the user visually estimates and inputs the occupancy of the image region 1404 relative to the image region of the entire body of the person 1450. When calculating the occupancy of the image region 1402 of the head, the occupancy of the image region 1402 is calculated as the product of the ratio of the area of ​​the image region 1402 to the area of ​​the image region 1404 and the occupancy of the image region 1404. When calculating the occupancy of the image region 1403 of the torso, the occupancy of the image region 1403 is calculated as the product of the ratio of the area of ​​the image region 1403 to the area of ​​the image region 1404 and the occupancy of the image region 1404. The user may also visually estimate and input the occupancy of the image region 1402 of the head and the image region 1403 of the torso.

[0097] In the example of FIG. 13(c), the ratio of the area of ​​image region 1501 to the area of ​​image region 1501 including the entire automobile 1550 is calculated as the occupancy of image region 1501. The ratio of the area of ​​image region 1502 of the automobile 1550's headlights to the area of ​​image region 1501 is calculated as the occupancy of image region 1502. The ratio of the area of ​​image region 1503 of the automobile 1550's headlights to the area of ​​image region 1501 is calculated as the occupancy of image region 1503. The ratio of the area of ​​image region 1504 of the automobile 1550's tires to the area of ​​image region 1501 is calculated as the occupancy of image region 1504. The ratio of the area of ​​image region 1505 of the automobile 1550's tires to the area of ​​image region 1501 is calculated as the occupancy of image region 1505. Also, the ratio of the area of ​​the image area 1506 of the windshield of the automobile 1550 to the area of ​​the image area 1501 is calculated as the occupancy of the image area 1506 .

[0098] In the example of FIG. 13(d), the ratio of the area of ​​image region 1601 of the whole body of cat 1650 to the area of ​​image region 1601 is calculated as the occupancy of image region 1601. The ratio of the area of ​​image region 1602 of the head of cat 1650 to the area of ​​image region 1601 is calculated as the occupancy of image region 1602. The ratio of the area of ​​image region 1603 of the right eye of cat 1650 to the area of ​​image region 1601 is calculated as the occupancy of image region 1603. The ratio of the area of ​​image region 1604 of the left eye of cat 1650 to the area of ​​image region 1601 is calculated as the occupancy of image region 1604. The ratio of the area of ​​image region 1605 of the body of cat 1650 to the area of ​​image region 1601 is calculated as the occupancy of image region 1605.

[0099] If you want to estimate a detection area by limiting the type (category) of the object to be detected, you can prepare learning data for the limited type of object. For example, if you want to detect a person, you can prepare learning data for people, and if you want to detect a car, you can prepare learning data for cars. If you want to estimate a detection area without limiting the type of object to be detected, you can prepare learning data for various types of objects.

[0100] For example, in addition to the people, cars, and cats shown in Figure 13, training data for various types of objects such as trains, airplanes, insects, birds, and dogs can be prepared. If training data for various types of objects is prepared in this way and the training process is performed appropriately, it becomes possible to detect object types that are not included in the training data. For example, even if there is no training data for fish, it is possible to detect fish-likeness, making it possible to detect fish.

[0101] Returning to FIG. 12, next, in step S802, the extraction unit 211 extracts object features from the training image acquired by the acquisition unit 703 in step S801, using the neural network under training.

[0102] In step S803, the estimation unit 212 estimates the entire image area of ​​the object in the training image or a partial image area of ​​the object as a detection area using the object features extracted from the training image in step S802 and the neural network being trained.

[0103] In step S804, the estimation unit 213 uses the neural network under training to estimate the occupancy rate for each detection region estimated from the training image by the extraction unit 212 in step S803.

[0104] In step S805, the calculation unit 707 calculates a region error based on the position and size of the detection region estimated from the training image by the estimation unit 212 and the position and size of the reference region indicated by the region information paired with the training image. For example, the calculation unit 707 calculates the region error by adding up the deviations (e.g., squared errors) of the center coordinates and size of the detection region estimated by the estimation unit 212 from the center coordinates and size of the corresponding reference region in the training data.

[0105] In step S806, the calculation unit 708 calculates the difference (e.g., square error) between the occupancy of each detection region in the training image estimated by the estimation unit 213 and the occupancy of a corresponding region paired with the detection region in the training image. The calculation unit 708 then calculates the sum of the differences calculated for each detection region in the training image as the occupancy error.

[0106] In step S807, the learning unit 709 updates the parameters of the neural networks used in the extraction unit 211, the estimation unit 212, and the estimation unit 213 so as to reduce the sum (loss value) of the region error calculated by the calculation unit 707 and the occupancy error calculated by the calculation unit 708. The parameters can be updated using, for example, the backpropagation method.

[0107] In step S808, the learning unit 709 determines whether a learning termination condition has been met. There are various conditions for terminating learning, and the condition is not limited to a specific one. For example, the learning termination condition may be that the loss value is equal to or less than a threshold, that the rate of change in the loss value is equal to or less than a threshold, or that the number of parameter updates is equal to or greater than a threshold. Alternatively, for example, learning data for accuracy verification may be prepared separately from learning data for parameter update, and the processes of steps S801 to S807 described above may be performed. When the sum of the loss values ​​is equal to or less than the threshold, it may be determined that the learning termination condition has been met.

[0108] If it is determined that the learning termination conditions have been met as a result of such a determination, the processing according to the flowchart of FIG. 12 ends, and if it is determined that the learning termination conditions have not been met, the processing proceeds to step S801.

[0109] Next, learning related to extraction unit 214 will be described. Storage unit 701 stores data for learning (learning data) for extraction unit 214. The learning data for extraction unit 214 includes a plurality of sets of two learning images containing the same object, area information indicating the position (center position, upper left corner position, etc.) and size (vertical size and horizontal size) of the image area of ​​the tracking target in the learning image, and the occupancy of the image area. The definition of occupancy is as described above. Hereinafter, one of the two learning images included in the learning data for extraction unit 214 will be referred to as the first image, and the other will be referred to as the second image.

[0110] Fig. 14 shows examples of the first image and the second image. Fig. 14(a) shows an example of the first image, and Fig. 14(b) shows an example of the second image. The first image and the second image include both a tracking target object 1701 and a non-tracking target object 1702. The non-tracking target is an object that has an appearance similar to the tracking target. In this way, the learning data of extraction unit 214 includes two learning images that include both the tracking target and a non-tracking target that has an appearance similar to the tracking target.

[0111] The positions and sizes of image region 1705 of tracking target 1701 in the first image and image region 1707 of tracking target 1701 in the second image are defined by region information included in the training data. The positions and sizes of image region 1706 of non-tracking target 1702 in the first image and image region 1708 of non-tracking target 1702 in the second image are obtained from the first image and the second image, respectively, by extraction unit 211 and estimation unit 212.

[0112] Extraction unit 214 then acquires tracking feature amounts of image region 1705 of tracking target 1701 and image region 1706 of non-tracking target 1702 from the first image, and acquires tracking feature amounts of image region 1707 of tracking target 1701 from the second image.

[0113] The learning unit 709 then updates the parameters of the neural network used in the extraction unit 214 so that the inter-feature distance between the tracking feature of the tracking target in the first image and the tracking feature of the tracking target in the second image is shortened, and so that the inter-feature distance between the tracking feature of the tracking target in the first image and the tracking feature of the non-tracking target in the first image is lengthened. The parameter update can be performed using, for example, backpropagation. In the example of FIG. 14 , the learning unit 709 updates the parameters of the neural network used in the extraction unit 214 so that the inter-feature distance between the tracking feature of image region 1705 and the tracking feature of image region 1707 is shortened, and so that the inter-feature distance between the tracking feature of image region 1705 and the tracking feature of image region 1706 is lengthened. Note that by preparing images including various types of objects as learning data, tracking features applicable to tracking of unspecified objects can be acquired.

[0114] As described above, according to this embodiment, even if the tracking target, which is an unspecified object, temporarily disappears, the tracking target can be redetected when it returns to the image. In this case, by using the occupancy of the tracking target during redetection, even if another object with similar tracking features exists in the image, the tracking target can be accurately redetected as long as the occupancy is different. Furthermore, by using the occupancy, if the tracking target is the entire object, the entire object can be redetected, and if the tracking target is only a part of the object, the part can be redetected. Therefore, it is possible to track and focus on the tracking target intended by the user.

[0115] [Second embodiment] The following describes the differences from the first embodiment, and unless otherwise specified below, it is assumed that the present embodiment is the same as the first embodiment. In the first embodiment, the detection area of ​​the tracking target is specified using the image coordinates and occupancy range input by the user, and the occupancy and tracking feature amount for the specified detection area are registered in the storage unit 218.

[0116] In this embodiment, the detection area of ​​the tracking target is identified using the occupancy corresponding to the imaging parameters in the imaging device 100 and the image coordinates input by the user, and the occupancy and tracking features for the identified detection area are registered in the memory unit 218.

[0117] An example of the functional configuration of the imaging device 100 according to this embodiment is shown in the block diagram of FIG. 15. In FIG. 15, functional units similar to those shown in FIG. 3 are assigned the same reference numerals, and descriptions of these functional units will be omitted. In the following, the functional units shown in FIG. 15 may be described as the subject of processing. However, in reality, the functions of each functional unit shown in FIG. 15, except for the tracking unit 219, the AF processing unit 220, and the storage unit 218, are realized by the arithmetic processing device 130 executing a computer program that causes the arithmetic processing device 130 to realize the functions of the functional units. Similarly, the functions of the tracking unit 219 and the AF processing unit 220 are realized by the arithmetic processing device 101 executing a computer program that causes the arithmetic processing device 101 to realize the functions of the tracking unit 219 and the AF processing unit 220, among the functional units shown in FIG. 15.

[0118] First, the imaging parameters of the imaging device 100 will be described. Examples of imaging parameters that can be used include aperture value, exposure time, AF frame size, ISO sensitivity, and Bv value. For the sake of specificity, the following description will be given, as an example, of a case where the imaging parameter of the imaging device 100 is the "aperture value of the lens of the imaging unit 105." However, the following description is equally applicable to imaging parameters of the imaging device 100 other than the "aperture value of the lens of the imaging unit 105." The aperture value is one of the setting values ​​that controls the brightness and blur of an image, and is expressed as F1.4, F2, F2.8, F4, F5.6, F8, F11, and F16. A smaller setting value results in a larger aperture size, resulting in a brighter captured image and a shallower depth of field. On the other hand, a larger setting value results in a smaller aperture size, resulting in a darker captured image and a deeper depth of field. By narrowing the aperture value and narrowing the depth of field, you can limit the range of focus, making the main subject stand out from the background and resulting in a more impressive image. Conversely, by widening the aperture value and deepening the depth of field, you can fit many subjects into the image without blurring them.

[0119] The selection unit 940 selects a detection area of ​​the tracking target from the detection areas estimated from the captured image by the estimation unit 212a. Then, the selection unit 940 stores (registers) the occupancy rate estimated by the estimation unit 213a for the selected detection area and the tracking feature amount extracted by the extraction unit 214a for the selected detection area in the storage unit 218. The selection unit 240 includes a selection unit 915, an input unit 216, and an input unit 917.

[0120] The selection unit 915 selects a detection area of ​​the tracking target from the detection areas estimated from the captured image by the estimation unit 212a, based on the image coordinates acquired by the input unit 216 and the aperture value as an imaging parameter acquired by the input unit 917. The selection unit 915 holds occupancies corresponding to various aperture values. For example, the selection unit 915 holds "0.1" as an occupancy corresponding to an aperture value of F1.4 or less, and "1.0" as an occupancy corresponding to an aperture value of F8 or more. The selection unit 915 also calculates occupancies corresponding to aperture values ​​between the aperture value F1.4 and the aperture value F8 by linear interpolation using the occupancy "0.1" corresponding to the aperture value F1.4 and the occupancy "1.0" corresponding to the aperture value F8.

[0121] The input unit 917 receives an aperture value input by the user using the input device 103. For example, a hardware dial of the imaging device 100 can be applied to the input device 103 operated by the user at this time.

[0122] The process of selecting detection areas for tracking by the selection unit 940 will be described below using Fig. 16(a) as an example. In a captured image 1800, detection areas 1802 to 1804 are each detection areas estimated from the captured image 1800 by the estimation unit 212a.

[0123] Detection area 1802 is a detection area that includes the entire horse 1801, detection area 1803 is a detection area for the head, which is part of the horse 1801, and detection area 1804 is a detection area for trees. Point 1805 indicates the indicated position indicated as the position of the tracking target by the user operating the input device 103. Here, it is assumed that the occupancy degree calculated by the estimation unit 213a for detection area 1802 is "1.0", and the occupancy degree calculated for detection area 1803 is "0.3".

[0124] The input unit 216 acquires image coordinates corresponding to the point 1805. Furthermore, the input unit 917 acquires the "aperture value" input by the user operating the input device 103. The selection unit 915 selects the occupancy OCC that is stored in association with the aperture value acquired by the input unit 917. TFurthermore, the selection unit 915 identifies, as a candidate, a detection area that includes the image coordinates corresponding to point 1805 from among the detection areas 1802 to 1804. In the case of FIG. 16(a), the detection area that includes the image coordinates corresponding to point 1805 is detection area 1802, and therefore detection area 1802 is identified as a candidate. Note that if there are multiple detection areas that include the image coordinates corresponding to point 1805, the detection area with the largest occupancy among the multiple detection areas is identified as a candidate. Then, the selection unit 915 selects the detection area identified as a candidate and the detection area included in the detection area identified as a candidate, with an occupancy of occupancy OCC T The selection unit 915 selects, as the detection region of the tracking target, a detection region where the occupancy degree is equal to or greater than the occupancy degree OCC T If there are a plurality of detection areas where the coordinates are the same, the detection area that is closest to the image coordinates acquired by the input unit 216 is determined to be the detection area of ​​the tracking target.

[0125] For example, if the aperture value acquired by the input unit 917 is F2.8, the occupancy rate corresponding to F2.8 is 0.3. Of the detection areas 1802 to 1804, only detection area 1802 includes the image coordinates corresponding to point 1805, and therefore detection area 1802 is identified as a candidate. Of detection area 1802 identified as a candidate and detection area 1803 included in detection area 1802, detection area 1803 is the only detection area with an occupancy rate of 0.3, and therefore the selection unit 915 selects detection area 1803 as the detection area of ​​the tracking target.

[0126] Furthermore, for example, if the aperture value acquired by the input unit 917 is F8, the occupancy rate corresponding to F8 is 1.0. Of the detection areas 1802 to 1804, only detection area 1802 includes the image coordinates corresponding to point 1805, and therefore detection area 1802 is identified as a candidate. Then, of detection area 1802 identified as a candidate and detection area 1803 included in detection area 1802, detection area 1802 is the only detection area with an occupancy rate of 1.0, and therefore the selection unit 915 selects detection area 1802 as the detection area of ​​the tracking target.

[0127] In addition, among the detection areas identified as candidates and the detection areas included in the detection areas identified as candidates, OCC T -α <OCC ID <OCC T +α Occupancy OCC ID The detection area may be selected as the detection area of ​​the tracking target.

[0128] In this way, when the aperture value is large, a detection area with a large occupancy is selected as the detection area for the tracking target, and when the aperture value is small, a detection area with a small occupancy is selected. This corresponds to the user's intention to limit the focus range when the aperture value is small and to widen the focus range when the aperture value is large.

[0129] 16(b), the horse 1801 in Fig. 16(a) moves to the right, is hidden by a tree located in the center of the captured image, and temporarily disappears from the captured image. The captured images shown in Fig. 18(c) and (d) are captured images acquired by the acquisition unit 210 after the captured image in Fig. 16(b), and are captured images in which the horse 1801 reappears from the right side of the tree.

[0130] When the aperture setting is F2.8, as shown in Fig. 16(c), a detection area 1850 of the head of the horse 1801 is redetected as the detection area of ​​the tracking target, and the focus is set on the redetected detection area 1850 of the head. As a result, a captured image is obtained in which the body of the horse 1801 and trees in the background direction relative to the head are blurred. In such a captured image, the head of the horse 1801 stands out from the background, making it an impressive captured image.

[0131] On the other hand, when the aperture setting is F8, the entire detection area 1860 of the horse 1801 is re-detected as the detection area of ​​the tracking target, as shown in Fig. 16(d). In such a captured image, the re-detected horse 1801 is clearly visible in its entirety, and as a result, a captured image that conveys the dynamic movement of the horse 1801 is obtained.

[0132] In this way, by linking the occupancy setting when selecting a tracking target with the aperture value, it is possible to reliably detect the tracking target and obtain a captured image that expresses the user's intended expression.

[0133] As described above, in this embodiment, the imaging parameter may be something other than the aperture value. For example, a method of changing the selection criteria for occupancy level according to the exposure time may be considered. As the exposure time becomes longer, the amount of subject blur and camera shake increases, and the influence of blur becomes more apparent when the captured image is viewed microscopically. Therefore, when the exposure time is longer, it is better to preferentially select a detection area with a larger occupancy level as the detection area for the tracking target.

[0134] Another possible method is to change the occupancy selection criteria depending on the AF frame size. When the AF frame size is small, it can be assumed that the user intends to focus on only a portion of the subject, so the smaller the AF frame size, the better it is to prioritize selecting a detection area with a smaller occupancy as the detection area for the tracking target. Conversely, when the AF frame size is large, it can be assumed that the user intends to focus on the entire subject, so the larger the AF frame size, the better it is to prioritize selecting a detection area with a larger occupancy as the detection area for the tracking target.

[0135] In this way, in this embodiment, even if the tracking target of an unspecified object temporarily disappears, the tracking target can be re-detected when it returns to the image. Furthermore, by linking the occupancy setting with the imaging parameters, it is possible to set the object as the tracking target so that a captured image that expresses the user's intended expression can be obtained.

[0136] [Third embodiment] In the first and second embodiments, cases have been described in which tracking processing and AF processing are performed on the detection area of ​​the tracking target. However, the processing performed on the detection area of ​​the tracking target is not limited to tracking processing and AF processing, and other processing such as auto exposure processing (AE processing) that controls exposure to a proper level and auto white balance processing (AWB processing) that performs color correction for a light source may also be used. Furthermore, multiple processing may be applied to the detection area of ​​the tracking target.

[0137] Furthermore, in the configuration of Figure 1, the learning device 700 is described as being a separate device from the imaging device 100, but the imaging device 100 and the learning device 700 may be integrated to form a single imaging device 100.

[0138] Furthermore, the operation of the imaging device described in the above embodiment can be similarly applied to "an image processing device that detects / redetects a detection area of ​​a tracking target from an image captured by an external imaging device." For example, such an image processing device can notify the imaging device of the detected / redetected detection area of ​​the tracking target, causing the imaging device to perform tracking processing, AF processing, etc. for the detection area of ​​the tracking target. Furthermore, such an image processing device may store information related to the detected / redetected detection area of ​​the tracking target in an external device.

[0139] In the above embodiment, the functional units (excluding the storage unit 218 and the storage unit 701) shown in Figures 3, 11, and 15 are implemented as software (computer programs). However, some or all of the functional units shown in Figures 3, 11, and 15 may be implemented as hardware.

[0140] Furthermore, the numerical values, processing timing, processing order, processing subject, destination / source / storage location of data (information) used in each of the above embodiments are given as examples to provide a concrete explanation, and are not intended to be limited to such examples.

[0141] In addition, some or all of the above-described embodiments may be used in appropriate combination, and some or all of the above-described embodiments may be selectively used.

[0142] (Other embodiments) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0143] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0144] 210: Acquisition unit 215: Selection unit 216: Input unit 217: Input unit 218: Storage unit 219: Tracking unit 220: AF processing unit 221: Determination unit 230: Acquisition unit 240: Selection unit 250: Redetection unit

Claims

1. a registration means for registering an occupancy indicating a ratio of an image area of ​​the tracking target in an image captured by the image capture device to an image area of ​​an object to which the tracking target belongs, and a feature amount of the tracking target; a re-detection means for re-detecting an image area of ​​the tracking target from the captured image based on the occupancy rate and the feature amount when it is determined that tracking of the tracking target in the captured image has not been successful; An image processing device comprising:

2. The re-detection means The image processing device described in claim 1, characterized in that the occupancy and feature amount are calculated for a detection area of ​​an object or a part of the object detected from a captured image, and an image area of ​​the detection area detected from the captured image whose occupancy amount is included in a range based on the occupancy amount registered by the registration means and whose feature amount has the highest correlation value with the feature amount registered by the registration means is re-detected as the image area to be tracked.

3. The registration means 3. The image processing device according to claim 1, wherein the image area of ​​the entire object or a part of the object is detected as a detection area from the captured image, and the occupancy rate and feature amount of a detection area selected from the detected detection areas in accordance with a user operation are registered.

4. The image processing device according to claim 3, characterized in that the registration means registers the occupancy and feature amount of a detection area whose occupancy is included in the occupancy range input by the user and which includes the image coordinates specified by the user.

5. The image processing device according to claim 3, characterized in that the registration means registers the occupancy and feature values ​​of a detection area, of which the occupancy degree corresponds to the imaging parameters input by the user, or a detection area of ​​which the occupancy degree is included in an occupancy degree range based on the occupancy degree, among the detection areas identified based on the image coordinates input by the user.

6. an imaging means for capturing an image; An image processing device according to any one of claims 1 to 5; a processing means for executing processing on an image area of ​​a tracking target in the captured image; An imaging device comprising:

7. 7. The imaging apparatus according to claim 6, wherein the processing includes tracking processing, AF processing, AE processing, and AWB processing.

8. An image processing method performed by an image processing device, a registration step in which a registration means of the image processing device registers an occupancy rate indicating a ratio of an image area of ​​the tracking target in an image area of ​​the tracking target in a captured image or an image area of ​​an object to which the tracking target belongs, and a feature amount of the tracking target; a re-detection step in which, when it is determined that the tracking of the tracking target in the captured image has not been successful, the re-detection means of the image processing device re-detects the image area of ​​the tracking target from the captured image based on the occupancy rate and the feature amount; An image processing method comprising:

9. A computer program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 5.

Images