Image processing device and control method thereof, imaging device, and program

Abstract

To provide an image processing device that enables detection results of multiple portions of subjects to be classified for each of the subjects with higher accuracy.SOLUTION: An imaging apparatus acquires consecutive images in time order and performs image processing on the images. The imaging apparatus detects a first portion (a face or the like) of a subject in the images (S201) and detects a second portion (a torso or the like) of a subject in the images (S202). The imaging apparatus uses a detection result obtained from a previously acquired image to perform processes of searching for a detection result obtained from a current image and classifying a detection result satisfying a condition for each of subjects. In the search process (S209), a same portion search using detection results of the same portion from the previously acquired image and the current image is preferentially performed over a different portion search using detection results of different portions from the previously acquired image and the current image.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an imaging device, and more particularly to detection of a part of a subject. [Background technology]

[0002] In subject detection processing in an imaging device, an area corresponding to a part such as a face is detected for each of images acquired continuously in time series. A search process is performed to determine the detection result of the current image by setting a predetermined range as a search range starting from the detection result of a previously acquired image. In the search process, detection results that satisfy a search condition are linked as detection results of the same subject based on the fact that the same subject exists in close positions in consecutive images. Patent Document 1 discloses a technology to suppress erroneous detection by combining and determining detection results of multiple frames that are linked as the same subject. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2019-41414 A [Patent Document 2] JP 2016-156934 A [Non-patent literature]

[0004] [Non-Patent Document 1] S. Haykin, “Neural Networks A Comprehensive Foundation 2nd Edition”, Prentice Hall, pp.156-255, July 1998 [Non-Patent Document 2] Alejandro Newell, Zhiao Huang, Jia Deng. Associative Embedding: End-to-End Learning for Joint Detection and Grouping. Neural Information Processing Systems, 2017 Summary of the Invention [Problem to be solved by the invention]

[0005] Here, we consider a shooting scene in which the subject moves vigorously, such as in gymnastics, and occlusion frequently occurs in a specific part. In this case, by providing a new means for detecting not only the face of the subject but also other parts such as the torso, it is possible to improve the detection rate of the subject.

[0006] In conventional technology, it is difficult to perform search processing that satisfies a predetermined accuracy condition in an embodiment having multiple detection means. When classifying multiple body parts by subject, if the detection result of a body part that should be classified as a first subject is erroneously classified as a body part of a second subject, the detection accuracy of the subject may decrease. The present invention aims to provide an image processing device that can classify the detection results of multiple body parts by subject with higher accuracy. [Means for solving the problem]

[0007] An image processing device according to an embodiment of the present invention includes: an acquisition unit that acquires a captured image; and a detection unit that detects a plurality of parts of a subject from the image. A part that satisfies a predetermined condition that is set based on a part detected by the detection means based on a first image previously acquired, is extracted from a plurality of parts detected by the detection means based on a second image currently acquired by the acquisition means, and the extracted part is associated with the subject and stored in a storage means. a search means for performing a different part search using a detection result of a different part between the first image and the second image, and a same part search using a detection result of a same part between the first image and the second image. And The method is characterized in that the search for the same site is performed with priority over the search for the different site. Effect of the Invention

[0008] According to the image processing device of the present invention, it is possible to classify the detection results of a plurality of parts for each subject with higher accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing a configuration of an imaging apparatus according to an embodiment of the present invention. [Diagram 2] 4 is a flowchart of an overall process in the present embodiment. [Diagram 3] FIG. 2 is a schematic diagram of an example of the overall processing in this embodiment. [Figure 4] 11 is a flowchart of a search process in the first embodiment. [Diagram 5] 5 is a flowchart of the process following FIG. 4. [Figure 6] 11 is a flowchart of a part search process according to the present embodiment. [Figure 7] 13 is a flowchart of a different part searching process in this embodiment. [Figure 8] FIG. 13 is a schematic diagram of an example of searching for the same part in this embodiment. [Figure 9] 11 is a schematic diagram of an example of searching for different parts using part vectors in this embodiment. FIG. [Figure 10] 11 is a schematic diagram of an example of searching for different parts without using part vectors in this embodiment. FIG. [Figure 11] 11 is a flowchart of a prohibition area update process in the present embodiment. [Figure 12] 10A to 10C are schematic diagrams illustrating an example of prohibition area elimination in the present embodiment. [Figure 13] 11 is a flowchart of a new subject adding process in this embodiment. [Figure 14] 14 is a flowchart of the process subsequent to FIG. 13. [Figure 15] 13 is a flowchart of a search process in the second and third embodiments. [Figure 16] 16 is a flowchart of the process subsequent to FIG. 15. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] An embodiment of the present invention will be described in detail below with reference to the drawings. In the embodiment, an example of an imaging device to which the image processing device of the present invention is applied is shown. In the imaging device, the search result of the part of the subject in the image is used, for example, for automatic focus adjustment control or subject tracking control, and is notified to the user by a display frame on the screen. For example, assume that the face area of ​​the subject in the image acquired in the past is used as a starting point to search the torso area of ​​the subject in the current image. Since the face and torso are originally flat images between the parts and there is a distance between the two parts, it is necessary to devise a search range. If the search range is simply expanded, there is a possibility that the part detection result of the subject will be erroneously linked to the detection result of another subject. Below, a process will be described that suppresses erroneous classification of the part detection result of the subject and performs highly accurate classification for each subject for the detection results.

[0011] [Example 1] An example of the configuration of an imaging device in this embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of an imaging device 100. The imaging device 100 is a digital still camera, video camera, or the like that can capture an image of a subject and record the image on a recording medium. Data of moving images and still images is recorded on a tape, solid-state memory, an optical disk, a magnetic disk, or the like. Each unit in the imaging device 100 is connected via a bus 260. Each unit is controlled by a CPU (Central Processing Unit) 151.

[0012] The lens unit 101 includes a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed third group lens 121, and a focus lens (lens for focus adjustment) 131. An aperture control unit 105 drives the aperture 103 via an aperture motor (AM) 104 in accordance with instructions from the CPU 151, thereby adjusting the aperture diameter of the aperture 103 and adjusting the amount of light during shooting. A zoom control unit 113 drives the zoom lens 111 via a zoom motor (ZM) 112 to change the focal length.

[0013] The focus control unit 133 determines the drive amount of the focus motor (FM) 132 based on the focus state detection result of the lens unit 101, that is, the amount of deviation in the focusing direction (defocus amount). The focus control unit 133 drives the focus lens 131 via the focus motor 132 based on the determined drive amount, thereby controlling the focus adjustment state. AF (autofocus) control is realized by the movement control of the focus lens 131 by the focus control unit 133 and the focus motor 132. Although the focus lens 131 is simply shown as a single lens in FIG. 1, it is usually made up of multiple lenses.

[0014] The subject image (optical image) formed on the image sensor 141 through the lens unit 101 is converted into an electric signal by the image sensor 141. The image sensor 141 includes a photoelectric conversion element that performs photoelectric conversion on the subject image. The image sensor 141 has a configuration in which m pixels are arranged horizontally and n pixels are arranged vertically. For example, a plurality of microlenses and two photoelectric conversion elements corresponding to each microlens are arranged in a light receiving area. The image signal photoelectrically converted by the image sensor 141 is arranged as predetermined image data by the image signal processing unit 142. By adding the outputs of the two photoelectric conversion elements, image data of the imaging surface can be obtained. In addition, two images (parallax images) having parallax can be obtained from the outputs of the two photoelectric conversion elements. In this embodiment, it is assumed that an A+B image signal is obtained by adding the outputs of the two photoelectric conversion elements, and an A image signal and a B image signal are obtained from the outputs of the two photoelectric conversion elements.

[0015] The monitor display 150 is equipped with a display device and displays the captured image and the rectangular frame of the object detection result. The image data output by the imaging signal processing unit 142 is sent to the imaging control unit 143 and temporarily stored in a RAM 154 (random access memory). The image data stored in the RAM 154 is compressed by the image compression / decompression unit 153 and then recorded in an image recording medium 157. In parallel with this, the image data stored in the RAM 154 is sent to the image processing unit 152.

[0016] The operation switch unit 156 is an input interface unit including a touch panel, buttons, etc. that are operated by the user. The user can give a desired instruction to the imaging device 100 by selecting various function icons displayed on the monitor display 150.

[0017] The CPU 151 performs setting processing of the image sensor 141 based on a user's operation instruction input from the operation switch unit 156 or the magnitude of a pixel signal related to image data temporarily stored in the RAM 154. For example, the CPU 151 determines the accumulation time of the image sensor 141 and the setting value of the gain when outputting from the image sensor 141 to the image signal processing unit 142. The imaging control unit 143 receives instructions on the accumulation time and the gain setting value from the CPU 151 and controls the image sensor 141. Regarding the control of the imaging optical system, the CPU 151 performs AF control for a specific object region by the focus control unit 133. In addition, the CPU 151 performs exposure control using the luminance value of the specific object region by the aperture control unit 105.

[0018] The image processing unit 152 acquires image data, and performs, for example, a reduction process or an enlargement process on the image data to an optimal size. The image data processed to the optimal size is appropriately sent to and displayed on the monitor display 150, thereby displaying a preview image or a through image. In addition, the object detection result of the part detection unit 161 can also be superimposed on the image on the monitor display 150. In addition, by using the RAM 154 as a ring buffer, it is possible to buffer data of multiple images captured within a predetermined period and the processing results of each unit corresponding to each image data.

[0019] A defocus map and a reliability map can be generated based on the data of the A and B images acquired from the imaging element 141. The defocus map is a map having information on the amount of defocus for each pixel, and each defocus amount is expressed in units of Fδ. The reliability map is a map in which each coordinate corresponds to the defocus map, and has reliability information that indicates the degree of reliability of the amount of defocus for each pixel. The defocus map is an example of information that indicates the depth distribution in the depth direction of an image, and information that indicates the distribution of distance information such as an image shift amount map acquired from a plurality of viewpoint images with different viewpoints, a distance map, and a distance image can be used. Note that the method of generating the defocus map and the reliability map can be the method disclosed in Patent Document 2, and detailed explanations thereof will be omitted.

[0020] The flash memory 155 stores control programs necessary for the operation of the imaging device 100, parameters used for the operation of each unit, etc. When the imaging device 100 is started by a user operation, that is, when the power is changed from an OFF state to an ON state, the control programs and parameters stored in the flash memory 155 are loaded into a part of the RAM 154. The CPU 151 controls the operation of the imaging device 100 in accordance with the control programs and constants loaded into the RAM 154. The image recording medium 157 records image data, image-related data, etc.

[0021] The battery 159 is appropriately managed by a power management unit 158, and provides a stable power supply to the entire imaging apparatus 100. The timer 160 is a timing device capable of acquiring relative time in milliseconds in synchronization with the processing of each unit.

[0022] The part detection unit 161 detects a plurality of regions corresponding to predetermined parts of the subject using the image signal. The detection process in the part detection unit 161 can be realized by, for example, feature extraction processing using Convolutional Neural Networks (hereinafter, referred to as CNN). More specifically, the method described in Non-Patent Document 1 can be used. Also, by switching the weight of the feature detection cell plane in the CNN, it is possible to detect various types of subjects and parts of the subject. Note that, in implementing the present invention, any method can be used for the part detection process.

[0023] The labeling processing unit 162 generates a label map for each image, which is separated for each subject in the image. More specifically, the method described in Non-Patent Document 2 can be used. When the part detection unit 161 obtains detection results of multiple parts, the detection results of multiple parts can be linked to each other by comparing with the label map.

[0024] The position and orientation change acquisition unit 163 includes position and orientation sensors such as a gyro sensor, an acceleration sensor, and an electronic compass, and measures a change in the position and orientation of the imaging device 100 relative to the captured scene. The acquired data on the change in position and orientation is stored in the RAM 154.

[0025] The motion vector calculation unit 164 performs a calculation process of a motion vector from image signals corresponding to two images acquired consecutively in time series. In the calculation process of the motion vector, a feature point matching process between a past frame and a current frame is performed for multiple positions in the image, and multiple motion vectors are detected.

[0026] The overall processing flow in this embodiment will be described with reference to Fig. 2. In this embodiment, it is assumed that body part detection unit 161 can detect the face region and torso region of a person who is a subject, and detection 1 is the detection of the face region, and detection 2 is the detection of the torso region. It is also assumed that the face region and torso region of a person have equal importance. The processing shown in each step is realized by CPU 151 executing a program.

[0027] In S200, the image data acquired by the imaging control unit 143 is supplied to each unit. In addition, time information at which the acquired image data was supplied is acquired from the timer 160 and recorded in the RAM 154.

[0028] In S201, part detection unit 161 performs detection processing on the acquired image. A plurality of detection areas (hereinafter referred to as detection 1 areas) related to the face area of ​​the subject are acquired. In S202, part detection unit 161 performs detection processing on the acquired image. A plurality of detection areas (hereinafter referred to as detection 2 areas) related to the torso area of ​​the subject are acquired.

[0029] In S203, the labeling processing unit 162 performs labeling processing on the acquired image. A label map with a different label for each subject region is acquired. In S204, the position and orientation change acquisition unit 163 acquires data on the position and orientation change of the imaging device 100 in synchronization with the acquired image.

[0030] In S205, the motion vector calculation unit 164 acquires a motion vector map for the acquired image. In S206, the image processing unit 152 acquires a defocus map and a reliability map. In S207, the CPU 151 performs a linking process on the multiple detection 1 regions acquired in S201 and the multiple detection 2 regions acquired in S202, using the label map acquired in S203. The linking process will be described later.

[0031] In S208, the CPU 151 determines whether or not there is a subject classification result for a previously acquired image. If it is determined that there is a subject classification result, the process proceeds to S209, and if it is determined that there is no subject classification result, the process proceeds to S210.

[0032] In S209, the CPU 151 performs a search process on each detection area obtained in S201 and S202, using the object classification result for the image acquired in the past as a reference. As a result of the search, the object classification result recorded in the RAM 154 is updated. The search process will be described later. After S209, the process proceeds to S210.

[0033] In S210, the CPU 151 performs processing for adding a new subject. Of the detection areas obtained in S201 and S202, areas that were not put into a used state in the search processing in S209 are added as areas of a new subject to the subject classification results recorded in the RAM 154. Adding a new subject will be described later.

[0034] In S211, the CPU 151 calculates a subject priority level for each subject in the subject classification results recorded in the RAM 154. The subject priority level is determined based on the size of the detection 1 region or detection 2 region. The larger the size, the higher the subject priority level.

[0035] In S212, the CPU 151 determines whether or not an end instruction has been issued by operating the operation switch unit 156. If it is determined that an end instruction has been issued, the entire process is terminated, and if it is determined that an end instruction has not been issued, the process returns to S200, and the series of processes are repeatedly executed.

[0036] An example of the operation of the overall processing will be described with reference to Fig. 3. Hereinafter, an image captured by the imaging device 100 will be referred to as an acquired image. Fig. 3(A) shows an example of detection regions obtained in S201 and S202 of Fig. 2 for an acquired image 300. In the processing of S201 in Fig. 2, face regions 311, 312, and 313 of three people are obtained. In the processing of S202 in Fig. 2, torso regions 321 and 323 of two people are obtained.

[0037] Fig. 3(B) shows an example of a label map acquired in S203 of Fig. 2 corresponding to the acquired image 300. The label map 301 has coordinates corresponding to the acquired image 300, and stores labels 331, 332, 333, and 334 that can distinguish between a subject region and a background region for each coordinate. The labels 331, 332, and 333 are identification information that respectively correspond to the subject regions of the three people. The label 334 is identification information that corresponds to the background region.

[0038] FIG. 3C shows an example of the result of detection association processing performed using a face region, a torso region, and a label map. In the detection association processing, the label map is referenced for each detection region, and a process is performed to search for the other detection region that belongs to the same label. For example, a torso region 321 that belongs to the same label 331 is associated with a face region 311 of a first person located on the left side. Since there is no torso region that belongs to the same label 332 as a face region 312 of a second person located in the center, the torso region is not associated. A torso region 323 that belongs to the same label 333 is associated with a face region 313 of a third person located in the foreground.

[0039] Fig. 3(D) shows an example of the subject classification result recorded in S209 or S210 in Fig. 2. The subject classification result consists of information on the subject ID, detection 1 area, and detection 2 area for each subject. The subject IDs A, B, and C are assigned to identify the first, second, and third persons, respectively.

[0040] The search process (FIG. 2: S209) in this embodiment will be described in detail with reference to FIG. 4 and FIG. 5. In S400, CPU 151 determines whether or not the object classification result is recorded in RAM 154 in the search process for the previously acquired image. If it is determined that the object classification result is recorded in RAM 154, the process proceeds to S401, and if it is determined that the object classification result is not recorded in RAM 154, the search process ends.

[0041] In S401, the CPU 151 determines a target subject from the subject classification results recorded in the RAM 154. When there are multiple candidates for the target subject, the subject with the higher subject priority calculated in S211 of FIG. 2 is given priority.

[0042] In S402, the CPU 151 determines whether or not the detection 1 area classified as the subject of interest in the previously acquired image is recorded in the RAM 154. If it is determined that the detection 1 area is recorded, the process proceeds to S403, and if it is determined that the detection 1 area is not recorded, the process proceeds to S406.

[0043] In S403, the same part search process is executed. The CPU 151 searches for the same part in the multiple detection 1 areas obtained from the current image, using the detection 1 area classified as the subject of interest in the previously acquired image as a reference. The same part search process will be described in detail later.

[0044] In S404, the CPU 151 determines whether an area satisfying the conditions has been found from the detection area of ​​the current image and whether the object classification result of the target object has been updated. If it is determined that the object classification result of the target object has been updated, the process proceeds to S410 in FIG. 5, and if it is determined that the object classification result has not been updated, the process proceeds to S405.

[0045] In S405, a different part search process is executed. The CPU 151 searches for different parts in the multiple detection 2 regions obtained from the current image, using the detection 1 region classified as the subject of interest in the previously acquired image as a reference. The different part search process will be described in detail later.

[0046] In S406, the CPU 151 determines whether or not the detected 2 region classified as the subject of interest in the previously obtained image is recorded in the RAM 154. If it is determined that the detected 2 region is recorded, the process proceeds to S407, and if it is determined that the detected 2 region is not recorded, the process proceeds to S410 in FIG.

[0047] In S407, the same part search process is executed. The CPU 151 searches for the same part in the two detected regions obtained from the current image, using the two detected regions classified as the subject of interest in the previously acquired image as a reference. The same part search process will be described in detail later.

[0048] In S408, CPU 151 determines whether an area satisfying the conditions has been found from the detection area of ​​the current image and whether the object classification result of the target object has been updated. If it is determined that the object classification result of the target object has been updated, the process proceeds to S410 in Fig. 5, and if it is determined that the object classification result of the target object has not been updated, the process proceeds to S409.

[0049] At S409, the different-site search process is executed. The CPU 151 searches for different sites with respect to a plurality of detected region 1 obtained from the current image, based on the detected region 2 classified as the subject of interest in the past acquired images. Details of the different-site search process will be described later. After S409, the process proceeds to S410 in FIG. 5.

[0050] At S410, the CPU 151 determines whether or not the subject classification result of the subject of interest has been updated with the detected region 1 and the detected region 2 obtained from the current image. If it is determined that the subject classification result of the subject of interest has been updated, the process proceeds to S411, and if it is determined that the subject classification result of the subject of interest has not been updated, the process proceeds to S412.

[0051] At S411, the CPU 151 calculates a site vector and records it in the RAM 154. The site vector is represented by a two-dimensional vector with respect to the image plane. The site vector can be obtained by calculating a difference value by subtraction from the coordinates of the detected region 1 and the detected region 2 recorded in the subject classification result of the subject of interest. The time information recorded in the RAM 154 at S200 in FIG. 2 is recorded in the RAM 154 as the recording time of the site vector. After S411, the process proceeds to S414.

[0052] At S412, the CPU 151 determines whether or not a predetermined period has elapsed since the time recorded at S411 in the search process for the past acquired images. The length of the predetermined period is a threshold time set according to the subject type of the detection target. For example, if the subject type is a person, T1 is set, and if the subject type is an animal, T2 (<T1) is set. If it is determined that the predetermined period has elapsed, the process proceeds to S413, and if it is determined that the predetermined period has not elapsed, the process proceeds to S414.

[0053] At S413, the CPU 151 deletes the data of the site vector recorded in the RAM 154. Then at S414, the CPU 151 updates the detection 1 prohibited region for the subject of interest using the detected region 1 that was not used for updating the subject classification result. Details of the update process will be described later. After S414, the process proceeds to S415.

[0054] In S415, CPU 151 updates the detection 2 prohibited area for the target object using the detection 2 area that was not used to update the object classification result. Details of the update process will be described later. In S416, CPU 151 determines whether or not there is an object that was not selected as the target object in S401 of Fig. 4 among the object classification results recorded in RAM 154 in the past acquired image. If it is determined that there is a corresponding object, the process returns to S401 and continues, and if it is determined that there is no corresponding object, the search process ends.

[0055] In this embodiment, the same-part search process with higher accuracy is executed with higher priority than the different-part search process with lower accuracy. When the subject classification result of the target subject is updated by the prioritized search process, the other search process is not executed, so that the more accurate search process can be executed.

[0056] Next, the same part search process (FIG. 4: S403 and S407) will be described in detail with reference to FIG. 6. In S500, CPU 151 determines whether or not a detection area to be searched for obtained from the current image exists. The detection area to be searched for is detection 1 area in S403 of FIG. 4, and detection 2 area in S407. If it is determined that the detection area to be searched for exists, the process proceeds to S501, and if it is determined that the detection area to be searched for does not exist, the same part search process ends.

[0057] In S501, the CPU 151 calculates the part search range. The part search range is calculated as a range having the same center of gravity as the reference detection area in the previously acquired image of the subject of interest, and a predetermined magnification (denoted as n1) of the range indicated by the reference detection area. The value of n1 is set according to the type of subject to be detected. For example, if the type of subject is a person, n 1a If it is an animal, n 1b (>n 1a 4, the reference detection area is set to the detection 1 area classified as the subject of interest in the previously acquired image in S403, and the reference detection area is set to the detection 2 area classified as the subject of interest in the previously acquired image in S407.

[0058] In S502, the CPU 151 determines a detection area of ​​interest from unused candidate detection areas obtained from the current image. The candidate detection area is the detection 1 area in S403 of FIG. 4, and is the detection 2 area in S407.

[0059] In S503, the CPU 151 performs a condition determination as to whether or not the target detection area belongs to the corresponding prohibited area. The corresponding prohibited area is the detection 1 prohibited area in S403 of FIG. 4, and is the detection 2 prohibited area in S407. Whether or not the target detection area belongs to the corresponding prohibited area can be determined by whether or not the center of gravity of the target detection area exists within the corresponding prohibited area. If this condition is met, the process returns to S502, and if the condition is not met, the process proceeds to S504.

[0060] In S504, the CPU 151 performs a condition determination as to whether or not the target detection region is included in the same part search range. If this condition is met, the process proceeds to S505, and if the condition is not met, the process proceeds to S510.

[0061] In S505, CPU 151 updates the classification result of the target subject in the target detection area, and in S506, changes the target detection area to a used state. In S507, CPU 151 determines whether or not there is a detection result associated with the target detection area in S207 of Fig. 2. If it is determined that there is such a detection result, the process proceeds to S508, and if it is determined that there is no such detection result, the process proceeds to S510.

[0062] In S508, CPU 151 updates the classification result of the target subject in the associated detection area, and in S509 updates the associated detection result to a used state. In S510, CPU 151 determines whether or not there is an unprocessed area that has not been selected as the target detection area among the detection areas to be searched. If it is determined that such an area exists, the process returns to S502 to repeat the series of processes, and if it is determined that such an area does not exist, the part search process ends.

[0063] Next, the different part search process (FIG. 4: 405 and S409) will be described in detail with reference to FIG. 7. In S600, CPU 151 determines whether or not a detection area to be searched for obtained from the current image exists. The detection area to be searched for is detection area 2 in S405 of FIG. 4, and detection area 1 in S409. If it is determined that the detection area to be searched for exists, the process proceeds to S601, and if it is determined that the detection area to be searched for does not exist, the different part search process ends.

[0064] In S601, CPU 151 determines whether or not there is a part vector for the subject of interest recorded in RAM 154. If it is determined that the part vector is recorded, the process proceeds to S602, and if it is determined that the part vector is not recorded, the process proceeds to S603.

[0065] In S602, CPU 151 calculates a different part search range using the part vector. The center of gravity of the different part search range in this step is found by adding or subtracting the part vector to a reference detection area in a previously acquired image of the subject of interest. The different part search range is found as a range obtained by multiplying the range of the reference detection area in a previously acquired image of the subject of interest by a predetermined magnification (denoted as n2). The value of n2 is set according to the type of subject to be detected. For example, if the type of subject is a person, n 2a If it is an animal, n 2b (>n 2a 4, the reference detection area is the detection 1 area included in the subject classification result of the target subject, and in S407, the reference detection area is the detection 2 area included in the subject classification result of the target subject.

[0066] In S603, CPU 151 calculates a different part search range without using a part vector. The different part search range in this step has the same center of gravity as the reference detection area in a previously acquired image of the subject of interest, and is calculated as a range obtained by multiplying the range of the reference detection area by a predetermined magnification factor (denoted as n3). The value of n3 is set according to the type of subject to be detected, and is set to "n3>n2". For example, if the type of subject is a person, n 3a If it is an animal, n3b (>n 3a The reference detection area is set to the same as in S602.

[0067] In S604, the CPU 151 determines a detection area of ​​interest from unused candidate detection areas obtained from the current image. The candidate detection area is the detection 2 area in S405 of FIG.

[0068] In S605, the CPU 151 performs a condition determination as to whether or not the target detection area belongs to the corresponding prohibited area. The corresponding prohibited area is the detection 2 prohibited area in S405 of FIG. 4, and is the detection 1 prohibited area in S409. Whether or not the target detection area belongs to the prohibited area can be determined by whether or not the center of gravity of the target detection area exists in the corresponding prohibited area. If the condition of S605 is satisfied, the process returns to S604, and if the condition of S605 is not satisfied, the process proceeds to S606. The processes from S606 to S612 are the same as the processes from S504 to S510 of FIG. 6, respectively, and therefore their description will be omitted.

[0069] The search process in this embodiment will be described in detail with reference to Fig. 8 to Fig. 10. In Fig. 8 to Fig. 10, (A) shows a previously acquired image, and (B) shows a current image. In the previously acquired image in Fig. 8(A), a detection 1 region 700 is acquired for the subject. In this case, a same part search range 731 is calculated in S501 of Fig. 6 for the current image in Fig. 8(B). The subject classification result of the target subject is updated using detection 1 region 730 that satisfies the conditions within same part search range 731.

[0070] Fig. 9 is a schematic diagram showing an example of different part search using part vectors. In the previously acquired image in Fig. 9(A), detection 1 region 800 and detection 2 region 810 have been acquired for the subject, and part vector 820 has been recorded in S411 in Fig. 5. In this case, different part search range 831 is calculated in S602 in Fig. 7 for the current image in Fig. 9(B). The subject classification result for the subject of interest is updated using detection 2 region 811 that satisfies the conditions within different part search range 831.

[0071] Fig. 10 is a schematic diagram showing an example of different part search without using part vectors. In the previously acquired image in Fig. 10(A), a detection 1 region 900 is acquired for the subject. In this case, a different part search range 931 is calculated in S603 in Fig. 7 for the current image in Fig. 10(B). The subject classification result of the target subject is updated using a detection 2 region 911 that satisfies the conditions within the different part search range 931.

[0072] The prohibited area update process (FIG. 5: S414 and S415) in this embodiment will be described in detail with reference to FIG. 11. In S1000, CPU 151 determines whether or not the prohibited area to be updated has been recorded in a previously acquired image. The prohibited area to be updated is the detection 1 prohibited area in S414 of FIG. 5, and is the detection 2 prohibited area in S415. If it is determined that the prohibited area to be updated has been recorded, the process proceeds to S1001, and if it is determined that the prohibited area to be updated has not been recorded, the process proceeds to S1009.

[0073] In S1001, the CPU 151 sets a prohibited area of ​​interest from among the prohibited areas to be updated. In S1002, the CPU 151 refers to the motion vector map acquired in S205 of Fig. 2 and performs a condition determination as to whether or not a motion vector greater than a predetermined value (threshold value) exists in the area corresponding to the prohibited area of ​​interest. If it is determined that a motion vector that satisfies this condition exists, the process proceeds to S1007, and if it is determined that a motion vector that satisfies the condition does not exist, the process proceeds to S1003.

[0074] In S1003, the CPU 151 compares the defocus map acquired in S206 of Fig. 2 for the previously acquired image with the defocus map acquired in S206 of Fig. 2 for the current image. A process is performed to determine whether or not a distance difference greater than a predetermined value exists in an area corresponding to the attention prohibition area. If it is determined that a distance difference that satisfies the condition exists, the process proceeds to S1007, and if it is determined that a distance difference that satisfies the condition does not exist, the process proceeds to S1004.

[0075] In S1004, CPU 151 determines whether or not there is a detection area that belongs to the attention prohibited area among the object detection areas obtained from the current image. If it is determined that the detection area exists, the process proceeds to S1005, and if the detection area does not exist, the process proceeds to S1007. However, the object detection area is limited to the detection 1 area that was not used to update the object classification result of the object of interest in S414 of Fig. 5 and was not registered in S1006. Also, the object detection area is limited to the detection 2 area that was not used to update the object classification result of the object of interest in S415 of Fig. 5 and was not registered in S1006.

[0076] In S1005, the CPU 151 updates the attention prohibition area recorded in the RAM 154. The area to be updated is an area that is processed so as to cancel the amount of change in position and orientation obtained in S204 of Fig. 2, based on the detection area determined to exist in S1004. After S1005, the process proceeds to S1006.

[0077] In S1006, the CPU 151 updates the detection area used for updating in S1005 to a registered state. In S1007, the CPU 151 deletes the attention prohibition area recorded in the RAM 154. After S1006 and S1007, the process proceeds to S1008.

[0078] In S1008, the CPU 151 performs a condition determination as to whether or not there is an area to be updated that has not been designated as a prohibited area in S1001. If this condition is met, the process returns to S1001 to continue, and if the condition is not met, the process proceeds to S1009.

[0079] In S1009, the CPU 151 performs a condition determination as to whether or not there is a target detection area that has not been registered (is in an unregistered state) in S1006. If this condition is met, the process proceeds to S1010, and if the condition is not met, the prohibition area update process is terminated.

[0080] In S1010, the CPU 151 adds all of the detection areas determined to exist in S1009 (unregistered) as new prohibited areas for the target subject to the RAM 154. Then, the series of processes ends.

[0081] An example of the removal of the prohibited area in this embodiment will be specifically described with reference to Fig. 12. Fig. 12(A) shows the detection state in a previously acquired image. Face areas 1100, 1102 and torso areas 1110, 1111 have been detected for each subject. The subject in the face area 1100 has a part of its torso hidden by the subject in the torso area 1110 who is in front (on the imaging device side).

[0082] Fig. 12(B) shows the detection state in the current image. Face regions 1120, 1121 and torso regions 1130, 1131 have been detected for each subject. The subject in face region 1120 hides the face region and part of the torso of the subject behind it. Fig. 12(C) shows a motion vector map corresponding to the current image. Data on the direction and magnitude of the vector is recorded for each coordinate.

[0083] Fig. 12(D) is a distance map corresponding to a previously captured image, and Fig. 12(E) is a distance map corresponding to a current image. In the distance map, higher brightness indicates closer to the object (closer to the imaging device).

[0084] Fig. 12(F) is a table showing prohibited areas when the foremost subject C in a previously acquired image is the subject of interest. Face areas 1100 and 1102 are detection 1 prohibited areas, and torso area 1111 is detection 2 prohibited area. Fig. 12(G) is a table showing the subject classification results for the current image. For example, for a subject with subject ID C, face area 1120 is detection 1 area, and torso area 1130 is detection 2 area.

[0085] As shown in Fig. 12(C), in the face region 1100 registered in the prohibited area, a motion vector having a magnitude equal to or greater than a predetermined value is calculated in the current image. Therefore, the face region 1100 satisfies the prohibited area removal condition determined in S1002 of Fig. 11. Also, as shown in Fig. 12(D) to (F), in the face region 1100 registered in the prohibited area, distance information fluctuates in the current image. Therefore, the face region 1100 satisfies the prohibited area removal condition determined in S1003 of Fig. 11.

[0086] By removing the prohibited area according to such a condition determination, it is possible to prevent the occurrence of a problem in which the subject classification result of the subject of interest is not updated in the search process when the detection result of the subject of interest appears in a prohibited area.

[0087] The new subject addition process (FIG. 2: S210) in this embodiment will be described in detail with reference to FIG. 13 and FIG. 14. In S1200, the CPU 151 determines whether or not an unused detection 1 area exists among the detection 1 areas obtained from the current image. If it is determined that an unused detection 1 area exists, the process proceeds to S1201, and if it is determined that an unused detection 1 area does not exist, the process proceeds to S1208 in FIG.

[0088] In S1201, CPU 151 sets the unused detection 1 region as a region of interest. In S1202, CPU 151 assigns a new object ID to the region of interest, adds it to the object classification result, and records it in RAM 154. In S1203, CPU 151 updates the region of interest as a used state, and records it in RAM 154.

[0089] In S1204, the CPU 151 performs a condition determination as to whether or not there is a detection 2 region that is associated with the region of interest in S207 of Fig. 2 and is in an unused state. If this condition is met, the process proceeds to S1205, and if the condition is not met, the process proceeds to S1207.

[0090] In S1205, the CPU 151 assigns the same subject ID to the detected 2 region associated with the region of interest as the subject ID assigned in S1202, adds the detected 2 region to the subject classification result, and records it in the RAM 154. In S1206, the CPU 151 records the detected 2 region associated with the region of interest in the RAM 154 as being in a used state.

[0091] In S1207, the CPU 151 performs a condition determination as to whether or not there is an unused area in the detection 1 area obtained from the current image. If this condition is met, the process returns to S1201 and continues, and if the condition is not met, the process proceeds to S1208 in FIG.

[0092] In S1208, the CPU 151 performs a condition determination as to whether or not there is an unused area among the two detection areas obtained from the current image. If this condition is met, the process proceeds to S1209, and if the condition is not met, the new subject addition process ends.

[0093] In S1209, CPU 151 sets the unused detected region 2 as a region of interest. In S1210, CPU 151 assigns a new subject ID to the region of interest, adds it to the subject classification result, and records it in RAM 154. In S1211, CPU 151 updates the region of interest as a used state, and records it in RAM 154.

[0094] In S1212, the CPU 151 performs a condition determination as to whether or not there is a detection 1 region that is associated with the region of interest in S207 of Fig. 2 and is in an unused state. If this condition is met, the process proceeds to S1213, and if the condition is not met, the process proceeds to S1215.

[0095] In S1213, CPU 151 assigns the same subject ID as the subject assigned in S1210 to the detection 1 region linked to the region of interest, adds the detection 1 region to the subject classification result, and records it in RAM 154. In S1214, CPU 151 updates the detection 1 region linked to the region of interest to a used state, and records it in RAM 154.

[0096] In S1215, the CPU 151 performs a condition determination as to whether or not there is an unused area among the two detection areas obtained from the current image. If this condition is met, the process returns to S1209 and continues, and if the condition is not met, the new subject addition process ends.

[0097] According to this embodiment, it is possible to suppress errors due to different part search, while increasing the success rate of subject detection in consecutive frames and continuing the search.

[0098] [Example 2] Next, a second embodiment of the present invention will be described. In this embodiment, the same matters as in the first embodiment will be omitted, and differences will be mainly described. The method of omitting such explanations will be the same in the embodiments described later. In this embodiment, it is assumed that the body part detection unit 161 can detect a person's face region and a person's torso region. The detection 1 region is the face region, and the detection 2 region is the torso region. It is also assumed that the face region, which is of great interest to the photographer, is more important than the torso region.

[0099] The search process (FIG. 2: S209) in this embodiment will be described in detail with reference to FIG. 15 and FIG. 16. In S1300 in FIG. 15, CPU 151 determines whether or not the object classification result is recorded in RAM 154 in the search process for the previously acquired image. If it is determined that the object classification result is recorded, the process proceeds to S1301, and if it is determined that the object classification result is not recorded, the search process ends.

[0100] In S1301, the CPU 151 determines a target subject from the subject classification results recorded in the RAM 154. When there are multiple candidates for the target subject, a process is performed in which a subject having a high subject priority calculated in S211 of FIG. 2 is given priority.

[0101] In S1302, the CPU 151 determines whether or not a detection 1 area classified as a subject of interest in a previously acquired image has been recorded in the RAM 154. If it is determined that the detection 1 area has been recorded, the process proceeds to S1303, and if it is determined that the detection 1 area has not been recorded, the process proceeds to S1305.

[0102] In S1303, CPU 151 performs a same part search for multiple detection 1 regions obtained from the current image, using the detection 1 region classified as the subject of interest in the previously acquired image as a reference. In S1304, CPU 151 performs a condition determination as to whether a subject that satisfies the conditions for the subject of interest has been found in the detection region of the current image and the subject classification result has been updated. If this condition is met, the process proceeds to S1313 in FIG. 16, and if the condition is not met, the process proceeds to S1305.

[0103] In S1305, the CPU 151 determines whether or not the detection 2 region classified as the subject of interest in the previously acquired image is recorded in the RAM 154. If it is determined that the detection 2 region is recorded, the process proceeds to S1306, and if it is determined that the detection 2 region is not recorded, the process proceeds to S1308.

[0104] In S1306, CPU 151 performs a different part search for multiple detection 1 regions obtained from the current image, using the detection 2 region classified as the target subject in the previously acquired image as a reference. In S1307, CPU 151 performs a condition determination as to whether a subject that satisfies the conditions for the target subject has been found in the detection region of the current image, and the subject classification result has been updated. If this condition is met, the process proceeds to S1313 in FIG. 16, and if the condition is not met, the process proceeds to S1308.

[0105] In S1308, the CPU 151 performs a condition determination as to whether or not a detected 2 region classified as a subject of interest in a previously acquired image is recorded in the RAM 154. If this condition is met, the process proceeds to S1309, and if the condition is not met, the process proceeds to S1311 in FIG.

[0106] In S1309, CPU 151 performs a same part search for multiple detection 2 regions obtained from the current image, using the detection 2 regions classified as the subject of interest in the previously acquired image as a reference. In S1310, CPU 151 performs a condition determination as to whether a subject that satisfies the conditions for the subject of interest has been found in the detection regions of the current image and the subject classification result has been updated. If this condition is met, the process proceeds to S1313 in FIG. 16, and if the condition is not met, the process proceeds to S1311 in FIG. 16.

[0107] 16, CPU 151 performs a condition determination as to whether or not a detection 1 region classified as a subject of interest in a previously obtained image is recorded in RAM 154. If this condition is met, the process proceeds to S1312, and if the condition is not met, the process proceeds to S1313. In S1312, CPU 151 performs a different part search for multiple detection 2 regions obtained from the current image, using the detection 1 region classified as a subject of interest in the previously obtained image as a reference.

[0108] 5 described in the first embodiment, the same part search process, different part search process, prohibited area update process, and new subject addition process in this embodiment are the same as those in the first embodiment.

[0109] In this embodiment, the same-part search process with higher accuracy is executed with higher priority than the different-part search process with lower accuracy. When the target subject is updated in the prioritized search process, the other search process is not executed, so that the highly accurate search process can be executed. In addition, with regard to the same-part search process, the process of S1303 is executed with higher priority than S1309 in FIG. 15, so that the probability that the target subject information is updated in a detection area with higher importance can be increased. According to this embodiment, it is possible to suppress errors due to different-part search, and to continue searching by increasing the success rate of subject detection in consecutive frames while prioritizing areas with higher importance.

[0110] [Example 3] Next, the differences between the second embodiment and the third embodiment of the present invention will be described. The part detection unit 161 of this embodiment is assumed to be capable of detecting a person's whole body region and a person's eye region. The detection 1 region is assumed to be the whole body region, and the detection 2 region is assumed to be the eye region. The size of the person's whole body is assumed to be sufficiently larger than the size of the person's eyes, and therefore has a high importance. Alternatively, the person's whole body region is assumed to be highly important, as it contains the person's eye regions. The search process in this embodiment is the same as in the second embodiment (descriptions of the same part search process, different part search process, prohibited area update process, new subject addition process, etc. will be omitted).

[0111] In this embodiment, it is possible to perform a highly accurate search process similar to that in the second embodiment. In addition, since the process of S1303 is performed with priority over S1309 in Fig. 15, it is possible to suppress search errors due to local regions (small-sized regions). According to this embodiment, it is possible to suppress errors due to different part search and to increase the success rate of subject detection in consecutive frames while suppressing errors due to local regions.

[0112] [Other Examples] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. [Explanation of symbols]

[0113] 100 Imaging device 141 Image sensor 151 CPU 152 Image processing section 161 Body part detection unit 162 Labeling Processing Unit

Claims

1. An acquisition means for acquiring a captured image; a detection means for detecting a plurality of parts of a subject from the image; a search means for extracting a part that satisfies a predetermined condition that is set based on a part detected by the detection means based on a first image previously acquired from among a plurality of parts detected by the detection means based on a second image currently acquired by the acquisition means, and for storing the extracted part in a storage means in association with the subject, The search means performs a different part search using a detection result of a different part between the first image and the second image, and a same part search using a detection result of a same part between the first image and the second image, and performs the same part search with priority over the different part search.

13. An image processing device comprising:

2. the detection means performs a first detection for detecting a first portion of a subject from the image, and a second detection for detecting a second portion of the subject from the image; When the importance of the detection result by the first detection is higher than the importance of the detection result by the second detection, the search means prioritizes a search using the detection result by the first detection.

2. The image processing device according to claim 1,

3. The detection means detects a face region of the subject in the first detection and detects a torso region of the subject in the second detection, or detects a whole body region of the subject in the first detection and detects an eye region of the subject in the second detection.

3. The image processing device according to claim 2.

4. The search means prioritizes a search using a detection result of a large-sized portion over a detection result of a small-sized portion.

3. The image processing device according to claim 2.

5. When the first portion includes the second portion, the search means prioritizes a search using a detection result of the first portion.

3. The image processing device according to claim 2.

6. A labeling processing means for acquiring a label map for each object in the acquired image; a linking processing means for linking a detection area obtained by the first detection with a detection area obtained by the second detection using the label map.

3. The image processing device according to claim 2.

7. The search means performs a process of classifying the plurality of detection results by subject when a detection result that satisfies a predetermined condition and is linked by the linking processing means is found among the plurality of detection results obtained from the second image.

7. The image processing device according to claim 6,

8. The search means performs a search process for each of the plurality of objects in the second image when there is a detection result in which the objects are classified into a plurality of objects in the first image.

8. The image processing device according to claim 1,

9. A calculation means for calculating a priority of a plurality of objects, The search means performs a search process by giving priority to the subject having a high priority.

9. The image processing device according to claim 1,

10. The apparatus further includes a registration unit for registering a detection result that is not classified by the search unit as a detection result of a new subject.

10. The image processing device according to claim 1,

11. The search means does not use a detection result classified as a first object among the detection results obtained from the second image in a search process for a second object.

11. The image processing device according to claim 1,

12. The search means performs the different part search using a part vector acquired using the detection results of the first and second detections.

3. The image processing device according to claim 2.

13. A recording means for recording the site vector is provided, When the detection results by the first and second detections obtained from the first image are classified as a detection result of a target object, the search means calculates a search range for searching for a different part in the second image using the part vector. The image processing device according to claim 12 .

14. A deletion means is provided for deleting the region vector from the recording means when the region vector is not updated within a predetermined period. The image processing device according to claim 13 .

15. a recording means for acquiring information indicating a change in a position or an attitude of the imaging means when the image is acquired, and for recording a detection result in which the searching means has not classified the first image as a subject of interest, and a prohibited area calculated using the acquired information, The searching means does not use any detection result that belongs to the prohibited area among the detection results obtained from the second image in the search process for the target subject.

13. The image processing device according to claim 1,

16. a motion vector calculation means for calculating a motion vector within an image based on the first and second images and the information; a deletion means for deleting the prohibited area from the recording means when the motion vector corresponding to the prohibited area exists.

16. The image processing device according to claim 15,

17. the acquiring means acquiring distance information associated with the image; The deletion means deletes the prohibited area from the recording means when a change in the distance information corresponding to the prohibited area is greater than a threshold value.

17. The image processing device according to claim 16,

18. The deletion means deletes the prohibited area from the recording means when a detection result belonging to the prohibited area is not obtained in the second image.

17. The image processing device according to claim 16,

19. An image processing device according to any one of claims 1 to 18, 1. An imaging device comprising:

20. A control method executed in an image processing device that acquires and processes a captured image, comprising: a detection step in which a detection means detects a plurality of parts of a subject from the image; a searching step of extracting a part that satisfies a predetermined condition that is set based on a part detected by the detection means based on a first image previously acquired from among a plurality of parts detected by the detection means based on a second image currently acquired, and storing the extracted part in a storage means in association with the subject, In the searching step, a different part search is performed using a detection result of a different part between the first image and the second image, and a same part search is performed using a detection result of a same part between the first image and the second image, and the same part search is performed with priority over the different part search.

23. A method for controlling an image processing apparatus comprising the steps of:

21. The steps according to claim 20 are executed by a computer of an image processing device. A program characterized by:

Images