Image processing device and image processing method

Abstract

To provide an image processing device and image processing method, which allow multiple different parts belonging to a single object to be accurately associated with each other.SOLUTION: An image processing device disclosed herein detects a first part and second part of a specific object from an image and estimates a movement direction of the specific object. The image processing device associates parts of the same object with each other, from among the detected first part and second part, on the basis of the estimated movement direction.SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing device and an image processing method, and more particularly to a technique for detecting a subject. [Background technology]

[0002] There is known a technology that uses machine learning to detect an area (subject area) in an image in which a specific subject is captured (Patent Document 1). In Patent Document 1, when tracking a part of a specific subject, the whole of the specific subject and the part are detected separately. Then, in order to improve tracking accuracy, it is determined whether the detection results belong to the same subject or not based on the relationship between the detection positions of the whole and the part. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2021-152578 A Summary of the Invention [Problem to be solved by the invention]

[0004] The method described in Patent Document 1 requires that a part is detected within the entire region, and cannot be used to associate multiple different parts belonging to the same subject.

[0005] The present invention has been made in view of the problems with the conventional techniques. In one aspect, the present invention provides an image processing apparatus and an image processing method capable of associating multiple different parts of the same subject with each other with high accuracy. [Means for solving the problem]

[0006] The above-mentioned object is achieved by an image processing device having a detection means for detecting a first part and a second part of a specific subject from an image, an estimation means for estimating a moving direction of the specific subject, and a matching means for matching parts of the same subject among the first part and the second part detected by the detection means based on the estimated moving direction. Effect of the Invention

[0007] According to the present invention, it is possible to provide an image processing device and an image processing method that are capable of associating multiple different parts of the same subject with each other with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing an example of the functional configuration of an imaging apparatus as an example of an image processing apparatus according to an embodiment; [Diagram 2] FIG. 11 is a block diagram showing an example of the functional configuration of a subject detection unit in the first and third embodiments; [Diagram 3] FIG. 13 is a diagram showing an example of a setting screen for a priority subject presented by the imaging device; [Figure 4] Flowchart for subject detection processing in the first embodiment [Diagram 5] FIG. 13 is a diagram showing an example of dictionary data switching operation. [Figure 6] Flowchart of movement direction estimation process in the embodiment [Figure 7] FIG. 2 is a diagram showing a schematic diagram of a method for associating parts in the first embodiment; [Figure 8] FIG. 13 is a block diagram showing an example of the functional configuration of a subject detection unit in the second embodiment. [Figure 9] Flowchart for subject detection processing in the second embodiment [Figure 10] FIG. 11 is a diagram illustrating a method for associating parts in a second embodiment. [Figure 11] Flowchart for subject detection processing in the third embodiment [Figure 12]FIG. 13 is a diagram showing a schematic diagram of a method for associating parts in a third embodiment; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] The present invention will be described in detail below based on its exemplary embodiments with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. In addition, although multiple features are described in the embodiments, not all of them are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated explanations are omitted.

[0010] In the following embodiment, the present invention will be described with respect to a case where the present invention is implemented in an imaging device such as a digital camera. However, the imaging function is not essential to the present invention, and the present invention can be implemented in any electronic device. Such electronic devices include video cameras, computer devices (personal computers, tablet computers, media players, PDAs, etc.), mobile phones, smartphones, game consoles, robots, drones, and drive recorders. These are merely examples, and the present invention can also be implemented in other electronic devices.

[0011] (Configuration of imaging device) 1 is a block diagram showing an example of a functional configuration of an imaging device 100 according to an embodiment. The imaging device 100 is capable of capturing and recording moving images and still images. The functional blocks of the imaging device 100 are communicably connected to each other via a bus 160. The operation of the imaging device 100 is realized by a main control unit (CPU) 151 loading a program stored in a ROM 155 into a RAM 154, executing the program, and controlling each functional block.

[0012] In the figure, functional blocks labeled "~ unit" may be realized by dedicated hardware such as an ASIC. Alternatively, they may be realized by a processor such as a CPU executing a program stored in memory. Note that multiple functional blocks may be realized by a common configuration (e.g., one ASIC). Furthermore, hardware that realizes part of the functions of a certain functional block may be included in hardware that realizes another functional block.

[0013] The subject detection unit 161 detects two or more body part regions for a subject to be detected (specific subject). For example, if the specific subject is a human or an animal, the subject detection unit 161 detects a face region and a torso region. In addition, the subject detection unit 161 associates the detected body parts that belong to the same subject. The configuration and operation of the subject detection unit 161 will be described in detail later.

[0014] The photographing lens (lens unit) 101 has a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed third group lens 121, a focus lens 131, a zoom motor 112, an aperture motor 104, and a focus motor 132. The fixed first group lens 102, the zoom lens 111, the aperture 103, the fixed third group lens 121, and the focus lens 131 constitute a photographing optical system. Although each lens is illustrated as a single lens for convenience, each lens may be composed of multiple lenses. The photographing lens 101 may be configured as a detachable lens unit. The aperture 103 may have a mechanical shutter function.

[0015] An aperture control unit 105 controls the operation of an aperture motor 104 that drives the aperture 103, and changes the opening diameter of the aperture 103. A zoom control unit 113 controls the operation of a zoom motor 112 that drives a zoom lens 111, and changes the focal length (angle of view) of the photographing lens 101.

[0016] The focus control unit 133 performs autofocus detection (AF) using an imaging surface phase difference detection method. That is, the focus control unit 133 calculates the defocus amount and defocus direction of the photographing lens 101 based on the phase difference between a pair of focus detection signals (images A and B) obtained from the image sensor 141. The focus control unit 133 then converts the defocus amount and defocus direction into a drive amount and drive direction of the focus motor 132. Based on the drive amount and drive direction, the focus control unit 133 controls the operation of the focus motor 132 and drives the focus lens 131, thereby controlling the focus state of the photographing lens 101.

[0017] The focus control unit 133 may calculate the defocus amount and defocus direction of the photographing lens 101 based on the phase difference between a pair of focus detection signals (images A and B) obtained from the AF sensor. The focus control unit 133 may also execute AF using a contrast detection method. In this case, the focus control unit 133 calculates a contrast evaluation value from an image signal obtained from the image sensor 141, and drives the focus lens 131 to a position where the contrast evaluation value is maximized.

[0018] The imaging element 141 may be, for example, a known CCD or CMOS color image sensor having a primary color Bayer array color filter. The imaging element 141 has a pixel array in which a plurality of pixels are arranged two-dimensionally, and a peripheral circuit for reading out signals from each pixel. Each pixel has a photoelectric conversion region and accumulates electric charge according to the amount of incident light. By reading out a signal having a voltage according to the amount of electric charge accumulated during the exposure period from each pixel, a group of pixel signals (analog image signals) representing the subject image formed on the imaging plane by the photographing lens 101 can be obtained.

[0019] In this embodiment, the image sensor 141 can generate a focus detection signal in addition to an analog image signal. Specifically, each pixel has a plurality of photoelectric conversion regions (sub-pixels). The image sensor 141 is configured to be able to read out a signal for each photoelectric conversion region. For example, each pixel has two photoelectric conversion regions A and B of the same size arranged in the horizontal direction. In this case, for the pixels included in the focus detection region, an image A is generated from a signal read out from the photoelectric conversion region A, and an image B is generated from a signal read out from the photoelectric conversion region B, and AF of the phase difference detection method can be performed. Therefore, a signal read out from one of the photoelectric conversion regions A and B can be used as a focus detection signal. A signal read out from both the photoelectric conversion regions A and B can be used as a normal pixel signal. The imaging control unit 143 controls how to read out signals from the image sensor 141 according to instructions from the CPU 151.

[0020] The analog image signal read out from the imaging element 141 is supplied to a signal processing unit 142. The signal processing unit 142 applies signal processing such as noise reduction processing, A / D conversion processing, and automatic gain control processing to the analog image signal. The signal processing unit 142 supplies a digital image signal (image data) obtained by applying the signal processing to an imaging control unit 143. The imaging control unit 143 stores the image signal data supplied from the signal processing unit 142 in a RAM (random access memory) 154.

[0021] The motion sensor 162 outputs a signal corresponding to the motion of the imaging device 100. The motion sensor 162 outputs a signal corresponding to the motion in the translation direction and the rotation direction in an orthogonal coordinate system in which the gravity direction is the Z axis. The motion sensor 162 may be, for example, a combination of an angular velocity sensor and an acceleration sensor. The motion sensor 162 stores the signal in the RAM 154 at regular intervals, for example. The subject detection unit 161 can acquire information related to the motion of the imaging device 100 by referring to the RAM 154.

[0022] The image processing unit 152 applies predetermined image processing to the image data stored in the RAM 154. The image processing applied by the image processing unit 152 includes, but is not limited to, so-called development processing such as white balance adjustment processing, color interpolation (demosaic) processing, and gamma correction processing, as well as signal format conversion processing, scaling processing, etc. In addition, the image processing unit 152 can also generate information on subject brightness to be used for automatic exposure control (AE).

[0023] The image processing unit 152 may use the detection result of the specific subject supplied from the subject detection unit 161 for, for example, white balance adjustment processing. When performing AF by a contrast detection method, the image processing unit 152 may generate an AF evaluation value. The image processing unit 152 stores the image data to which the image processing has been applied in the RAM 154.

[0024] When recording image data stored in RAM 154, CPU 151 generates a data file according to the recording format by, for example, adding a predetermined header to the image data. At this time, CPU 151 can reduce the amount of data by encoding the image data using CODEC 153 as necessary. CPU 151 records the generated data file on recording medium 157, such as a memory card.

[0025] Furthermore, when displaying image data stored in RAM 154, CPU 151 generates image data for display by scaling the image data in image processing unit 152 so as to fit the display size of display 150. CPU 151 then writes the image data for display to an area (VRAM area) of RAM 154 used as a video memory. Display 150 reads the image data for display from the VRAM area of ​​RAM 154 and displays it.

[0026] When in standby for shooting or during video recording, imaging device 100 causes display 150 to function as an electronic viewfinder (EVF) by instantly displaying a captured video on display 150. The video and its frame images displayed to cause display 150 to function as an EVF are called live view images or through images. When imaging device 100 captures a still image, it displays the still image captured immediately before on display 150 for a certain period of time so that the user can check the shooting results. These display operations are also realized under the control of CPU 151.

[0027] The input device 156 is a switch, a button, a key, a touch panel, an eye-gaze input device, etc., provided on the imaging device 100. The input through the input device 156 is detected by the CPU 151 via the bus 160, and the CPU 151 controls each functional block to realize an operation according to the input. Note that, if the display 150 is a touch display, the touch panel of the display 150 is included in the input device 156.

[0028] The CPU 151 controls each functional block by, for example, loading a program stored in the ROM 155 into the RAM 154 and executing it, thereby realizing the functions of the imaging device 100. The CPU 151 also executes AE processing for automatically determining exposure conditions (shutter speed or accumulation time, aperture value, sensitivity) based on information on the subject brightness. Information on the subject brightness can be acquired, for example, from the image processing unit 152. The CPU 151 may determine the exposure conditions based on brightness information on the area of ​​a specific subject detected by the subject detection unit 161, such as a person's face.

[0029] Based on the determined exposure conditions, the CPU 151 controls the operations of the imaging control unit 143 and the aperture control unit 105. The shutter speed is used to control the opening and closing of the aperture 103 when shooting a still image, and to control the accumulation time of the image sensor 141 when shooting a moving image. The shooting sensitivity is provided to the imaging control unit 143, and the imaging control unit 143 controls the gain of the image sensor 141 according to the shooting sensitivity.

[0030] The detection result of the part of the specific subject by the subject detection unit 161 can be used for automatic setting of the focus detection area by the CPU 151. By automatically setting the focus detection area following the detection result of the same part, a tracking AF function can be realized. In addition, it is also possible to perform AE processing based on the luminance information of the focus detection area, and to perform image processing (e.g., gamma correction processing, white balance adjustment processing, etc.) based on the pixel value of the focus detection area. Note that the CPU 151 may superimpose an index (e.g., a rectangular frame surrounding the focus detection area) indicating the position of the currently set focus detection area on the live view image.

[0031] The battery 159 is managed by a power management unit 158 ​​and supplies power to the entire imaging device 100 .

[0032] The RAM 154 is used to load the programs executed by the CPU 151 and to temporarily store variables and the like while the programs are being executed. The RAM 154 is also used as a temporary storage location for image data processed by the image processing unit 152, image data being processed, and image data that has already been processed. Furthermore, a part of the RAM 154 is also used as a video memory (VRAM) for the display 150.

[0033] The ROM 155 is a rewritable non-volatile memory that stores programs executed by the CPU 151, various setting values ​​of the imaging device 100, GUI data, and the like.

[0034] For example, when an instruction to switch from a power-off state to a power-on state is given by operating the input device 156, the CPU 151 loads a program stored in the ROM 155 into a part of the RAM 154. The CPU 151 executes the program, causing the imaging device 100 to switch to a shooting standby state. When the imaging device 100 switches to the standby state, the CPU 151 executes processing in the shooting standby state, such as live view display.

[0035] (Configuration of subject detection unit) Fig. 2 is a block diagram mainly showing an example of the functional configuration of subject detection unit 161. Subject detection unit 161 has a dictionary data selection unit 201, a dictionary data storage unit 202, a part detection unit 203, a history storage unit 204, a movement direction estimation unit 205, a part correlation unit 206, and a determination unit 207. Although subject detection unit 161 is shown as an independent functional block in Fig. 1, it may actually be implemented by CPU 151 executing a program, or it may be implemented by image processing unit 152.

[0036] The body part detection unit 203 detects multiple body parts of a specific subject using a convolutional neural network (CNN) in which learned parameters are set. The specific subject to be detected and the learned parameters for each body part are stored as dictionary data in the dictionary data storage unit 202. The body part detection unit 203 may have separate CNNs depending on the combination of the type of specific subject to be detected and the body part to be detected. The body part detection unit 203 may be realized using a GPU (Graphics Processing Unit) or a circuit for quickly executing CNN calculations (NPU (Neural Processing Unit)).

[0037] Machine learning of CNN parameters can be performed by any known method depending on the structure. For example, the CNN has a structure in which a fully connected layer and an output layer are connected to a stacked structure in which a plurality of convolutional layers and pooling layers are alternately arranged. In this case, the machine learning of the CNN can be performed by the backpropagation method. In addition, when the CNN is a neocognitron CNN in which a feature detection layer (S layer) and a feature integration layer (C layer) are set, for example, a learning method called "Add-if Silent" can be used. Note that the configuration and learning method of the CNN described here are merely examples, and are not intended to limit the configuration and learning method of the CNN.

[0038] The machine learning of the CNN can be executed by a computer, such as a server, separate from the imaging device 100. In this case, the imaging device 100 can acquire the trained CNN from the computer and use it. Here, the machine learning is assumed to be supervised learning. Specifically, the machine learning of the CNN used in the part detection unit 203 is executed by using image data for learning in which a specific subject is captured and teacher data (annotation) corresponding to the image data for learning. The teacher data includes at least position information of the part of the specific subject to be detected by the part detection unit 203. The machine learning of the CNN may be executed by the imaging device 100.

[0039] Part detection unit 203 inputs image data captured using image sensor 141 to a trained CNN (trained model), and outputs the position and size of a part of a specific subject, detection reliability, and the like as detection results. Part detection unit 203 does not detect a specific subject and then detects a part, but instead detects the part directly, so information regarding which subject the detected part belongs to is not included in the detection result. In addition, detection is performed separately for each part.

[0040] Note that the part detection unit 203 is not limited to a configuration using a trained CNN. For example, the part detection unit 203 may be realized using a trained model generated by machine learning such as a support vector machine or a decision tree.

[0041] In addition, the part detection unit 203 does not have to be a trained model generated by machine learning. For example, dictionary data generated by a rule base without machine learning may be used. The dictionary data generated by the rule base is, for example, image data of a part of a specific subject or data of feature amounts specific to a part of a specific subject determined by a designer. By comparing the image data or feature amount data included in the dictionary data with the captured image data or its feature amount, the part of a specific subject can be detected. The rule-based dictionary data is simpler and has a smaller amount of data than the trained model generated by machine learning. Therefore, subject detection using the rule-based dictionary data can be performed faster with a lower processing load than when a trained model is used.

[0042] History storage unit 204 stores the detection results of part detection unit 203 and information on subject parts associated by part correlation unit 206. Furthermore, history storage unit 204 supplies the stored history to dictionary data selection unit 201. History storage unit 204 stores, as detection history, dictionary data used for detection, the position and size of the detected subject area, detection reliability, and information on correlated parts. However, without being limited to these, other information related to detection, such as the number of detections and identification information (such as a file name) of the image data used for detection, may also be stored.

[0043] The dictionary data storage unit 202 stores learned parameters for detecting body parts of a specific subject as dictionary data. The dictionary data storage unit 202 stores separate dictionary data for each combination of the type and body part of a specific subject. For example, the dictionary data storage unit 202 can store dictionary data for detecting the "head" and dictionary data for detecting the "torso" for the specific subject "human". In addition, dictionary data may be stored for a part of a body part as a separate body part. For example, dictionary data for detecting a face included in the head of a human or animal, and dictionary data for detecting facial parts (eyes, pupils, etc.) may be stored.

[0044] Dictionary data selection unit 201 reads out dictionary data corresponding to a detection target of part detection unit 203 from dictionary data storage unit 202, and supplies the dictionary data to part detection unit 203. Dictionary data selection unit 201 can supply dictionary data to part detection unit 203 in an order based on detection history stored in history storage unit 204, for example.

[0045] The movement direction estimation unit 205 estimates the movement direction of the specific subject based on the detection result of the part detection unit 203, the detection history stored in the history storage unit 204, and the movement of the imaging device 100 detected by the movement sensor 162.

[0046] A part correlation section 206 takes into consideration the moving direction of the subject estimated by the moving direction estimation section 205, and specifies and associates parts belonging to the same subject from among the parts detected by the part detection section 203.

[0047] The determination unit 207 determines the main subject from among the specific subjects including the parts associated by the part correlation unit 206. If there is one specific subject, the determination unit 207 determines the specific subject as the main subject. If there are multiple specific subjects, the determination unit 207 determines one of them as the main subject. The determination unit 207 can determine the main subject by any known method based on the position and / or size of the subject area, user settings, etc.

[0048] 3 shows an example of a setting screen for a specific subject (priority subject) that should be preferentially detected by the subject detection function of the imaging device 100. The setting screen 300 can be called up, for example, from a menu screen through the operation of the input device 156. The setting screen 300 includes a list 310 that displays selectable types of priority subjects. The list 310 includes types of specific subjects that the subject detection unit 161 can detect, and "Auto" indicating that there is no priority subject. The list 310 also includes "None" which disables detection of a specific subject.

[0049] In addition, when the specific subjects are classified hierarchically, the priority subject may be set for any hierarchy. For example, when the living subject has humans and animals in a lower hierarchy, and the animals have dogs, cats, horses, and birds in a lower hierarchy, any of "horses," "animals," and "living things" may be set as the priority subject.

[0050] The user can operate the input device 156 (e.g., a direction key) to move the cursor 315 within the list 310. The user can execute the setting by operating the input device 156 (e.g., a setting button) while the desired setting is selected by the cursor 315. The CPU 151 stores the item selected when the setting button is operated in, for example, the ROM 155 as a setting related to the priority subject.

[0051] When a priority subject is set, the determination unit 207 determines the priority subject as the main subject by giving priority to the priority subject. When the priority subject is set to a classification having a lower layer such as "animals", the determination unit 207 determines the main subject from among the subjects of the lowest layer type. For example, the priority subject is set to "animals" and the lowest layer of "animals" has "dogs and cats, horses, and birds". In this case, the determination unit 207 determines one of the detected dogs and cats, horses, and birds as the main subject. When multiple subjects are detected, the determination unit 207 can determine the main subject by a known method such as the subject whose detection position is closest to the center of the image, the subject whose detection size is the largest, or the subject whose reliability is the highest.

[0052] (Subject detection processing) The subject detection process will be described with reference to the flowchart shown in Fig. 4. Note that the operation described below is assumed to be performed while the imaging device 100 is powered on and in a shooting standby state. In the shooting standby state, the device is assumed to be waiting for an instruction to shoot (prepare) a still image or video while continuously executing live view display.

[0053] A series of processes from S401 to S408 is executed within one frame period of a moving image for live view display by the imaging control unit 143 of the imaging device 100, but may be executed over a predetermined period of multiple frames. For example, a result of subject detection in the first frame may be reflected from any of the second and subsequent frames.

[0054] In S401, the CPU 151 controls the imaging control unit 143 to capture one frame of image.

[0055] In S402, the dictionary data selection unit 201 of the subject detection unit 161 selects dictionary data to be used for subject detection. As described above, the dictionary data is parameters generated by performing learning in an external device, and is set in the CNN of the body part detection unit 203 for use.

[0056] The dictionary data switching operation by the dictionary data selection unit 201 will be described with reference to Fig. 5. As described above, the dictionary data storage unit 202 stores separate dictionary data for each combination of the type of subject and the type of body part to be detected. By switching the dictionary data set in the CNN, it is possible to change the subject and body part to be detected by the CNN. Therefore, for example, when detecting a human head and torso in an image of one frame, it is necessary to apply a detection process using dictionary data for a human head and a detection process using dictionary data for a human torso to the image data of the same frame.

[0057] On the other hand, the time during one frame period (one vertical synchronization period) during which the subject detection process can be used is limited by the frame rate, exposure time, etc. Therefore, particularly when the subject detection process is performed for each frame, it may happen that only a limited number of dictionary data can be used.

[0058] Therefore, the dictionary data selection unit 201 determines the type and use order of dictionary data to be used in the subject detection process, taking into consideration the presence or absence of a specific subject to be prioritized, the detection history, etc. An example of the operation of the dictionary data selection unit 201 will be described with reference to Figs. 5(a) and 5(b).

[0059] Here, it is assumed that the dictionary data can be switched three times in one frame period to execute the subject detection process. Also, it is assumed that "animals" are set as the priority subject. V0, V1, and V2 indicate the vertical synchronization periods of the first to third frames, respectively.

[0060] 5(a) shows an example of the switching operation of dictionary data supplied by dictionary data selection unit 201 to body part detection unit 203 when a specific subject is not detected. In this case, in the first frame, dictionary data selection unit 201 supplies to body part detection unit 203 dictionary data for a person's head, dictionary data for animals (dog / cat head), and dictionary data for animals (dog / cat body) in this order. In addition, in the second frame, dictionary data selection unit 201 supplies to body part detection unit 203 dictionary data for a person's head, dictionary data for animals (horse head), and dictionary data for animals (horse body) in this order. Then, in the third frame, dictionary data selection unit 201 supplies to body part detection unit 203 dictionary data for a person's head, dictionary data for animals (bird head), and dictionary data for animals (bird body).

[0061] In this embodiment, during a period when a specific subject is not detected, the dictionary data selection unit 201 supplies dictionary data for detecting a person's head and dictionary data for detecting a priority subject in each frame. Here, since "animals" are set as the priority subject, the dictionary data selection unit 201 sequentially supplies dictionary data for detecting the heads and bodies of "dogs, cats, horses, and birds," which are lower in rank than "animals," to the body part detection unit 203. As a result, detection processing for all types of detectable animals is performed over a period of three frames.

[0062] Among the dictionary data for detecting different body parts of the same type of specific subject, dictionary data for detecting large body parts is preferentially selected for use during a period when a specific subject is not detected. For example, if dictionary data exists for four body parts, "head," "torso," "face," and "eyes," dictionary data for detecting "torso" and "head" is selected with priority over dictionary data for detecting "face" and "eyes."

[0063] In addition, the selection priority of dictionary data of subjects that are unlikely to exist at the same time can be lowered. For example, if the priority subject is set to an animal, dictionary data related to subjects such as "airplanes" and "trains" can be set not to be selected (not detected). This makes it possible to increase the frequency of detection processing for priority subjects.

[0064] 5(b) shows an example of dictionary data selection operation when a horse's body and / or head is detected in the previous frame. In the first frame, the dictionary data selection unit 201 supplies dictionary data to the body part detection unit 203 in the order of animal (horse head), animal (horse eye), and animal (horse body). When priority subjects have been detected in all frames, the dictionary data selection unit 201 supplies dictionary data relating to the detected priority subjects to the body part detection unit 203.

[0065] By sequentially supplying dictionary data for detecting different parts of the same type of subject to the part detection unit 203, even if one part is no longer detected, tracking of the same subject can be continued if another part is detected. In the example of FIG. 5(b), only dictionary data related to the detected priority subject is supplied, but supply of dictionary data related to other subjects is not excluded. For example, the dictionary data supplied last for each vertical synchronization period (or a predetermined number of vertical synchronization periods) may be changed to dictionary data for detecting a person (head). This makes it possible to detect subjects other than the priority subject, such as a person riding a horse in the example of FIG. 5(b).

[0066] Note that even in a configuration in which body part detection unit 203 executes three object detection processes in parallel, thereby performing object detection processing three times in one vertical synchronization period, dictionary data selection unit 201 can select three dictionary data in a similar manner. However, since there is no priority order for the execution order of the object detection processes, the three dictionary data are input in parallel from dictionary data selection unit 201 to body part detection unit 203.

[0067] Returning to Fig. 4, in S403, the image processing unit 152 processes the image data into a state suitable for subject detection processing. For example, the image processing unit 152 reduces the image size in order to reduce the amount of processing. The image processing unit 152 may reduce the entire image, or may reduce the image size by cropping the image. Depending on the detection part, cropping can improve the accuracy of the subject detection processing.

[0068] The image processor 152 can crop the image according to the detected part, for example. For example, when detecting a small part such as an eye, cropping the area including the eye rather than reducing the entire image can reduce the image size without reducing the eye area. In addition, because unnecessary areas are reduced by cropping, it is expected that the detection accuracy will improve.

[0069] Image processing unit 152 can specify the detected part by acquiring information on dictionary data to be supplied to part detection unit 203 from dictionary data selection unit 201 or dictionary data storage unit 202. Image processing unit 152 can also acquire information on the detected position and size of the part from the detection results in frames older than the current frame that are stored in history storage unit 204. Image processing unit 152 can determine the cropping range of the image based on this information.

[0070] For example, when detecting the eyes of a horse, the area in which the horse's eyes are captured can be trimmed by determining the trimming area centered on the detection position of the horse's head. The size of the area to be trimmed can be the input image size of the CNN if the input image size of the CNN is equal to or larger than the detection size. If the input image size of the CNN is smaller than the detection size, the image may be trimmed to a size based on the detection size and then reduced to the input image size of the CNN. Note that these are merely examples, and trimming may be performed by other methods.

[0071] The image processing unit 152 supplies data of the image whose size has been adjusted to the part detection unit 203 of the subject detection unit 161 .

[0072] In S404, part detection unit 203 obtains the dictionary data selected by dictionary data selection unit 201 from dictionary data storage unit 202, and sets it in CNN. Then, part detection unit 203 inputs image data supplied from image processing unit 152 to CNN, and applies subject detection processing. Part detection unit 203 outputs the detection result to part correlation unit 206 and history storage unit 204. The detection result may include, but is not limited to, the position and size of the part of the detected specific subject, the detection reliability, information specifying the dictionary data and image data used, and the like.

[0073] History storage section 204 stores the detection result when it receives it from part detection section 203. Note that history storage section 204 may be configured to delete old history that satisfies a predetermined condition.

[0074] In S405, the dictionary data selection unit 201 judges whether or not the subject detection process has been performed for all parts to be detected in the current frame. This judgment corresponds to the judgment of whether or not the subject detection process to be performed in the Vn period (n is 0, 1, or 2) has been performed (whether the subject detection process has been performed using three dictionary data) in FIG. 5(a).

[0075] If it is determined that the object detection process to be executed for the current frame has not been completed, the process from S402 is executed again, and the dictionary data selection unit 201 selects the next dictionary data. On the other hand, if it is determined that the object detection process to be executed for the current frame has been completed, S406 is executed.

[0076] In S406, the dictionary data selection unit 201 determines whether or not there is any dictionary data to be used in the next frame among the dictionary data not used in the current frame. This determination can be judged as Yes when the subject detection process for all parts to be detected is executed over multiple frames as in FIG. 5(a). Specifically, when the current frame corresponds to the first or second frame in FIG. 5(a), the dictionary data selection unit 201 determines as Yes. On the other hand, when the current frame corresponds to the third frame in FIG. 5(a) or the first or second frame in FIG. 5(b), the dictionary data selection unit 201 determines as No.

[0077] If the determination in S406 is Yes, the processes in S407 to S409 are skipped, and the process proceeds to the next frame. On the other hand, if the determination in S406 is No, S407 is executed. Note that even if the determination in S406 is Yes, the processes from S407 onwards may be executed if it is necessary to use the subject detection result. For example, this applies to cases where quick response is required, such as when performing autofocus processing to focus on a detected subject.

[0078] In S407, the movement direction estimation unit 205 estimates the movement direction of the specific subject detected in the current frame based on the detection history stored in the history storage unit 204 and the movement of the imaging device acquired by the movement sensor 162. Details will be described later.

[0079] In S408, the part correlation section 206 associates the parts detected in the current frame with each subject, taking into account the movement direction estimated by the movement direction estimation section 205. Details will be described later.

[0080] In S409, the determination unit 207 determines the main subject from among the specific subjects detected in the current frame. When, for example, a person and a horse are detected in the current frame, the determination unit 207 determines the horse as the main subject if the priority subject is set to an animal. On the other hand, if the priority subject is set to a person or automatic, the determination unit 207 determines the person as the main subject.

[0081] When the set priority subject is not detected or when multiple priority subjects are detected, the determination unit 207 can determine the main subject based on one or more of the detection position, size, and reliability. In S409, the CPU 151 may cause the display 150 to display part or all of the information related to the main subject determined by the determination unit 207.

[0082] (Movement direction estimation process) The movement direction estimation process in S407 will be described with reference to FIG. 6 and FIG. Fig. 7(a) shows a schematic diagram of the nth frame of a video, and Fig. 7(b) shows a schematic diagram of the n+1th frame following the nth frame. Assume that animal heads 701 and 702 and animal torsos 703 and 704 have been detected by subject detection processing for the nth frame. Assume that animal heads 705 and 706 and animal torsos 707 and 708 have been detected by subject detection processing for the n+1th frame. Assume that animal head 706 has been detected closer to animal torso 707 than animal head 705.

[0083] The details of the movement direction estimation process when the (n+1)th frame in FIG. 7(b) is the current frame will be described with reference to the flowchart shown in FIG. In S601, the movement direction estimation unit 205 associates parts of the same type between frames based on the detection result history for the current frame (n+1th frame) and the previous frame (nth frame) stored in the history storage unit 204.

[0084] The moving direction estimating unit 205 associates the animal's head detected in the n+1th frame with the animal's head detected in the nth frame that is closest in distance (detected position). Specifically, the moving direction estimating unit 205 associates the animal's head 705 detected in the n+1th frame with the animal's head 701 detected in the nth frame. The moving direction estimating unit 205 also associates the animal's head 706 detected in the n+1th frame with the animal's head 702 detected in the nth frame. The moving direction estimating unit 205 performs similar association for the animal's torso.

[0085] However, the association may be performed by other methods, for example, a correlation calculation may be performed between the region of the part detected in the nth frame and the region of the part detected in the (n+1)th frame as a template, and the part may be associated with the region with the highest correlation.

[0086] In S602, the movement direction estimation unit 205 calculates the amount of movement between frames for each part based on the detected positions of the parts associated in S601. Here, the movement direction estimation unit 205 calculates, as the amount of movement, a vector whose starting point is the detected position in the nth frame and whose ending point is the detected position in the (n+1)th frame.

[0087] In S603, the movement direction estimation unit 205 calculates the amount of background movement between frames (the amount of movement of the entire frames). Here, the movement direction estimation unit 205 calculates the amount of background movement based on the focal length (angle of view) of the photographing lens 101 and the movement of the imaging device 100 obtained from the motion sensor 162, as shown in the following formulas (1) and (2). GlobalVec(x) = f×tan(Yaw)×imagewidth (1) GlobalVec(y) = f×tan(Pitch)×imageheight (2)

[0088] GlobalVec(x) and GlobalVec(y) calculated by formula (1) and formula (2) are the horizontal and vertical components of a vector indicating the amount of background movement. Of the focal length f and the movement of the image capture device 100 obtained from the motion sensor 162, the amount of rotation around the y axis is Yaw (°) and the amount of rotation around the x axis is Pitch (°). Additionally, imagewidth and imageheight are coefficients indicating the horizontal and vertical sizes of the image.

[0089] The movement amount of the entire frame may be calculated by other known methods. For example, a background area of ​​the nth frame may be detected, and template matching using a part of the background area as a template may be applied to the n+1th frame to obtain the inter-frame movement amount of the template as the background movement amount. Alternatively, a motion vector between frames may be detected for each image, and the value with the highest frequency in a histogram for each directional component of the motion vector may be obtained as the directional component of the background movement amount.

[0090] In S604, the movement direction estimation unit 205 estimates the movement direction of the subject including the part based on the inter-frame movement amount for each part calculated in S602 and the background movement amount calculated in S603, based on the following equations (3) to (5). T.H. <TargetVec(x) - GlobalVec(x) (3) TargetVec(x) - GlobalVec(x)< -TH (4) -TH≦TargetVec(x) - GlobalVec(x)≦TH (5)

[0091] TargetVec(x) is the horizontal component of the inter-frame movement amount of the part calculated in S602, and GlobalVec(x) is the horizontal component of the background movement amount calculated in S603. Furthermore, TH is a threshold value having a positive value.

[0092] If the formula (3) is true, the movement direction estimation unit 205 estimates that the subject is moving to the right of the screen. If the formula (4) is true, the movement direction estimation unit 205 estimates that the subject is moving to the left of the screen. If the formula (5) is true, the movement direction estimation unit 205 estimates that the subject is not moving in the left / right direction of the screen.

[0093] The vertical movement amount of the subject can be estimated as follows by replacing TargetVec(x) and GlobalVec(x) in equations (3) to (5) with TargetVec(y) and GlobalVec(y). If the formula (3) is true, the movement direction estimation unit 205 estimates that the subject is moving in the downward direction of the screen. If the formula (4) is true, the movement direction estimation unit 205 estimates that the subject is moving in the upward direction of the screen. If the formula (5) is true, the movement direction estimation unit 205 estimates that the subject is not moving in the vertical direction of the screen.

[0094] The movement amount of the subject may be a representative value of the movement amount calculated for each associated part, or may be a movement amount calculated for a set of associated parts. The representative value may be an average value, a median value, or other value. When calculating the movement amount for only a set of parts, the movement direction estimation unit 205 uses the part corresponding to the subject determined to be the main subject in the previous frame.

[0095] When multiple parts of the main subject are detected, the movement direction estimation unit 205 calculates the amount of movement for each part. Then, among the vectors representing the amount of movement, the horizontal component of the vector with the smallest amount of vertical movement is estimated as the amount of horizontal movement. This makes it possible to obtain a stable estimation result.

[0096] For example, when an animal subject moves horizontally, the vertical movement of the torso is small, but the vertical movement of the head can be large due to the movement of the neck. Therefore, by adopting the amount of movement obtained for parts with little vertical movement, a stable estimation result can be obtained. Note that, if the reliability of the amount of movement can be determined based on the type of part, it is possible to obtain only the amount of movement of parts with high reliability without calculating the vertical movement. For example, if the head and torso are detected for an animal subject, the amount of movement of the torso may be calculated.

[0097] The moving direction may be estimated based on the estimation results for a plurality of frames. For example, the estimated moving direction may be output only when the same moving direction is estimated for a plurality of consecutive frames. If the same moving direction is not estimated for a plurality of consecutive frames even after a certain period of time has elapsed, the moving direction estimating unit 205 may output a result indicating that the moving direction cannot be estimated.

[0098] (Part matching process) Next, the part association process in S408 will be described with reference to FIG. In S408, the part correlation section 206 estimates and associates different types of parts belonging to the same subject based on the movement direction estimated in S407.

[0099] In S407, it is assumed that all of the animal subjects are estimated to be moving to the right of the screen, and that heads 705 and 706 of two animals are detected within a range where the distance from body 707 of an animal detected in the current frame (the n+1th frame in FIG. 7(b)) is less than a threshold.

[0100] In this case, part correlation unit 206 associates body 707 of the animal detected in the current frame with head 705 of two animal heads 705 and 706 whose distance is less than the threshold. This is because, of heads 705 and 706, head 705 satisfies the positional relationship between the body and the head specified by the estimated moving direction. Head 706 is closer to body 707 than head 705, but does not satisfy the positional relationship between the body and the head specified by the estimated moving direction.

[0101] That is, based on the estimated moving direction of the subject, part correlation section 206 estimates that, of heads 705 and 706 whose distance from torso 707 is less than the threshold, head 705 belongs to the same subject as torso 707, and head 706 belongs to a different subject. Then, part correlation section 206 associates torso 707 and head 705.

[0102] Part correlation unit 206 also associates head 706, which is closer to torso 707 in the right direction, with torso 708. This is because the moving directions of the subjects are both estimated to be rightward, and the head corresponding to torso 707 is expected to be located to the right of torso 707, whereas head 706, which is located to the left of torso 707, is expected not to correspond to torso 707.

[0103] In addition, if the subject's moving direction is estimated to be stationary in S407, if it is determined that the moving direction cannot be estimated, or if an estimation result is not obtained, the part cannot be associated with the part in consideration of the estimated moving direction. In this case, the part correlation unit 206 can associate other types of parts whose distance is less than the threshold value based on, for example, the detected position of the part. In this case, if there are multiple candidates for association, the part correlation unit 206 may prioritize not performing an erroneous association and may not perform the association.

[0104] For example, two animal heads 705 and 706 are present within a predetermined distance range for an animal's torso 707. In this case, the part correlation unit 206 does not associate the animal's torso 707 with either head. Note that, if the number of candidates for the head of the torso 707 is narrowed down to one based on the results of the association for other parts, the head may be associated with the torso 707 at that point in time. Also, the association may be performed by narrowing down the multiple candidates to one, taking into consideration other conditions, such as referring to the association result in the previous frame.

[0105] As described above, according to this embodiment, when different parts detected for the same type of subject are associated with parts belonging to the same subject, the movement direction of the subject is taken into consideration, thereby improving the accuracy of the association. Note that, although an animal subject has been described here, the same can be applied to the association of parts for other types of subjects in which the positional relationship of parts can be specified by the movement direction.

[0106] <Second embodiment> Next, a second embodiment of the present invention will be described. This embodiment is similar to the first embodiment except for the configuration and operation of the subject detection unit. Therefore, a description of the configuration other than the subject detection unit will be omitted.

[0107] (Configuration of subject detection unit) Fig. 8 shows an example of the configuration of a subject detection unit 161' in the second embodiment, similar to Fig. 2. The same components as in the first embodiment are given the same reference numerals as in Fig. 2. Subject detection unit 161' differs from the first embodiment in that it does not have a movement direction estimation unit.

[0108] (Subject detection processing) The operation of the subject detection unit 161' will be described with reference to Figures 9 and 10. In the flowchart of Figure 9, the same processing steps as in the first embodiment are denoted by the same reference numerals as in Figure 4.

[0109] Since S401 to S403 are the same as in the first embodiment, the description will be omitted. In this embodiment, the dictionary data is learned so that a vector indicating a position where there is a high probability that another part of the same subject is present is included in the detection result. For example, such dictionary data can be obtained by including a vector from the torso to the position of another part of the same subject, such as the position of the head, in addition to the position of the torso, in the training data when learning parameters for detecting the torso of an animal subject. Similarly, a vector from the head to the position of another part of the same subject, such as the position of the torso, in addition to the position of the head, can be included in the training data when learning parameters for detecting the head of an animal subject.

[0110] In S901, part detection unit 801 executes subject detection processing using the dictionary data selected by dictionary data selection unit 201. This is the same as S404, except that, because the dictionary data is different, the detection result includes a position estimation vector indicating a position where there is a high probability that another part of the same subject exists. Since part detection unit 801 of this embodiment outputs a position estimation vector, it also functions as a means for estimating the position where another part exists.

[0111] 10 is a diagram for explaining the detection result obtained when the torso of an animal subject is detected in S901. As a result of applying a process for detecting the torso of an animal subject to a current frame 1000, a torso 1002 and a position estimation vector 1003 are detected.

[0112] Part detection section 801 outputs the detection result including position estimation vector 1003 to history storage section 204. Thereafter, the processes of S405 and S406 are the same as those in the first embodiment.

[0113] If the result of S406 is No, S902 is executed. In S902, the part correlation unit 802 uses the position estimation vector to associate the parts detected in the current frame with each subject.

[0114] The association of parts in S902 will be described with reference to FIG. It is assumed that the body 1002 and head 1001 of an animal subject have been detected by subject detection processing on the current frame 1000. It is also assumed that the position estimation vector 1003 for the head has been obtained as a result of detecting the body 1002, as described above.

[0115] In this case, part correlation unit 802 sets a search range 1004 for a position estimation vector centered on the detected position of head 1001. Then, part correlation unit 802 searches for a position estimation vector for the head whose end point is within the set search range 1004. In the example of Fig. 10, the end point of position estimation vector 1003 is within search range 1004. Therefore, part correlation unit 802 associates torso 1002, which includes position estimation vector 1003 as a detection result, with head 1001.

[0116] The search range of the position estimation vector can be determined according to the type and size of the part (here, the head) that includes the center of the search range. Here, the search range of the position estimation vector is set based on another part, not on the part (here, the torso) that includes the position estimation vector in the detection result. However, the search range of the part may be set based on the end point of the position estimation vector.

[0117] 10, part correlation section 802 sets a head search range centered on the end point of head position estimation vector 1003. Then, part correlation section 802 can associate, among the heads of the same type of subject detected in current frame 1000, those whose detected positions are included in the search range with torso 1002.

[0118] According to this embodiment, since there is no need to estimate the moving direction of the subject, there is an effect that accurate association can be achieved while reducing the processing load related to matching of parts.

[0119] <Third embodiment> Next, a third embodiment of the present invention will be described. This embodiment is the same as the first embodiment except for the configuration and operation of the subject detection unit. Therefore, a description of the configuration other than the subject detection unit will be omitted.

[0120] (Configuration of subject detection unit) Subject detection section 161 in the third embodiment may have the same configuration as in Fig. 2, with part correlation section 206 operating differently. As in the second embodiment, dictionary data is assumed to have been learned so that vectors indicating positions where there is a high probability that other parts of the same subject are present are included in the detection results. Therefore, part detection section 203 operates in the same way as part detection section 801 in the second embodiment.

[0121] (Subject detection processing) The operation of the subject detection unit 161 will be described with reference to Figures 11 and 12. In the flowchart of Figure 11, the same reference numerals as in Figure 4 are used for the processing steps that are the same as in the first embodiment, and the same reference numerals as in Figure 9 are used for the processing steps that are the same as in the second embodiment.

[0122] Since S401 to S403 and S405 to S407 are the same as in the first embodiment, and S901 is the same as in the second embodiment, the description thereof will be omitted.

[0123] After estimating the movement direction of the subject in S407, in S1201 part correlation section 206 uses the movement direction estimated in S407 and the position estimation vector detected in S901 to associate parts detected in the current frame with each subject.

[0124] The association of parts in S1202 will be described with reference to FIG. Assume that torsos 1303 and 1304 and heads 1301 and 1302 of an animal subject have been detected by subject detection processing on current frame 1300. Assume also that position estimation vector 1305 has been obtained as the detection result of torso 1303, and position estimation vector 1306 has been obtained as the detection result of torso 1304. Assume also that the movement directions of both subjects have been estimated in S407 to be rightward.

[0125] In this case, part correlation section 206 sets search range 1307 for a position estimation vector centered on the detected position of head 1301 and search range 1308 for a position estimation vector centered on the detected position of head 1302 .

[0126] Part correlation unit 206 searches for a position estimation vector whose end point is within each of set search ranges 1307 and 1308. In the example of Fig. 12, there is no position estimation vector whose end point is within search range 1307. Therefore, it is not possible to identify a body associated with head 1301. On the other hand, for search range 1308, the end points of two position estimation vectors 1305 and 1306 are present.

[0127] When there are multiple position estimation vectors with end points within the search range, part correlation unit 206 takes into consideration the moving direction of the subject. Here, it is estimated that all of the moving directions of the subject are to the right. Therefore, part correlation unit 206 associates head 1302 with torso 1304 based on position estimation vector 1306 whose end point is to the right of the starting point and does not contradict the estimated moving direction.

[0128] Note that, by associating head 1302 with torso 1304, the only parts that are not associated among the parts detected in current frame 1300 are head 1301 and torso 1303. In this case, part correlation section 206 may associate head 1301 with torso 1303, taking into consideration that the distance between head 1301 and torso 1303 is less than a threshold value, and that the positional relationship between the head and torso is consistent with the estimated movement direction, etc.

[0129] According to this embodiment, by matching parts taking into consideration both the estimated movement direction of the subject and the position estimation vector of the part, it is possible to further increase the reliability of the matching.

[0130] (Other embodiments) In the above embodiment, in order to facilitate explanation and understanding, the case where two parts are detected for one type of subject has been described. However, the case where three or more parts are detected for one type of subject can also be handled in a similar manner by associating two parts at a time. Moreover, when detecting multiple types of subjects, the above-mentioned part association may be performed for each type of subject.

[0131] The present invention is not limited to being applied to real-life images, but can also be applied to CG images, for example, to images obtained by cutting out a predetermined angle of view from the viewpoint of a user (avatar) in a virtual space.

[0132] The disclosure of the present embodiment includes the following image processing device, image processing method, and program. (Item 1) a detection means for detecting a first portion and a second portion of a specific subject from an image; An estimation means for estimating a moving direction of the specific subject; a matching means for matching the first and second regions of the same subject detected by the detection means based on the estimated moving direction; 13. An image processing device comprising: (Item 2) a positional relationship between the first portion and the second portion can be specified based on a moving direction of the specific subject; 2. The image processing device according to item 1, wherein the association means associates the first region with the second region so as to satisfy the positional relationship specified by the estimated direction of movement. (Item 3) The image processing device according to item 1 or 2, characterized in that the matching means matches the first part and the second part that satisfy the positional relationship specified by the estimated movement direction and whose distance is less than a threshold. (Item 4) 4. The image processing device according to any one of items 1 to 3, characterized in that, when the estimation means is unable to estimate the direction of movement, the association means associates the first part and the second part whose distance is less than a threshold. (Item 5) 5. The image processing device according to item 4, wherein the association means does not associate the second part with the first part for which there are multiple second parts whose distance is less than a threshold value when the estimation means is unable to estimate the moving direction. (Item 6) 6. The image processing device according to any one of items 1 to 5, wherein the estimation means estimates the movement direction based on a vector that has the smallest vertical movement amount among vectors indicating the movement amount for each of the specific subjects. (Item 7) when detecting the first region, the detection means detects a vector indicating a position where the second region is highly likely to exist; The image processing device described in item 1, characterized in that the association means associates the first part, for which a vector that does not contradict the estimated movement direction is detected, with the second part, among the vectors having an end point within a search range set for the second part. (Item 8) A detection means for detecting a first part and a second position of a specific subject from an image, the detection result of the first part including a vector indicating a position where there is a high probability that a corresponding second part exists; a matching means for matching the first and second regions of the same subject detected by the detection means based on the vector; 13. An image processing device comprising: (Item 9) 9. The image processing device according to item 8, characterized in that the association means associates the first part in which the vector having an end point within a search range set for the second part is detected with the second part. (Item 10) The method further includes estimating a moving direction of the specific subject, 10. The image processing device according to item 8 or 9, characterized in that the association means associates the first part, for which a vector that does not contradict the estimated movement direction is detected among the vectors having an end point within a search range set for the second part, with the second part. (Item 11) 11. The image processing device according to any one of items 1 to 10, wherein the detection means detects the first region and the second region separately. (Item 12) The image processing device according to any one of items 1 to 11, characterized in that the detection means detects the first part and the second part using a neural network in which dictionary data corresponding to a combination of the type of specific subject and the part to be detected is set. (Item 13) 13. The image processing device according to any one of items 1 to 12, wherein the specific subject is a human being or an animal. (Item 14) Item 14. The image processing device according to item 13, wherein the first part is a torso and the second part is a head. (Item 15) An image processing method executed by an image processing device, comprising: Detecting a first portion and a second portion of a specific object from the image; Estimating a moving direction of the specific subject; Correlating the first body part and the second body part of the same subject detected by the detection based on the estimated moving direction; 13. An image processing method comprising: (Item 16) An image processing method executed by an image processing device, comprising: Detecting a first part and a second position of a specific object from an image, the detection result of the first part including a vector indicating a position where a corresponding second part is highly likely to exist; Correlating the first region and the second region detected by the detection based on the vector with respect to the same subject; 13. An image processing method comprising: (Item 17) A program for causing a computer to function as each of the means possessed by the image processing device according to any one of items 1 to 14.

[0133] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0134] The present invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Therefore, the following claims are appended to disclose the scope of the invention. [Explanation of symbols]

[0135] 100: imaging device, 151: CPU, 152: image processing unit, 161: subject detection unit, 201: dictionary data selection unit, 202: dictionary data storage unit, 203: body part detection unit, 204: history storage unit, 205: movement direction estimation unit, 206: body part correlation unit, 207: determination unit

Claims

1. a detection means for detecting a first portion and a second portion of a specific subject from an image; An estimation means for estimating a moving direction of the specific subject; a matching means for matching the first and second parts of the same subject detected by the detection means based on the estimated moving direction; 13. An image processing device comprising:

2. a positional relationship between the first portion and the second portion can be specified based on a moving direction of the specific subject; The image processing apparatus according to claim 1 , wherein the association means associates the first region with the second region so as to satisfy the positional relationship specified by the estimated movement direction.

3. The image processing device according to claim 1 , wherein the association means associates the first part and the second part, which satisfy a positional relationship specified by the estimated movement direction and whose distance is less than a threshold value.

4. The image processing device according to claim 1 , wherein the association means associates the first part and the second part, the distance of which is less than a threshold value, with each other when the estimation means is unable to estimate the moving direction.

5. The image processing device according to claim 4, characterized in that, when the estimation means is unable to estimate the direction of movement, the matching means does not match the second part for the first part to which there are multiple second parts whose distance is less than a threshold value.

6. 2 . The image processing apparatus according to claim 1 , wherein the estimation means estimates the movement direction based on a vector that has a smallest amount of vertical movement among vectors that indicate the amount of movement of each of the specific subjects.

7. the detection means detects a vector indicating a position where the second portion is highly likely to exist when detecting the first portion; The image processing device according to claim 1, characterized in that the matching means matches the first part, among the vectors having an end point within a search range set for the second part, with the second part, for which a vector that does not contradict the estimated movement direction has been detected.

8. a detection means for detecting a first part and a second position of a specific subject from an image, the detection result of the first part including a vector indicating a position where there is a high probability that a corresponding second part exists; a matching means for matching the first and second parts of the same subject detected by the detection means based on the vector; 13. An image processing device comprising:

9. 9. The image processing device according to claim 8, wherein the association means associates the first part in which the vector having an end point within a search range set for the second part is detected with the second part.

10. The method further includes estimating a moving direction of the specific subject, The image processing device according to claim 8, characterized in that the matching means matches the first part for which a vector that does not contradict the estimated movement direction is detected among the vectors having an end point within a search range set for the second part, to the second part.

11. 2. The image processing apparatus according to claim 1, wherein the detection means detects the first region and the second region separately.

12. 2. The image processing device according to claim 1, wherein the detection means detects the first region and the second region using a neural network in which dictionary data corresponding to a combination of a type of specific subject and a region to be detected is set.

13. 2. The image processing apparatus according to claim 1, wherein the specific subject is a human being or an animal.

14. 14. The image processing apparatus according to claim 13, wherein the first region is a torso and the second region is a head.

15. An image processing method executed by an image processing device, comprising: Detecting a first portion and a second portion of a specific subject from the image; Estimating a moving direction of the specific subject; Correlating the first body part and the second body part of the same subject detected by the detection based on the estimated moving direction; 13. An image processing method comprising:

16. An image processing method executed by an image processing device, comprising: Detecting a first part and a second position of a specific subject from an image, the detection result of the first part including a vector indicating a position where a corresponding second part is highly likely to exist; Correlating the first region and the second region detected by the detection based on the vector, the first region and the second region being parts of a same subject; 13. An image processing method comprising:

17. A program for causing a computer to function as each of the means included in the image processing device according to any one of claims 1 to 14.

Images