Image processing apparatus, image processing method, and computer program

Abstract

This invention provides an image processing device, an image processing method, and a computer program that accurately estimate whether or not two images depict the same person. [Solution] The image processing device 1 includes a first region detection unit 11 that detects a first region from an image that includes at least a part of a person, a first feature point detection unit 12 that detects a first feature point from the first region, a second region detection unit 13 that detects a second region from an image that includes at least a part of a person, overlaps with at least a part of the first region, and is of a different size from the first region, a second feature point detection unit 14 that detects a second feature point from the second region, and an estimation unit 15 that estimates whether the person included in the first region and the person included in the second region are the same person based on the first feature point and the second feature point.

Description

Technical Field

[0001] This disclosure relates to the technical fields of image processing apparatuses, image processing methods, and computer programs.

Background Art

[0002] There is described in Patent Document 1 a technique including an image acquisition step of acquiring a person image to be determined, an attribute determination region detection step of detecting at least two attribute determination regions selected from the group consisting of a head region, a face region, and other regions from the person image to be determined, and an attribute determination step of determining an attribute from images of the at least two attribute determination regions, which enables high detection accuracy of a person and determination of a person's attributes even when characteristic parts of the face are hidden. There is described in Patent Document 2 a technique of selecting, as a final detection result of a feature region, one of detection results of a predetermined feature region for a predetermined partial region detected from image data by a first method and detection results of a predetermined feature region for a predetermined partial region detected from image data by a second method. In an image search apparatus having a face region extraction program for extracting a face region from video data, there is described in Patent Document 3 a technique of extracting a face region from video data by a first algorithm, extracting a head region from video data by a second algorithm, performing face detection while changing the image quality for a region that is extracted as a head region but not extracted as a face region, and extracting the head region in which a face image is detected as a face region.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Patent Document 3

Summary of the Invention

[0004] This disclosure aims to provide an image processing apparatus, an image processing method, and a recording medium that improve upon the technologies described in prior art documents. [Means for solving the problem]

[0005] One embodiment of an image processing apparatus includes: a first region detection means for detecting a first region from an image that includes at least a portion of a person; a first feature point detection means for detecting a first feature point from the first region; a second region detection means for detecting a second region from the image that includes at least a portion of a person, overlaps with at least a portion of the first region, and is of a different size from the first region; a second feature point detection means for detecting a second feature point from the second region; and an estimation means for estimating whether a person included in the first region and a person included in the second region are the same person, based on the first feature point and the second feature point.

[0006] One aspect of the image processing method involves detecting a first region from an image that includes at least a part of a person, detecting a first feature point from the first region, detecting a second region from the image that includes at least a part of a person, overlaps with at least a part of the first region, and is of a different size from the first region, detecting a second feature point from the second region, and estimating whether the person included in the first region and the person included in the second region are the same person based on the first feature point and the second feature point.

[0007] One embodiment of a recording medium contains a computer program that causes a computer to perform an image processing method which involves detecting a first region from an image that includes at least a part of a person, detecting a first feature point from the first region, detecting a second region from the image that includes at least a part of a person, overlaps with at least a part of the first region, and is of a different size from the first region, detecting a second feature point from the second region, and estimating whether the person included in the first region and the person included in the second region are the same person based on the first and second feature points. [Brief explanation of the drawing]

[0008] [Figure 1] Figure 1 is a block diagram showing the configuration of the image processing apparatus in the first embodiment. [Figure 2] Figure 2 is a block diagram showing the configuration of the image processing apparatus in the second embodiment. [Figure 3] Figure 3 is a flowchart showing the flow of image processing operations performed by the image processing device in the second embodiment. [Figure 4] Figure 4 is a conceptual diagram of the image processing operations performed by the image processing device in the second embodiment. [Figure 5] Figure 5 is a block diagram showing the configuration of the image processing apparatus in the third embodiment. [Figure 6] Figure 6 is a flowchart showing the flow of image processing operations performed by the image processing device in the third embodiment. [Figure 7] Figure 7 is a conceptual diagram of the image processing operations performed by the image processing device in the third embodiment. [Figure 8] Figure 8 is a block diagram showing the configuration of the image processing apparatus in the fifth embodiment. [Figure 9] Figure 9 is a flowchart showing the flow of image processing operations performed by the image processing device in the fifth embodiment. [Modes for carrying out the invention]

[0009] Hereinafter, embodiments of an image processing apparatus, an image processing method, and a recording medium will be described with reference to the drawings.

[0010] [1: First Embodiment]

[0011] The first embodiment of an image processing apparatus, an image processing method, and a recording medium will be described. Hereinafter, the first embodiment of an image processing apparatus, an image processing method, and a recording medium will be described using an image processing apparatus 1 to which the first embodiment of an image processing apparatus, an image processing method, and a recording medium is applied.

[0012] [1-1: Configuration of Image Processing Apparatus 1]

[0013] The configuration of the image processing apparatus 1 in the first embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the image processing apparatus 1 in the first embodiment.

[0014] As shown in FIG. 1, the image processing apparatus 1 includes a first region detection unit 11, a first feature point detection unit 12, a second region detection unit 13, a second feature point detection unit 14, and an estimation unit 15. The first region detection unit 11 detects a first region including at least a part of a person from an image. The first feature point detection unit 12 detects first feature points from the first region. The second region detection unit 13 detects a second region that includes at least a part of a person from the image, overlaps at least a part of the first region, and has a different size from the first region. The second feature point detection unit 14 detects second feature points from the second region. The estimation unit 15 estimates whether the person included in the first region and the person included in the second region are the same person based on the first feature points and the second feature points.

[0015] [1-2: Technical Effects of Image Processing Apparatus 1]

[0016] Since the image processing apparatus 1 in the first embodiment estimates whether the person included in the first region and the person included in the second region are the same person based on the first feature points and the second feature points, it is possible to accurately estimate whether they are the same person.

[0017] [2: Second Embodiment]

[0018] The second embodiment of an image processing apparatus, an image processing method, and a recording medium will be described. Hereinafter, the second embodiment of an image processing apparatus, an image processing method, and a recording medium will be described using an image processing apparatus 2 to which the second embodiment of an image processing apparatus, an image processing method, and a recording medium is applied.

[0019] <000009​​​​​​​​​​​​The arithmetic unit 21 includes, for example, at least one of a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), and an FPGA (Field Programmable Gate Array). The arithmetic unit 21 reads a computer program. For example, the arithmetic unit 21 may read a computer program stored in the storage device 22. For example, the arithmetic unit 21 may read a computer program stored in a computer-readable and non-temporary recording medium using a recording medium reading device (not shown) provided by the image processing device 2 (for example, an input device 24 described later). The arithmetic unit 21 may obtain a computer program from a device (not shown) located outside the image processing device 2 via a communication device 23 (or other communication device) (i.e., it may download or read the program). The arithmetic unit 21 executes the read computer program. As a result, logical functional blocks for performing the operations that the image processing device 2 should perform are realized within the arithmetic unit 21. In other words, the arithmetic unit 21 can function as a controller for realizing logical functional blocks necessary for the image processing device 2 to perform the operations (in other words, processing).

[0023] Figure 2 shows an example of a logical functional block implemented within the arithmetic unit 21 to perform image processing operations. As shown in Figure 2, the arithmetic unit 21 implements a first region detection unit 211, which is a specific example of the "first region detection means"; a first feature point detection unit 212, which is a specific example of the "first feature point detection means"; a second region detection unit 213, which is a specific example of the "second region detection means"; a second feature point detection unit 214, which is a specific example of the "second feature point detection means"; and an estimation unit 215, which is a specific example of the "estimation means". The operations of the first region detection unit 211, the first feature point detection unit 212, the second region detection unit 213, the second feature point detection unit 214, and the estimation unit 215 will be described later with reference to Figures 3 and 4.

[0024] The storage device 22 is capable of storing desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic unit 21. The storage device 22 may temporarily store data that the arithmetic unit 21 uses temporarily when it is executing a computer program. The storage device 22 may store data that the image processing device 2 stores long-term. The storage device 22 may include at least one of the following: RAM (Random Access Memory), ROM (Read Only Memory), hard disk drive, magneto-optical disk drive, SSD (Solid State Drive), and disk array drive. In other words, the storage device 22 may include a non-temporary recording medium.

[0025] The communication device 23 can communicate with devices outside the image processing device 2 via a communication network (not shown). The communication device 23 may acquire images to be used in image processing operations from, for example, an imaging device via the communication network.

[0026] The input device 24 is a device that receives information input to the image processing device 2 from outside the image processing device 2. For example, the input device 24 may include an operating device (e.g., at least one of a keyboard, mouse, and touch panel) that can be operated by an operator of the image processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be attached externally to the image processing device 2.

[0027] The output device 25 is a device that outputs information to the outside of the image processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (so-called display) capable of displaying an image that shows the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (so-called speaker) capable of outputting sound. For example, the output device 25 may output information onto paper. That is, the output device 25 may include a printing device (so-called printer) capable of printing desired information onto paper.

[0028] [2-2: Image processing operations performed by the image processing device 2]

[0029] Referring to Figures 3 and 4, the flow of image processing operations performed by the image processing device 2 in the second embodiment will be explained. Figure 3 is a flowchart showing the flow of image processing operations performed by the image processing device 2 in the second embodiment. Figure 4 is a conceptual diagram of the image processing operations of the image processing device 2 in the second embodiment.

[0030] As shown in Figure 3, the first region detection unit 211 detects a first region from the image that includes at least a part of a person (step S20). The first region may be a face region that includes the person's face. The first region detection unit 211 may detect a face region that includes the person's face from the image as the first region. The first region detection unit 211 may detect a face region 1R, for example, as illustrated in Figure 4(a).

[0031] The first region detection unit 211 may detect face regions by applying known face detection processing to image data. The first region detection unit 211 may also detect regions having facial features as face regions. Regions having facial features may be characteristic parts that constitute a face, such as eyes, nose, and mouth. There are no particular restrictions on the method by which the first region detection unit 211 detects face regions. For example, the first region detection unit 211 may detect face regions based on the extraction of edges or patterns characteristic of face regions.

[0032] The first region detection unit 211 may use a neural network that has been trained on facial regions. The first region detection unit 211 may be composed of a convolutional neural network (CNN). The first region detection unit 211 may be composed of a two-stage CNN. The first region detection unit 211 may detect facial regions 1R originating from the first stage of the CNN and facial regions 1R originating from the second stage.

[0033] In step S20, if the first region detection unit 211 detects multiple face regions 1R as first regions, it may select one face region 1R based on the confidence level of each face region 1R. If the first region detection unit 211 detects multiple face regions 1R, it may sort each face region 1R in order of its confidence score. The confidence score may indicate the likelihood that it is a face region 1R. The first region detection unit 211 may adopt the face region 1R with the highest confidence score as the face region 1R to be used in subsequent processing.

[0034] The first region detection unit 211 may suppress overlapping regions using Non-Maximum Suppression (NMS). The first region detection unit 211 may adopt, for example, 0.45 as the threshold for NMS.

[0035] The first feature point detection unit 212 detects a first feature point from the face region 1R (step S21). The first feature point detection unit 212 may also detect the position of the first feature point from the face region 1R. The first feature point detection unit 212 may also detect the positions of multiple first feature points from the face region 1R. The first feature point detection unit 212 may also detect multiple face feature points 1F, as illustrated in Figure 4(a), for example. The first feature point detection unit 212 may also detect multiple characteristic points in the eye region and the nose region as face feature points 1F, as illustrated in Figure 4(a), for example. Multiple characteristic points in the eye region may include the edge of the eye and the pupil. Multiple characteristic points in the nose region may include the nasal protuberance and the nasal ala margin.

[0036] There are no particular restrictions on the method used by the first feature point detection unit 212 to detect feature points. The first feature point detection unit 212 may, for example, use pattern matching. The first feature point detection unit 212 may use a neural network that has been trained on the eye region, nose region, etc. The first feature point detection unit 212 may be composed of a CNN.

[0037] The second region detection unit 213 detects a second region from the image that includes at least a part of the person, overlaps with at least a part of the first region, and is of a different size from the first region (step S22). The second region detection unit 213 may also detect a second region from the image that includes at least a part of the person and encompasses the first region. The second region may be a head region that includes the person's head. The second region detection unit 213 may also detect a head region that includes the person's head as the second region from the image. The second region detection unit 213 may detect a head region 2R, for example, as illustrated in Figure 4(b).

[0038] The second region detection unit 213 may detect the head region 2R by applying known head detection processing to the image data. The second region detection unit 213 may detect a region having head features as the head region 2R. There are no particular restrictions on the method used by the second region detection unit 213 to detect the head region 2R. For example, the second region detection unit 213 may detect a region that includes a head-specific part such as hair. For example, the second region detection unit 213 may detect a region with a predetermined shape, such as an Ω shape. The second region detection unit 213 may detect the head region 2R mainly based on the shape of the contour. The second region detection unit 213 may also detect the head region 2R by using the detection of human body parts such as hands, feet, and torso in combination. Unlike the first region detection unit 211, the second region detection unit 213 does not have to rely on the detection of characteristic parts that constitute the face. In this case, the second region detection unit 213 may be able to detect the head region 2R even if the first region detection unit 211 fails to detect the face region 1R. On the other hand, unlike the first region detection unit 211, the second region detection unit 213 may depend on detecting characteristic parts that make up the face.

[0039] The second region detection unit 213 may use a neural network that has been machine-trained on the head region 2R. The second region detection unit 213 may be composed of a CNN. The second region detection unit 213 may be composed of a two-stage CNN. The second region detection unit 213 may detect head region 2R originating from the first stage of the CNN and head region 2R originating from the second stage.

[0040] In step S22, if the second region detection unit 213 detects multiple head regions 2R as second regions, it may select one head region 2R based on the confidence level of each head region 2R. If the second region detection unit 213 detects multiple head regions 2R, it may sort each head region 2R in order of its confidence score. The confidence score may indicate the likelihood that a region is a head region 2R. The second region detection unit 213 may adopt the head region 2R with the highest confidence score as the head region 2R to be used in subsequent processing.

[0041] The second region detection unit 213 may use Non-Maximum Suppression (NMS) to suppress overlapping regions. The second region detection unit 213 may adopt, for example, 0.40 as the threshold for NMS.

[0042] The second feature point detection unit 214 detects a second feature point from the head region 2R (step S23). The second feature point detection unit 214 may detect the position of the second feature point from the head region 2R. The second feature point detection unit 214 may detect the positions of multiple second feature points from the head region 2R. The second feature point detection unit 214 may detect multiple head feature points 2F, for example, as illustrated in Figure 4(b). The second feature point detection unit 214 may detect multiple characteristic points in the eye region and nose region as head feature points 2F, for example, as illustrated in Figure 4(b). There are no particular restrictions on the method of feature point detection performed by the second feature point detection unit 214. The second feature point detection unit 214 may use, for example, pattern matching. The second feature point detection unit 214 may use a neural network that has been machine-trained on the eye region and nose region, etc. The second feature point detection unit 214 may be composed of a CNN.

[0043] The estimation unit 215 integrates the face region 1R and the head region 2R (step S24). The estimation unit 215 may match the face region 1R and the head region 2R. The estimation unit 215 estimates whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the face feature point 1F and the head feature point 2F. The estimation unit 215 may estimate whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the position of the face feature point 1F and the position of the head feature point 2F. The estimation unit 215 may estimate whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the positional relationship between the feature points. The estimation unit 215 may estimate that the smaller the distance between the feature points, the greater the match. The estimation unit 215 may estimate that the person included in the face region 1R and the person included in the head region 2R are the same person based on the degree of match of each feature point.

[0044] The estimation unit 215 may estimate whether the person included in face region 1R and the person included in head region 2R are the same person, based on a first circumscribing shape that circumscribes each of the face feature points 1F and a second circumscribing shape that circumscribes each of the head feature points 2F. In this case, the circumscribing shape that circumscribes each of the feature points may be a circumscribing rectangle that circumscribes each of the feature points. That is, the estimation unit 215 may estimate whether the person included in face region 1R and the person included in head region 2R are the same person, based on the state of overlap between face feature point region 1FR, which is a rectangle that circumscribes each of the face feature points 1F, and head feature point region 2FR, which is a rectangle that circumscribes each of the head feature points 2F. The estimation unit 215 may also estimate whether the person included in face region 1R and the person included in head region 2R are the same person, based on face feature point region 1FR and head feature point region 2FR, as illustrated in Figure 4(c), for example.

[0045] The estimation unit 215 may estimate whether the person included in face region 1R and the person included in head region 2R are the same person, based on the overlap rate between face feature region 1FR and head feature region 2FR. The estimation unit 215 may use, for example, IoU (Intersection over Union, sometimes called the "Jaccard coefficient") as the overlap rate. IoU is the ratio of the overlap portion to the union of the two regions. IoU = Size of the overlap of the two regions / Size of the union of the two regions It can be expressed as follows.

[0046] If the face region 1R and head region 2R are not associated, the number of false detections of people may increase. In contrast, the estimation unit 215 associates the face region 1R and head region 2R, thereby suppressing false detections of people. Furthermore, in the comparative example where the face region 1R and head region 2R are directly associated, the association may be difficult, but the estimation unit 215 performs the association based on the first and second feature points, making the association easier compared to the comparative example.

[0047] Furthermore, the estimation unit 215 does not need to perform a mapping operation if the face region 1R is not detected and only the head region 2R is detected. Similarly, the estimation unit 215 does not need to perform a mapping operation if the head region 2R is not detected and only the face region 1R is detected.

[0048] [2-3: Effects of the image processing device 2]

[0049] For example, CNNs may misdetect or fail to detect the target region when the image resolution is low. On the other hand, using high-resolution images in a CNN improves detection accuracy, but increases computational complexity and slows down detection speed. For example, edge servers have a demand for lightweight and efficient detection engines, but there is a trade-off between detection accuracy and detection speed.

[0050] In contrast, the image processing device 2 in the second embodiment performs at least one of estimation based on the position of a first feature point and the position of a second feature point, and estimation based on the first circumscribing shape and the second circumscribing shape. Since the image processing device 2 uses features detected from the detection results, it can accurately estimate whether the person included in the first region and the person included in the second region are the same person. As a result, the image processing device 2 can suppress false detection of the target region. Furthermore, by detecting the second region that encompasses the first region, the image processing device 2 can suppress the failure to detect the target region. Therefore, the image processing device 2 can accurately detect the target region even when the image is low resolution. Since it is not necessary to increase the image resolution, the image processing device 2 can improve the detection accuracy without reducing the detection speed.

[0051] [3: Third Embodiment]

[0052] A third embodiment of the image processing apparatus, image processing method, and recording medium will be described below. In the following description, the third embodiment of the image processing apparatus, image processing method, and recording medium will be described using an image processing apparatus 3 to which the third embodiment of the image processing apparatus, image processing method, and recording medium is applied.

[0053] [3-1: Configuration of Image Processing Device 3]

[0054] The configuration of the image processing apparatus 3 in the third embodiment will be described with reference to Figure 5. Figure 5 is a block diagram showing the configuration of the image processing apparatus 3 in the third embodiment.

[0055] As shown in Figure 5, the image processing device 3 in the third embodiment includes a processing unit 21 and a storage device 22, similar to the image processing device 2 in the second embodiment. Furthermore, the image processing device 3 may also include a communication device 23, an input device 24, and an output device 25, similar to the image processing device 2 in the second embodiment. However, the image processing device 3 does not need to include at least one of the communication device 23, the input device 24, and the output device 25. The image processing device 3 in the third embodiment differs from the image processing device 2 in the second embodiment in that the processing unit 21 includes an output control unit 316 and a third detection unit 317. Other features of the image processing device 3 may be the same as other features of the image processing device 2 in the second embodiment.

[0056] [3-2: Image processing operations performed by the image processing device 3]

[0057] Referring to Figures 6 and 7, the flow of image processing operations performed by the image processing device 3 in the third embodiment will be explained. Figure 6 is a flowchart showing the flow of image processing operations performed by the image processing device 3 in the third embodiment. Figure 7 is a conceptual diagram of the image processing operations of the image processing device 3 in the third embodiment.

[0058] As shown in Figure 6, the first region detection unit 211 detects a face region 1R from the image that includes at least a part of the person (step S20). The first feature point detection unit 212 detects a face feature point 1F from the face region 1R (step S21). The second region detection unit 213 detects a head region 2R from the image that includes at least a part of the person and encompasses the face region 1R (step S22). The second feature point detection unit 214 detects a head feature point 2F from the head region 2R (step S23). The estimation unit 215 integrates the face region 1R and the head region 2R (step S24).

[0059] The output control unit 316 determines whether or not a face region 1R exists (step S30). The output control unit 316 may also determine whether or not the first region detection unit 211 has detected the face region 1R. If the face region 1R is not detected and only the head region 2R is detected, step S30 may be omitted and the process may proceed to step S32.

[0060] If face region 1R exists (step S30: Yes), that is, if the first region detection unit 211 detects face region 1R, the output control unit 316 outputs face region 1R and face feature point 1F (step S31). If face region 1R exists, this may include cases where the first region detection unit 211 detects face region 1R and the second region detection unit 213 does not detect head region 2R, and cases where the first region detection unit 211 detects face region 1R and the second region detection unit 213 detects head region 2R. The output control unit 316 may output face region 1R and face feature point 1F that have been integrated with head region 2R in step S24. The output control unit 316 may search for the largest pair with an IoU of 0.01 or more and output the corresponding face region 1R and face feature point 1F. The output control unit 316 may output the face region 1R and the face feature point 1F, as illustrated in Figure 7(a).

[0061] On the other hand, if face region 1R does not exist (step S30: No), that is, if the first region detection unit 211 does not detect face region 1R, the third detection unit 317 detects a corresponding region 3R from the head region 2R that corresponds to face region 1R (step S32). The case where face region 1R does not exist may also include the case where the first region detection unit 211 does not detect face region 1R and the second region detection unit 213 detects head region 2R. The third detection unit 317 may detect a corresponding region 3R that corresponds to face region 1R, as illustrated in Figure 7(b). For example, the third detection unit 317 may detect a square with sides equal to the larger of twice the width and four times the height of the head feature point region 2FR as a new face region 1R.

[0062] The output control unit 316 outputs the corresponding region 3R and the feature points included in the corresponding region 3R (step S33). The feature points included in the corresponding region 3R may be the same points as the head feature point 2F. The output control unit 316 may also output the corresponding region 3R and the head feature point 2F, as illustrated in Figure 7(c).

[0063] The output control unit 316 may integrate multiple rectangles to perform data shaping and output the data shaping result. The data shaping operation may include, for example, sorting, rounding, NMS processing, inclusion relationship processing, and class assignment processing. The sorting process may, for example, sort all detected regions in order of confidence score. The rounding process may, for example, round coordinates that extend beyond the image into the image. The NMS processing may, for example, adopt an NMS threshold of, for example, 0.45 to suppress overlapping regions. The inclusion relationship processing may, for example, keep the region with the higher confidence score in regions with a complete inclusion relationship. The class assignment processing may, for example, assign the same class ID to results originating from the head as to results originating from the face.

[0064] [3-3: Effects of the image processing device 3]

[0065] In the third embodiment, if the first region is not detected, the image processing device 3 detects a corresponding region from the second region that corresponds to the first region, and outputs the corresponding region and the feature points included in the corresponding region. This makes it possible to obtain the first region of a person and the feature points included in the first region even if the first region is not detected.

[0066] The head is larger than the face. Therefore, the head region 2R can be detected as larger than the face region 1R, and the head region 2R may contain more useful information than the face region 1R. The first region detection unit 211 and the second region detection unit 213 use different detection methods. Therefore, even if the face region 1R is not detected by the first region detection unit 211, the head region 2R may be detected by the second region detection unit 213.

[0067] The image processing device 3 can prevent the target region from being undetected by detecting the head region 2R and converting the head region 2R to the face region 1R when the face region 1R is not detected. As a result, the image processing device 3 can detect the region corresponding to the face region 1R even in situations where the face region 1R could not be detected before. In other words, the image processing device 3 can improve the undetection of the face region 1R when the image is low resolution. The image processing device 3 can also improve the undetection of the face region 1R when the image size is near the minimum size limit for detection. That is, the image processing device 3 can lower the minimum image size limit for detection and expand the detectable size.

[0068] The first region detection unit 211 and the first feature point detection unit 212 may be composed of the same CNN. Similarly, the second region detection unit 213 and the second feature point detection unit 214 may be composed of the same CNN. Furthermore, the first region detection unit 211, the first feature point detection unit 212, the second region detection unit 213, and the second feature point detection unit 214 may all be composed of the same CNN. Also, the detection of the first region and the detection of the second region may be performed simultaneously.

[0069] [4: Fourth Embodiment]

[0070] A fourth embodiment of the image processing apparatus, image processing method, and recording medium will be described below. In the following description, the fourth embodiment of the image processing apparatus, image processing method, and recording medium will be described using an image processing apparatus 4 to which the fourth embodiment of the image processing apparatus, image processing method, and recording medium is applied.

[0071] In the fourth embodiment, the image processing apparatus 4 differs from the image processing apparatus 2 in the second embodiment and the image processing apparatus 3 in the third embodiment in that the first region detected by the first region detection unit 211 and the second region detected by the second region detection unit 213 are different. Other features of the image processing apparatus 4 may be the same as other features of at least one of the image processing apparatus 2 and the image processing apparatus 3.

[0072] In the fourth embodiment, the first region detection unit 211 may detect a head region including the head of a person as the first region. The second region detection unit 213 may detect at least one of the upper body region and the whole body region of a person as the second region.

[0073] Head region detection is useful in person tracking operations. Person tracking operations may be applied to so-called gateless authentication, etc., where person authentication is performed while a person stays within a predetermined area. Furthermore, upper body region and / or full body region detection is useful in cover operations for person tracking operations, such as when head region detection fails. For example, even if the first region detection unit 211 fails to detect the head region, if the second region detection unit 213 detects the upper body region or the full body region, it can determine that it is the same person based on the characteristics of the person included in the upper body region or full body region, and the person tracking operation can continue.

[0074] In the fourth embodiment, the first region detection unit 211 may detect a face region including a person's face as the first region, and the second region detection unit 213 may detect the upper body region and / or the whole body region of a person as the second region.

[0075] [5: Fifth Embodiment]

[0076] A fifth embodiment of the image processing apparatus, image processing method, and recording medium will be described below. The fifth embodiment of the image processing apparatus, image processing method, and recording medium will be described using an image processing apparatus 5 to which the fifth embodiment of the image processing apparatus, image processing method, and recording medium is applied.

[0077] [5-1: Configuration of Image Processing Device 5]

[0078] The configuration of the image processing apparatus 5 in the fifth embodiment will be described with reference to Figure 8. Figure 8 is a block diagram showing the configuration of the image processing apparatus 5 in the fifth embodiment.

[0079] As shown in Figure 8, the image processing apparatus 5 in the fifth embodiment differs from the image processing apparatus 3 in the third embodiment in that the arithmetic unit 21 includes a third region detection unit 518 and a third feature point detection unit 519. Other features of the image processing apparatus 5 may be the same as other features of the image processing apparatus 3 in the third embodiment.

[0080] [5-2: Image processing operations performed by the image processing device 5]

[0081] Referring to Figure 9, the flow of image processing operations performed by the image processing device 5 in the fifth embodiment will be explained. Figure 9 is a flowchart showing the flow of image processing operations performed by the image processing device 5 in the fifth embodiment.

[0082] As shown in Figure 9, the first region detection unit 211 detects the face region as the first region (step S20). The first feature point detection unit 212 detects face feature points as the first feature points from the face region (step S21). The second region detection unit 213 detects the head region as the second region (step S22). The second feature point detection unit 214 detects head feature points as the second feature points from the head region (step S23).

[0083] The third region detection unit 518 detects a third region from the image that includes at least a part of the person, overlaps with at least a part of the second region, and is of a different size from the second region (step S51). The third region detection unit 518 may also detect a third region from the image that includes at least a part of the person and encompasses the second region. The third region may be the upper body region of the person or the whole body region. The third region detection unit 518 may detect the upper body region or the whole body region of the person as the third region from the image.

[0084] The third feature point detection unit 519 detects a third feature point from the upper body region or the whole body region (step S52). The third feature point detection unit 519 may also detect the position of the third feature point from the upper body region or the whole body region. The third feature point detection unit 519 may also detect the positions of multiple third feature points from the upper body region or the whole body region.

[0085] The estimation unit 215 integrates the face region, head region, and upper body region, or full body region (step S53). The estimation unit 215 may match the face region, head region, and upper body region, or full body region. The estimation unit 215 estimates whether the person included in the face region, the person included in the head region, and the person included in the upper body region or full body region are the same person, based on the face feature points, head feature points, and a third feature point. The estimation unit 215 may estimate whether the person included in the face region, the person included in the head region, and the person included in the upper body region or full body region are the same person, based on the positions of the face feature points, the head feature points, and the third feature point. The estimation unit 215 may estimate whether the person included in the face region, the person included in the head region, and the person included in the upper body region or full body region are the same person, based on the positional relationship between the feature points. The estimation unit 215 may estimate that the smaller the distance between the feature points, the greater the match. The estimation unit 215 may estimate, based on the degree of agreement between each feature point, whether the person included in the face region, the person included in the head region, and the person included in the upper body region or the whole body region are the same person. Also, similar to the above embodiment, the estimation unit 215 may estimate, based on the circumscribing shape that circumscribes each of the feature points, whether the person included in the face region, the person included in the head region, and the person included in the upper body region or the whole body region are the same person.

[0086] The output control unit 316 determines whether or not a face region exists (step S30). If a face region exists (step S30: Yes), the output control unit 316 outputs the face region and face feature points (step S31).

[0087] If the face region does not exist (step S30: No), the output control unit 316 determines whether or not the head region exists (step S54). If the head region exists (step S54: Yes), the third detection unit 317 detects the corresponding region originating from the head region that corresponds to the face region (step S32). The output control unit 316 outputs the corresponding region originating from the head region and the feature points included in the corresponding region originating from the head region (step S33).

[0088] If the head region does not exist (step S54: No), the output control unit 316 and the third detection unit 317 detect a corresponding region originating from the upper body or whole body region that corresponds to the face region (step S55). The output control unit 316 outputs the corresponding region originating from the upper body or whole body region, and the feature points included in the corresponding region originating from the upper body or whole body region (step S56).

[0089] The operations performed by the image processing apparatus in each of the embodiments described above can be rephrased as detecting objects of multiple classes that are in a containment relationship, detecting feature points from the detected objects, estimating pairs of objects based on the detection results, and estimating whether the detection results of different classes belong to the same person. In particular, the operations performed by the image processing apparatus in the third embodiment can be rephrased as estimating the region of objects of the contained class.

[0090] The image processing devices in each of the embodiments described above are preferably applied to tracking operations and facial recognition operations.

[0091] [6: Addendum]

[0092] The following additional information is disclosed regarding the embodiments described above.

[0093] [Note 1] A first region detection means for detecting a first region from an image that includes at least a portion of a person, A first feature point detection means for detecting a first feature point from the first region, A second region detection means for detecting a second region from the aforementioned image that includes at least a portion of a person, overlaps with at least a portion of the first region, and is of a different size from the first region, A second feature point detection means for detecting a second feature point from the second region, Estimation means for estimating whether the person included in the first region and the person included in the second region are the same person, based on the first and second feature points. An image processing device equipped with the following features.

[0094] [Note 2] The estimation means estimates whether the person included in the first region and the person included in the second region are the same person, based on the first circumscribing shape that circumscribing each of the first feature points and the second circumscribing shape that circumscribing each of the second feature points. The image processing apparatus described in Appendix 1.

[0095] [Note 3] Output means for outputting the first region and the feature points included in the first region, The system further comprises a third detection means for detecting a corresponding region from the second region that corresponds to the first region, The output means is When the first region detection means detects the first region, it outputs the first region and the first feature point. If the first region detection means does not detect the first region, the corresponding region and the feature points included in the corresponding region are output. The image processing apparatus described in Appendix 1 or 2.

[0096] [Note 4] The second region detection means detects a second region from the image that includes at least a portion of a person and encompasses the first region. An image processing device as described in any one of the appendices 1 to 3.

[0097] [Note 5] The first region detection means detects a face region including the person's face from the image as the first region, The second region detection means detects the head region, including the head of the person, from the image as the second region. The image processing apparatus described in Appendix 4.

[0098] [Note 6] When the first region detection means detects a plurality of first regions, it selects one first region based on the reliability of each first region. If the second region detection means detects multiple second regions, it selects one second region based on the reliability of each second region. An image processing device as described in any one of the appendices 1 to 5.

[0099] [Note 7] A first region containing at least a part of a person is detected from the image. A first feature point is detected from the first region, From the aforementioned image, a second region is detected that includes at least a part of a person and encompasses the first region. A second feature point is detected from the second region, Based on the first and second feature points, it is estimated whether the person included in the first region and the person included in the second region are the same person. Image processing methods.

[0100] [Note 8] On the computer, A first region containing at least a part of a person is detected from the image. A first feature point is detected from the first region, From the aforementioned image, a second region is detected that includes at least a part of a person and encompasses the first region. A second feature point is detected from the second region, Based on the first and second feature points, it is estimated whether the person included in the first region and the person included in the second region are the same person. A recording medium on which a computer program for executing an image processing method is stored.

[0101] At least some of the constituent elements of each embodiment described above can be appropriately combined with at least some other constituent elements of each embodiment described above. Some of the constituent elements of each embodiment described above may not be used. Furthermore, to the extent permitted by law, all of the disclosures of the documents cited above in this disclosure (e.g., published gazettes) shall be incorporated as part of the description of this disclosure.

[0102] This disclosure may be modified as appropriate, insofar as it does not contradict the technical idea that can be inferred from the claims and the entire specification. Image processing apparatus, image processing methods, and recording media, with such modifications, are also included in the technical idea of ​​this disclosure. [Explanation of Symbols]

[0103] 1,2,3 Image processing device 11,211 First Region Detection Unit 12,212 First Feature Point Detection Unit 13,213 Second Region Detection Unit 14,214 Second Feature Point Detection Unit 15,215 Estimation Department 1R Face Area 1F Facial feature points 1FR Facial Feature Point Region 2R head area 2F Head feature points 2FR Head Feature Point Region 316 Output Control Unit 317 Third detection unit 3R (Reduce, Reuse, Recycle) areas 518 Third Region Detection Unit 519 Third Feature Point Detection Unit

Claims

1. A first region detection means for detecting a first region from an image that includes at least a part of a person, A first feature point detection means for detecting a first feature point from the first region, A second region detection means for detecting a second region from the aforementioned image that includes at least a portion of a person, overlaps with at least a portion of the first region, and is of a different size from the first region, A second feature point detection means for detecting a second feature point from the second region, If the first region detection means does not detect the first region, a third detection means detects a corresponding region from the second region that corresponds to the first region, If the first region detection means detects the first region, the output means outputs the first region and the first feature points; if the first region detection means does not detect the first region, the output means outputs the corresponding region and the feature points included in the corresponding region. An image processing device equipped with the following features.

2. When the first region detection means detects the first region, the system further includes estimation means for estimating whether the person included in the first region and the person included in the second region are the same person, based on a first external shape that circumscribes each of the first feature points and a second external shape that circumscribes each of the second feature points. The image processing apparatus according to claim 1.

3. The second region detection means detects a second region from the image that includes at least a portion of a person and encompasses the first region. The image processing apparatus according to claim 1 or 2.

4. The first region detection means detects a face region including the person's face from the image as the first region, The second region detection means detects the head region, including the head of the person, from the image as the second region. The image processing apparatus according to claim 3.

5. When the first region detection means detects a plurality of first regions, it selects one first region based on the reliability of each first region. When the second region detection means detects multiple second regions, it selects one second region based on the reliability of each second region. The image processing apparatus according to claim 1 or 2.

6. A first region containing at least a part of a person is detected from the image. A first feature point is detected from the first region, From the aforementioned image, a second region is detected that includes at least a part of a person and encompasses the first region. A second feature point is detected from the second region, If the first region is not detected, a corresponding region corresponding to the first region is detected from the second region. If the first region is detected, the first region and the first feature points are output; if the first region is not detected, the corresponding region and the feature points included in the corresponding region are output. A method of image processing performed by a computer.

7. On the computer, A first region containing at least a part of a person is detected from the image. A first feature point is detected from the first region, From the aforementioned image, a second region is detected that includes at least a part of a person and encompasses the first region. A second feature point is detected from the second region, If the first region is not detected, a corresponding region corresponding to the first region is detected from the second region. If the first region is detected, the first region and the first feature points are output. If the first region is not detected, the corresponding region and the feature points included in the corresponding region are output. A computer program used to execute image processing methods.

Images