Image processing device, image processing method, and recording medium

Abstract

An image processing device 1 comprises: a first area detection unit 11 that detects from an image a first area including at least a portion of a person; a first feature detection unit 12 that detects a first feature from the first area; a second area detection unit 13 that detects from an image a second area which includes at least a portion of a person, which overlaps the first area at least partially, and which is different in size from the first area; a second feature detection unit 14 that detects a second feature from the second area; and an estimation unit 15 that estimates, on the basis of the first and second features, whether or not the person included in the first area and the person included in the second area are the same person.

Description

[Technical Field]

[0001] The present disclosure relates to the technical fields of an image processing device, an image processing method, and a recording medium. [Background technology]

[0002] Patent Document 1 describes a technology that includes an image acquisition step of acquiring an image of a person to be determined, an attribute determination region detection step of detecting at least two attribute determination regions selected from a group consisting of a head region, a face region, and other regions from the image of the person to be determined, and an attribute determination step of determining attributes from images of the at least two attribute determination regions, and that enables high person detection accuracy and person attribute determination even when characteristic facial features are hidden. Patent Document 2 describes a technology that selects, as a final characteristic region detection result, either a detection result of a predetermined characteristic region for a predetermined partial region detected from image data by a first method or a detection result of a predetermined characteristic region for a predetermined partial region detected from image data by a second method. Patent Document 3 describes a technology in an image retrieval device having a face region extraction program that extracts a face region from video data, extracts a face region from the video data by a first algorithm, extracts a head region from the video data by a second algorithm, and performs face detection on a region that was extracted as a head region but not as a face region while changing the image quality, and extracts the head region in which a face image is detected as the face region. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2012 / 053311 [Patent Document 2] Patent Publication No. 2021-082136 [Patent Document 3] International Publication No. 2018 / 173947 Summary of the Invention [Problem to be solved by the invention]

[0004] An object of this disclosure is to provide an image processing device, an image processing method, and a recording medium that aim to improve upon the techniques described in prior art documents. [Means for solving the problem]

[0005] One aspect of the image processing device includes a first area detection means for detecting a first area from an image that includes at least a portion of a person, a first feature point detection means for detecting a first feature point from the first area, a second area detection means for detecting a second area from the image that includes at least a portion of the person, overlaps with at least a portion of the first area, and is a different size from the first area, a second feature point detection means for detecting a second feature point from the second area, and an estimation means for estimating whether the person included in the first area and the person included in the second area are the same person based on the first feature point and the second feature point.

[0006] One aspect of the image processing method includes detecting a first area from an image that includes at least a portion of a person, detecting first feature points from the first area, detecting a second area from the image that includes at least a portion of the person, overlaps with at least a portion of the first area, and is a different size from the first area, detecting second feature points from the second area, and estimating whether the person included in the first area and the person included in the second area are the same person based on the first feature points and the second feature points.

[0007] In one embodiment of the recording medium, a computer program is recorded to cause a computer to execute an image processing method of detecting a first area from an image that includes at least a portion of a person, detecting first feature points from the first area, detecting a second area from the image that includes at least a portion of the person, overlaps with at least a portion of the first area, and is a different size from the first area, detecting second feature points from the second area, and estimating, based on the first feature points and the second feature points, whether the person included in the first area and the person included in the second area are the same person. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing the configuration of an image processing apparatus according to the first embodiment. [Figure 2] FIG. 2 is a block diagram showing the configuration of an image processing apparatus according to the second embodiment. [Figure 3] FIG. 3 is a flowchart showing the flow of image processing operations performed by the image processing device in the second embodiment. [Figure 4] FIG. 4 is a conceptual diagram of the image processing operation performed by the image processing device in the second embodiment. [Figure 5] FIG. 5 is a block diagram showing the configuration of an image processing apparatus according to the third embodiment. [Figure 6] FIG. 6 is a flowchart showing the flow of image processing operations performed by the image processing device in the third embodiment. [Figure 7] FIG. 7 is a conceptual diagram of the image processing operation performed by the image processing device in the third embodiment. [Figure 8] FIG. 8 is a block diagram showing the configuration of an image processing apparatus according to the fifth embodiment. [Figure 9] FIG. 9 is a flowchart showing the flow of image processing operations performed by the image processing device in the fifth embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of an image processing device, an image processing method, and a recording medium will be described with reference to the drawings. [1: First embodiment]

[0010] An image processing device, an image processing method, and a recording medium according to a first embodiment will be described below. The image processing device, the image processing method, and the recording medium according to the first embodiment will be described below using an image processing device 1 to which the image processing device, the image processing method, and the recording medium according to the first embodiment are applied. [1-1: Configuration of image processing device 1]

[0011] The configuration of an image processing device 1 in the first embodiment will be described with reference to Fig. 1. Fig. 1 is a block diagram showing the configuration of the image processing device 1 in the first embodiment.

[0012] As shown in FIG. 1, the image processing device 1 includes a first region detection unit 11, a first feature point detection unit 12, a second region detection unit 13, a second feature point detection unit 14, and an estimation unit 15. The first region detection unit 11 detects a first region from an image that includes at least a portion of a person. The first feature point detection unit 12 detects first feature points from the first region. The second region detection unit 13 detects a second region from the image that includes at least a portion of a person, overlaps with at least a portion of the first region, and is a different size from the first region. The second feature point detection unit 14 detects second feature points from the second region. The estimation unit 15 estimates whether the person included in the first region and the person included in the second region are the same person based on the first feature points and the second feature points. [1-2: Technical Effects of Image Processing Device 1]

[0013] The image processing device 1 in the first embodiment estimates whether the person included in the first area and the person included in the second area are the same person based on the first feature point and the second feature point, and can therefore accurately estimate whether they are the same person. [2: Second embodiment]

[0014] A second embodiment of an image processing device, an image processing method, and a recording medium will be described below. The second embodiment of the image processing device, the image processing method, and the recording medium will be described below using an image processing device 2 to which the second embodiment of the image processing device, the image processing method, and the recording medium is applied. [2-1: Configuration of image processing device 2]

[0015] The configuration of the image processing device 2 in the second embodiment will be described with reference to Fig. 2. Fig. 2 is a block diagram showing the configuration of the image processing device 2 in the second embodiment.

[0016] 2, the image processing device 2 includes a calculation device 21 and a storage device 22. The image processing device 2 may further include a communication device 23, an input device 24, and an output device 25. However, the image processing device 2 does not necessarily include at least one of the communication device 23, the input device 24, and the output device 25. The calculation device 21, the storage device 22, the communication device 23, the input device 24, and the output device 25 may be connected via a data bus 26.

[0017] The arithmetic device 21 includes, for example, at least one of a central processing unit (CPU), a graphics processing unit (GPU), and a field programmable gate array (FPGA). The arithmetic device 21 reads a computer program. For example, the arithmetic device 21 may read a computer program stored in the storage device 22. For example, the arithmetic device 21 may read a computer program stored in a computer-readable, non-transitory recording medium using a recording medium reading device (e.g., an input device 24 described later) not shown in the figure that is provided in the image processing device 2. The arithmetic device 21 may acquire (i.e., download or read) the computer program from a device (not shown) located outside the image processing device 2 via the communication device 23 (or another communication device). The arithmetic device 21 executes the read computer program. As a result, logical functional blocks for executing operations to be performed by the image processing device 2 are realized within the arithmetic device 21. That is, the arithmetic device 21 can function as a controller for realizing logical functional blocks for executing the operations (in other words, processing) that the image processing device 2 should perform.

[0018] 2 shows an example of logical functional blocks implemented in the arithmetic device 21 to perform image processing operations. As shown in Fig. 2, implemented in the arithmetic device 21 are a first area detection unit 211 which is a specific example of a "first area detection means," a first feature point detection unit 212 which is a specific example of a "first feature point detection means," a second area detection unit 213 which is a specific example of a "second area detection means," a second feature point detection unit 214 which is a specific example of a "second feature point detection means," and an estimation unit 215 which is a specific example of an "estimation means." The operations of the first area detection unit 211, the first feature point detection unit 212, the second area detection unit 213, the second feature point detection unit 214, and the estimation unit 215 will be described below with reference to Figs. 3 and 4.

[0019] The storage device 22 can store desired data. For example, the storage device 22 may temporarily store a computer program executed by the arithmetic device 21. The storage device 22 may temporarily store data that the arithmetic device 21 temporarily uses when the arithmetic device 21 is executing a computer program. The storage device 22 may store data that the image processing device 2 stores long-term. The storage device 22 may include at least one of a RAM (Random Access Memory), a ROM (Read Only Memory), a hard disk device, a magneto-optical disk device, an SSD (Solid State Drive), and a disk array device. In other words, the storage device 22 may include a non-temporary recording medium.

[0020] The communication device 23 is capable of communicating with devices external to the image processing device 2 via a communication network (not shown). The communication device 23 may acquire images to be used in the image processing operation from, for example, an imaging device via the communication network.

[0021] The input device 24 is a device that accepts information input to the image processing device 2 from outside the image processing device 2. For example, the input device 24 may include an operation device (for example, at least one of a keyboard, a mouse, and a touch panel) that can be operated by an operator of the image processing device 2. For example, the input device 24 may include a reading device that can read information recorded as data on a recording medium that can be externally attached to the image processing device 2.

[0022] The output device 25 is a device that outputs information to the outside of the image processing device 2. For example, the output device 25 may output information as an image. That is, the output device 25 may include a display device (a so-called display) that can display an image showing the information to be output. For example, the output device 25 may output information as sound. That is, the output device 25 may include an audio device (a so-called speaker) that can output sound. For example, the output device 25 may output information on paper. That is, the output device 25 may include a printing device (a so-called printer) that can print desired information on paper. [2-2: Image processing operation performed by image processing device 2]

[0023] The flow of image processing operations performed by the image processing device 2 in the second embodiment will be described with reference to Figures 3 and 4. Figure 3 is a flowchart showing the flow of image processing operations performed by the image processing device 2 in the second embodiment. Figure 4 is a conceptual diagram of the image processing operations of the image processing device 2 in the second embodiment.

[0024] As shown in FIG. 3, the first area detection unit 211 detects a first area including at least a part of a person from the image (step S20). The first area may be a face area including the face of the person. The first area detection unit 211 may detect a face area including the face of the person from the image as the first area. The first area detection unit 211 may detect, for example, a face area 1R as shown in FIG. 4(a).

[0025] The first area detection unit 211 may detect a face area by applying a known face detection process to the image data. The first area detection unit 211 may detect an area having facial features as a face area. The area having facial features may be characteristic parts that make up a face, such as the eyes, nose, and mouth. There are no particular limitations on the method of detecting a face area performed by the first area detection unit 211. The first area detection unit 211 may detect a face area, for example, based on extraction of edges or patterns that are characteristic of a face area.

[0026] The first region detection unit 211 may use a neural network that has learned face regions through machine learning. The first region detection unit 211 may be configured with a convolutional neural network (hereinafter also referred to as "CNN"). The first region detection unit 211 may be configured with two stacked CNNs. The first region detection unit 211 may detect a face region 1R derived from the first stage of the CNN and a face region 1R derived from the second stage.

[0027] In step S20, if the first area detection unit 211 detects multiple face areas 1R as first areas, it may select one face area 1R based on the reliability of each face area 1R. If the first area detection unit 211 detects multiple face areas 1R, it may sort each face area 1R in order of the reliability score of the corresponding face area 1R. The reliability score may indicate the likelihood that the face area 1R is a face area 1R. The first area detection unit 211 may adopt the face area 1R with the highest reliability score as the face area 1R to be used in subsequent processing.

[0028] The first region detection unit 211 may suppress overlapping regions using Non-Maximum Suppression (NMS). The first region detection unit 211 may use, for example, 0.45 as a threshold value for NMS.

[0029] The first feature point detection unit 212 detects first feature points from the face region 1R (step S21). The first feature point detection unit 212 may detect the position of the first feature point from the face region 1R. The first feature point detection unit 212 may detect the positions of multiple first feature points from the face region 1R. The first feature point detection unit 212 may detect multiple facial feature points 1F, for example, as illustrated in FIG. 4(a). The first feature point detection unit 212 may detect multiple characteristic points in the eye region and nose region as facial feature points 1F, for example, as illustrated in FIG. 4(a). The multiple characteristic points in the eye region may include the edge of the eye and the pupil. The multiple characteristic points in the nose region may include the nasal protuberance and the edge of the nostrils.

[0030] There are no particular limitations on the method of detecting feature points performed by the first feature point detection unit 212. The first feature point detection unit 212 may use, for example, pattern matching. The first feature point detection unit 212 may use a neural network that has machine-learned the eye region, nose region, etc. The first feature point detection unit 212 may be configured as a CNN.

[0031] The second region detection unit 213 detects a second region from the image that includes at least a part of a person, overlaps with at least a part of the first region, and is different in size from the first region (step S22). The second region detection unit 213 may detect a second region from the image that includes at least a part of a person and encompasses the first region. The second region may be a head region that includes the person's head. The second region detection unit 213 may detect a head region that includes the person's head as the second region from the image. The second region detection unit 213 may detect, for example, a head region 2R as illustrated in FIG. 4(b).

[0032] The second region detection unit 213 may detect the head region 2R by applying a known head detection process to the image data. The second region detection unit 213 may detect a region having characteristics of the head as the head region 2R. There are no particular limitations on the method used by the second region detection unit 213 to detect the head region 2R. For example, the second region detection unit 213 may detect a region including a characteristic part of the head, such as hair. The second region detection unit 213 may detect a region having a predetermined shape, such as an Ω shape. The second region detection unit 213 may detect the head region 2R mainly based on the shape of the outline. The second region detection unit 213 may detect the head region 2R by also detecting human body parts such as the limbs and torso. Unlike the first region detection unit 211, the second region detection unit 213 does not need to rely on the detection of characteristic parts that make up the face. In this case, the second region detection unit 213 may be able to detect the head region 2R even if the first region detection unit 211 fails to detect the face region 1R. On the other hand, unlike the first area detection section 211, the second area detection section 213 may rely on the detection of characteristic parts that make up the face.

[0033] The second region detection unit 213 may use a neural network that has learned the head region 2R through machine learning. The second region detection unit 213 may be configured with a CNN. The second region detection unit 213 may be configured with two stacked CNNs. The second region detection unit 213 may detect the head region 2R derived from the first stage of the CNN and the head region 2R derived from the second stage.

[0034] In step S22, if second region detection unit 213 detects multiple head regions 2R as second regions, it may select one head region 2R based on the reliability of each head region 2R. If second region detection unit 213 detects multiple head regions 2R, it may sort each head region 2R in order of the reliability score of the corresponding head region 2R. The reliability score may indicate the likelihood that the head region 2R is a head region 2R. Second region detection unit 213 may adopt the head region 2R with the highest reliability score as the head region 2R to be used in subsequent processing.

[0035] The second region detection unit 213 may suppress overlapping regions using Non-Maximum Suppression (NMS). The second region detection unit 213 may use, for example, 0.40 as a threshold for NMS.

[0036] The second feature point detection unit 214 detects second feature points from the head region 2R (step S23). The second feature point detection unit 214 may detect the position of the second feature point from the head region 2R. The second feature point detection unit 214 may detect the positions of multiple second feature points from the head region 2R. The second feature point detection unit 214 may detect multiple head feature points 2F, for example, as illustrated in FIG. 4(b). The second feature point detection unit 214 may detect multiple feature points in the eye region and nose region as head feature points 2F, for example, as illustrated in FIG. 4(b). There are no particular limitations on the method of feature point detection performed by the second feature point detection unit 214. The second feature point detection unit 214 may use, for example, pattern matching. The second feature point detection unit 214 may use a neural network that has machine-learned the eye region, nose region, etc. The second feature point detection unit 214 may be configured as a CNN.

[0037] The estimation unit 215 integrates the face region 1R and the head region 2R (step S24). The estimation unit 215 may match the face region 1R and the head region 2R. The estimation unit 215 estimates whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the face feature point 1F and the head feature point 2F. The estimation unit 215 may estimate whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the position of the face feature point 1F and the position of the head feature point 2F. The estimation unit 215 may estimate whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the positional relationship between the feature points. The estimation unit 215 may estimate that the closer the distance between the feature points is, the more they match. The estimation unit 215 may estimate that the person included in the face region 1R and the person included in the head region 2R are the same person based on the degree of match between the feature points.

[0038] The estimation unit 215 may estimate whether the person included in the face region 1R and the person included in the head region 2R are the same person based on a first circumscribing shape circumscribing each of the face feature points 1F and a second circumscribing shape circumscribing each of the head feature points 2F. In this case, the circumscribing shape circumscribing each of the feature points may be a circumscribing rectangle circumscribing each of the feature points. That is, the estimation unit 215 may estimate that the person included in the face region 1R and the person included in the head region 2R are the same person based on the overlap state between the face feature point region 1FR, which is a rectangle circumscribing each of the face feature points 1F, and the head feature point region 2FR, which is a rectangle circumscribing each of the head feature points 2F. For example, as illustrated in FIG. 4(c), the estimation unit 215 may estimate whether the person included in the face region 1R and the person included in the head region 2R are the same person based on the face feature point region 1FR and the head feature point region 2FR.

[0039] The estimation unit 215 may estimate whether the person included in the face area 1R and the person included in the head area 2R are the same person based on the overlap rate between the face feature point area 1FR and the head feature point area 2FR. The estimation unit 215 may use, for example, IoU (Intersection over Union, sometimes called "Jaccard coefficient") as the overlap rate. IoU is the ratio of the overlapping portion to the union of the two areas, IoU = size of overlap of two regions / size of union of two regions It can be expressed as:

[0040] If the face region 1R and the head region 2R are not associated, there is a possibility that the number of false detections of people will increase. In contrast, the estimation unit 215 associates the face region 1R and the head region 2R, thereby making it possible to suppress false detections of people. Furthermore, in the comparative example in which the face region 1R and the head region 2R are directly associated, association may be difficult, but the estimation unit 215 associates the face region 1R and the head region 2R based on the first feature point and the second feature point, making association easier than in the comparative example.

[0041] Note that the estimation unit 215 may not perform the association operation if the face region 1R is not detected and only the head region 2R is detected. Similarly, the estimation unit 215 may not perform the association operation if the head region 2R is not detected and only the face region 1R is detected. [2-3: Effects of image processing device 2]

[0042] For example, when the image resolution is low, CNN may misdetect or fail to detect the target area. On the other hand, when a high-resolution image is used in CNN, the detection accuracy improves, but the amount of calculation increases and the detection speed slows down. For example, there is a demand for a lightweight detection engine in edge servers, but there is a trade-off between detection accuracy and detection speed.

[0043] In contrast, the image processing device 2 in the second embodiment performs at least one of estimation based on the positions of the first feature points and the second feature points, and estimation based on the first circumscribing shape and the second circumscribing shape. The image processing device 2 uses features detected from the detection results, thereby enabling accurate estimation of whether the person included in the first area and the person included in the second area are the same person. This allows the image processing device 2 to suppress erroneous detection of the target area. Furthermore, by detecting the second area that encompasses the first area, the image processing device 2 can suppress non-detection of the target area. Therefore, the image processing device 2 can accurately detect the target area even when the image has a low resolution. Since it is not necessary to increase the image resolution, the image processing device 2 can improve detection accuracy without reducing the detection speed. [3: Third embodiment]

[0044] A third embodiment of an image processing device, an image processing method, and a recording medium will be described below. The third embodiment of the image processing device, the image processing method, and the recording medium will be described below using an image processing device 3 to which the third embodiment of the image processing device, the image processing method, and the recording medium is applied. [3-1: Configuration of image processing device 3]

[0045] The configuration of the image processing device 3 in the third embodiment will be described with reference to Fig. 5. Fig. 5 is a block diagram showing the configuration of the image processing device 3 in the third embodiment.

[0046] As shown in FIG. 5 , the image processing device 3 in the third embodiment includes a calculation device 21 and a storage device 22, similar to the image processing device 2 in the second embodiment. Furthermore, the image processing device 3 may include a communication device 23, an input device 24, and an output device 25, similar to the image processing device 2 in the second embodiment. However, the image processing device 3 does not have to include at least one of the communication device 23, the input device 24, and the output device 25. The image processing device 3 in the third embodiment differs from the image processing device 2 in the second embodiment in that the calculation device 21 includes an output control unit 316 and a third detection unit 317. Other features of the image processing device 3 may be the same as other features of the image processing device 2 in the second embodiment. [3-2: Image processing operation performed by image processing device 3]

[0047] The flow of image processing operations performed by the image processing device 3 in the third embodiment will be described with reference to Figures 6 and 7. Figure 6 is a flowchart showing the flow of image processing operations performed by the image processing device 3 in the third embodiment. Figure 7 is a conceptual diagram of the image processing operations of the image processing device 3 in the third embodiment.

[0048] As shown in FIG. 6, the first region detection unit 211 detects a face region 1R including at least a part of a person from the image (step S20). The first feature point detection unit 212 detects face feature points 1F from the face region 1R (step S21). The second region detection unit 213 detects a head region 2R including at least a part of a person and encompassing the face region 1R from the image (step S22). The second feature point detection unit 214 detects a head feature point 2F from the head region 2R (step S23). The estimation unit 215 integrates the face region 1R and the head region 2R (step S24).

[0049] The output control unit 316 determines whether or not a face region 1R is present (step S30). The output control unit 316 may determine whether or not the first region detection unit 211 has detected a face region 1R. Note that if the face region 1R is not detected and only the head region 2R is detected, step S30 may be omitted and the process may proceed to step S32.

[0050] If a face region 1R is present (step S30: Yes), that is, if the first region detection unit 211 detects the face region 1R, the output control unit 316 outputs the face region 1R and facial feature points 1F (step S31). The presence of a face region 1R may include a case where the first region detection unit 211 detects the face region 1R and the second region detection unit 213 does not detect the head region 2R, and a case where the first region detection unit 211 detects the face region 1R and the second region detection unit 213 detects the head region 2R. The output control unit 316 may output the face region 1R and facial feature points 1F integrated with the head region 2R in step S24. The output control unit 316 may search for a pair whose IoU is greater than or equal to 0.01 and is the largest, and output the corresponding face region 1R and facial feature points 1F. The output control unit 316 may output a face region 1R and face feature points 1F, as exemplified in FIG. 7(a).

[0051] On the other hand, if a face region 1R does not exist (step S30: No), that is, if the first region detection unit 211 has not detected a face region 1R, the third detection unit 317 detects a corresponding region 3R corresponding to the face region 1R from the head region 2R (step S32). The absence of a face region 1R may include a case where the first region detection unit 211 does not detect a face region 1R and the second region detection unit 213 detects a head region 2R. The third detection unit 317 may detect a corresponding region 3R corresponding to the face region 1R, as exemplified in FIG. 7(b). The third detection unit 317 may detect, for example, a square whose sides are twice the width or four times the height of the head feature point region 2FR, whichever is longer.

[0052] The output control unit 316 outputs the corresponding region 3R and the feature points included in the corresponding region 3R (step S33). The feature points included in the corresponding region 3R may be the same points as the head feature points 2F. The output control unit 316 may output the corresponding region 3R and the head feature points 2F, as exemplified in FIG. 7(c).

[0053] The output control unit 316 may integrate multiple rectangles to perform data shaping and output the data shaping results. The data shaping operation may include, for example, sorting, rounding, NMS, inclusion, and class assignment. The sorting may be, for example, sorting all detected regions in order of confidence score. The rounding may be, for example, rounding coordinates that extend outside the image to within the image. The NMS may be, for example, suppressing overlapping regions by using an NMS threshold of, for example, 0.45. The inclusion may be, for example, retaining the region with the higher confidence score for regions with a complete inclusion relationship. The class assignment may be, for example, assigning the same class ID to head-derived results as to face-derived results. [3-3: Effects of image processing device 3]

[0054] When the first area is not detected, the image processing device 3 in the third embodiment detects a corresponding area corresponding to the first area from the second area and outputs the corresponding area and the feature points included in the corresponding area. This makes it possible to obtain the first area of ​​a person and the feature points included in the first area even when the first area is not detected.

[0055] The head is larger than the face. For this reason, the head region 2R can be detected as larger than the face region 1R, and the head region 2R may contain more useful information than the face region 1R. The first region detection unit 211 and the second region detection unit 213 use different detection methods. Therefore, even if the first region detection unit 211 does not detect the face region 1R, the second region detection unit 213 may detect the head region 2R.

[0056] When a face region 1R has not been detected, the image processing device 3 detects the head region 2R and converts the head region 2R into a face region 1R, thereby preventing the target region from going undetected. This allows the image processing device 3 to detect the region corresponding to the face region 1R even in situations where the face region 1R has not been detected until now. In other words, the image processing device 3 can improve the failure to detect the face region 1R when the image has a low resolution. The image processing device 3 can improve the failure to detect the face region 1R when the image is near the minimum size of the image that is the detection limit. In other words, the image processing device 3 can lower the minimum size of the image that is the detection limit, thereby expanding the detectable size.

[0057] The first region detection unit 211 and the first feature point detection unit 212 may be configured by the same CNN. Furthermore, the second region detection unit 213 and the second feature point detection unit 214 may be configured by the same CNN. Furthermore, the first region detection unit 211, the first feature point detection unit 212, the second region detection unit 213, and the second feature point detection unit 214 may be configured by the same CNN. Furthermore, the detection of the first region and the detection of the second region may be performed simultaneously. [4: Fourth embodiment]

[0058] A fourth embodiment of an image processing device, an image processing method, and a recording medium will be described below. The fourth embodiment of the image processing device, the image processing method, and the recording medium will be described below using an image processing device 4 to which the fourth embodiment of the image processing device, the image processing method, and the recording medium is applied.

[0059] The image processing device 4 in the fourth embodiment is different from the image processing device 2 in the second embodiment and the image processing device 3 in the third embodiment in the first region detected by the first region detection unit 211 and the second region detected by the second region detection unit 213. Other features of the image processing device 4 may be the same as other features of at least one of the image processing device 2 and the image processing device 3.

[0060] In the fourth embodiment, the first region detection unit 211 may detect a head region including the head of a person as the first region, and the second region detection unit 213 may detect at least one of an upper body region and a whole body region of a person as the second region.

[0061] Detection of the head region is useful in tracking a person. The tracking operation of a person may be applied to so-called gateless authentication, which performs person authentication while a person is staying in a predetermined area. Furthermore, detection of the upper body region and / or the whole body region is useful in covering the tracking operation of a person when, for example, detection of the head region fails. For example, even if the first region detection unit 211 fails to detect the head region, if the second region detection unit 213 detects the upper body region or the whole body region, it is possible to determine that the person is the same person based on the characteristics of the person included in the upper body region or the whole body region, and continue tracking the person.

[0062] In the fourth embodiment, the first area detection unit 211 may detect a face area including a person's face as the first area, and the second area detection unit 213 may detect the person's upper body area and / or whole body area as the second area. [5: Fifth embodiment]

[0063] A fifth embodiment of an image processing device, an image processing method, and a recording medium will be described below. The fifth embodiment of an image processing device, an image processing method, and a recording medium will be described below using an image processing device 5 to which the fifth embodiment of the image processing device, the image processing method, and the recording medium is applied. [5-1: Configuration of image processing device 5]

[0064] The configuration of the image processing device 5 in the fifth embodiment will be described with reference to Fig. 8. Fig. 8 is a block diagram showing the configuration of the image processing device 5 in the fifth embodiment.

[0065] 8, the image processing device 5 in the fifth embodiment differs from the image processing device 3 in the third embodiment in that the calculation device 21 includes a third region detection unit 518 and a third feature point detection unit 519. Other features of the image processing device 5 may be the same as other features of the image processing device 3 in the third embodiment. [5-2: Image processing operation performed by image processing device 5]

[0066] The flow of the image processing operation performed by the image processing device 5 in the fifth embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing the flow of the image processing operation performed by the image processing device 5 in the fifth embodiment.

[0067] 9, the first region detection unit 211 detects a face region as a first region (step S20). The first feature point detection unit 212 detects face feature points as first feature points from the face region (step S21). The second region detection unit 213 detects a head region as a second region (step S22). The second feature point detection unit 214 detects head feature points as second feature points from the head region (step S23).

[0068] The third region detection unit 518 detects a third region from the image that includes at least a part of a person, overlaps with at least a part of the second region, and is different in size from the second region (step S51). The third region detection unit 518 may detect a third region from the image that includes at least a part of a person and encompasses the second region. The third region may be an upper body region or a whole body region of the person. The third region detection unit 518 may detect the upper body region or the whole body region of the person from the image as the third region.

[0069] The third feature point detection unit 519 detects a third feature point from the upper body region or the whole body region (step S52). The third feature point detection unit 519 may detect the position of the third feature point from the upper body region or the whole body region. The third feature point detection unit 519 may detect the positions of multiple third feature points from the upper body region or the whole body region.

[0070] The estimation unit 215 integrates the face region, the head region, and the upper body region or the whole body region (step S53). The estimation unit 215 may match the face region, the head region, and the upper body region or the whole body region. The estimation unit 215 estimates whether a person included in the face region, a person included in the head region, and a person included in the upper body region or the whole body region are the same person based on the face feature points, the head feature points, and the third feature points. The estimation unit 215 may estimate whether a person included in the face region, a person included in the head region, and a person included in the upper body region or the whole body region are the same person based on the positions of the face feature points, the head feature points, and the third feature points. The estimation unit 215 may estimate whether a person included in the face region, a person included in the head region, and a person included in the upper body region or the whole body region are the same person based on the positional relationship between the feature points. The estimation unit 215 may estimate that the smaller the distance between the feature points, the more similar the person is. The estimation unit 215 may estimate the person included in the face region, the person included in the head region, and the person included in the upper body region or the whole body region based on the degree of match of each feature point. Also, similar to the above embodiment, the estimation unit 215 may estimate whether the person included in the face region, the person included in the head region, and the person included in the upper body region or the whole body region are the same person based on the circumscribing shapes that circumscribe each of the feature points.

[0071] The output control unit 316 determines whether a face region exists (step S30). If a face region exists (step S30: Yes), the output control unit 316 outputs the face region and face feature points (step S31).

[0072] If a face region does not exist (step S30: No), output control unit 316 determines whether a head region exists (step S54). If a head region exists (step S54: Yes), third detection unit 317 detects a corresponding region derived from the head region that corresponds to the face region (step S32). Output control unit 316 outputs the corresponding region derived from the head region and feature points included in the corresponding region derived from the head region (step S33).

[0073] If no head region is present (step S54: No), the output control unit 316 causes the third detection unit 317 to detect a corresponding region derived from the upper body or whole body region that corresponds to the face region (step S55).The output control unit 316 outputs the corresponding region derived from the upper body or whole body region and the feature points included in the corresponding region derived from the upper body or whole body region (step S56).

[0074] The operations performed by the image processing device in each of the above-described embodiments can be rephrased as detecting objects of multiple classes that are in an inclusion relationship, detecting feature points from the detected objects, estimating object pairs based on the detection results, and estimating whether the detection results of different classes belong to the same person. In particular, the operations performed by the image processing device in the third embodiment can be rephrased as estimating the areas of objects of included classes.

[0075] The image processing device in each of the above-described embodiments is suitable for application to tracking operations and face authentication operations. [6: Note]

[0076] The following additional notes are provided regarding the above-described embodiment. [Appendix 1] a first area detection means for detecting a first area including at least a part of a person from an image; a first feature point detection means for detecting a first feature point from the first region; a second region detection means for detecting a second region from the image, the second region including at least a part of a person, overlapping at least a part of the first region, and having a different size from the first region; second feature point detection means for detecting second feature points from the second region; an estimation means for estimating whether a person included in the first area and a person included in the second area are the same person based on the first feature point and the second feature point; An image processing device comprising: [Appendix 2] The estimation means estimates whether a person included in the first area and a person included in the second area are the same person based on a first circumscribing shape that circumscribing each of the first feature points and a second circumscribing shape that circumscribing each of the second feature points. 2. The image processing device according to claim 1. [Appendix 3] an output means for outputting the first region and feature points included in the first region; a third detection means for detecting a corresponding area corresponding to the first area from the second area; The output means When the first area detection means detects the first area, the first area and the first feature point are output; If the first region detection means does not detect the first region, the corresponding region and the feature points included in the corresponding region are output. 3. The image processing device according to claim 1 or 2. [Appendix 4] The second region detection means detects a second region that includes at least a part of a person and encompasses the first region from the image. 4. The image processing device according to any one of claims 1 to 3. [Appendix 5] the first area detection means detects a face area including a face of the person from the image as the first area; The second region detection means detects a head region including the head of the person from the image as the second region. 5. The image processing device according to claim 4. [Appendix 6] When the first region detection means detects a plurality of the first regions, the first region detection means selects one of the first regions based on the reliability of each of the first regions; When the second region detection means detects a plurality of the second regions, the second region detection means selects one second region based on the reliability of each second region. 6. The image processing device according to any one of appendices 1 to 5. [Appendix 7] Detecting a first region including at least a portion of a person from the image; Detecting a first feature point from the first region; detecting a second region from the image that includes at least a portion of a person and encompasses the first region; Detecting a second feature point from the second region; Based on the first feature point and the second feature point, it is estimated whether the person included in the first area and the person included in the second area are the same person. Image processing methods. [Appendix 8] On the computer, Detecting a first region including at least a portion of a person from the image; Detecting a first feature point from the first region; detecting a second region from the image that includes at least a portion of a person and encompasses the first region; Detecting a second feature point from the second region; Based on the first feature point and the second feature point, it is estimated whether the person included in the first area and the person included in the second area are the same person. A recording medium on which a computer program for executing an image processing method is recorded.

[0077] At least some of the constituent elements of each of the above-described embodiments can be appropriately combined with at least some of the other constituent elements of each of the above-described embodiments. Some of the constituent elements of each of the above-described embodiments may not be used. Furthermore, to the extent permitted by law, the disclosures of all documents (e.g., published patent applications) cited in this disclosure are incorporated by reference as part of the description of this disclosure.

[0078] This disclosure may be modified as appropriate within the scope of the claims and the technical idea that can be read from the entire specification. Image processing devices, image processing methods, and recording media that incorporate such modifications are also included in the technical idea of ​​this disclosure. [Explanation of symbols]

[0079] 1,2,3 Image processing device 11,211 First area detection unit 12,212 First feature point detection unit 13,213 Second area detection unit 14,214 Second feature point detection unit 15,215 Estimation Department 1R Face area 1F Facial feature points 1FR Facial Feature Area 2R head area 2F Head feature points 2FR head feature area 316 Output control section 317 Third detection unit 3R response area 518 Third Area Detection Unit 519 Third feature point detection unit

Claims

1. a first area detection means for detecting a first area including at least a part of a person from an image; a first feature point detection means for detecting a first feature point from the first region; a second region detection means for detecting a second region from the image, the second region including at least a part of a person, overlapping at least a part of the first region, and having a different size from the first region; second feature point detection means for detecting second feature points from the second region; an estimation means for estimating, when the first area detection means detects the first area, whether a person included in the first area and a person included in the second area are the same person, based on a first circumscribing shape circumscribing each of the first feature points and a second circumscribing shape circumscribing each of the second feature points; An image processing device comprising:

2. an output means for outputting the first region and feature points included in the first region; a third detection means for detecting a corresponding area corresponding to the first area from the second area; The output means When the first area detection means detects the first area, the first area and the first feature point are output; If the first area detection means does not detect the first area, the corresponding area and the feature points included in the corresponding area are output. The image processing device according to claim 1 .

3. The second region detection means detects the second region that includes at least a part of a person and encompasses the first region from the image. The image processing device according to claim 1 .

4. the first area detection means detects a face area including a face of the person from the image as the first area; The second region detection means detects a head region including the head of the person from the image as the second region. The image processing device according to claim 3 .

5. When the first region detection means detects a plurality of the first regions, the first region detection means selects one of the first regions based on the reliability of each of the first regions; When the second region detection means detects a plurality of the second regions, the second region detection means selects one second region based on the reliability of each second region. The image processing device according to claim 1 .

6. Detecting a first region including at least a portion of a person from the image; Detecting a first feature point from the first region; detecting a second region from the image that includes at least a portion of a person and encompasses the first region; Detecting a second feature point from the second region; When the first area is detected, it is estimated whether or not the person included in the first area and the person included in the second area are the same person, based on a first circumscribing shape that circumscribing each of the first feature points and a second circumscribing shape that circumscribing each of the second feature points. A computer-implemented image processing method.

7. On the computer, Detecting a first region including at least a portion of a person from the image; Detecting a first feature point from the first region; detecting a second region from the image that includes at least a portion of a person and encompasses the first region; Detecting a second feature point from the second region; When the first area is detected, it is estimated whether or not the person included in the first area and the person included in the second area are the same person based on a first circumscribing shape that circumscribing each of the first feature points and a second circumscribing shape that circumscribing each of the second feature points. A computer program for executing an image processing method.

Images