Image processing device, image processing method and program

Abstract

To determine an area of an object even when the object is partially masked.SOLUTION: An image processing apparatus includes: first detection means of detecting a first feature point group in association with regions of an object from an image obtained by imaging the object having a plurality of regions; acquisition means of acquiring reliability indicating a degree of probability that a position of each feature point is the region corresponding to the feature point, for each first feature point detected by the first detection means; second detection means of detecting a second feature point on the basis of a part of the first feature point group, for the region corresponding to a feature point with reliability lower than a predetermined value, out of the first feature point group; and determination means of determining an area where the object exists on the basis of a part of the first feature point group and the second feature point.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to detecting people in images. [Background technology]

[0002] In surveillance camera systems, there is a technology that detects objects such as people from camera images and determines whether they are the same as objects detected by other cameras. When the object to be identified is a person, the object is first detected from the camera image. Next, matching features that represent the object's unique characteristics are extracted from the object's region. Then, by comparing the matching features of objects detected by different cameras, it is possible to identify whether these objects are the same. In Patent Document 1, feature points of the object are extracted, and the object region is determined from the circumscribing rectangle of the feature point group. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2014-197386 Summary of the Invention [Problem to be solved by the invention]

[0004] "Occlusion," in which a part of a subject is occluded by another object, is known to be a cause of reduced accuracy in determining an object region, image processing, and image recognition. In feature point detection, the image features of the subject cannot be correctly extracted from the occluded surrounding region, making it impossible to correctly estimate the feature points. Furthermore, in the case of extracting matching features for person matching, information for identifying a person cannot be correctly extracted from the occluded surrounding region. In such cases, the method of Patent Document 1 is unable to determine the object region. The present invention has been made in consideration of the above-mentioned problems, and aims to determine the object region even when part of the object is occluded. [Means for solving the problem]

[0005] In order to achieve the object of the present invention, an image processing device includes a first detection means for acquiring, from an image of a person having a plurality of body parts, a plurality of feature points indicating the positions of the plurality of body parts and a reliability indicating the likelihood of the positions indicated by each feature point, and when the reliability of a feature point indicating the position of an ankle among the plurality of body parts acquired by the first detection means is lower than a predetermined value, a feature point that is on a line connecting feature points of the neck and waist among the plurality of body parts acquired by the first detection means and where the ratio of the distance between the neck and the waist to the distance between the neck and the ankle is a predetermined value, before The apparatus is characterized by having a second detection means for detecting the ankle as a feature point, and a determination means for determining an area including the head feature point acquired by the first detection means and the ankle feature point detected by the second detection means as an area in which the person exists. [Effects of the Invention]

[0006] According to the present invention, the region of an object can be determined even when part of the object is occluded. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a functional configuration of an image display device according to an embodiment of the present invention; [Figure 2] Block diagram showing an example of the functional configuration of an image feature determination unit [Figure 3] Block diagram showing an example of hardware configuration [Figure 4] 1 is a flowchart showing a flow of processing executed by an image processing apparatus according to an embodiment; [Figure 5] 1 is a flowchart showing a flow of processing executed by an image processing apparatus; [Figure 6] 1 is a flowchart showing a flow of processing executed by an image processing apparatus; [Figure 7] FIG. 10 is a diagram illustrating an example of correction of waist feature points. [Figure 8] FIG. 1 is a diagram illustrating an example of correction of foot feature points. [Figure 9] A diagram explaining the process of determining the area of ​​an object. [Figure 10] 1 is a flowchart showing a flow of processing executed by an image processing apparatus; [Figure 11] FIG. 10 is a diagram illustrating a process for correcting feature points outside a partial image area. [Figure 12] A diagram explaining an example of a neural network configuration [Figure 13] Flowchart showing the process of training a neural network [Figure 14] Diagram explaining an example of screen display [Figure 15] A diagram explaining examples of facial parts [Figure 16] Diagram explaining an example of a subnetwork configuration [Figure 17] A diagram illustrating an example of the configuration of an image integration subnetwork. [Figure 18] A diagram illustrating an example of an occluded portion of a person. DETAILED DESCRIPTION OF THE INVENTION

[0009] <Embodiment 1> FIG. 3 shows an example of the hardware configuration of this embodiment. In FIG. 3, reference numeral 301 denotes an image sensor (imaging means) configured with a CCD, CMOS, or the like, for converting a subject image from light into an electrical signal. Reference numeral 302 denotes a signal processing circuit that processes a time-series signal relating to the subject image obtained from the image sensor 301 and converts it into a digital signal. 301 and 302 are connected to a bus as a camera. Reference numeral 303 denotes a CPU that controls the entire device by executing a control program stored in a ROM 304. Reference numeral 304 denotes a ROM that stores the control program executed by the CPU 303 and various parameter data. The control program, when executed by the CPU 303, causes the device to function as various means for executing the processes shown in the flowcharts described below. Reference numeral 305 denotes a RAM that stores images and various information. The RAM 305 also functions as a work area for the CPU 303 and a temporary data save area. Reference numeral 306 denotes a display. Reference numeral 307 denotes an input device, such as a pointing device like a mouse or a keyboard, that accepts input from the user. Reference numeral 308 denotes a communication device such as a network or bus, which communicates data and control signals with other communication devices. In this embodiment, the processes corresponding to the steps of the flowcharts described below are implemented by software using the CPU 303, but some or all of the processes may be implemented by hardware such as electronic circuits. The image display device of the present invention may be implemented by using a general-purpose PC without the image sensor 301 and signal processing circuit 302, or may be implemented as a dedicated device. Software (programs) acquired via a network or various storage media may be executed by a processing device (CPU, processor) such as a personal computer.

[0010] Before describing the embodiments, we will explain terminology. A feature point is a point associated with a component of an object consisting of multiple parts. In the following description, a feature point is specifically defined as the position (two-dimensional coordinates) of a person's joints in an image. Reliability is calculated for each detected feature point and is a real number between 0 and 1 indicating the likelihood that the part corresponding to that feature point is present in the image. For example, when detecting the position of a person's head as a feature point, if the person's head is clearly visible in the image, the reliability is high. Conversely, if the head is blurred or obscured by another object, the reliability of the feature point corresponding to the head is low. In other words, the reliability indicates the likelihood that the position indicated by the feature point is the part corresponding to that feature point. This embodiment will be described using a person as an example of a monitored object, but this is not limited to this, and other objects such as animals and cars may also be used. In other words, the present invention is applicable to any structure consisting of multiple parts. In this embodiment, a person is identified using the features of the person's entire body. On the other hand, a person may also be identified using a face, in which case this is known as "face recognition," "face matching," "face search," or the like.

[0011] The configuration of this embodiment is shown in Fig. 1. This embodiment is composed of an image acquisition unit 101, a first detection unit 102, a feature group unit 103, a second detection unit 104, a feature point storage unit 105, an area determination unit 106, an image extraction unit 107, an image feature extraction unit 108, a recognition unit 109, a display unit 110, a learning unit 111, and an object storage unit 112.

[0012] The image acquisition unit 101 acquires image frames from a camera capturing an object having multiple parts. The first detection unit 102 detects the positions and reliability of feature points of the object from the image frames. A method for detecting the positions and reliability of a person's joints in an image will be described in detail later. The feature group determination unit 103 determines a feature group for detecting feature points whose reliability is lower than a predetermined value, based on the positions and reliability of the feature points detected by the first detection unit 102. Combinations of feature points are prepared in advance, and a feature group is determined from these according to the condition of the feature point reliability. A specific method will be described later. If the reliability of a specific feature point detected by the first detection unit is lower than a predetermined value, the second detection unit 104 detects the specific feature point from the image using a method different from the first detection means. Feature points are detected using the relative positional relationship between feature points. A specific method will be described later. The feature point storage unit 105 stores the detected feature points. The area determination unit 106 determines the area in which an object exists from the feature points. A predetermined combination of specific feature points from the detected feature points is used to determine the region where the object to be subjected to image feature extraction exists. The image extraction unit 107 cuts out the region determined by the region determination unit from the image frame. The image feature extraction unit 108 extracts image features for identifying a person from the cut-out partial image using a neural network or the like. The recognition unit 109 performs image recognition using the extracted image features. In this embodiment, image recognition is performed to identify a person. Specifically, by comparing extracted image features with each other, it is determined whether these features belong to the same person. Details will be described later. The display unit 110 displays the results of image recognition on the screen. The learning unit 111 trains the neural network and the like used by the image feature extraction unit 108 for image feature extraction. The object storage means 112 stores information on objects used by the recognition unit 109.

[0013] 2 shows an example of the configuration of the image feature extraction unit 108 in Fig. 1. The image feature extraction unit 108 is made up of an outside-region feature point correction unit 202, an object part extraction unit 203, an intermediate image feature extraction unit 204, a reliability conversion unit 205, a feature integration unit 206, and an image feature output unit 207.

[0014] An outside-region feature point correction unit 202 corrects feature points outside the partial image region, among the feature points extracted by the feature point extraction unit 102 in Figure 1. An object part extraction unit 203 extracts object parts from the image. An intermediate image feature extraction unit 204 extracts intermediate image features from the image and the object parts. A reliability conversion unit 205 applies conversion processing to the reliability of the feature points extracted by the feature point extraction unit 102. A feature integration unit 206 integrates the output of the intermediate image feature extraction unit 204 and the output of the reliability conversion unit 205. An image feature output unit 207 generates image features from the output of the feature integration unit 206.

[0015] The operation of this image processing apparatus will be described with reference to the flowchart in Fig. 4. The processing shown in the flowchart in Fig. 4 is executed by the CPU 303 of Fig. 3, which is a computer, in accordance with a computer program stored in the ROM 304.

[0016] Step 401 acquires an image frame from the camera. This step corresponds to the operation of the image acquisition unit 101 in FIG.

[0017] Step 402 detects multiple feature points associated with the parts of an object from the image of the object having multiple parts acquired in step 401 (first detection method). This step corresponds to the operation of the first detection unit 102 in Figure 1. Step 402 also receives an image frame as input and extracts multiple feature points of a person present in the image and their reliability. For each detected feature point, a reliability indicating the likelihood that the feature point appears in the image is obtained. If the image processing target is a person, the joint positions of the human body can be used as feature points. The five feature points detected in this step are the top of the head, neck, waist, right ankle, and left ankle. Convolutional Pose Machines are used to detect feature points (Shih-En Wei et al., "Convolutional Pose Machines," IEEE, 2016). This method uses a trained model (neural network) to calculate a reliability map indicating the location of each joint position in the image. The confidence map is a two-dimensional map, and if the number of articulation points is P, then there are P+1 confidence maps (one map corresponds to the background). In the confidence map for a certain articulation point, the position with the highest confidence is considered to be the position where that articulation point exists. The confidence is a real number between 0 and 1 that indicates the likelihood that the feature point exists. The closer the confidence is to 1, the higher the probability that the articulation point exists. Articulation points that are occluded by other objects are extracted from objects that are not human, and therefore are less likely to be human joints. Therefore, the confidence of the joint position is lower than that of joints that are not occluded by other objects. On the other hand, joints that are not hidden by other objects are well extracted from the human, and therefore have a higher confidence level.

[0018] Note that methods other than Convolutional Pose Machines may be used to detect feature points of an object and their reliability. For example, a rule-based method may be used to identify each joint point of a human body using image features extracted from the joint points. Alternatively, image features of the head may be extracted from an image, and the position of the torso may be estimated from the extracted position of the head. In addition, although the present embodiment uses joint points of a human body as feature points, if the target of image processing is a face, facial feature points may be used. As facial feature points, the centers and endpoints of features such as the eyes, eyebrows, nose, mouth, and ears, points on the contour, points on the contour of the entire face, etc. may be used.

[0019] In step 403, a feature point group to be used in the second detection method is determined. Step 403 corresponds to the operation of the feature point determination unit 103 in Figure 1. The feature point group determined in step 403 is used in the second detection method. A plurality of combination patterns of feature points are prepared, and one is selected and determined from these depending on the condition of the reliability of the feature points. This is used in the second detection method in step 404 later. The feature point group includes feature points (here, head, neck, or waist) used to determine the position after correction. In this embodiment, the feature points to be corrected as predetermined feature points are the waist, right ankle, and left ankle. Correction of the right ankle and left ankle is performed using the same procedure, so only correction of the right ankle will be explained. Hereinafter, the ankle on one side to be processed will be simply referred to as the "ankle."

[0020] The operation of step 403 will be explained using the flowchart in Figure 5. As will be described later, six types of feature point groups A1, A2, A3, B1, B2, and B3 are prepared in advance as candidate feature point groups to be used for correction. One of the feature point groups A1, A2, and A3 related to correction of the waist and one of the feature point groups B1, B2, and B3 related to detection of the ankles using the second detection method are determined according to conditions.

[0021] Details will be described later, but feature point group A1 is an empty set, and the detection results of the first detection unit are used as is. Feature point group A2 is used to detect the position of the waist from the positions of the head and neck in the current frame. Feature point group A3 is used to detect the current position of the waist from the positions of the head and waist in previous frames. Feature point group B1 is an empty set, and the detection results of the first detection unit are used as is. Feature point group B2 is used to detect the position of the ankles from the positions of the neck and waist in the current frame. Feature point group B3 is used to detect the position of the ankles in the current frame from the positions of the neck and ankles in previous frames.

[0022] 5, step 501 evaluates whether the reliability of the waist in the current frame determined in step 402 is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold, the process proceeds to step 503; if not, the process proceeds to step 502.

[0023] In step 502, it is evaluated whether the reliability of the waist in a past frame stored in feature point storage unit 105 is equal to or greater than a threshold value. If it is equal to or greater than the threshold value, the process proceeds to step 505; if it is not, the process proceeds to step 504. A past frame is the image frame acquired in step 401 of the previous repeat loop in the repeat loop of the flowchart in Figure 4. However, if feature points in a past frame are not stored in feature point storage unit 105, that is, if step 403 in Figure 4 is being executed for the first time, the process proceeds to step 504.

[0024] In step 503, feature point group A1 is determined as the feature point group to be used in the second detection method, and the process proceeds to step 506. When feature point group A1 is determined, it means that the waist feature points in the current frame are reliable, and there is no need to re-detect the waist feature points in the subsequent processing.

[0025] In step 504, feature point group A2 is determined as the feature point group to be used in the second detection method, and the process proceeds to step 506. If feature point group A2 is determined, the waist joint points in both the current frame and previous frames are unreliable, and the position of the waist in the current frame is detected from the positions of the head and neck in the current frame in subsequent processing.

[0026] In step 505, feature point group A3 is selected as the feature point group to be used for correction, and the process proceeds to step 506. When feature point group A3 is selected, the waist feature points in the current frame are unreliable, but the waist feature points in past frames are reliable, and the current waist position is corrected in the subsequent processing from the head and waist positions in past frames.

[0027] Step 506 evaluates whether the reliability of the ankle in the current frame determined in step 402 is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold, the process proceeds to step 508; if not, the process proceeds to step 507.

[0028] In step 507, it is evaluated whether the reliability of the ankle in the past frame stored in feature point storage unit 105 is equal to or greater than a predetermined threshold. If it is equal to or greater than the threshold, the process proceeds to step 510; if it is not, the process proceeds to step 509. However, if feature points in the past frame are not stored in feature point storage unit 105, that is, if step 403 in FIG. 4 is being executed for the first time, the process proceeds to step 509.

[0029] In this embodiment, the threshold values ​​used in S501, S502, S506, and S507 are different from each other, but they may be the same value.

[0030] In step 508, feature point group B1 is selected as the feature point group to be used for correction, and the processing of the flowchart in Fig. 5 is terminated. When feature point group B1 is selected, it means that the foot feature points in the current frame are reliable, and there is no need to detect the foot positions in later processing.

[0031] In step 509, feature point group B2 is selected as the feature point group to be used for correction, and the processing of the flowchart in Fig. 5 is terminated. When feature point group B2 is selected, the foot positions in both the current frame and previous frames are unreliable, and the foot positions in the current frame are detected in the subsequent processing from the foot and hip positions in the current frame.

[0032] In step 510, feature point group B3 is selected as the feature point group to be used for correction, and the processing of the flowchart in Figure 5 is terminated. When feature point group B3 is selected, this means that the foot feature points in the current frame are unreliable but the foot feature points in the previous frame are reliable, and the positions of the current frame are detected from the positions of the neck and feet in the previous frame in the subsequent processing.

[0033] Although the above description of steps 506, 507, 508, 509, and 510 has been made with respect to only one ankle (the right ankle), the feature point group to be used in the second detection method is determined in a similar manner for the other ankle (the left ankle). Note that, to detect the position of the ankle, it is preferable to be able to estimate the position of the ankle from feature points as close to the ankle position as possible. Therefore, if the position of the waist can be used (high reliability of the waist position), the ankle position is detected using the position of the waist. If the position of the waist is unknown (low reliability of the waist position), the ankle position is detected using the position of the neck, which is the next closest to the ankle after the waist. The following processing order is based on the above intention, but the order may be changed. Furthermore, the feature point group may be determined so as to detect only the ankle position without detecting the waist position.

[0034] In step 404 of Fig. 4, predetermined feature points are detected by a second detection method using the feature point group determined in step 403. The processing of step 404 corresponds to second detection unit 104 in Fig. 1. The operation of step 404 will be explained using the flowchart in Fig. 6. In the processing of Fig. 6, predetermined feature points (ankle positions) are detected based on the feature point groups A1, A2, A3, B1, B2, and B3 determined in the processing of the flowchart in Fig. 5.

[0035] As with step 403 in Fig. 4, the correction of the right ankle and the left ankle is performed using the same procedure, so only the detection of the right ankle will be described. Hereinafter, the ankle on one side to be processed will be simply referred to as "ankle."

[0036] 6, it is determined which of the feature point groups A1, A2, and A3 relating to the waist has been selected. If feature point group A1 has been selected, the process proceeds to step 602; if feature point group A2 has been selected, the process proceeds to step 603; and if feature point group A3 has been selected, the process proceeds to step 604. In steps 602, 603, and 604, the positions of the waist feature points are detected using the second detection method.

[0037] Step 602 does not detect the location of the waist feature points because the reliability of the waist feature points in previous processing is greater than a certain threshold and is therefore considered reliable.

[0038] In step 603, the position of the waist is detected from the positions of the head and neck detected in the current image frame. This process will be explained using FIG. 7. As shown in FIG. 7(a), feature points—top of the head 701, neck 702, waist 703, right ankle 704, and left ankle 705—are detected in step 402 of FIG. 4. First, as shown in FIG. 7(b), a line 706 connecting the head and neck is calculated. The distance between the head and neck is calculated from their respective position coordinates. Here, it can be assumed that the ratio between the head-neck distance and the head-waist distance is roughly the same for each individual, although this varies from person to person. Therefore, the waist position is detected so that it is on the line connecting the head and neck, and the ratio between the head-neck distance and the head-waist distance is a predetermined value. FIG. 7(c) shows an example of a corrected waist feature point 707. This predetermined ratio can be determined, for example, from the ratio of the body parts of an average adult.

[0039] In step 604, the current waist position is detected from the positions of the head and waist in previous frames. First, the distance between the head and waist is calculated from the feature points of previous frames stored in the feature point storage unit 105. Next, in the current frame, a straight line connecting the head and neck is calculated, as in FIG. 7(b). Here, it is assumed that the distance between the head and waist in the previous frame is approximately the same as the distance between the head and waist in the current frame. Then, the waist position in the current frame is detected so that the waist position is on the straight line connecting the head and neck, and the distance between the head and waist in the current frame is equal to the distance between the head and waist in the previous frame.

[0040] In step 605 of Fig. 6, it is determined which of the ankle-related feature point groups B1, B2, and B3 has been selected. If feature point group B1 has been selected, the process proceeds to step 606; if feature point group B2 has been selected, the process proceeds to step 607; and if feature point group B3 has been selected, the process proceeds to step 608. In steps 607 and 608, the positions of the ankle feature points are detected. In step 606, the positions of the ankle feature points are not detected.

[0041] In step 607, the position of the ankles is detected from the positions of the neck and waist in the current frame. This process will be described with reference to FIG. 8. As shown in FIG. 8(a), the feature points of the top of the head 801, neck 802, waist 803, right ankle 804, and left ankle 805 are detected in step 402 of FIG. 4. First, as shown in FIG. 8(b), a line 806 (body axis) connecting the neck and waist is calculated. The distance between the neck and waist is also calculated from the coordinates of each position. It can be assumed that the ratio between the distance between the neck and waist and the distance between the neck and right ankle is roughly the same for all people, although this varies from person to person. Therefore, the ankle position is detected so that it is on the line connecting the neck and waist, and the ratio between the distance between the neck and waist and the distance between the neck and ankle is a predetermined value. An example of the ankle 807 after feature point detection is shown in FIG. 8(c) on the left.

[0042] In step 608, the position of the ankle in the current frame is detected from the positions of the neck and ankle in the previous frame. First, the distance between the neck and waist is calculated from the feature points of the previous frame stored in the feature point storage unit 105. Next, in the current frame, a line connecting the neck and waist (body axis) is calculated as in FIG. 8(b). Here, it is assumed that the distance between the neck and ankle in the previous frame is approximately the same as the distance between the neck and ankle in the current frame. Then, the position of the ankle in the current frame is detected so that the position of the ankle is on the body axis and the distance between the neck and ankle in the current frame is equal to the distance between the neck and ankle in the previous frame.

[0043] Although the above explanation of steps 605, 606, 607, and 608 has focused on the right ankle, the left ankle is also detected in the same manner. This process makes it possible to detect a more accurate position of the ankle even if the first detection unit is unable to properly detect the ankle due to occlusion or noise.

[0044] In step 405 of FIG. 4, a region where the object exists is determined based on the detected feature points. This partial image region indicates the region where a person exists in the captured image and is used to specify the region from which a person image is extracted from the image frame in later processing. The operation of step 405 corresponds to the region determination unit 106 in FIG. 1. The processing of step 405 will be explained using FIG. 9(a). As shown in FIG. 9(a), feature points for the top of the head, neck, waist, right ankle, and left ankle exist in image frame 903. First, the midpoints of the right and left ankles are calculated. Then, a line 901 (body axis) connecting the head and the midpoint is calculated. In this embodiment, the partial image region is a rectangle with a predetermined aspect ratio. A rectangle 902 is determined so that the vertical direction of the rectangle is parallel to the body axis, the central axis of the rectangle is equal to the body axis, the top side of the rectangle contacts the head, and the bottom side of the rectangle contacts the ankles. At this time, margins may be provided between the top side of the rectangle and the head, and between the bottom side of the rectangle and the ankles. For example, a margin of a size calculated by multiplying the distance between the head and ankles (height) by a certain coefficient may be provided. That is, the partial image area is determined based on the circumscribed rectangle of the feature points. In this embodiment, the aspect ratio of the rectangle is fixed to facilitate subsequent input to the neural network, but it may not be fixed depending on the configuration of subsequent processing. Note that when corrected joint positions are used, the area determined here may contain occlusions of the person's body parts or may contain a lot of noise. For example, as shown in Figure 18, even if the ankle is hidden by an occluding object 1803, the area is determined as including the person's body parts. By determining the area in this manner, a partial image area can be determined in which the arrangement of the human body parts within the rectangle is consistent. By ensuring consistent arrangement of the body parts, the feature amount of each part can be extracted in a subsequent feature amount extraction process that more accurately reflects the characteristics of each part.

[0045] In step 406 of Fig. 4, the partial image area determined in step 405 is cut out from the image frame as a person image. If the rectangle of the partial image area determined in step 405 is tilted, the image is rotated so that the rectangle is upright. An example of cutting out from Fig. 9(a) is shown in Fig. 9(b). The operation of step 406 corresponds to the image extraction unit 107 in Fig. 1.

[0046] The corrected portion in the current frame is stored in step 407. The operation in step 407 corresponds to the feature point storage unit 105 in FIG.

[0047] In step 408, feature amounts are extracted from the partial image region (person image). The operation in step 408 corresponds to the image feature extraction unit 108 in Fig. 1 and Fig. 2. The operation in step 408 will be described with reference to the flowchart in Fig. 10.

[0048] In step 1001 of FIG. 10, the out-of-region feature point correction unit 202 corrects the reliability of feature points outside the partial image region based on the coordinates of the partial image region and feature points. Step 1001 corresponds to the out-of-region feature point correction unit 202 in FIG. 2. When the aspect ratio of the rectangle of the partial image region is fixed, feature points may not be included in the partial image region, for example, when the limbs are spread apart. Body parts outside the partial image region are outside the range of feature extraction, which can reduce the accuracy of feature extraction in these areas. Therefore, to mitigate this effect, adjustments are made to reduce the reliability of feature points outside the partial region in a later step. For example, in FIG. 11, the right ankle 1104 is outside the range of the rectangle 1106, and the reliability of this feature point of the right ankle is reduced. In this embodiment, the corrected reliability is calculated by multiplying the original reliability by a predetermined real value less than 1. In this way, by reducing the reliability of feature points outside the partial region, the problem of reduced accuracy of feature extraction due to the placement of human body parts outside the partial region and the problem of reduced accuracy of feature extraction due to occlusion can be addressed by common processing that will be described later.

[0049] In step 1002, features are extracted from the partial image region and the reliability of the feature points. A neural network can be used to extract the features. FIG. 12 shows an example of the configuration of a neural network. The neural network in FIG. 12 receives an image 1201 and feature point reliability 1206 as input, and outputs image features 1210. The neural network is composed of an image conversion subnetwork 1202, a reliability conversion subnetwork 1207, an integration subnetwork 1208, and a feature output subnetwork 1209. The image conversion subnetwork 1202 corresponds to the intermediate image feature extraction unit 204 in FIG. 2. The reliability conversion subnetwork 1207 corresponds to the reliability conversion unit 205 in FIG. 2. The integration subnetwork 1208 corresponds to the feature integration unit 206 in FIG. 2. The feature output subnetwork 1209 corresponds to the image feature output unit 207 in FIG. 2.

[0050] The input data, intermediate data, and output data handled by neural networks are treated as tensors. Tensors are data represented as multidimensional arrays, and the number of dimensions is called the rank. A tensor with rank 0 is called a scalar, a tensor with rank 1 is called a vector, and a tensor with rank 2 is called a matrix. For example, an image with one channel (such as a grayscale image) can be treated as a rank 2 tensor with size H×W, or a rank 3 tensor with size H×W×1. Additionally, an image with RGB components can be treated as a rank 3 tensor with size H×W×3.

[0051] The data obtained by cutting a tensor at a certain position in a certain dimension and the operations performed on it are called slices. For example, by slicing a rank-3 tensor of size H×W×C at the cth position in the third dimension, we obtain a rank-2 tensor of size H×W or a rank-3 tensor of size H×W×1.

[0052] A layer that performs a convolution operation on a tensor is called a convolution layer (abbreviated as Conv). The filter coefficients used in the convolution operation are called "weights." For example, a convolution layer generates an output tensor of H×W×D from an input tensor of H×W×C.

[0053] A layer that multiplies a vector by a weight matrix and adds a bias vector is called a fully connected layer (abbreviated as FC). For example, a vector of length D is generated by applying a fully connected layer to a vector of length C.

[0054] The operation of dividing a tensor into intervals and taking the maximum value of each interval to reduce the size of the tensor is called max pooling. When the average value of the interval is taken instead of the maximum value, it is called average pooling. In this embodiment, max pooling is used, and the layer of the neural network that performs this is simply called the pooling layer (abbreviated as "Pooling"). In this embodiment, the pooling layer outputs a tensor whose first and second dimensions are half the size of the input. Specifically, an output tensor of H / 2×W / 2×C is generated from an input tensor of H×W×C.

[0055] In neural networks, a nonlinear function typically applied after a convolution layer is called an activation function. Activation functions include the rectified linear function (abbreviated as ReLU) and the sigmoid function. In particular, the sigmoid function has the property that its output value ranges from 0 to 1. In this embodiment, ReLU is used as the activation function unless otherwise specified.

[0056] In neural networks, the operation of arranging and connecting tensors in a certain dimension is called "concatenation."

[0057] Let us explain Global average pooling. In a rank-3 tensor of size H×W×C, for each slice at every position in the third dimension, we take the average value of all elements contained in the slice. Then, by arranging these C average values, we generate a vector of length C. This operation is called Global average pooling.

[0058] 12, the size of an image 1201 that is input to the neural network is width W1, height H1, and the number of channels is 3. In other words, the image can be regarded as a tensor of H1×W1×3.

[0059] The image transformation subnetwork 1202 transforms the image 1201 into a feature map. The image transformation subnetwork 1202 is further composed of a preprocessing subnetwork 1203, a part estimation subnetwork 1204, and an image synthesis subnetwork 1205.

[0060] The image conversion subnetwork 1202 extracts features for identifying an object for each part corresponding to the detected feature points. Specifically, as in the paper by L. Zhao et al., it includes a module that estimates parts and extracts part features. The image conversion subnetwork 1202 corresponds to the object part extraction unit 203 in Figure 2. (L. Zhao et al., "Deeply-Learned Part-Aligned Representations for Person Re-Identification," IEEE, 2017.) In this embodiment, the object part extraction unit 203 operates within the neural network that performs feature extraction. However, the object part extraction unit 203 may operate outside this neural network and receive information on the position and size of parts from an external source. Furthermore, the object part extraction unit 203 and the first detection unit 102 in Figure 1 may share the same purpose. Information derived from the output of the first detection unit 102 may be used as the output of the object part extraction unit 203, or vice versa. The features extracted for each part are integrated as overall features in subsequent processing. In this process, the feature values ​​for each part are weighted according to the reliability of each feature point so that they are reflected in the overall feature value. In other words, feature values ​​extracted from parts corresponding to feature points with low reliability are prevented from contributing to the final recognition result. This is because feature values ​​with low reliability may be occluded by the object or may be noisy, and the feature values ​​extracted from such parts do not necessarily represent the characteristics of that part of the object. By performing this type of processing, feature values ​​that better reflect the characteristics of the object can be generated, which is expected to improve object recognition accuracy.

[0061] The image transformation subnetwork 1202 can be configured as a sequence of one or more convolution layers (Conv) and max pooling layers (Pooling). In this embodiment, it is configured as a sequence such as "Conv, Conv, Pooling, Conv, Pooling, Conv, Pooling, Conv." The configuration is outlined in Figure 16(a). Applying the image transformation subnetwork to an image results in a tensor of H2 × W2 × C2.

[0062] The part estimation subnetwork 1204 receives the output of the image conversion subnetwork 1202 as input and outputs a tensor of size H2 × W2 × P1, which is a feature map. Here, P1 is the number of parts to be estimated and may be any predetermined number. A slice of this tensor at position p in the third dimension (a tensor of size H2 × W2 × 1) is a mask image indicating the location of the pth part. Each pixel takes a value between 0 and 1, and the closer the value is to 1, the higher the likelihood that the part is present at that position. The part estimation subnetwork 1204 is composed of one convolution layer and a sigmoid function. An outline of the configuration is shown in Figure 16(b). The configuration of the part estimation network is not limited to this; it may be composed of multiple convolution layers.

[0063] The image integration subnetwork 1205 integrates the outputs of the image conversion subnetwork 1202 and the part estimation subnetwork 1204. Figure 17 shows the processing flow. First, slice 1702 (a tensor of size H2 × W2 × 1) at position p in the third dimension of output tensor 1701 of the part estimation subnetwork is copied C2 times and concatenated in the third dimension to expand it into tensor 1703 of size H2 × W2 × C2. Each element of this tensor is then multiplied by each element of output tensor 1704 of the image conversion subnetwork 1202 to generate new tensor 1705 (of size H2 × W2 × C2). Global average pooling is then applied to this tensor to generate vector 1706 of length C2, and a fully connected layer is further applied to generate vector 1707 of length C3. This process is applied to channel p of all parts, and vector 1708 is generated by concatenating each of the generated vectors. That is, the length of vector 1708 generated by the image integration sub-network is (C3)P1. In this embodiment, the data to be integrated is a vector, but a vector is a type of tensor, and even if the data to be integrated is a tensor of second or higher order, it may be integrated by combining in the same way.

[0064] The feature point reliability 1206 is a vector of length C4. In this embodiment, the number of feature points detected in step 402 in FIG.

[0065] The confidence conversion sub-network 1207 converts the feature point confidence 1206 into a vector of length C5. The confidence conversion sub-network 1207 can be configured with zero or more fully connected layers. In this embodiment, it has one fully connected layer. An outline of the configuration is shown in Figure 16(c).

[0066] The integrating subnetwork 1208 integrates the output vector of the image integration subnetwork 1205 and the output vector of the confidence conversion subnetwork 1207. The integrating subnetwork 1208 outputs a vector of length C6. In this embodiment, these two vectors are concatenated. The configuration is outlined in Figure 16(d). Therefore, C6 = (C3)P1 + C5.

[0067] The feature output subnetwork 1209 receives the output vector of the integrating subnetwork 1208 as input and outputs image features 1210, which are vectors of length C7. The feature output subnetwork 1209 can be configured with one or more fully connected layers. In this embodiment, it is configured with two fully connected layers. An overview of the configuration is shown in Figure 16(e). These image features are also called "matching features," "person features," "descriptors," or "embeddings."

[0068] In step 409 of FIG. 4, the feature quantities of the person image extracted in step 408 are compared with the feature quantities stored in a person database. The person database is a storage means in which cropped images and feature quantities of N people to be identified are pre-registered. Images of the people to be identified are captured in advance, and the images are cropped and feature quantities are extracted and stored using the same method as in steps 402 to 408. The person database corresponds to the object storage means 112 in FIG. 1. In step 409, the distance between the feature quantities of the people in the person database and the feature quantities of the person image extracted in step 408 is calculated. Then, the people in the person database are sorted in order of distance, with the person with the smallest distance placed at the top of the list. Step 409 corresponds to the processing of the recognition unit 109 in FIG. 1. In this embodiment, Euclidean distance is used to compare the feature quantities. Other methods for comparing the feature quantities may be used, such as other distance indices such as L1 distance or cosine distance, or machine learning such as metrics learning or neural networks may be used for comparison.

[0069] In step 410 of FIG. 4, the person matched in step 409 is displayed on the screen. Step 410 corresponds to the processing of the image display unit 110 in FIG. 1. An example of the display screen is shown in FIG. 14. The display screen 1401 is composed of a query 1402 and a gallery 1403. The query 1402 is an image of the person to be searched for, and displays the person image extracted in step 406. The gallery 1403 is a list of search results, and displays the top five people in the images in the person database sorted by distance in step 409. In this case, the top five people may be displayed, or only people whose distance is less than a predetermined threshold may be displayed from among the five. The images displayed in the gallery may be extracted using the same method as in steps 401 to 407 of FIG. 4, or may be extracted using another method. Markers indicating the positions of detected feature points may be superimposed on the images of the person in the query and gallery, as shown in FIG. 14.

[0070] 4, step 411 determines whether or not to end the processing of the flowchart. In this embodiment, if the number of times step 411 is executed reaches or exceeds a predetermined number, it is determined that the processing is to end. If not, the processing proceeds to step 401, and the processing of the flowchart continues.

[0071] <Neural network training> A method of learning the neural network used in the image feature extraction unit 108 in Fig. 1 will be described with reference to the flowchart in Fig. 13. The processing in the flowchart in Fig. 13 corresponds to the learning means 111 in Fig. 1.

[0072] As mentioned above, the structure of the neural network is shown in Fig. 12. The neural network receives an image 1201 and feature point reliability 1206 as input, and outputs image features 1210.

[0073] The neural network is trained using triplet loss (F. Shroff et al. “Face Net: A Unified Embedding for Face Recognition and Clustering,” arXiv:1503.03832). Triplet loss uses triplets consisting of a sample called an anchor sample, a sample from the same person as the anchor called a positive sample, and a sample from a different person than the anchor called a negative sample. The network is updated by comparing the features obtained from the anchor sample, positive sample, and negative sample to calculate a loss function.

[0074] 13, the weights of the convolution layer and the fully connected layer that make up the network are initialized. In this embodiment, random numbers are used as the initial values ​​of the weights.

[0075] In step 1302, training data is randomly acquired from the training data group. One training data is a triplet, containing one anchor sample, one positive sample, and one negative sample. The anchor sample, positive sample, and negative sample each consist of an image and feature point reliability. The images and feature point reliability are generated using the same procedure as those input to the neural network used in the flowchart of Figure 4.

[0076] Step 1303 updates the network with the training data. First, the current state of the network is applied to the anchor sample, positive sample, and negative sample to calculate the feature values ​​for each. Losses are calculated for these three feature values ​​using triplet loss. Then, the weights in the network are updated using the backpropagation method.

[0077] In step 1304, it is determined whether learning should be terminated. If step 1304 has been executed a specified number of times, it is determined that learning should be terminated, and the series of processes in the flowchart of Fig. 13 is terminated. If it is determined that learning should not be terminated, the process proceeds to step 1302.

[0078] According to this embodiment, it is possible to detect unsatisfactory feature points from among the good feature points again in the feature group determination unit 103 and the second detection unit 104. Therefore, even in a situation where part of the object is occluded by another object or is subject to external disturbance, it is expected that there will be an effect of reducing errors in the object region determination by the region determination unit 106.

[0079] In areas where an object is partially occluded by another object or is subject to external disturbance, it can be assumed that the reliability of the feature points acquired by the first detection unit 102 will be lower than normal and output. In this case, it is believed that the quality of the image features for image recognition extracted from these local areas will also decrease at the same time. Therefore, by using information on the reliability of the feature points as an index representing the reliability of a certain local area in the image feature extraction unit 108, it is expected that the effect of reducing the decrease in the quality of the image features can be expected. Therefore, it is expected that the effect of improving the accuracy of image recognition can be expected.

[0080] Step 1001 in Figure 10 reduces the reliability of feature points outside the partial image region. Body parts outside the partial image region are outside the range of feature extraction, which can lead to a problem of reduced accuracy in feature extraction in these areas. Therefore, in order to mitigate this effect in later steps, reducing the reliability of feature points outside the partial region is expected to have the effect of reducing the degradation of image feature quality.

[0081] In steps 403 and 404, feature points from not only the current frame but also past frames are used to select a group of feature points to be used for correction and to correct the feature points. By using feature points from past frames, it is expected that the accuracy of feature point correction can be improved even when the reliability of feature points in the current frame is low.

[0082] In step 403, feature points are selected in a predetermined order. By preferentially selecting feature points that are expected to have higher accuracy in correcting the feature point positions in step 404, it is expected that the feature point positions can be corrected more accurately.

[0083] In step 404, feature points are corrected in a predetermined order. Here, feature points are corrected in the order of the waist, then the legs. This is because a person's body parts are connected in the order of neck, waist, and legs. First, the position of the waist is corrected, and then the legs can be corrected using this more accurate waist position. In this way, comparing feature points in a predetermined order is expected to have the effect of enabling more accurate correction of feature point positions.

[0084] In step 404, the positions of the feature points are corrected based on the relative positional relationship between the feature points. In this embodiment, the feature points are corrected based on the ratio of the distances between the feature points and the straight line (body axis) obtained from the feature points. In this way, by using prior knowledge about the structure of the object, it is expected that the positions of the feature points can be corrected more accurately.

[0085] <Modification of the First Embodiment> The feature points extracted in step 402 are not limited to the top of the head, neck, waist, right ankle, and left ankle, but may also be other parts such as wrists, elbows, knees, etc. Furthermore, they do not necessarily have to be on body parts, but may be other points determined based on the positional relationship of body parts, such as the midpoint between the right ankle and the left ankle, or the intersection of the line connecting the body axis and the left ankle and the right ankle.

[0086] In step 604, the position of the waist in the current frame is corrected based on the distance between the head and waist in the previous frame, but other methods may be used. The position of the waist in the current frame may also be corrected based on the difference in position coordinates of the head and waist in the previous frame. For example, the difference in position coordinates of the head and waist in the previous frame may be such that the x and y coordinates of the waist are larger by X pixels and Y pixels than the x and y coordinates of the head. The position of the waist in the current frame may be corrected to be equal to the difference in position coordinates of the head and waist in the previous frame. Furthermore, the difference in position coordinates of the neck and waist may be used instead of the difference in position coordinates of the head and waist.

[0087] In step 607, the ratio of the distance between the neck and waist of the human body to the distance between the neck and the right ankle (or the left ankle) is used, but this is not limiting and other ratios between feature points may be used. As an example, the head may be used instead of the neck, such as the ratio of the distance between the head and waist to the distance between the head and the right ankle (or the left ankle). As another example, the ratio of the distance between the head and neck to the distance between the waist and the right ankle (or the left ankle) may be used. The same applies to step 608.

[0088] In step 607, the right ankle and the left ankle are corrected so that they are on the body axis. However, this is not limiting, and correction may be made by moving the right ankle (or the left ankle) in the direction of the body axis so that the ratio between the feature points becomes a predetermined value. The same applies to step 608.

[0089] In the region determination unit 106, the partial image region is set to a rectangle, but other shapes are also acceptable. For example, it may be a polygon, or may be surrounded by curves. Instead of a figure, a mask image that distinguishes the object region from other regions may also be used.

[0090] The structure of the neural network of the first embodiment is not limited to this. For example, another sub-network may be inserted between sub-networks. The branching structure of the network may also be different. The configuration of the sub-networks may also be different in terms of the types and number of components, such as convolution layers, pooling layers, and fully connected layers.

[0091] Although the integration subnetwork 1208 in Figure 12 integrates two vectors by combining them, other computation methods may be used. For example, if the two vectors are the same size, multiplication or addition of the vector elements may be used instead.

[0092] 2 is implemented as a reliability conversion sub-network 1207 as shown in Fig. 12, the reliability conversion unit 205 may be provided outside the neural network. For example, normalization processing, conversion processing, etc. may be performed on the reliability of feature points outside the neural network, and the processing result may be used as one of the inputs to the neural network.

[0093] In step 403 and step Feature Point Correction in Fig. 4, a group of feature points to be used for correction is selected from the current frame and the immediately preceding frame, and the feature points are corrected. It is also possible to select a group of feature points and correct the feature points using not only the immediately preceding frame, but also frames prior to that. Furthermore, it is also possible to use three or more frames, including the current frame.

[0094] Although the image feature extraction unit 108 is configured using a neural network, methods other than neural networks may be used. For example, HOG (Histogram of Oriented Gradients) features or LBP (Local Binary Pattern) features may be extracted and image features may be determined based on these. Alternatively, parts may be estimated from HOG features or LBP features.

[0095] In step 603 of Figure 6, line 706 in Figure 7 was calculated from the head and neck, but it is also possible to calculate the line from just the head or neck. For example, if it is assumed that the body axis of a person is parallel to the y-axis of the image frame, it is also possible to assume that the line is parallel to the y-axis of the image frame, and the line can be calculated from a single point on either the neck or the head. Similarly, in step 405 of Figure 4, line 901 in Figure 9 was calculated from multiple points, but it is also possible to calculate it from a single point.

[0096] 10, the corrected reliability is calculated by multiplying the original reliability by a predetermined real number value smaller than 1, but other methods may be used. The method of updating the reliability is not limited to this, and the reliability may be set to 0, a predetermined real number value may be subtracted from the reliability, or other methods may be used.

[0097] As described above, by the process described in the first embodiment, even when a part of an object is occluded in an image or when there is a lot of noise, it is possible to detect feature points of a part that is difficult to see.

[0098] <Embodiment 2> In the first embodiment, the entire body of a person is the subject of image processing, but the face may be the subject of image processing instead. In the second embodiment, only the differences from the first embodiment will be explained.

[0099] When the target is a face, facial feature points are detected in step 402 in Fig. 4. This is illustrated in Fig. 15. Here, the feature points detected are a right eye 1501, a left eye 1502, a nose 1503, a right edge of the mouth 1504, and a left edge of the mouth 1505.

[0100] In the second embodiment, a case will be considered in which the feature points of the right eye are corrected from the nose and mouth in steps 403 and 404. The left eye is processed in the same way as the right eye.

[0101] The processing of step 403 will be described. First, the reliability of the feature points for the right eye is evaluated. If the reliability is equal to or greater than a threshold, feature point group C1 is selected. If the reliability is less than the threshold, feature point group C2 is selected if the reliability of the right eye in the past frame was not equal to or greater than the threshold, and if it was equal to or greater than the threshold, feature point group C3 is selected.

[0102] The processing of step 404 will now be described. If the feature point group used for correction is feature point group C1, the position of the right eye is not corrected. If it is feature point group C2, the position of the right eye in the current frame is corrected so that it approximates the arrangement of facial features of an average person, based on the positional relationship of the right edge of the nose and the right edge of the mouth and the left edge of the mouth in the current frame. If it is feature point group C3, the position of the right eye in the current frame is corrected so that it approximates the arrangement of the right eye, nose, right edge of the mouth, and left edge of the mouth in the previous frame.

[0103] The processing of other steps is also the same as that of the first embodiment, if the feature points extracted from the whole body are replaced with the feature points of the face.

[0104] In the second embodiment, the facial feature points are the right eye, left eye, nose, right edge of the mouth, and left edge of the mouth, but other parts may be used as feature points, such as the corners of the eyes, inner corners of the eyes, pupils, right edge of the nose, bottom edge of the nose, eyebrows, facial contours, etc. The processing in steps 403 and 404 may be changed accordingly.

[0105] According to the second embodiment, it is expected that the performance of cutting out a face image from an image frame and of face recognition can be improved. For example, this is effective in cases where a part of the face is covered by an accessory such as sunglasses or a mask, or where a part of the face is temporarily hidden by a hand or the like.

[0106] The present invention can also be realized by executing the following process. That is, software (programs) that realize the functions of the above-described embodiments are supplied to a system or device via a data communication network or various storage media. Then, a computer (or CPU, MPU, etc.) of the system or device reads and executes the programs. The programs may also be provided by recording them on a computer-readable storage medium. [Explanation of symbols]

[0107] 101 Image acquisition unit 102 first detection unit 103 Feature group determination unit 104 Second detection unit 105 Feature point memory unit 106 Area determination part 107 Image Extraction Unit 108 Image feature extraction unit 109 Recognition part 110 Display section 111 Learning Department 112 Object memory section

Claims

1. a first detection means for acquiring, from an image of a person having a plurality of body parts, a plurality of feature points indicating the positions of the plurality of body parts and a reliability indicating the accuracy of the positions indicated by each of the feature points; a second detection means for detecting, when the reliability of a feature point indicating the position of the ankle among the plurality of body parts acquired by the first detection means is lower than a predetermined value, a feature point that is on a straight line connecting the feature points of the neck and waist among the plurality of body parts acquired by the first detection means and for which the ratio of the distance between the neck and the waist to the distance between the neck and the ankle is a predetermined value, as the feature point of the ankle; a determining means for determining, as an area in which the person is present, an area including the head feature point acquired by the first detecting means and the ankle feature point detected by the second detecting means.

2. the first detection means acquires a reliability of a feature point of the person's ankle in a previous image captured before the image; 2. The image processing device according to claim 1, wherein, when the reliability of the feature point of the person's ankle in the past image is higher than a predetermined value, the second detection means detects the position of the ankle in the image from the positions of the person's neck and ankle in the past image, and sets the position as the feature point of the ankle.

3. 2. The image processing device according to claim 1, wherein, when the reliability of the waist feature point acquired by the first detection means is lower than a predetermined value, the second detection means detects, as the waist feature point, a feature point that is on a straight line connecting the head and neck feature points among the multiple parts acquired by the first detection means and for which the ratio of the distance between the head and the neck to the distance between the neck and the waist is a predetermined value.

4. the first detection means acquires a reliability of a feature point of the waist of the person in a previous image captured before the image; 4. The image processing device according to claim 3, wherein, when the reliability of the feature point of the person's waist in the past image is higher than a predetermined value, the second detection means detects the position of the waist in the past image from the positions of the person's neck and waist in the past image, and sets the position as the feature point of the waist.

5. The image processing device according to any one of claims 1 to 4, characterized in that the first detection means detects the multiple feature points by inputting the image into a trained model that has been trained to output multiple feature points corresponding to the multiple parts of a person in response to an input image.

6. an extracting means for extracting a feature amount for recognizing the person based on a partial image obtained by cutting out the area determined by the determining means from the image; 5. The image processing device according to claim 1, further comprising: a recognition unit that recognizes the person captured in the image by comparing the extracted feature amount with the feature amount of a person registered in advance.

7. A program for causing a computer to function as each of the means included in the image processing apparatus according to any one of claims 1 to 4.

8. a first detection step of acquiring, from an image of a person having a plurality of body parts, a plurality of feature points indicating the positions of the plurality of body parts and a reliability indicating the likelihood of the positions indicated by each of the feature points; a second detection step of detecting, when the reliability of the feature point indicating the position of the ankle among the plurality of parts acquired in the first detection step is lower than a predetermined value, a feature point that is on a straight line connecting the feature points of the neck and waist among the plurality of parts acquired in the first detection step and has a predetermined ratio of the distance between the neck and the waist to the distance between the neck and the ankle, as the feature point of the ankle; a determining step of determining, as an area in which the person is present, an area that includes the head feature point acquired in the first detection step and the ankle feature point detected in the second detection step.

Images