Information processing device, matching program, and matching method

Abstract

To accurately match face information recognized by face recognition and posture information estimated by posture estimation with respect to an image on which a plurality of persons are reflected.SOLUTION: An information processing device 1 detects a face of each person from a captured image including a plurality of persons, and detects a person from the captured image. The information processing device 1 estimates a posture from the captured image on the basis of person detection data for every piece of person detection data corresponding to the plurality of detected persons. The information processing device 1 calculates face candidate data from skeleton information based on posture data for the estimated posture. The information processing device 1 identifies face detection data with the largest overlap area among pieces of face detection data of each detected person as the same person with respect to the face candidate data to be calculated for every piece of person detection data.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device and the like. [Background technology]

[0002] In recent years, data has been utilized in various fields. In particular, personal data is important because it can lead to services tailored to an individual's behavior and preferences. Personal data includes, for example, captured images of an individual.

[0003] Facial recognition technology is used to identify people in captured images and recognize their facial expressions. Facial recognition technology, for example, recognizes the facial expression of a person from a facial image of a single person. Also, posture estimation technology is used to recognize the posture of a person in a captured image. For example, posture estimation technology estimates the posture of a person from an image of a single person and estimates skeletal information from the posture.

[0004] Technologies using face recognition technology and posture estimation technology have been disclosed (see, for example, Patent Documents 1 and 2). For example, in Patent Document 1, face information and skeletal information are extracted from an image of a subject, and a positional relationship between the face position based on the face information and the skeletal position based on the skeletal information is judged to be normal or abnormal, and if it is judged to be abnormal, a message indicating that the subject cannot be authenticated is output. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2020 / 152917 [Patent Document 2] JP 2014-053965 A Summary of the Invention [Problem to be solved by the invention]

[0006] However, there is a problem that it is difficult to accurately match face information recognized by face recognition with posture information estimated by posture estimation for an image that includes multiple people. For example, when the target image is an image in which multiple people are close to each other, such as a group photo, matching between face information and posture information may fail. Here, a case in which matching fails will be described with reference to FIG. 18.

[0007] FIG. 18 is a diagram for explaining a case where matching fails. FIG. 18 shows a group photo in which multiple people are close to each other. From the group photo, for example, face information indicated by the symbol a0 is recognized by face recognition. Also, person information on which posture information indicated by the symbol b0 is based is estimated by posture estimation. However, in an image showing the group photo, face information indicated by the symbol a1 overlaps with other people's personal information, so matching between face information and person information may fail. That is, matching between face information and posture information may fail.

[0008] In one aspect, the present invention aims to accurately match face information recognized by face recognition with pose information estimated by pose estimation for an image containing multiple people. [Means for solving the problem]

[0009] In one aspect, an information processing device has a first detection unit that detects the faces of multiple people from an captured image including the multiple people, a second detection unit that detects people from the captured image, a posture estimation unit that estimates a posture from the captured image based on the person detection data for each of the multiple people detected by the second detection unit, a calculation unit that calculates face candidate data from skeletal information based on the posture data for the posture estimated by the posture estimation unit, and an identification unit that identifies, for the face candidate data calculated for each of the person detection data, face detection data with the largest overlap area among the face detection data of each person detected by the first detection unit as the same person. Effect of the Invention

[0010] According to one embodiment, for an image containing a plurality of people, face information recognized by face recognition and posture information estimated by posture estimation can be matched with high accuracy. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of an information processing system including an information processing device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of a target image. [Diagram 3] FIG. 3 is a diagram showing an example of face detection data. [Figure 4] FIG. 4 is a diagram illustrating an example of human detection data. [Diagram 5] FIG. 5 is a diagram illustrating an example of face detection. [Figure 6] FIG. 6 is a diagram illustrating an example of human detection. [Figure 7] FIG. 7 is a diagram illustrating an example of the posture estimation data. [Figure 8] FIG. 8 is a diagram showing an example of human body proportions. [Figure 9] FIG. 9 is a diagram showing an example of face candidate data. [Figure 10] FIG. 10 is a diagram for explaining coordinates according to the embodiment. [Figure 11A] FIG. 11A is a diagram (1) for explaining landscape orientation determination according to the embodiment. [Figure 11B] FIG. 11B is a diagram (2) for explaining landscape orientation determination according to the embodiment. [Figure 12A] FIG. 12A is a diagram (1) for explaining face candidate extraction according to the embodiment. [Figure 12B] FIG. 12B is a diagram (2) for explaining face candidate extraction according to the embodiment. [Figure 13A] FIG. 13A is a diagram (1) for explaining matching according to an embodiment. [Figure 13B]FIG. 13B is a diagram (2) for explaining matching according to the embodiment. [Figure 14] FIG. 14 is a diagram illustrating an example of matching according to the embodiment. [Figure 15] FIG. 15 is a diagram illustrating an example of an overall flowchart of information processing according to the embodiment. [Figure 16] FIG. 16 is a diagram illustrating an example of a flowchart of the matching process according to the embodiment. [Figure 17] FIG. 17 is a diagram illustrating an example of a computer that executes a matching program. [Figure 18] FIG. 18 is a diagram for explaining a case where matching fails. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an information processing device, a matching program, and a matching method disclosed in the present application will be described in detail with reference to the accompanying drawings. Note that the present invention is not limited to the embodiments. EXAMPLES

[0013] [Configuration of a system including an information processing device] 1 is a block diagram showing a functional configuration of an information processing system including an information processing device according to an embodiment. The information processing system 9 has an information processing device 1 and a camera 3. The information processing device 1 and the camera 3 are connected, for example, by a network 5. The network 5 may be wireless communication or wired communication, as long as image data can be transmitted from the camera 3 to the information processing device 1. As an example, in the case of wireless communication, the network 5 may be WiFi (registered trademark) or Bluetooth (registered trademark).

[0014] The information processing device 1 performs face detection and pose estimation from image data containing multiple people transmitted from the camera 3, and matches the face and pose for each person from the face detection and pose estimation information. An example of a target image used in the embodiment is shown in FIG. 2. FIG. 2 is a diagram showing an example of a target image. As shown in FIG. 2, a group photo containing multiple people is the target image. In such a target image, one person's face overlaps with another person's body, and multiple people are in close proximity to each other.

[0015] The information processing device 1 has a communication unit 10, a control unit 20, and a storage unit 30. The communication unit 10 controls communication with other devices. For example, the communication unit 10 receives image data of an image in which a plurality of people are captured from the camera 3. The communication unit 10 may also receive video data of an image in which a plurality of people are captured from the camera 3. In the embodiment, the video data is a bundle of image data, so that each image data will be described.

[0016] The storage unit 30 stores various data and programs executed by the control unit 20. For example, the storage unit 30 stores an image storage 31, face detection data 32, person detection data 33, pose estimation data , face candidate data 35, and a matching list .

[0017] The control unit 20 is a processing unit that controls the entire information processing device 1, and includes a storage unit 21, a face detection unit 22, a person detection unit 23, a posture estimation unit 24, a landscape orientation determination unit 25, a face candidate extraction unit 26, and a matching unit 27. The storage unit 21 stores image data. For example, the storage unit 21 receives image data transmitted from the camera 3 via the communication unit 10, and stores the image data in the image storage 31.

[0018] The face detection unit 22 detects faces from a captured image in which multiple people are captured. For example, the face detection unit 22 acquires target image data from the image storage 31. The image data is image data of an image in which multiple people are captured. The face detection unit 22 detects individual faces from the acquired image data. An example of a tool used for face detection is OpenCV, but any tool capable of detecting faces may be used. The face detection unit 22 generates face detection data 32 in which a face ID is assigned to the face coordinates and face range for each detected face.

[0019] Here, an example of the face detection data 32 will be described with reference to FIG. 3. FIG. 3 is a diagram showing an example of the face detection data. As shown in FIG. 3, the face detection data 32 is information in which a face ID (IDentifier), an x ​​coordinate, a y coordinate, a width, and a height are associated with each other. The face ID is an identifier that uniquely identifies a face. The x coordinate and the y coordinate are coordinates of a predetermined corner position when the detected face is shown as a rectangle. The coordinates are coordinates when the origin is the upper left of the captured image. The position of the predetermined corner may be the lower left, upper left, lower right, or upper right of the rectangle, and may be a predetermined corner position. The width and the height are the horizontal length and the height of the rectangle when the detected face is shown as a rectangle. This rectangle represents a face range. The face coordinates and the face range are called a bounding box.

[0020] As an example, when the face ID is "0", "1333" is stored as the x coordinate, "98" as the y coordinate, "413" as the width, and "413" as the height.

[0021] Returning to FIG. 1, the person detection unit 23 detects people from a captured image in which multiple people are captured. For example, the person detection unit 23 detects people from target image data. One example of a tool used for person detection is YOLO (You Only Look Once), but any tool that can detect people may be used. The person detection unit 23 generates person detection data 33 in which a body ID is assigned to the coordinates and range of the person for each detected person.

[0022] Here, an example of the person detection data 33 will be described with reference to FIG. 4. FIG. 4 is a diagram showing an example of the person detection data. As shown in FIG. 4, the person detection data 33 is information in which a body ID, an x ​​coordinate, a y coordinate, a width, and a height are associated with each other. The body ID is an identifier that uniquely identifies a person. The x coordinate and the y coordinate are coordinates of a predetermined corner position when the detected person is shown as a rectangle. The coordinates are coordinates when the origin is the upper left of the captured image. The position of the predetermined corner may be the lower left, upper left, lower right, or upper right of the rectangle, and may be a predetermined corner position. The width and the height are the horizontal length and the height of the rectangle when the detected person is shown as a rectangle. This rectangle represents the range of the person. The coordinates of the person and the range of the person are called a bounding box.

[0023] As an example, when the body ID is "0", the x coordinate is stored as "1176", the y coordinate is stored as "109", the width is stored as "754", and the height is stored as "680".

[0024] Fig. 5 is a diagram showing an example of face detection. As shown in Fig. 5, a bounding box of a face detected by the face detection unit 22 is shown for a target image. The face detection unit 22 detects individual faces for image data of the target image using a face detection tool. Then, the face detection unit 22 generates face detection data 32 in which a face ID is assigned to the face coordinates and face range for each detected face. The face coordinates are, for example, the lower left coordinates of a rectangle when the upper left corner of the captured image is taken as the origin, and the bounding box is shown by the face coordinates and face range (width, height).

[0025] FIG. 6 is a diagram showing an example of person detection. As shown in FIG. 6, the bounding box of a person detected by the person detection unit 23 is shown for the target image. Note that in FIG. 6, for convenience of explanation, the bounding box of only one person is shown, but in reality, the bounding boxes of multiple detected people are shown. The person detection unit 23 detects each person using a person detection tool for the image data of the target image. Then, the person detection unit 23 generates person detection data 33 in which a body ID is assigned to the coordinates of the person and the range of the person for each detected person. The coordinates of the person are, for example, the coordinates of the lower left corner of a rectangle when the upper left corner of the captured image is the origin, and the bounding box is expressed by the coordinates of the person and the range of the person (width, height).

[0026] Returning to FIG. 1, the posture estimation unit 24 estimates the posture of a person from a captured image based on the person detection data 33 for each person detection data 33 corresponding to a plurality of people detected by the person detection unit 23. For example, the posture estimation unit 24 extracts an image (person detection image) corresponding to each person detection data 33 from the captured image. The posture estimation unit 24 estimates position information (skeletal information) of a person's joint points using a posture estimation tool based on each person detection image. An example of the posture estimation tool is DeepPose, but any tool that can estimate position information (skeletal information) of a person's joint points may be used. The posture estimation unit 24 generates posture estimation data 34 by allocating the body ID of the person detection data 33 on which the posture estimation unit 24 is based to the estimated skeletal information.

[0027] Here, an example of the posture estimation data 34 will be described with reference to FIG. 7. FIG. 7 is a diagram showing an example of the posture estimation data. As shown in FIG. 7, the posture estimation data 34 is information in which a body ID and skeletal information are associated with each other. The body ID corresponds to the body ID of the person detection data 33. The skeletal information is information in which a joint number, an x ​​coordinate, and a y coordinate are associated with each other. The joint number is a number corresponding to a joint point. As an example, "0" as the joint number indicates the nose. "1" indicates the left eye. "2" indicates the right eye. "3" indicates the left ear. "4" indicates the right ear. "5" indicates the left shoulder. "6" indicates the right shoulder. "7" indicates the left elbow. "8" indicates the right elbow. "9" indicates the left wrist. "10" indicates the right wrist. "11" indicates the left hip. "12" indicates the right hip. "13" indicates the left knee. "14" indicates the right knee. "15" indicates the left ankle. "16" indicates the right ankle. The joint numbers correspond to the numbers in the joint model of the person in the right diagram of FIG. 7. The x and y coordinates are the coordinates of the estimated joint point. The coordinates are those when the origin is the upper left corner of the captured image. Note that the definition of the joint number is not limited to that described above.

[0028] For example, when the body ID is "0", the joint number is "5" (left shoulder), the x coordinate is "1173", and the y coordinate is "481". The joint number is "6" (right shoulder), the x coordinate is "953", and the y coordinate is "464".

[0029] Returning to FIG. 1, landscape orientation determination section 25 uses the human body ratio to determine whether or not the person's posture estimated by posture estimation section 24 is landscape orientation.

[0030] For example, the sideways determination unit 25 performs the following process for each posture of the person estimated by the posture estimation unit 24. The sideways determination unit 25 uses the human body ratio to determine whether the person is facing sideways from the skeletal information based on the posture estimation data 34 corresponding to the posture. The human body ratio is calculated by calculating the ratio (coefficient) of a part in the entire human body from the publicly known ratio of the length of each part of the human body. The details of the human body ratio will be described later. As an example, the sideways determination unit 25 calculates the length of a predetermined part using the skeletal information based on the posture estimation data 34. The predetermined part refers to, for example, the upper arm, the lower arm, the torso, the thigh, and the calf. The sideways determination unit 25 uses the human body ratio to multiply the length of the predetermined part by the ratio according to the part, and estimates the length of the torso from the length of the part. Then, the sideways determination unit 25 determines the length of the part having the maximum value as the length of the torso. Here, the body part ratios used to estimate the torso length are 9 / 5 for the upper arm, 9 / 4 for the lower arm, 1 for the torso, 3 / 2 for the thigh, and 3 / 2 for the calf. The body part coefficients and body part ratios are obtained from the human body proportions.

[0031] Furthermore, the sideways orientation determination unit 25 calculates the length of the shoulder width using skeletal information based on the posture estimation data 34. The sideways orientation determination unit 25 then determines whether the length of the shoulder width is shorter than 1 / 3 of the length of the torso, and if it is determined that the shoulder width is shorter, determines that the body is sideways. The sideways orientation determination unit 25 then associates a sideways orientation flag with the posture estimation data 34 of the person (body ID) whose body has been determined to be sideways. The sideways orientation determination unit 25 then passes the posture estimation data 34 of the person whose body has been determined not to be sideways to the face candidate extraction unit 26 in order to exclude the person whose body has been determined to be sideways from the target of face candidate extraction. The reason why the person whose body has been determined to be sideways is excluded from the target of face candidate extraction is that in the case of a sideways body, the x coordinate of the position of one shoulder overlaps with the x coordinate of the range of the face, and therefore face candidates cannot be correctly extracted by the face candidate extraction unit 26 described later.

[0032] An example of human body ratios will now be described with reference to FIG. 8. FIG. 8 is a diagram showing an example of human body ratios. The left diagram of FIG. 8 shows publicly known ratios of lengths of each part of the human body. The right diagram of FIG. 8 shows ratios (coefficients) of parts in the entire human body calculated from the ratios of lengths of each part of the human body. As shown in the right diagram of FIG. 8, the coefficient of the torso is "9", and the coefficient of shoulder width is "6". The coefficients of the chest and waist in the torso are "5" and "4". The coefficients of the thighs and calves are "6" and "6".

[0033] Here, the reason why the body is determined to be sideways when the shoulder width is shorter than 1 / 3 of the torso length is as follows. That is, when the human body is facing forward (or backward), the ratio of the torso to the shoulder width is approximately 3:2 according to the human body proportions. When the ratio of the torso to the shoulder width is less than 3:1, it is presumed that the shoulder width is narrow and the body is no longer facing forward (or backward). That is, it is presumed that the x coordinate of the position of one shoulder overlaps with the x coordinate of the range of the face. For this reason, the sideways determination unit 25 determines that the body is sideways when the shoulder width is shorter than 1 / 3 of the torso length, and excludes it from the targets for face candidate extraction.

[0034] The face candidate extraction unit 26 extracts data of the face candidate using the human body ratio and posture estimation data 34. For example, the face candidate extraction unit 26 acquires the x-coordinate and y-coordinate of the left shoulder (joint No. is "5") and the x-coordinate and y-coordinate of the right shoulder (joint No. is "6") from the skeletal information for the body ID of the target. When the origin is set to the upper left of the captured image of the target, the face candidate extraction unit 26 sets the smaller value of the x-coordinate of the left shoulder and the x-coordinate of the right shoulder as the x-coordinate of the rectangle of the face candidate. The face candidate extraction unit 26 sets the larger value of the y-coordinate of the left shoulder and the y-coordinate of the right shoulder as the y-coordinate of the rectangle of the face candidate. Then, the face candidate extraction unit 26 sets the length of the horizontal component of the shoulder width as the width of the rectangle of the face candidate. The face candidate extraction unit 26 sets 5 / 9 of the length of the torso as the height of the rectangle of the face candidate using the human body ratio. The height of the face candidate rectangle is set to 9 / 5 of the torso length because, according to the human body proportions shown in Fig. 8, the ratio of the torso to the head including the neck is 9 to 5. Then, face candidate extraction unit 26 generates the coordinates, width and height of the face candidate rectangle as face candidate data 35 for each target body ID.

[0035] Here, an example of the face candidate data 35 will be described with reference to FIG. 9. FIG. 9 is a diagram showing an example of the face candidate data. As shown in FIG. 9, the face candidate data 35 is information in which a body ID, an x ​​coordinate, a y coordinate, a width, and a height are associated with each other. The body ID corresponds to the body ID of the person detection data 33. The x coordinate and the y coordinate are coordinates of a predetermined corner position when the extracted face candidate is shown as a rectangle. The coordinates are coordinates when the origin is the upper left of the captured image. The position of the predetermined corner may be the lower left, upper left, lower right, or upper right of the rectangle, and may be a predetermined corner position. The width and the height are the horizontal length and the height of the rectangle when the extracted face candidate is shown as a rectangle. This rectangle represents the range of the face candidate. The coordinates of the face candidate and the range of the face candidate are called a bounding box.

[0036] Returning to FIG. 1, the matching unit 27 matches (links) the face detection data 32 having the largest overlapping area among the multiple face detection data 32 with the face candidate data 35 as the same person. For example, the matching unit 27 performs the following process for each face candidate data 35. The matching unit 27 obtains the overlapping area between the bounding box obtained from the target face candidate data 35 and the bounding box obtained from the face detection data 32 of all faces detected by the face detection unit 22. The matching unit 27 determines that the body ID of the target face candidate data 35 is matched with the face ID of the face detection data 32 having the largest overlapping area. That is, the matching unit 27 specifies the matched face ID and body ID as the same person. Then, the matching unit 28 outputs the specified face ID and body ID to the matching list 36.

[0037] Here, the coordinates according to the embodiment will be described with reference to FIG. 10. FIG. 10 is a diagram for explaining the coordinates according to the embodiment. For convenience of explanation, FIG. 10 uses an image in which a part of the captured image is extracted. The upper left corner of the captured image is the coordinates (0,0) of the origin. The right direction from the origin is the x coordinate, and the downward direction from the origin is the y coordinate. Here, for example, the coordinates (skeleton information) of the joint points of a person estimated by the posture estimation unit 24 are shown. As an example, the coordinates of the right shoulder are (14,34). The coordinates of the right shoulder are coordinates corresponding to the joint number "6" in the joint model of the person shown in FIG. 7. The coordinates of the left shoulder are (28,24). The coordinates of the left shoulder are coordinates corresponding to the joint number "5" in the joint model of the person shown in FIG. 7. The coordinates of the right hip are (28,55). The coordinates of the right hip are coordinates corresponding to the joint number "12" in the joint model of the person shown in FIG. 7. The coordinates of the left hip are (44, 52), which correspond to joint number "11" in the human joint model shown in FIG.

[0038] 11A and 11B are diagrams for explaining sideways orientation determination according to an embodiment. FIG. 11A shows the bounding boxes of face candidates of a person whose body is turned to the side. The black circles indicate the coordinates of the left and right shoulders. When the body is turned to the side, the x coordinate of the position of one shoulder (here, the right shoulder) overlaps with the x coordinate of the range of the face, and therefore face candidates extracted by the face candidate extraction unit 26 are not correctly extracted. Therefore, the sideways orientation determination unit 25 excludes people whose body is turned to the side from targets for face candidate extraction.

[0039] As shown in FIG. 11B, some of the coordinates (skeletal information) of a person's joint points estimated by posture estimation unit 24 are shown. The coordinates of the right shoulder are (14, 34). The coordinates of the left shoulder are (28, 24). The coordinates of the right hip are (28, 55). The coordinates of the left hip are (44, 52). Although not shown, there are also the coordinates of the left elbow and the left wrist. The space between the right shoulder and the right hip, and the space between the left shoulder and the left hip, respectively, indicate the torso. The space between the left elbow and the left wrist indicates the lower arm.

[0040] Sideways orientation determination unit 25 calculates the length of each part using the skeletal information of posture estimation data 34. Here, as an example, the length of the torso and the lower arm are calculated. Using the coordinates (14,34) of the right shoulder and the coordinates (28,55) of the right hip, the length of the torso is calculated to be "25." Furthermore, using the coordinates (28,24) of the left shoulder and the coordinates (44,52) of the left hip, the length of the torso is calculated to be "32." Furthermore, the length of the lower arm is calculated to be "8" using the coordinates of the left elbow and the left wrist.

[0041] The sideways orientation determination unit 25 uses the human body ratio to multiply the length of a predetermined part by a ratio according to the part, and determines the maximum multiplication result as the length of the torso. Here, the length of the torso calculated from the right shoulder and right hip is calculated as "25" because the ratio is 1. The length of the torso calculated from the left shoulder and left hip is calculated as "32" because the ratio is 1. The length of the torso calculated from the lower arms is calculated as "18" (=8×9 / 4) because the ratio is 9 / 4. Then, the sideways orientation determination unit 25 calculates max(25,32,18) and determines the length of the torso as "32". Here, the sideways orientation determination unit 25 determines the length of the torso from the torso and lower arms as an example, but in reality, the length of the torso is determined by adding the upper arms, thighs, calves, etc.

[0042] Furthermore, sideways orientation determination unit 25 calculates the shoulder width using the coordinates of the right shoulder and the left shoulder. In this example, the coordinates of the right shoulder are (14, 34) and the coordinates of the left shoulder are (28, 24), so the shoulder width is calculated to be "17."

[0043] Then, the sideways orientation determination unit 25 determines whether the shoulder width is shorter than 1 / 3 of the torso length, and if it is determined that the shoulder width is shorter, it determines that the body is sideways. Here, the shoulder width is "17" and the torso length is "32", so the shoulder width "17" is longer than 1 / 3 of the torso length (approximately 10). Therefore, it is determined that the body is not sideways. As a result, the sideways orientation determination unit 25 sets the posture estimation data 34 of this person as a target for face candidate extraction by the face candidate extraction unit 26. Note that, if the sideways orientation determination unit 25 determines that the body is sideways, it does not set the posture estimation data 34 of the person whose body is determined to be sideways as a target for face candidate extraction by the face candidate extraction unit 26.

[0044] 12A and 12B are diagrams for explaining face candidate extraction according to an embodiment. FIG. 12A shows the bounding box of a human face candidate. The large black circle indicates the position of the left shoulder and the right shoulder. The small black circle indicates the position of the left hip and the right hip. The length between the shoulder and the hip indicated by the arrow is the length of the torso. The face candidate extraction unit 26 determines the length of the horizontal component of the shoulder width as the width of the bounding box of the face candidate. The face candidate extraction unit 26 determines the height of the bounding box of the face candidate to be 5 / 9 of the length of the torso using the human body ratio. This is because the ratio of the length of the torso to the length of the head and neck is 9:5 according to the human body ratio in FIG. 8.

[0045] As shown in FIG. 12B, a part of the coordinates (skeletal information) of the joint points of the person estimated by the posture estimation unit 24 is shown. The coordinates of the right shoulder are (14, 34). The coordinates of the left shoulder are (28, 24). The face candidate extraction unit 26 sets the smaller value of the x coordinate of the left shoulder and the x coordinate of the right shoulder as the x coordinate of the bounding box of the face candidate. Here, the x coordinate of the bounding box indicates "14". The face candidate extraction unit 26 sets the larger value of the y coordinate of the left shoulder and the y coordinate of the right shoulder as the y coordinate of the bounding box of the face candidate. Here, the y coordinate of the bounding box indicates "34". Then, the face candidate extraction unit 26 sets the length of the horizontal component of the shoulder width as the width of the bounding box of the face candidate. Here, the width of the bounding box indicates "14" since the length of the horizontal component of the shoulder width is calculated to be 14. Then, using the human body ratio, the face candidate extraction unit 26 determines the height of the bounding box of the face candidate to be 5 / 9 of the length of the torso. In this case, since the length of the torso is "32" as calculated in FIG. 11B, the height of the bounding box is 18 (=32×5 / 9). As a result, the bounding box of the face candidate is extracted with the coordinates of the face candidate being (14, 34), the width of the face candidate range being 14, and the height of the face candidate range being 18. The dotted rectangle is the bounding box of the face candidate.

[0046] Fig. 13A and Fig. 13B are diagrams for explaining matching according to an embodiment. Fig. 13A shows the overlap between a bounding box bbox1 obtained from one face candidate data 35 and a bounding box bbox2 obtained from one face detection data 32 detected by the face detection unit 22. The coordinates of the face candidate in bbox1 are (bbox1_x, bbox1_y). The width of bbox1 is bbox1_w. The height of bbox1 is bbox1_h. In addition, the coordinates of the face candidate in bbox2 are (bbox2_x, bbox2_y). The width of bbox2 is bbox2_w. The height of bbox2 is bbox2_h.

[0047] The matching unit 27 determines the four points of the overlapping rectangle between bounding boxes bbox1 and bbox2, the bottom left, top left, bottom right, and top right, as follows: "left" indicates the x coordinate values ​​of the top and bottom of the left of the overlap; "right" indicates the x coordinate values ​​of the top and bottom of the right of the overlap; "bottom" indicates the y coordinate values ​​of the left and right of the bottom of the overlap; and "top" indicates the y coordinate values ​​of the left and right of the top of the overlap. left = max(bbox1_x,bbox2_x) top=max(bbox1_y-bbox1_h,bbox2_y-bbox2_h) right=min(bbox1_x+bbox1_w,bbox2_x+bbox2_w) bottom = min(bbox1_y, bbox2_y) That is, the coordinates of the bottom left corner of the overlap rectangle are (left, bottom). The coordinates of the top left corner of the overlap rectangle are (left, top). The coordinates of the bottom right corner of the overlap rectangle are (right, bottom). The coordinates of the top right corner of the overlap rectangle are (right, top).

[0048] Then, if an overlapping rectangle exists, the matching unit 27 calculates the area of ​​the overlap. Here, if the left is smaller than the right and the top is smaller than the bottom, it is determined that an overlapping rectangle exists. Then, the area of ​​the overlap is calculated as follows. Overlap area = (right-left) x (bottom-top)

[0049] 13B, a bounding box (dashed line) obtained from face candidate data 35 and a bounding box (solid line) obtained from face detection data 32 are shown. For the bounding box obtained from face candidate data 35, the coordinates of the face candidate are represented as (14, 34), and the width and height of the face candidate are represented as 14 and 18, respectively. For the bounding box obtained from face detection data 32, the coordinates of the face are represented as (14, 28), and the width and height of the face are represented as 12 and 13, respectively.

[0050] Then, the matching unit 27 finds the four points (lower left, upper left, lower right, and upper right) of the overlapping rectangle as follows. left=max(14,14)=14 top=max(28-13,34-18)=max(15,16)=16 right=min(26,28)=26 bottom=min(28,34)=28 That is, the coordinates of the bottom left corner of the overlap rectangle are (14,28). The coordinates of the top left corner of the overlap rectangle are (14,16). The coordinates of the bottom right corner of the overlap rectangle are (26,28). The coordinates of the top right corner of the overlap rectangle are (26,16).

[0051] Then, since left (14) is smaller than right (26) and top (16) is smaller than bottom (28), an overlapping rectangle exists, so matching unit 27 calculates the overlapping area as 144 (=(26-14)×(28-16)).

[0052] Fig. 14 is a diagram showing an example of matching according to the embodiment. It is assumed that the rectangle enclosed by dashed lines shown in Fig. 14 is a bounding box obtained from the face candidate data 35 indicating the body ID "1".

[0053] The matching unit 27 calculates the overlapping area between the bounding box obtained from the target face candidate data 35 and the bounding boxes obtained from the face detection data 32 of all faces detected by the face detection unit 22. Here, the bounding boxes obtained from the face detection data 32 that overlap the bounding box of the face candidate with a body ID of "1" are shown. The overlapping face IDs of the face detection data 32 are "1", "2", and "3". The overlapping areas between the body ID "1" and the face IDs "1", "2", and "3" are assumed to be "20", "30", and "10".

[0054] Then, the matching unit 27 determines that the body ID of the target face candidate data 35 matches the face ID of the face detection data 32 with the largest overlapping area. Here, the overlapping area between the body ID "1" and the face ID "2" is the largest. Therefore, it is determined that the body ID "1" and the face ID "2" match. Then, the matching unit 27 links the body ID "1" and the face ID "2" and outputs them to the matching list 36.

[0055] Fig. 15 is a diagram showing an example of an overall flowchart of information processing according to an embodiment. As shown in Fig. 15, the control unit 20 inputs a video in which a plurality of people are shown (step S11). The control unit 20 executes the following processes for the input video on a frame-by-frame basis (step S12).

[0056] The face detection unit 22 detects faces in the image data, assigns a face ID to the face coordinates and face area for each detected face, and generates face detection data 32 (step S13). Then, the face detection unit 22 proceeds to step S20 to perform matching processing.

[0057] Meanwhile, the human detection unit 23 detects humans from the image data, assigns a body ID to the coordinates and range of each detected human, and generates human detection data 33 (step S14).

[0058] Next, posture estimation unit 24 extracts images (person detection images) corresponding to each person detection data 33 from the images (step S15). Then, posture estimation unit 24 performs posture estimation for all extracted person detection images, and estimates skeletal information obtained by posture estimation. Posture estimation unit 24 assigns the body ID of person detection data 33 on which the posture estimation is based to the skeletal information, and generates posture estimation data 34 (step S16). Skeletal information here refers to position information of a person's joint points.

[0059] Next, sideways determination unit 25 uses the human body ratio and posture estimation data 34 to determine whether or not there is a person whose body is sideways (step S17). For example, sideways determination unit 25 calculates the length of the shoulder width using the skeletal information of posture estimation data 34, determines whether or not the length of the shoulder width is shorter than 1 / 3 of the length of the torso, and determines that the body is sideways if it is determined to be shorter. The length of the torso is calculated from the lengths of the upper arm, lower arm, torso, thigh, and calf calculated from the skeletal information and the human body ratio.

[0060] If it is determined that there is no person with a sideways body (step S17; No), the sideways orientation determination unit 25 proceeds to step S19 to generate face candidates. On the other hand, if it is determined that there is a person with a sideways body (step S17; Yes), the sideways orientation determination unit 25 associates a sideways orientation flag with the posture estimation data 34 of the person with a sideways body (step S18). Then, the sideways orientation determination unit 25 proceeds to step S19 to generate face candidates.

[0061] In step S19, the face candidate extraction unit 26 calculates the coordinates and range of the face candidate from the posture estimation data 34 (skeletal information) not associated with the human body ratio and the landscape flag to generate face candidate data 35 (step S19). For example, the face candidate extraction unit 26 acquires the x and y coordinates of the left shoulder and the x and y coordinates of the right shoulder from the skeleton information for the body ID of the posture estimation data 34. The face candidate extraction unit 26 sets the smaller value of the x coordinate of the left shoulder and the x coordinate of the right shoulder as the x coordinate of the rectangle of the face candidate, and sets the larger value of the y coordinate of the left shoulder and the y coordinate of the right shoulder as the y coordinate of the rectangle of the face candidate. Then, the face candidate extraction unit 26 sets the length of the horizontal component of the shoulder width as the width of the rectangle of the face candidate. The face candidate extraction unit 26 uses the human body ratio to set 5 / 9 of the length of the torso as the height of the rectangle of the face candidate. The length of the torso is calculated by the landscape determination unit 25. Then, face candidate extraction unit 26 generates face candidate data 35 including the coordinates, width, and height of the face candidate for each body ID in posture estimation data 34.

[0062] Then, the matching unit 27 executes a matching process for each body ID in the posture estimation data 34 (step S20). A flowchart of the matching process will be described later. Then, if there is an unprocessed image, the control unit 20 proceeds to step S12 (step S21). On the other hand, if there is no unprocessed image, the control unit 20 ends the information processing.

[0063] Fig. 16 is a diagram showing an example of a flowchart of the matching process according to the embodiment. As shown in Fig. 16, the matching unit 27 executes the process for each body ID (step S31). The matching unit 27 judges whether or not a landscape flag is associated with the posture estimation data 34 corresponding to the body ID (step S32). If it is judged that a landscape flag is associated with the posture estimation data 34 corresponding to the body ID (step S32; Yes), the matching unit 27 proceeds to step S37 to execute the process corresponding to the next body ID.

[0064] On the other hand, if it is determined that the landscape flag is not associated with the posture estimation data 34 corresponding to the body ID (step S32; No), the matching unit 27 executes processing on a face ID basis (step S33). The matching unit 27 calculates the overlapping area of ​​the bounding boxes (coordinates and range) of the body ID and the face ID (step S34). If there is an unprocessed face ID, the matching unit 27 proceeds to step S33 (step S35). On the other hand, if there is no unprocessed face ID, the matching unit 27 proceeds to step S36 to match the body ID and the face ID.

[0065] In step S36, the matching unit 27 performs matching by pairing the body ID and the face ID with the largest overlapping area of ​​the bounding box (coordinates and range) (step S36). If there is an unprocessed body ID, the matching unit 27 proceeds to step S31 (step S37).

[0066] On the other hand, if there is no unprocessed body ID, the matching unit 27 outputs the matching result to the matching list 36 (step S38). Then, the matching unit 27 ends the matching process.

[0067] The matching process executed by the information processing device 1 according to the embodiment can be applied to a video distribution service. For example, the video distribution service matches the faces and bodies (postures) of multiple people in a video. In addition, the video distribution service considers a change in facial expression or body movement to be a characteristic of a moment when a person's emotions change, and distributes an image of that moment and videos before and after that moment to the user. One example is a video of a part where everyone is high-fiving after getting a strike in bowling. The video distribution service may also highlight a target person and distribute an image of that moment and videos before and after that moment to the user.

[0068] [Effects of the embodiment] In this way, in the above embodiment, the information processing device 1 detects the faces of each person from a captured image including multiple people. The information processing device 1 detects people from a captured image. For each person detection data corresponding to the multiple detected people, the information processing device 1 estimates a posture from the captured image based on the person detection data. The information processing device 1 calculates face candidate data from skeletal information based on posture data for the estimated posture. For the face candidate data calculated for each person detection data, the information processing device 1 identifies face detection data with the largest overlap area among the face detection data of each detected person as the same person. With this configuration, the information processing device 1 can improve the matching accuracy between a face detected from face detection and a posture (body) estimated from posture estimation. That is, the information processing device 1 can match a face and a body with high accuracy even on an image where people are crowded together.

[0069] In the above embodiment, the information processing device 1 calculates face candidate data based on the human body ratio indicating the ratio of parts in the whole human body and the skeleton information obtained from the posture data. With this configuration, the information processing device 1 can calculate candidates for the face position from the posture data by using the human body ratio.

[0070] In the above embodiment, the information processing device 1 estimates the length of the torso from the length of a specific part obtained from the skeletal information using the human body ratio. The information processing device 1 calculates face candidate data based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso. With this configuration, the information processing device 1 can calculate candidates for the position of the face by using the position of the shoulders and the length of the torso.

[0071] In the above embodiment, the information processing device 1 further determines whether the body is turned sideways using the shoulder width length and the torso length. When the information processing device 1 determines that the body is turned sideways, it does not calculate face candidate data from skeletal information. With this configuration, the information processing device 1 can determine whether the body is turned sideways by using the human body ratio. Then, the information processing device 1 can avoid calculating face candidates with low accuracy by excluding skeletal information in which the body is turned sideways from the calculation of face candidates.

[0072] [others] In the embodiment, the coordinates of the face detection data 32, the person detection data 33, the posture estimation data 34, and the face candidate data 35 are described as having the origin at the top left of the captured image. However, the origin may be located at the bottom left of the captured image. In such a case, for example, the face candidate extraction unit 26 may set the smaller of the x coordinate of the left shoulder and the x coordinate of the right shoulder as the x coordinate of the rectangle of the face candidate, and may set the smaller of the y coordinate of the left shoulder and the y coordinate of the right shoulder as the y coordinate of the rectangle of the face candidate.

[0073] Moreover, each component of the illustrated information processing device 1 does not necessarily have to be physically configured as illustrated. That is, the specific manner of distribution and integration of each device is not limited to that illustrated, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions. For example, the landscape determination unit 25 and the face candidate extraction unit 26 may be integrated into one unit. The landscape determination unit 25 may be distributed into a determination unit that determines the length of the torso and a determination unit that determines landscape orientation using the length of the torso. The image storage 31, the storage unit 30 that stores the face detection data 32, etc. may be connected via a network as an external device of the information processing device 1. Furthermore, the control unit 20 and the storage unit 30 may be placed on a cloud, and the object to be used may connect to the cloud via a network from a personal computer or the like to use the processing executed by the control unit 20. In this way, the processing of the present invention can be provided as a service.

[0074] Moreover, the various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, an example of a computer that executes a matching program that realizes the same functions as the information processing device 1 shown in Fig. 1 will be described below. Fig. 17 is a diagram showing an example of a computer that executes a matching program.

[0075] 17, the computer 200 has a CPU 203 that executes various arithmetic processes, an input device 215 that accepts data input from a user, and a display device 209. The computer 200 also has a drive device 213 that reads programs and the like from a storage medium, and a communication I / F (Interface) 217 ​​that transmits and receives data to and from other computers via a network. The computer 200 also has a memory 201 that temporarily stores various information, and a HDD (Hard Disk Drive) 205. The memory 201, the CPU 203, the HDD 205, the display device 209, the drive device 213, the input device 215, and the communication I / F 217 are connected by a bus 219.

[0076] The drive device 213 is, for example, a device for the removable disk 211. The HDD 205 stores a matching program 205a and matching process related information 205b. The communication I / F 217 manages the interface between the network and the inside of the device, and controls input and output of data from other computers. For example, a modem or a LAN adapter can be used as the communication I / F 217.

[0077] The display device 209 is a display device that displays a cursor, an icon, a tool box, and data such as documents, images, function information, etc. The display device 209 can be, for example, a liquid crystal display or an organic EL (Electroluminescence) display.

[0078] The CPU 203 reads out the matching program 205a, develops it in the memory 201, and executes it as a process. Such a process corresponds to each functional unit of the information processing device 1. The matching processing related information 205b corresponds to the image storage 31, the face detection data 32, and the like. For example, the removable disk 211 stores each piece of information such as the matching program 205a.

[0079] Incidentally, the matching program 205a does not necessarily have to be stored in the HDD 205 from the beginning. For example, the program may be stored in a "portable physical medium" such as a flexible disk (FD), a CD-ROM, a DVD disk, a magneto-optical disk, or an IC card that is inserted into the computer 200. Then, the computer 200 may read out the matching program 205a from these and execute it. [Explanation of symbols]

[0080] 1. Information processing device 3. Camera 5. Network 9. Information Processing Systems 10. Communications Department 20 Control section 21 Preservation Department 22 Face detection section 23 Person detection section 24 Posture estimation section 25 Landscape orientation determination section 26 Face Candidate Extraction Unit 27 Matching Section 30 Storage section 31 Image Storage 32 Face detection data 33 Person detection data 34 Pose Estimation Data 35 Face candidate data 36 Matching List

Claims

1. a first detection unit that detects a face of each person from a captured image including a plurality of people; a second detection unit that detects each person from the captured image; a posture estimation unit that estimates, for each person detection data corresponding to each person detected by the second detection unit, a posture corresponding to the person detection data from the captured image; a first calculation unit that uses a human body ratio indicating a ratio of a part in the entire human body to estimate a length of a torso according to a length of a predetermined part obtained from skeletal information based on a posture estimated by the posture estimation unit, calculates a shoulder width using the skeletal information, and calculates face candidate data from the skeletal information based on posture data for the posture estimated by the posture estimation unit; a determination unit that determines whether or not the body is lying sideways using the shoulder width and the torso length; a second calculation unit that calculates the face candidate data based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso when it is determined that the body is not turned sideways, and does not calculate the face candidate data from the skeletal information when it is determined that the body is turned sideways; an identification unit that identifies, for face candidate data calculated for each of the person detection data, face detection data having a largest overlapping area among the face detection data of each person detected by the first detection unit, as the same person; 13. An information processing device comprising:

2. Detecting the faces of multiple people from a captured image containing each person, Detecting each person from the captured image; For each person detection data corresponding to each of the detected people, estimating a posture corresponding to the person detection data from the captured image; Using a human body ratio indicating the ratio of a part to the whole human body, a torso length is estimated according to the length of a predetermined part obtained from skeletal information based on the estimated posture, and a shoulder width is calculated using the skeletal information. Face candidate data is calculated from the skeletal information based on posture data for the estimated posture. determining whether the body is lying sideways using the shoulder width and the torso length; When it is determined that the body is not turned sideways, the face candidate data is calculated based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso, and when it is determined that the body is turned sideways, the face candidate data is not calculated from the skeletal information, identifying, for the face candidate data calculated for each of the person detection data, the face detection data having the largest overlapping area among the face detection data of each of the detected persons as the same person; A matching program that causes a computer to execute the processing.

3. Detecting the faces of multiple people from a captured image containing each person, Detecting each person from the captured image; For each person detection data corresponding to each of the detected people, estimating a posture corresponding to the person detection data from the captured image; Using a human body ratio indicating the ratio of a part to the whole human body, a torso length is estimated according to the length of a predetermined part obtained from skeletal information based on the estimated posture, and a shoulder width is calculated using the skeletal information. Face candidate data is calculated from the skeletal information based on posture data for the estimated posture. determining whether the body is lying sideways using the shoulder width and the torso length; When it is determined that the body is not turned sideways, the face candidate data is calculated based on the positions of the right and left shoulders obtained from the skeletal information, the length of the horizontal component of the shoulder width, and the length of the torso, and when it is determined that the body is turned sideways, the face candidate data is not calculated from the skeletal information, identifying, for the face candidate data calculated for each of the person detection data, the face detection data having the largest overlapping area among the face detection data of each of the detected persons as the same person; A matching method in which processing is performed by a computer.

Images