Keypoint association device, keypoint association method, and program

Abstract

The key point association device (20000) acquires a target image (10) in which one or more persons are imaged, and for each person, detects a reference key point (20) and one or more target key points (30) from the target image (10). The reference key point (20) of a person indicates the position of the reference part of that person. The target key point (30) of a person indicates the position of the target part of that person. The key point association device (2000) generates a feature map of each target part based on the target image (10). The feature map of the target part indicates a region connecting the reference part and the target part belonging to the same person as the reference part. The key point association device (2000) associates the reference key point (20) with the target key point (30) belonging to the same person as the reference key point (20) based on the feature map.

Description

[Technical Field]

[0001] The present disclosure generally relates to a keypoint association apparatus, a keypoint association method, and a non-transitory computer-readable storage medium. [Background technology]

[0002] There are various types of analyses that can be performed on images that contain one or more people. Some of these analyses (e.g., pose estimation) use keypoints of people (e.g., body joints). Specifically, keypoints are detected from an image and divided into groups, with each group containing keypoints that belong to the same person. This process of dividing keypoints into groups is called "keypoint association." Non-Patent Document 1 discloses one algorithm for keypoint association. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh, "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields," [online], December 18, 2018, [Retrieved April 29, 2022],<arXiv, https: / / arxiv.org / pdf / 1812.08008.pdf> Retrieved from Summary of the Invention [Problem to be solved by the invention]

[0004] In Non-Patent Document 1, adjacent body parts need to be defined in advance. For example, the neck and right wrist, the right wrist and right knee, and the right knee and right foot can be defined as adjacent body parts, respectively. One of the objectives of this disclosure is to disclose a new technique for keypoint association. [Means for solving the problem]

[0005] The present disclosure discloses a keypoint association device having at least one memory element storing instructions and at least one processor, wherein the at least one processor is configured to execute the instructions to acquire target images in which one or more persons are captured, detect, for each person, a reference keypoint and one or more target keypoints from the target image, the reference keypoint of the person indicating a location of a reference feature of the person, the target keypoint of the person indicating a location of a target feature of the person, the target feature being different from the reference feature, generate a feature map for each target feature based on the target image, the feature map for the target feature indicating, for each reference feature in the target image, a region connecting the reference feature to the target feature belonging to the same person as the reference feature, and associate the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint based on the feature map.

[0006] The present disclosure further discloses a computer-implemented method for keypoint association. The keypoint association method includes: acquiring a target image in which one or more persons are captured; detecting a reference keypoint and one or more target keypoints from the target image for each person; the reference keypoint for the person indicates the position of a reference feature of the person; the target keypoint for the person indicates the position of a target feature of the person, the target feature being different from the reference feature; generating a feature map for each target feature based on the target image; the feature map for the target feature indicates, for each reference feature in the target image, a region connecting the reference feature to the target feature belonging to the same person as the reference feature; and associating the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint based on the feature map.

[0007] The present disclosure further discloses a non-transitory computer-readable storage medium having a program stored thereon. The program causes a computer to acquire a target image in which one or more people are captured, detect a reference keypoint and one or more target keypoints from the target image for each person, the reference keypoint for the person indicating the position of a reference part of the person, the target keypoint for the person indicating the position of a target part of the person, the target part being different from the reference part, generate a feature map for each target part based on the target image, the feature map for the target part indicating, for each reference part in the target image, a region connecting the reference part to the target part belonging to the same person as the reference part, and associate the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint based on the feature map. [Effects of the Invention]

[0008] According to the present disclosure, a novel technique for keypoint association is provided. [Brief explanation of the drawings]

[0009] [Figure 1]FIG. 1 is a diagram showing an overview of a keypoint association device. [Figure 2] FIG. 2 is a diagram illustrating an example of a BCF feature map. [Figure 3] FIG. 3 is a block diagram illustrating an example of the functional configuration of the keypoint association device. [Figure 4] FIG. 4 is a block diagram illustrating an example of the hardware configuration of the keypoint association device. [Figure 5] FIG. 5 is a flowchart illustrating an example of the flow of processing executed by the keypoint association device. [Figure 6] FIG. 6 is a flowchart showing an example of the flow of processing performed by the keypoint association unit 2080 to associate keypoints. [Figure 7] FIG. 7 is a diagram showing candidate links and their midpoints. DETAILED DESCRIPTION OF THE INVENTION

[0010] Embodiments according to the present disclosure will be described below with reference to the drawings. The same elements are assigned the same reference numerals throughout the drawings, and redundant descriptions will be omitted as necessary. Furthermore, unless otherwise specified, predetermined information (e.g., predetermined values ​​or predetermined threshold values) is pre-stored in a storage device accessible by a computer that uses the information.

[0011] <Summary> 1 is a diagram illustrating an overview of a keypoint association device 2000 according to an embodiment. The overview illustrated in FIG. 1 is an example of the operation of the keypoint association device 2000, with the aim of making it easier to understand the keypoint association device 2000, and is not intended to limit or narrow the range of operations that the keypoint association device 2000 can perform.

[0012] The keypoint association device 2000 acquires a target image 10 in which one or more people are captured, detects keypoints from the target image 10, and performs keypoint association for the detected keypoints. The target image 10 is any type of image data (e.g., an RGB image or a grayscale image) captured in a manner that is visible to humans. The keypoints may indicate characteristic points (e.g., joints) of a person's body.

[0013] The keypoints belonging to a specific person include a reference keypoint 20 and one or more target keypoints 30. The reference keypoint 20 of a specific person indicates the position of a predetermined reference part of the person (i.e., coordinates on the target image 10), while the target keypoints 30 of a specific person indicate the positions of different predetermined target parts of the person. The reference part may be a representative one of the characteristic parts of the person's body, such as the neck. The target part may be a characteristic part of the person's body other than the reference part, such as the right eye or the left shoulder.

[0014] For example, suppose the reference part is the neck and the target parts include 16 parts of a person's body (right eye, right ear, right shoulder, right elbow, right hand, right wrist, right knee, right foot, left eye, left ear, left shoulder, left elbow, left hand, left wrist, left knee, and left foot). In this case, the keypoint association device 2000 detects, for each person in the target image 10, the neck point as the reference keypoint 20 and the points of these 16 target parts as the target keypoints 30.

[0015] After detecting the keypoints, the keypoint association device 2000 performs keypoint association. Keypoint association is a process of associating each reference keypoint 20 detected from the target image 10 with a target keypoint 30 that belongs to the same person as the reference keypoint 20. In other words, keypoint association is a process of creating, for each person, a group of keypoints that belong to that person. Hereinafter, a group of keypoints that belong to the same person will be referred to as a "keypoint group."

[0016] In the keypoint association process, the keypoint association device 2000 analyzes the target image 10 and generates a map called a "BCF (Body Crosscutting Field) feature map" for each target body part. For example, if the 16 target body parts described above are defined, the keypoint association device 2000 generates a BCF feature map for each of the 16 target body parts, such as a BCF feature map for the right eye and a BCF feature map for the right shoulder.

[0017] The BCF feature map for a specific body part indicates, for each reference body part included in the target image 10, a region (called a "BCF region") connecting the reference body part and the target body part belonging to the same person. FIG. 2 shows an example of a BCF feature map. In this example, the neck is defined as the reference body part. The target image 10 for which the BCF feature map 70 is generated includes three people 40-1 to 40-3. The necks 50-1 to 50-3 are the reference body parts of people 40-1 to 40-3, respectively.

[0018] 2, a BCF feature map 70 for the right knee is generated. Therefore, the BCF feature map 70 shows a BCF region 80 that connects the neck 50 and the right knee 60 that belong to the same person for each head 50 included in the target image 10. For example, BCF region 80-1 connects the neck 50-1 and the right knee 60-1 that belong to person 40-1.

[0019] The keypoint association device 2000 uses the BCF feature map to associate a reference keypoint 20 with a target keypoint 30 that belongs to the same person as the reference keypoint 20. As a result of the keypoint association, the keypoint association device 2000 obtains, for each reference keypoint 20, a keypoint group that includes the reference keypoint 20 and the target keypoint 30 that are associated with each other. This means that the keypoint group includes the reference keypoint 20 and the target keypoint 30 that belong to the same person.

[0020] <Examples of effects> The keypoint association device 2000 introduces a new concept called "BCF feature map" into keypoint association. Specifically, the keypoint association device 2000 generates a BCF feature map 70 for each target body part and uses it to associate reference keypoints 20 and target keypoints 30 that belong to the same person. This provides a new technique for keypoint association.

[0021] The keypoint association using the BCF feature map performed by the keypoint association device 2000 has the following advantages over the keypoint association performed in Non-Patent Document 1. Non-Patent Document 1 provides a concept called "PAF (Part Affinity Field)" to associate keypoints. The PAF is the region between two adjacent keypoints on a human body. Each pixel included in the PAF is associated with a unit vector pointing from one keypoint to the other. After generating a PAF feature map from the original image through a trained neural network, the integral of the vectors of each pixel in the PAF is called a prediction of the association between two keypoints.

[0022] In Non-Patent Document 1, PAF is defined only between adjacent keypoints. Due to this restriction, even a single error in associating adjacent keypoints can cause a fatal failure in keypoint association. For example, suppose two people P1 and P2 exist in the image to be analyzed, and the keypoints of the neck and right wrist, the keypoints of the right wrist and right knee, and the keypoints of the right knee and right foot are defined as keypoint pairs, respectively.

[0023] In this situation, due to the low quality of the PAF, if the neck keypoint of person P1 is associated with the right wrist keypoint of person P2, the neck keypoint of person P1 will not be associated with any of the keypoints of person P1. Specifically, the right wrist keypoint of person P2 may be associated with the right knee keypoint of person P2. Furthermore, the right knee keypoint of person P2 may be associated with the right foot keypoint of person P2. As a result, the neck keypoint of person P1, the right wrist keypoint of person P2, the right knee keypoint of person P2, and the right foot keypoint of person P2 are connected in this order.

[0024] On the other hand, since the BCF feature map 70 is generated for each target portion to represent the spatial relationship between the reference portion and the target portion, the keypoint association device 2000 can associate each target keypoint 30 with the reference keypoint 20. Therefore, one error in the association between the target keypoint 30 and the reference keypoint 20 does not result in further errors in the association between the target keypoint 30 and the reference keypoint 20. This means that the keypoint association device 2000 can perform keypoint association more accurately than the system of Non-Patent Document 1.

[0025] A more detailed description of the keypoint association device 2000 is provided below.

[0026] <Example of functional configuration> FIG. 3 is a block diagram illustrating an example of the functional configuration of a keypoint association device 2000 according to an embodiment. The keypoint association device 2000 includes an acquisition unit 2020, a keypoint detection unit 2040, a feature map generation unit 2060, and a keypoint association unit 2080. The acquisition unit 2020 acquires a target image 10. The keypoint detection unit 2040 detects one or more reference keypoints 20 and one or more target keypoints 30 from the target image 10. The feature map generation unit 2060 uses the target image 10 to generate a BCF feature map 70 for each target region, the BCF map including a BCF region for each reference region included in the target image 10. The keypoint association unit 2080 uses the BCF feature map to associate a reference keypoint 20 with a target keypoint 30 belonging to the same person as the reference keypoint 20.

[0027] <Example of hardware configuration> The keypoint association device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer manufactured for realizing the keypoint association device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0028] The keypoint association device 2000 may be realized by installing an application on one or more computers. The application is realized by a program that causes one or more computers to function as the keypoint association device 2000. In other words, the program is an implementation of the functional components of the keypoint association device 2000.

[0029] 4 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes the keypoint association device 2000. In FIG. 4, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0030] The bus 1020 is a data transmission path through which the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120 transmit and receive data to and from each other. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main memory element such as a random access memory (RAM) or a read-only memory (ROM). The storage device 1080 is an auxiliary memory element such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0031] The hardware configuration of computer 1000 is not limited to that shown in Fig. 4. For example, as described above, keypoint association device 2000 may be realized by multiple computers, which may be connected to each other via a network.

[0032] <Processing flow> 5 is a flowchart illustrating an example of the flow of processing executed by the keypoint association device 2000 according to the embodiment. The acquisition unit 2020 acquires a target image 10 (S102). The keypoint detection unit 2040 detects keypoints from the target image 10 (S104). The feature map generation unit 2060 generates a BCF feature map 70 for each target region (S106). For each reference keypoint 20, the keypoint association unit 2080 associates the reference keypoint 20 with target keypoints 30 belonging to the same person as the reference keypoint 20 (S108).

[0033] <Acquisition of target image 10: S102> The acquisition unit 2020 acquires the target image 10 (S102). There are various methods for acquiring the target image 10. In some embodiments, the target image 10 is stored in advance in a storage device in a manner that allows it to be acquired by the keypoint association device 2000. In this case, the acquisition unit 2020 may access the storage device to acquire the target image 10. In other embodiments, the target image 10 is transmitted by another device, such as a camera that generated the target image 10. In this case, the acquisition unit 2020 may acquire the target image 10 by receiving it.

[0034] In some embodiments, the target image 10 may be one of a time series of images (e.g., a time series of video frames constituting a video). In this case, the keypoint association device 2000 may acquire all or part of the time series of images as the target image 10 and perform keypoint detection and keypoint association for each target image 10.

[0035] <Keypoint detection: S104> The keypoint detection unit 2040 detects the reference keypoints 20 and the target keypoints 30 from the target image 10 (S104). There are various methods for detecting one or more positions of a predetermined part of a human body as keypoints from an image, and the keypoint detection unit 2040 can detect the reference keypoints 20 and the target keypoints 30 from the target image 10 by using one of these methods.

[0036] In some embodiments, the keypoint detection unit 2040 has a machine learning-based model (e.g., a neural network), which is configured to acquire an image as an input, and is pre-trained to detect one or more reference keypoints 20 and one or more target keypoints 30 for each target part from the input image in response to the input of the input image. Hereinafter, this model is referred to as a "keypoint detection model".

[0037] The keypoint detection model acquires the target image 10 as an input, extracts feature amounts from the target image 10, and based on the extracted feature amounts, detects one or more positions of each predetermined part (reference part and target part) of a human body, and outputs a pair of a position and a label as a keypoint. The label of the keypoint indicates which part of the human body is indicated by the keypoint. In this case, the keypoint detection model may have a first model pre-trained to extract feature amounts from the target image 10, and a second model pre-trained to detect one or more positions of each predetermined part of a human body based on the feature amounts extracted by the first model. Each of the first model and the second model may be configured as a machine learning-based model such as a neural network. Here, there are various types of machine learning-based models that can detect keypoints from an input image, and the keypoint detection model can be configured as one of these models.

[0038] <Generation of BCF feature map: S106> For each predetermined target region, the feature map generator 2060 generates a BCF feature map 70 (S106). As described above, the BCF feature map 70 for a particular target region includes, for each reference region, a BCF region 80 that connects the reference region and the target region that belong to the same person. The BCF feature map 70 may be image data with dimensions (i.e., height and width) that are the same as the dimensions of the target image 10. The values ​​of pixels within the BCF region 80 are set to be different (e.g., larger) than the values ​​of pixels outside the BCF region 80. For example, the values ​​of pixels within the BCF region 80 may be set to 1, while the values ​​of pixels outside the BCF region 80 may be set to 0.

[0039] To generate the BCF feature map 70, the feature map generation unit 2060 may have a machine learning-based model called a “feature map generation model” for each predetermined target region. The feature map generation model for a specific target region is configured to acquire images and is pre-trained to generate a BCF feature map 70 for the target region in response to an input image. If the values ​​of pixels in a BCF region are defined to be greater than the values ​​of pixels outside the BCF region, the feature map generation model for a specific target region generates the BCF feature map 70 for that target region such that the pixel values ​​of pixels that are more likely to be included in the BCF region 80 of that target region are greater. The feature map generation unit 2060 may obtain the BCF feature map 70 for each target region from the corresponding feature map generation model by inputting the target image 10 to each feature map generation model.

[0040] The feature map generation model is trained using multiple training datasets that include training input images and ground truth BCF feature maps. The training input images are image data that capture one or more people, similar to the target image 10. The ground truth BCF feature map is an ideal BCF feature map that should be output from the trained feature map generation model in response to the input of the corresponding training input image. A training dataset is prepared for each target body part.

[0041] The ground truth BCF feature map may be generated in advance by an administrator or the like of the keypoint association device 2000. For example, the administrator or the like operates a computer called a "dataset generator" to display training images on a display device. The administrator or the like specifies the type of target body part for which the BCF feature map 70 is to be generated. Furthermore, for each person included in the training input images, the administrator or the like specifies the positions of the reference body part and the target body part belonging to that person. Based on the specification of one or more pairs of reference body part and target body part, the dataset generator generates the BCF feature map 70 for the selected target body part.

[0042] Specifically, the dataset generator may generate the BCF feature map 70 so that the BCF feature map 70 has the same dimensions as the training input images and has pixels with a predetermined first value (e.g., zero) indicating that the corresponding pixel is outside the BCF region 80. Additionally, the dataset generator may identify one or more BCF regions 80 based on one or more designations of reference and target region pairs, and set values ​​of pixels within the BCF region 80 to a predetermined second value (e.g., one) indicating that the corresponding pixel is within the BCF region 80.

[0043] The BCF region 80 may be drawn in a predetermined shape (e.g., a rectangle or an ellipse), where the width of the BCF region 80 (i.e., the length in a direction perpendicular to the direction from the reference site to the target site) may be fixed or may be dynamically determined based on (e.g., proportional to) the distance between the reference site and the target site.

[0044] The feature map generation model may be trained by the keypoint association device 2000 or another device. Hereinafter, a device that trains a feature map generation model is referred to as a "training device." In some embodiments, a feature map generation model for a particular target area may be trained as follows: The training device selects one of the training datasets for the target area, inputs training input images from the selected training dataset into the feature map generation model for that target area, and obtains an output therefrom. The training device then calculates a loss by applying the obtained output and the ground truth BCF feature map of the selected training dataset to a predetermined loss function. The training device updates the trainable parameters (e.g., weights and biases of a neural network) of the feature map generation model for the target area. part The feature map generation model is trained by repeatedly performing the above process.

[0045] <Keypoint Association: S108> The keypoint association unit 2080 associates the base keypoint 20 with the target keypoints 30 that belong to the same person as the base keypoint 20 (S108). In other words, the keypoint association unit 2080 generates a keypoint group for each base keypoint 20. Specifically, the keypoint association unit 2080 may initialize a keypoint group for each base keypoint 20. Furthermore, for each base keypoint 20, the keypoint association unit 2080 identifies target keypoints 30 that belong to the same person as the base keypoint 20, and assigns the identified target keypoints 30 to the keypoint group of the base keypoint 20.

[0046] As described above, the keypoint association unit 2080 uses the BCF feature map 70 for keypoint association. The BCF feature map 70 can be used as follows. FIG. 6 is a flowchart illustrating an example of the processing flow for keypoint association performed by the keypoint association unit 2080. Steps S202 to S218 constitute a loop process L1 that is executed for each base keypoint 20. In step S202, the keypoint association unit 2080 determines whether the loop process L1 has already been executed for all base keypoints 20. If the loop process L1 has already been executed for all base keypoints 20, the keypoint association unit 2080 terminates keypoint association. On the other hand, if the loop process L1 has not yet been executed for all base keypoints, the keypoint association unit 2080 selects one of the base keypoints 20 for which the loop process L1 has not yet been executed. Hereinafter, the base keypoint 20 selected here will be referred to as "base keypoint B."

[0047] Steps S204 to S216 constitute a loop L2 that is executed for each target body part. Each execution of the loop L2 in one iteration of the loop L1 causes the keypoint association unit 2080 to identify target keypoints 30 that belong to the same person as the reference keypoint B corresponding to that iteration of the loop L1.

[0048] In step S204, the keypoint association unit 2080 determines whether the loop process L2 has already been executed for all target regions in the current iteration of the loop process L1. If the loop process L2 has already been executed for all target regions in the current iteration of the loop process L1, the keypoint association unit 2080 ends the loop process L2 in the current iteration of the loop process L1. Furthermore, the keypoint association unit 2080 ends the current iteration of the loop process L1 (S218), thereby proceeding to the next iteration of the loop process L1 (S202).

[0049] On the other hand, if the loop process L2 has not yet been executed for all target parts in the current iteration of the loop process L1, the keypoint association unit 2080 selects one of the target parts for which the loop process L2 has not yet been executed in the current iteration of the loop process L1. Hereinafter, the target part selected here will be referred to as "target part P."

[0050] The keypoint association unit 2080 generates a candidate link representing a line connecting each target keypoint 30 corresponding to the target portion P with the reference keypoint B (S206). The keypoint association unit 2080 generates a midpoint for each candidate link (S208). The midpoint of a particular candidate link may be a point that divides the candidate link into multiple lines each having the same length. The number of midpoints in one candidate link may be predetermined.

[0051] Figure 7 is a diagram showing candidate links and their midpoints. The target image 10 shown in Figure 7 includes two people, 40-1 and 40-2. The reference keypoint B is the reference keypoint 20-1 (i.e., the neck keypoint of the person 40-1). The target part P is the right knee.

[0052] In this example, the keypoint association unit 2080 generates two candidate links (candidate link 100-1 connecting reference keypoint B (reference keypoint 20-1) to target keypoint 30-1, which is the keypoint of the right knee of person 40-1, and candidate link 100-2 connecting reference keypoint B to target keypoint 30-2, which is the keypoint of the right knee of person 40-2).

[0053] The keypoint associating unit 2080 generates three midpoints for each candidate link: candidate link 100-1 has midpoints 110-1 to 110-3, and candidate link 100-2 has midpoints 110-4 to 110-6.

[0054] The keypoint association unit 2080 calculates a BCF score for each midpoint (S210). Specifically, the BCF score for a particular midpoint is the pixel value in the BCF feature map 70 of the target region P at the same coordinate as the midpoint. For example, if the target image 10 has a midpoint at (x1, y1), the BCF score for that midpoint is obtained from the pixel at (x1, y1) in the BCF feature map 70 of the target region P.

[0055] Based on the BCF scores calculated for the waypoints, the keypoint association unit 2080 identifies a target link that connects the base keypoint B to a target keypoint 30 that belongs to the same person as the base keypoint B (S212). Furthermore, the keypoint association unit 2080 assigns the target keypoint 30 of the target link to the keypoint group of the base keypoint B (S214). Since S216 is the end of the loop process L2, the keypoint association unit 2080 ends the current iteration of the loop process L2 and moves on to the next iteration of the loop process L2 (S204).

[0056] At S212, the keypoint associating unit 2080 may calculate the sum of the BCF scores of the midpoints on the candidate links (hereinafter referred to as the "total BCF score") for each candidate link, and then identify the candidate link with the highest total BCF score as the target link.

[0057] Suppose there are two candidate links, L1 and L2. Candidate link L1 contains three intermediate points (I11 with a BCF score of S11, I12 with a BCF score of S12, and I13 with a BCF score of S13). Candidate link L2 contains three intermediate points (I21 with a BCF score of S21, I22 with a BCF score of S22, and I23 with a BCF score of S23). 2In this case, the total BCF score TS1 of candidate link L1 is S11+S12+S13, and the total BCF score TS2 of candidate link L2 is S21+S22+S23.

[0058] If TS1 is greater than TS2, the keypoint associating unit 2080 identifies the candidate link L1 as the target link, whereas if TS2 is greater than TS1, the keypoint associating unit 2080 identifies the candidate link L2 as the target link.

[0059] A lower threshold may be set for the total BCF score. In this case, the keypoint association unit 2080 determines whether the maximum BCF score is greater than the lower threshold. If the maximum total BCF score is equal to or greater than the lower threshold, the keypoint association unit 2080 identifies the candidate link with the maximum total BCF score as the target link. On the other hand, if the maximum total BCF score is not greater than the lower threshold, the keypoint association unit 2080 determines that there is no candidate link that should be identified as the target link. In this case, none of the target keypoints 30 in the target portion P are assigned to the keypoint group of the reference keypoint B.

[0060] The keypoint associating unit 2080 may further consider the variability of the BCF scores of the candidate links. In this case, an upper threshold for the variability of the BCF scores is predetermined. The keypoint associating unit 2080 may identify one or more candidate links whose total BCF score is equal to or greater than a lower threshold for the total BCF score and whose variability of the BCF scores is equal to or less than an upper threshold for the variability of the BCF scores. Then, the keypoint associating unit 2080 may select the candidate link with the largest total BCF score as the target link from among the identified candidate links.

[0061] Note that the appropriate value of the lower threshold of the total BCF score may depend on the number of intermediate points. Therefore, the keypoint association unit 2080 may use the average BCF score instead of the total BCF score. In this case, the lower threshold of the average BCF score is used instead of the lower threshold of the total BCF score.

[0062] <Output from Keypoint Associator 2000> The keypoint association device 2000 may be configured to output information (called output information) indicating the result of the keypoint association. For example, the output information includes an identifier (e.g., a frame number) of the target image 10 and keypoint information. For each keypoint group, the keypoint information includes an identifier of the keypoint group and keypoint information for each keypoint included in the keypoint group. The keypoint information indicates the identifier of the keypoint, the position indicated by the keypoint, and the identifier of the part of the human body indicated by the keypoint.

[0063] The output information can be output in a variety of ways, and in some embodiments, the output information is stored in a storage device, displayed on a display device, or transmitted to another computer, such as a PC or smartphone of a user of the keypoint association device 2000.

[0064] <How to use key point groups> The results of keypoint association (i.e., keypoint groups) can be used in various ways. For example, keypoint groups can be used for pose estimation. As a result of pose estimation, for each keypoint group, the type of pose assumed by the person corresponding to that keypoint group is estimated.

[0065] Furthermore, by performing pose estimation for each target image 10 in the time series data (e.g., video frames in a video), a time series of poses is obtained for each person captured in the target image 10. The time series of a person's poses can be used to identify actions or a time series of actions taken by that person.

[0066] The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and RAMs). The program may also be provided to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0067] Although the present disclosure has been described with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various modifications that are understandable to those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the invention. [Explanation of symbols]

[0068] 10 Target image 20 Reference Key Points 30 Key Points of Interest 40 people 50 necks 60 Right knee 70 BCF feature maps 80 BCF area 100 candidate links 110 Midpoint 1000 computers 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 2000 Keypoint Associator 2020 Acquisition Department 2040 Keypoint Detector 2060 Feature map generation unit 2080 Keypoint Association

Claims

1. An acquisition unit that acquires a target image in which one or more people are captured; a keypoint detection unit that detects, for each person, a reference keypoint and one or more target keypoints from the target image; the reference keypoint of the person indicates a position of a reference part of the person; the target keypoints of the person indicate the positions of target parts of the person; the target site is different from the reference site; a feature map generation unit that generates a feature map of each target region based on the target image; the feature map of the target region indicates, for each of the reference regions in the target image, a region connecting the reference region to the target region belonging to the same person as the reference region; a keypoint association unit that associates the reference keypoint with one or more target keypoints that belong to the same person as the reference keypoint based on the feature map;

2. 2. The keypoint association apparatus of claim 1, wherein each pixel within the region of the feature map has a value greater than the value of pixels outside the region of the feature map.

3. The association between the reference keypoint and one or more of the target keypoints is as follows for each of the reference keypoints: For each of the target keypoints indicating the position of the target portion, generating a candidate link that is a straight line connecting the reference keypoint and the target keypoint; generating a plurality of intermediate points that divide the candidate link into a predetermined number of straight lines; For each of the intermediate points, obtain the value of the pixel at the position of the intermediate point from the feature map of the target area as a score of the intermediate point; identifying one of the candidate links as a target link based on one or more statistics of the scores of the midpoints of each of the candidate links; The keypoint associating device according to claim 2 , further comprising: associating the reference keypoint with the target keypoint of the target link for each target portion.

4. The keypoint associating apparatus according to claim 3 , wherein the target link is the candidate link having the largest sum or average of the scores of the waypoints among the candidate links.

5. The keypoint association device according to claim 4 , wherein the target link is the candidate link for which the sum or average of the scores of the waypoints is equal to or greater than a predetermined first threshold.

6. The keypoint association device according to claim 5 , wherein the candidate links are those for which the variance of the scores of the waypoints is equal to or less than a predetermined second threshold.

7. the keypoint association device has a machine learning-based model for each target region configured to receive the target image as an input and output the feature map for the target region in response to the input target image; The generation of the feature map of the target area includes: inputting the target image into the model of the target region; and obtaining the feature map of the target body part output from the model of the target body part.

8. an acquisition unit step of acquiring a target image in which one or more people are captured; a keypoint detection step of detecting, for each person, a reference keypoint and one or more target keypoints from the target image; the reference keypoint of the person indicates a position of a reference part of the person; the target keypoints of the person indicate the positions of target parts of the person; the target site is different from the reference site; a feature map generation unit that generates a feature map of each target region based on the target image; the feature map of the target region indicates, for each of the reference regions in the target image, a region connecting the reference region to the target region belonging to the same person as the reference region; a keypoint associating step of associating the reference keypoint with one or more target keypoints that belong to the same person as the reference keypoint based on the feature map;

9. 9. The method of claim 8, wherein each pixel within the region of the feature map has a value greater than the value of pixels outside the region of the feature map.

10. an acquisition step of acquiring a target image in which one or more people are captured; a keypoint detection step of detecting, for each person, a reference keypoint and one or more target keypoints from the target image; the reference keypoint of the person indicates a position of a reference part of the person; the target keypoints of the person indicate the positions of target parts of the person; the target site is different from the reference site; causing the computer to execute a feature map generation step of generating a feature map for each target region based on the target image; the feature map of the target region indicates, for each reference region in the target image, a region connecting the reference region to the target region belonging to the same person as the reference region; a program that causes the computer to execute a keypoint associating step of associating the reference keypoint with one or more target keypoints that belong to the same person as the reference keypoint, based on the feature map;

Images