Keypoint Association Device, Keypoint Association Method, and Program

Abstract

The key point association device (20000) acquires a target image (10) in which one or more persons are imaged, and for each person, detects a reference key point (20) and one or more target key points (30) from the target image (10). The reference key point (20) of a person indicates the position of the reference part of that person. The target key point (30) of a person indicates the position of the target part of that person. The key point association device (2000) generates a feature map of each target part based on the target image (10). The feature map of the target part indicates a region connecting the reference part and the target part belonging to the same person as the reference part. The key point association device (2000) associates the reference key point (20) with the target key point (30) belonging to the same person as the reference key point (20) based on the feature map.

Description

Technical Field

[0001] The present disclosure relates generally to a keypoint association device, a keypoint association method, and a non-transitory computer-readable storage medium.

Background Art

[0002] There are various types of analyses performed on images in which one or more persons are imaged. Some of those analyses (e.g., pose estimation) use keypoints of a person (e.g., body joints). Specifically, keypoints are detected from an image and divided into groups such that the keypoints belonging to the same person are included in respective groups. This process of dividing keypoints into groups is called "keypoint association". Non-Patent Document 1 discloses one of the algorithms for keypoint association.

Prior Art Documents

Non-Patent Documents

[0003]

Non-Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In Non-Patent Document 1, it is necessary to define adjacent body parts in advance. For example, the neck and the right wrist, the right wrist and the right knee, and the right knee and the right foot can be defined as adjacent body parts, respectively. One of the objectives of the present disclosure is to disclose a novel technique for associating key points.

Means for Solving the Problem

[0005] The present disclosure discloses a key point association device having at least one memory element in which instructions are stored and at least one processor. By executing the instructions, the at least one processor acquires a target image in which one or more persons are imaged, and for each person, detects a reference key point and one or more target key points from the target image. The reference key point of the person indicates the position of the reference part of the person, and the target key point of the person indicates the position of the target part of the person. The target part is different from the reference part. A feature map of each target part is generated based on the target image. The feature map of the target part indicates a region connecting the reference part to the target part belonging to the same person as the reference part for each reference part in the target image. Based on the feature map, the reference key point is associated with one or more target key points belonging to the same person as the reference key point.

[0006] The present disclosure further discloses a key point association method executed by a computer. The keypoint association method includes obtaining a target image in which one or more persons are imaged, detecting, for each person, a reference keypoint and one or more target keypoints from the target image, the reference keypoint of the person indicating the position of a reference part of the person, the target keypoint of the person indicating the position of a target part of the person, the target part being different from the reference part, generating a feature map of each target part based on the target image, the feature map of the target part indicating, for each reference part in the target image, a region connecting the reference part to the target part belonging to the same person as the reference part, and associating the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint based on the feature map.

[0007] The present disclosure further discloses a non-transitory computer-readable storage medium storing a program. The program causes a computer to obtain a target image in which one or more persons are imaged, detect, for each person, a reference keypoint and one or more target keypoints from the target image, the reference keypoint of the person indicating the position of a reference part of the person, the target keypoint of the person indicating the position of a target part of the person, the target part being different from the reference part, generate a feature map of each target part based on the target image, the feature map of the target part indicating, for each reference part in the target image, a region connecting the reference part to the target part belonging to the same person as the reference part, and associate the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint.

Advantages of the Invention

[0008] According to the present disclosure, a novel technique for keypoint association is provided.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

[0010] Embodiments according to the present disclosure will be described below with reference to the drawings. The same reference numerals are assigned to the same elements throughout the drawings, and redundant descriptions will be omitted as necessary. Also, predetermined information (for example, a predetermined value or a predetermined threshold) is stored in advance in a storage device accessible by a computer that uses the information, unless otherwise specified.

[0011] <Overview> FIG. 1 is a diagram showing an overview of the keypoint association device 2000 according to the embodiment. Here, the overview shown in FIG. 1 is an example of the operation of the keypoint association device 2000 for the purpose of facilitating understanding of the keypoint association device 2000, and does not limit or narrow the range of operations that the keypoint association device 2000 can perform.

[0012] The keypoint association device 2000 acquires a target image 10 in which one or more persons are imaged, detects keypoints from the target image 10, and performs keypoint association for the detected keypoints. The target image 10 is any type of image data (e.g., RGB image or grayscale image) imaged in a manner visible to a person. A keypoint can indicate a characteristic point (e.g., a joint) of a person's body.

[0013] The keypoints belonging to a specific person include a reference keypoint 20 and one or more target keypoints 30. The reference keypoint 20 of a specific person indicates the position of a predetermined reference part of that person (i.e., the coordinates on the target image 10), while the target keypoints 30 of a specific person indicate the positions of different predetermined target parts of that person. The reference part can be a representative one of the characteristic parts of a person's body, such as the neck. The target part can be a characteristic part of a person's body other than the reference part, such as the right eye or the left shoulder.

[0014] For example, assuming that the reference part is the neck and the target parts include 16 parts of a person's body (right eye, right ear, right shoulder, right elbow, right hand, right wrist, right knee, right foot, left eye, left ear, left shoulder, left elbow, left hand, left wrist, left knee, and left foot). In this case, the keypoint association device 2000 detects, for each person in the target image 10, the point of the neck as the reference keypoint 20 and the points of these 16 target parts as the target keypoints 30.

[0015] After detecting the keypoints, the keypoint association device 2000 performs keypoint association. Keypoint association is a process of associating each reference keypoint 20 detected from the target image 10 with the target keypoints 30 belonging to the same person as that reference keypoint 20. In other words, keypoint association is a process of creating a group of keypoints belonging to each person. Hereinafter, a group of keypoints belonging to the same person is called a "keypoint group".

[0016] In the keypoint association process, the keypoint association device 2000 analyzes the target image 10 and generates a map called the "BCF (Body Crosscutting Field) feature map" for each target part. For example, when the above-mentioned 16 target parts are defined, the keypoint association device 2000 generates a BCF feature map for each of these 16 target parts, such as the BCF feature map of the right eye and the BCF feature map of the right shoulder.

[0017] The BCF feature map of a specific part shows the area (referred to as the "BCF area") that connects the reference part and the target part belonging to the same person for each reference part included in the target image 10. FIG. 2 shows an example of the BCF feature map. In this example, the neck is defined as the reference part. The target image 10 for which the BCF feature map 70 is generated includes three persons 40-1 to 40-3. The necks 50-1 to 50-3 are the reference parts of the persons 40-1 to 40-3 respectively.

[0018] In FIG. 2, the BCF feature map 70 of the right knee is generated. Therefore, the BCF feature map 70 shows the BCF area 80 that connects the neck 50 and the right knee 60 belonging to the same person for each neck 50 included in the target image 10. For example, the BCF area 80-1 connects the neck 50-1 and the right knee 60-1 belonging to the person 40-1.

[0019] The keypoint association device 2000 uses the BCF feature map to associate the reference keypoint 20 with the target keypoint 30 belonging to the same person as the reference keypoint 20. As a result of the keypoint association, the keypoint association device 2000 obtains a keypoint group that includes the reference keypoint 20 and the target keypoint 30 that are associated with each other for each reference keypoint 20. This means that the keypoint group includes the reference keypoint 20 and the target keypoint 30 belonging to the same person.

[0020] <Examples of effects> According to the keypoint association device 2000, a new concept called the "BCF feature map" is introduced into keypoint association. Specifically, the keypoint association device 2000 generates a BCF feature map 70 for each target part and uses them to associate the reference keypoint 20 and the target keypoint 30 belonging to the same person with each other. Therefore, a new technology for keypoint association is provided.

[0021] The keypoint association using the BCF feature map executed by the keypoint association device 2000 is advantageous as follows compared to the keypoint association performed in Non-Patent Document 1. Non-Patent Document 1 provides a concept called "PAF (Part Affinity Field)" for associating keypoints. PAF is the area between two adjacent keypoints in a person's body. Each pixel included in PAF is associated with a unit vector from one keypoint to the other. After generating the PAF feature map from the original image through a trained neural network, the integral of the vectors of each pixel in PAF is called the prediction of the association of the two keypoints.

[0022] In Non-Patent Document 1, PAF is defined only between adjacent keypoints. Due to this limitation, even one error in the association of adjacent keypoints can cause a fatal failure in keypoint association. For example, assume that two persons P1 and P2 exist in the image to be analyzed, and the keypoints of the neck and the right wrist, the right wrist and the right knee, and the right knee and the right foot are each a pair of defined keypoints.

[0023] In this situation, due to the low quality of the PAF, when the key point of person P1's neck is associated with the key point of person P2's right wrist, the key point of person P1's neck is not associated with any key point of person P1. Specifically, the key point of person P2's right wrist can be associated with the key point of person P2's right knee. Furthermore, the key point of person P2's right knee can be associated with the key point of person P2's right foot. As a result, the key point of person P1's neck, the key point of person P2's right wrist, the key point of person P2's right knee, and the key point of person P2's right foot are connected in this order.

[0024] On the other hand, since the BCF feature map 70 is generated for each target part so as to represent the spatial relationship between the reference part and the target part, the key point association device 2000 associates the target key points 30 with the reference key points 20 individually. Therefore, one error in the association between the target key point 30 and the reference key point 20 does not cause further errors in the association between the target key point 30 and the reference key point 20. This means that the key point association device 2000 can perform key point association more accurately compared to the system of Non-Patent Document 1.

[0025] A more detailed description of the key point association device 2000 is described below.

[0026] <Example of Functional Configuration> FIG. 3 is a block diagram showing an example of the functional configuration of the keypoint association device 2000 according to the embodiment. The keypoint association device 2000 includes an acquisition unit 2020, a keypoint detection unit 2040, a feature map generation unit 2060, and a keypoint association unit 2080. The acquisition unit 2020 acquires the target image 10. The keypoint detection unit 2040 detects one or more reference keypoints 20 and one or more target keypoints 30 from the target image 10. The feature map generation unit 2060 generates a BCF feature map 70 including a BCF region for each reference part included in the target image 10 for each target part using the target image 10. The keypoint association unit 2080 associates the reference keypoint 20 with a target keypoint 30 belonging to the same person as the reference keypoint 20 using the BCF feature map.

[0027] <Example of hardware configuration> The keypoint association device 2000 may be realized by one or more computers. Each of the one or more computers may be a dedicated computer manufactured to realize the keypoint association device 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0028] The keypoint association device 2000 may also be realized by installing an application on one or more computers. The application is realized by a program for causing the one or more computers to function as the keypoint association device 2000. In other words, the program is implemented with the functional components of the keypoint association device 2000.

[0029] FIG. 4 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes the keypoint association device 2000. In FIG. 4, the computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0030] The bus 1020 is a data transmission path for the processor 1040, the memory 1060, the storage device 1080, the input / output interface 1100, and the network interface 1120 to transmit and receive data to and from each other. The processor 1040 is a processor such as a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a DSP (Digital Signal Processor), or an FPGA (Field-Programmable Gate Array). The memory 1060 is a main memory element such as a RAM (Random Access Memory) or a ROM (Read Only Memory). The storage device 1080 is an auxiliary memory element such as a hard disk, an SSD (Solid State Drive), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices (such as a keyboard, a mouse, or a display device). The network interface 1120 is an interface between the computer 1000 and a network. The network may be a LAN (Local Area Network) or a WAN (Wide Area Network).

[0031] The hardware configuration of the computer 1000 is not limited to that shown in FIG. 4. For example, as described above, the keypoint association device 2000 may be realized by a plurality of computers. In this case, those computers may be connected to each other via a network.

[0032] <Processing Flow> FIG. 5 is a flowchart showing an example of the processing flow executed by the keypoint association device 2000 according to the embodiment. The acquisition unit 2020 acquires the target image 10 (S102). The keypoint detection unit 2040 detects keypoints from the target image 10 (S104). The feature map generation unit 2060 generates a BCF feature map 70 for each target part (S106). For each reference keypoint 20, the keypoint association unit 2080 associates the reference keypoint 20 with a target keypoint 30 belonging to the same person as the reference keypoint 20 (S108).

[0033] <Acquisition of Target Image 10: S102> The acquisition unit 2020 acquires the target image 10 (S102). There are various methods for acquiring the target image 10. In some embodiments, the target image 10 is stored in advance in a storage device in a manner that can be acquired from the keypoint association device 2000. In this case, the acquisition unit 2020 can access the storage device to acquire the target image 10. In other embodiments, the target image 10 is transmitted by another device such as a camera that generated the target image 10. In this case, the acquisition unit 2020 can acquire the target image 10 by receiving it.

[0034] In some embodiments, the target image 10 can be one of a series of images (e.g., the time-series video frames constituting a video). In this case, the keypoint association device 2000 can acquire all or part of the time-series images as the target image 10 and perform keypoint detection and keypoint association for each target image 10.

[0035] <Detection of Keypoints: S104> The keypoint detection unit 2040 detects the reference keypoints 20 and the target keypoints 30 from the target image 10 (S104). There are various methods for detecting one or more positions of a predetermined part of a human body as keypoints from an image, and the keypoint detection unit 2040 can detect the reference keypoints 20 and the target keypoints 30 from the target image 10 using one of these methods.

[0036] In some embodiments, the keypoint detection unit 2040 has a machine learning-based model (e.g., a neural network), which is configured to acquire an image as an input, and is pre-trained to detect one or more reference keypoints 20 and one or more target keypoints 30 for each target part from the input image in response to the input image being input. Hereinafter, this model is referred to as the "keypoint detection model".

[0037] The keypoint detection model acquires the target image 10 as an input, extracts feature amounts from the target image 10, and based on the extracted feature amounts, detects one or more positions of each predetermined part (reference part and target part) of a human body, and outputs a pair of a position and a label as a keypoint. The label of the keypoint indicates which part of the human body is indicated by the keypoint. In this case, the keypoint detection model may have a first model pre-trained to extract feature amounts from the target image 10 and a second model pre-trained to detect one or more positions of each predetermined part of a human body based on the feature amounts extracted by the first model. Each of the first model and the second model may be configured as a machine learning-based model such as a neural network. Here, there are various types of machine learning-based models that can detect keypoints from an input image, and the keypoint detection model can be configured as one of these models.

[0038] <Generation of BCF feature map: S106> For each of the specified target parts, the feature map generation unit 2060 generates a BCF feature map 70 (S106). As described above, the BCF feature map 70 of a specific target part includes a BCF region 80 that connects the reference part and the target part belonging to the same person for each reference part. The BCF feature map 70 can be image data having the same dimensions (i.e., height and width) as the target image 10. The values of the pixels within the BCF region 80 are set to be different (e.g., larger) from the values of the pixels outside the BCF region 80. For example, the values of the pixels within the BCF region 80 can be set to 1 while the values of the pixels outside the BCF region 80 can be set to 0.

[0039] To generate the BCF feature map 70, the feature map generation unit 2060 can have a machine learning-based model called a "feature map generation model" for each predetermined target part. The feature map generation model for a specific target part is configured to acquire an image and is pre-trained to generate a BCF feature map 70 for the target part in response to the input of the input image. When the values of the pixels in the BCF region are defined to be larger than the values of the pixels outside the BCF region, the feature map generation model for a specific target part generates the BCF feature map 70 of that target part such that the pixel values are larger for the pixels with a higher probability of being included in the BCF region 80 of that target part. The feature map generation unit 2060 can obtain the BCF feature map 70 of each target part from the corresponding feature map generation model by inputting the target image 10 into each feature map generation model.

[0040] The feature map generation model is trained using a plurality of training datasets including training input images and ground truth BCF feature maps. The training input images are image data in which one or more persons are imaged, similar to the target image 10. The ground truth BCF feature map is an ideal BCF feature map that should be output from the trained feature map generation model in response to the corresponding training input image being input. The training datasets are prepared for each target part.

[0041] The ground truth BCF feature map can be pre-generated by an administrator or the like of the keypoint association device 2000. For example, an administrator or the like operates a computer called a "dataset generation device" to display a training image on a display device. The administrator or the like specifies the type of target part for which the BCF feature map 70 is to be generated. Further, for each person included in the training input image, the administrator or the like specifies the positions of the reference part and the target part belonging to that person. Based on one or more specifications of pairs of the reference part and the target part, the dataset generation device generates the BCF feature map 70 of the selected target part.

[0042] Specifically, the dataset generation device can generate the BCF feature map 70 such that the BCF feature map 70 has the same dimensions as the training input image and has pixels of a predetermined first value (for example, zero) indicating that the corresponding pixel is outside the BCF region 80. Further, the dataset generation device identifies one or more BCF regions 80 based on one or more specifications of pairs of the reference part and the target part, and can set a predetermined second value (for example, 1) indicating that the corresponding pixel is within the BCF region 80 as the value of the pixel within the BCF region 80.

[0043] The BCF region 80 can be drawn in a predetermined shape (for example, a rectangle or an ellipse). Here, the width of the BCF region 80 (that is, the length in the direction perpendicular to the direction from the reference part to the target part) may be determined by a fixed value, or may be dynamically determined based on the distance between the reference part and the target part (for example, proportionally).

[0044] The feature map generation model can be trained by the keypoint association device 2000 or other devices. Hereinafter, the device that trains the feature map generation model is referred to as the "training device". In some embodiments, the feature map generation model for a specific target part can be trained as follows. The training device selects one of the training data sets of the target part, inputs the training input image of the selected training data set into the feature map generation model of the target part, and then obtains the output. Further, the training device calculates the loss by applying the obtained output and the ground truth BCF feature map of the selected training data set to a predetermined loss function. The training device updates the trainable parameters (for example, the weights and biases of the neural network) of the feature map generation model of the target part. The feature map generation model of the target type is trained by repeatedly executing the above process.

[0045] <Keypoint Association: S108> The keypoint association unit 2080 associates the reference keypoint 20 with the target keypoint 30 belonging to the same person as the reference keypoint 20 (S108). In other words, the keypoint association unit 2080 generates a keypoint group for each reference keypoint 20. Specifically, the keypoint association unit 2080 can initialize a keypoint group for each reference keypoint 20. Further, for each reference keypoint 20, the keypoint association unit 2080 identifies the target keypoint 30 belonging to the same person as the reference keypoint 20, and assigns the identified target keypoint 30 to the keypoint group of the reference keypoint 20.

[0046] As described above, the keypoint association unit 2080 uses the BCF feature map 70 for keypoint association. The BCF feature map 70 can be used as follows. FIG. 6 is a flowchart showing an example of the process flow in which the keypoint association unit 2080 performs keypoint association. Steps S202 to S218 constitute a loop process L1 executed for each reference keypoint 20. In step S202, the keypoint association unit 2080 determines whether the loop process L1 has already been executed for all the reference keypoints 20. If the loop process L1 has already been executed for all the reference keypoints 20, the keypoint association unit 2080 ends the keypoint association. On the other hand, if the loop process L1 has not yet been executed for all the reference keypoints, the keypoint association unit 2080 selects one of the reference keypoints 20 for which the loop process L1 has not yet been executed. Hereinafter, the reference keypoint 20 selected here is referred to as "reference keypoint B".

[0047] Steps S204 to S216 constitute a loop process L2 executed for each target part. By each execution of the loop process L2 in one iteration of the loop process L1, the keypoint association unit 2080 identifies the target keypoint 30 belonging to the same person as the reference keypoint B corresponding to that iteration of the loop process L1.

[0048] In step S204, the keypoint association unit 2080 determines whether the loop process L2 has already been executed for all the target parts in the current iteration of the loop process L1. If the loop process L2 has already been executed for all the target parts in the current iteration of the loop process L1, the keypoint association unit 2080 ends the loop process L2 in the current iteration of the loop process L1. Further, the keypoint association unit 2080 ends the current iteration of the loop process L1 (S218), thereby proceeding to the next iteration of the loop process L1 (S202).

[0049] On one hand, in the current iteration of the loop process L1, if the loop process L2 has not yet been executed for all target parts, the keypoint association unit 2080 selects one of the target parts for which the loop process L2 has not yet been executed in the current iteration of the loop process L1. Hereinafter, the target part selected here is referred to as "target part P".

[0050] For each target keypoint 30 corresponding to the target part P, the keypoint association unit 2080 generates a candidate link representing a line connecting the target keypoint 30 and the reference keypoint B (S206). The keypoint association unit 2080 generates an intermediate point for each candidate link (S208). The intermediate point of a specific candidate link can be a point that divides the candidate link into a plurality of lines of the same length. The number of intermediate points in one candidate link can be predetermined.

[0051] FIG. 7 is a diagram showing a candidate link and its intermediate points. In the target image 10 shown in FIG. 7, there are two people, namely, persons 40-1 and 40-2. The reference keypoint B is the reference keypoint 20-1 (i.e., the keypoint of the neck of person 40-1). The target part P is the right knee.

[0052] In this example, the keypoint association unit 2080 generates two candidate links (a candidate link 100-1 connecting the reference keypoint B (reference keypoint 20-1) to the target keypoint 30-1 which is the keypoint of the right knee of person 40-1, and a candidate link 100-2 connecting the reference keypoint B to the target keypoint 30-2 which is the keypoint of the right knee of person 40-2).

[0053] The keypoint association unit 2080 generates three intermediate points for each candidate link. Specifically, the candidate link 100-1 has intermediate points 110-1 to 110-3, and the candidate link 100-2 has intermediate points 110-4 to 110-6.

[0054] The keypoint association unit 2080 calculates a BCF score for each intermediate point (S210). Specifically, the BCF score of a specific intermediate point is the pixel value of the BCF feature map 70 of the target part P at the same coordinates as that intermediate point. For example, if there is an intermediate point at (x1, y1) in the target image 10, the BCF score of that intermediate point is obtained from the pixel at (x1, y1) of the BCF feature map 70 of the target part P.

[0055] Based on the BCF scores calculated for the intermediate points, the keypoint association unit 2080 identifies a target link that connects the reference keypoint B to the target keypoint 30 belonging to the same person as that reference keypoint B (S212). Further, the keypoint association unit 2080 assigns the target keypoint 30 of the target link to the keypoint group of the reference keypoint B (S214). Since S216 is the end of the loop process L2, the keypoint association unit 2080 ends the current iteration of the loop process L2 and moves to the next iteration of the loop process L2 (S204).

[0056] In S212, the keypoint association unit 2080 can calculate the total value of the BCF scores of the intermediate points on the candidate link (hereinafter referred to as the "total BCF score") for each candidate link. Then, the keypoint association unit 2080 identifies the candidate link with the largest total BCF score as the target link.

[0057] Suppose there are two candidate links, L1 and L2. Candidate link L1 includes three intermediate points (I11 with a BCF score of S11, I12 with a BCF score of S12, and I13 with a BCF score of S13). Candidate link L2 includes three intermediate points (I21 with a BCF score of S21, I22 with a BCF score of S22, and I13 with a BCF score of S23). In this case, the total BCF score TS1 of candidate link L1 is S11 + S12 + S13, and the total BCF score TS2 of candidate link L2 is S21 + S22 + S23.

[0058] If TS1 is greater than TS2, the keypoint association unit 2080 identifies candidate link L1 as the target link. On the other hand, if TS2 is greater than TS1, the keypoint association unit 2080 identifies candidate link L2 as the target link.

[0059] Note that a lower threshold value may be defined for the total BCF score. In this case, the keypoint association unit 2080 determines whether the maximum BCF score is greater than the lower threshold value. If the maximum total BCF score is equal to or greater than the lower threshold value, the keypoint association unit 2080 identifies the candidate link with the maximum total BCF score as the target link. On the other hand, if the maximum total BCF score is not greater than the lower threshold value, the keypoint association unit 2080 determines that there is no candidate link to be identified as the target link. In this case, none of the target keypoints 30 of the target part P are assigned to the keypoint group of the reference keypoint B.

[0060] The keypoint association unit 2080 may further consider the variation in the BCF scores of the candidate links. In this case, an upper threshold value for the variation in the BCF scores is predetermined. The keypoint association unit 2080 identifies one or more candidate links whose total BCF score is equal to or greater than the lower threshold value of the total BCF score and whose variation in the BCF score is equal to or less than the upper threshold value of the variation in the BCF score. Then, the keypoint association unit 2080 may select, as the target link, the candidate link having the maximum total BCF score from among those identified candidate links.

[0061] Note that an appropriate value for the lower threshold value of the total BCF score may depend on the number of intermediate points. Therefore, the keypoint association unit 2080 may use the average value of the BCF scores instead of the total BCF score. In this case, instead of the lower threshold value of the total BCF score, the lower threshold value of the average value of the BCF scores is used.

[0062] <Output from the keypoint association device 2000> The keypoint association device 2000 may be configured to output information indicating the result of keypoint association (referred to as output information). For example, the output information includes an identifier (e.g., frame number) of the target image 10 and keypoint information. The keypoint information includes, for each keypoint group, an identifier of the keypoint group and keypoint information of each keypoint included in the keypoint group. The keypoint information indicates an identifier of the keypoint, a position indicated by the keypoint, and an identifier of a body part of a person indicated by the keypoint.

[0063] There are various methods for outputting the output information. In some embodiments, the output information is stored in a storage device, displayed on a display device, or transmitted to another computer such as the user's PC or smartphone of the keypoint association device 2000.

[0064] <Method of using the keypoint group> There are various ways to use the results of keypoint association (i.e., keypoint groups). For example, a keypoint group can be used for pose estimation. As a result of pose estimation, for each keypoint group, the type of pose taken by the person corresponding to that keypoint group is estimated.

[0065] Furthermore, by performing pose estimation for each target image 10 (e.g., video frame in a video) in the time-series data, a time series of poses can be obtained for each person captured in the target image 10. The time series of a person's pose can be used to identify the actions and the time series of actions taken by that person.

[0066] The program can be stored using various types of non-transitory computer readable media and provided to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer readable media include magnetic recording media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (e.g., magneto-optical disks), CD-ROM, CD-R, CD-R / W, semiconductor memories (e.g., mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM). Also, the program may be provided to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. Transitory computer readable media can supply the program to the computer via wired communication paths such as electric wires and optical fibers, or wireless communication paths.

[0067] Although the present disclosure has been described with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various changes that can be understood by those skilled in the art can be made to the configuration and details of the present disclosure within the scope of the invention.

Explanation of Signs

[0068] 10 Target image 20 Reference keypoints 30 Target keypoints 40 Person 50 Neck 60 Right knee 70 BCF feature map 80 BCF region 100 Candidate link 110 Intermediate point 1000 Computer 1020 Bus 1040 Processor 1060 Memory 1080 Storage device 1100 Input / output interface 1120 Network interface 2000 Keypoint association device 2020 Acquisition unit 2040 Keypoint detection unit 2060 Feature map generation unit 2080 Keypoint association unit

Claims

1. One or more memory elements configured to store commands, By executing the command, Obtain a target image in which one or more persons are imaged, For each person, from the target image, detect a reference keypoint and one or more target keypoints, the reference keypoint of the person indicating the position of the reference part of the person, and the target keypoint of the person indicating the position of the target part of the person, the target part being different from the reference part, Generate a feature map for each target part based on the target image, the feature map of the target part indicating, for each reference part in the target image, a region connecting the reference part to the target part belonging to the same person as the reference part, Based on the feature map, associate the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint, One or more processors configured as described above, and a keypoint association device having the same.

2. Each pixel in the region of the feature map has a value greater than the value of the pixel outside the region of the feature map. The keypoint association device according to claim 1.

3. The association of the reference keypoint with one or more target keypoints is, for each reference keypoint, For each target keypoint indicating the position of the target part, generating a candidate link that is a straight line connecting the reference keypoint and the target keypoint, Generating a plurality of intermediate points that divide the candidate link into a predetermined number of straight lines, For each intermediate point, obtaining, from the feature map of the target part, the value of the pixel at the position of the intermediate point as the score of the intermediate point, Identifying one of the candidate links as a target link based on one or more statistical values of the scores of the intermediate points of each candidate link, And associating the reference keypoint with the target keypoint of the target link, which is executed for each target part. The keypoint association device according to claim 2.

4. The target link is the candidate link in which the sum or average of the scores of the intermediate points is the largest among the candidate links. The keypoint association device according to claim 3.

5. The target link is the candidate link in which the sum or average of the scores of the intermediate points is equal to or greater than a predetermined first threshold value, according to the keypoint association device of claim 4.

6. The candidate link is the candidate link in which the variation of the scores of the intermediate points is equal to or less than a predetermined second threshold value, according to the keypoint association device of claim 5.

7. The at least one storage element stores, for each target part, a machine learning-based model configured to acquire the target image as an input and output the feature map of the target part in response to the input of the target image. The generation of the feature map of the target part includes inputting the target image into the model of the target part and acquiring the feature map of the target part output from the model of the target part, according to the keypoint association device of any one of claims 1 to 6.

8. Acquire a target image in which one or more persons are imaged. For each person, detect a reference keypoint and one or more target keypoints from the target image. The reference keypoint of the person indicates the position of the reference part of the person, and the target keypoint of the person indicates the position of the target part of the person. The target part is different from the reference part. Generate a feature map of each target part based on the target image. The feature map of the target part indicates a region connecting the reference part to the target part belonging to the same person as the reference part for each reference part in the target image. A keypoint association method executed by a computer, including associating the reference keypoint with one or more target keypoints belonging to the same person as the reference keypoint based on the feature map.

9. Each pixel in the region of the feature map has a value greater than the value of the pixel outside the region of the feature map, according to the keypoint association method of claim 8.

10. The association between the reference keypoint and one or more target keypoints is, for each reference keypoint, generating a candidate link, which is a straight line connecting the reference keypoint and the target keypoint, for each target keypoint indicating the position of the target part. generating a plurality of intermediate points that divide the candidate link into a predetermined number of straight lines. ​ For each intermediate point, obtaining, from the feature map of the target part, the value of the pixel at the position of the intermediate point as the score of the intermediate point; Identifying, based on one or more statistical values of the scores of the intermediate points of each candidate link, one of the candidate links as the target link; Associating the reference keypoint with the target keypoint of the target link, which is included in the keypoint association method according to claim 9, and is executed for each target part.

11. The keypoint association method according to claim 10, wherein the target link is the candidate link in which the sum or average of the scores of the intermediate points is the largest among the candidate links.

12. The keypoint association method according to claim 11, wherein the target link is the candidate link in which the sum or average of the scores of the intermediate points is equal to or greater than a predetermined first threshold.

13. The keypoint association method according to claim 12, wherein the candidate link is the candidate link in which the variation of the scores of the intermediate points is equal to or less than a predetermined second threshold.

14. The computer stores, for each target part, a machine learning-based model configured to obtain the target image as an input and output the feature map of the target part in response to the input of the target image. The generation of the feature map of the target part includes inputting the target image into the model of the target part; and obtaining the feature map of the target part output from the model of the target part, which is included in the keypoint association method according to any one of claims 8 to 13.

15. Obtaining a target image in which one or more persons are imaged; For each person, detecting a reference keypoint and one or more target keypoints from the target image, wherein the reference keypoint of the person indicates the position of the reference part of the person, and the target keypoint of the person indicates the position of the target part of the person, and the target part is different from the reference part; Generating a feature map of each target part based on the target image, wherein the feature map of the target part indicates, for each reference part in the target image, a region connecting the reference part to the target part belonging to the same person as the reference part. A non-transitory computer-readable storage medium storing a program that causes a computer to perform: associating the reference keypoints with one or more target keypoints belonging to the same person as the reference keypoints based on the feature map.

16. The storage medium according to claim 15, wherein each pixel in the region of the feature map has a value greater than the value of a pixel outside the region of the feature map.

17. The association of the reference keypoints with one or more target keypoints includes, for each reference keypoint, generating a candidate link, which is a straight line connecting the reference keypoint and the target keypoint, for each target keypoint indicating the position of the target part; generating a plurality of intermediate points that divide the candidate link into a predetermined number of straight lines; for each intermediate point, obtaining, from the feature map of the target part, the value of the pixel at the position of the intermediate point as the score of the intermediate point; identifying, based on one or more statistical values of the scores of the intermediate points of each candidate link, one of the candidate links as a target link; and associating the reference keypoint with the target keypoint of the target link, for each target part. The storage medium according to claim 16.

18. The storage medium according to claim 17, wherein the target link is the candidate link in which the sum or average of the scores of the intermediate points is the largest among the candidate links.

19. The storage medium according to claim 18, wherein the target link is the candidate link in which the sum or average of the scores of the intermediate points is equal to or greater than a predetermined first threshold.

20. The storage medium according to claim 19, wherein the candidate link is the candidate link in which the variation of the scores of the intermediate points is equal to or less than a predetermined second threshold.

21. The program includes, for each target part, a machine learning-based model configured to receive the target image as an input and output the feature map of the target part in response to the input of the target image. The generation of the feature map of the target part includes inputting the target image into the model of the target part; and obtaining the feature map of the target part output from the model of the target part. The storage medium according to any one of claims 15 to 20.

Images