Image processing device, image processing method, and program

Abstract

The keypoint association device (2000) acquires a target image (10) in which one or more people (80) are captured, detects keypoints (20) from the target image (10), and generates a spatial feature map (30) for each pair of body parts. The spatial feature map (30) includes a first direction region (32) for each keypoint (20) representing a first body part of the pair and a second direction region (32) for each keypoint (20) representing a second body part of the pair. The first direction region (32) and the second direction region (32) belonging to the same person (80) represent directions from the keypoint (20) corresponding to the first direction region (32) to the keypoint (20) corresponding to the second direction region (32). The keypoint association device (2000) generates a keypoint group (40) in which the keypoints (20) belong to the same person (80) for each person (80) captured in the target image (10).

Description

[Technical Field]

[0001] The present disclosure generally relates to a keypoint alignment apparatus, a keypoint alignment method, and a non-transitory computer-readable storage medium. [Background technology]

[0002] There are various types of analyses that can be performed on images of one or more people. Some of these analyses, such as pose estimation, use keypoints from people. Specifically, keypoints are detected from an image and then divided into groups, each containing keypoints that belong to the same person. This process of dividing keypoints into groups is called "keypoint correspondence."

[0003] Non-Patent Document 1 discloses one algorithm for keypoint matching. The system in Non-Patent Document 1 generates a feature map for each person from an input image for each pair of predetermined body parts, including a region called a Part Affinity Field (PAF) corresponding to the pair of body parts. The PAF corresponding to the pair of body parts connects two keypoints that represent the pair of body parts, and the two keypoints belong to the same person. The PAF is filled with pixel values ​​that represent the direction between the two keypoints. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Zhe Cao, Gines Hidalgo, Tomas Simon, Shih-En Wei, and Yaser Sheikh, "OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields," [online], December 18, 2018, arXiv, [Retrieved April 29, 2022],<URL:https: / / arxiv.org / pdf / 1812.08008.pdf> Summary of the Invention [Problem to be solved by the invention]

[0005] Since the PAF needs to connect two corresponding keypoints, it may include regions far from either of the two keypoints, for example, a region near the midpoint between the two keypoints. The purpose of this disclosure is to provide a novel technique for keypoint correspondence. [Means for solving the problem]

[0006] The present disclosure provides a keypoint alignment apparatus that includes at least one memory configured to store instructions and at least one processor configured to execute the instructions. The at least one processor executes the instructions to: acquire a target image in which one or more people are captured; detect keypoints from the target image for each of the people's body parts; generate a spatial feature map for each predetermined pair of the body parts using the target image; the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the keypoints representing a first body part of the pair and a second direction region corresponding to each of the keypoints representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the keypoint in the first direction region to the keypoint in the second direction region; and generate, for each of the people captured in the target image, a keypoint group including the keypoints belonging to the same person.

[0007] The keypoint correspondence method provided in this disclosure is computer-implemented. The keypoint matching method includes: acquiring a target image in which one or more persons are captured; detecting keypoints from the target image for each of the persons' body parts; generating a spatial feature map for each of predetermined pairs of the body parts using the target image; the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the keypoints representing a first body part of the pair and a second direction region corresponding to each of the keypoints representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing a direction from the keypoint in the first direction region to the keypoint in the second direction region; and generating a keypoint group for each of the persons captured in the target image, the keypoints including the keypoints belonging to the same person.

[0008] The present disclosure provides a non-transitory computer-readable storage medium for storing a program. The program causes a computer to perform the following steps: acquire a target image in which one or more people are captured; detect key points from the target image for each of the people's body parts; generate a spatial feature map for each predetermined pair of the body parts using the target image; the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing a direction from the key point in the first direction region to the key point in the second direction region; and generate a key point group for each of the people captured in the target image, the key point group including the key points belonging to the same person. [Effects of the Invention]

[0009] According to the present disclosure, a novel technique for point matching is provided. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an overview of a keypoint association device. [Figure 2] FIG. 2 is a diagram illustrating an example of a spatial feature map. [Figure 3] FIG. 3 is a block diagram illustrating an example of the functional configuration of the keypoint association device. [Figure 4] FIG. 4 is a block diagram showing an example of the hardware configuration of the keypoint association device. [Figure 5] FIG. 5 is a flowchart showing an example of the flow of processing executed by the keypoint association device. [Figure 6] FIG. 6 is a diagram illustrating an example of the configuration of the feature generation unit. [Figure 7] FIG. 7 is a diagram showing an example of a pair of horizontal and vertical spatial feature maps in which the directions between keypoints are represented in three-dimensional space. [Figure 8]FIG. 8 is a diagram illustrating an example of the configuration of the feature generation unit when the positions of the key points are expressed in three-dimensional coordinates. [Figure 9] FIG. 9 is a diagram showing an example of a method for keypoint association. DETAILED DESCRIPTION OF THE INVENTION

[0011] Examples of embodiments according to the present disclosure will be described below with reference to the drawings. Throughout the drawings, identical elements are designated by the same reference numerals, and duplicate descriptions will be omitted as necessary. Furthermore, unless otherwise specified, predetermined information (e.g., predetermined values ​​or predetermined threshold values) is pre-stored in a storage device accessible by a computer that uses the information.

[0012] <Summary> 1 is a diagram illustrating an overview of a keypoint matching device 2000 according to one embodiment. It should be noted that the overview illustrated in FIG. 1 shows an example of the operation of the keypoint matching device 2000 to facilitate understanding of the keypoint matching device 2000, and is not intended to limit or narrow the range of possible operations of the keypoint matching device 2000.

[0013] The keypoint association device 2000 acquires a target image 10 in which one or more people are captured, detects keypoints 20 from the target image 10, and performs keypoint association for the detected keypoints 20. The target image 10 may be any type of image data, such as an RGB image or a grayscale image, in which people can be visibly captured.

[0014] The key points 20 may indicate the positions of body parts of a person captured in the target image 10. The positions of the body parts may be represented by two-dimensional (2D) coordinates on the image plane of the target image 10 or three-dimensional (3D) coordinates in a predetermined three-dimensional (3D) space. The key point correspondence device 2000 is configured to detect one or more key points 20 from the target image 10 for each of the predetermined body parts. The predetermined body parts may include the neck, right and left eyes, right and left ears, right and left shoulders, right and left elbows, right and left wrists, waist, right and left knees, and right and left feet.

[0015] Keypoint association is a process that generates a group called a "keypoint group 40" for each person included in the target image 10. A keypoint group 40 for a particular person includes only keypoints 20 that belong to that particular person.

[0016] To generate keypoint groups 40 for each person, the keypoint correspondence apparatus 2000 generates spatial feature maps 30 for each predetermined pair of body parts based on the target image 10. The predetermined pair of body parts may include adjacent pairs of body parts, such as a pair of right eye and neck, a pair of neck and right shoulder, a pair of right shoulder and right elbow, and a pair of right elbow and right wrist. However, the body parts of a particular pair do not have to be adjacent to each other.

[0017] The spatial feature map 30 for a particular pair of body parts may be image data of the same dimensions as the target image 10, and may include regions called "directional regions" for each keypoint 20 representing one of the body parts of the particular pair. Directional regions belonging to the same person are generated to indicate the direction between the keypoints 20 (the direction from one keypoint 20 to the other). In one embodiment, different colors (i.e., pixel values) may be assigned to different directions. In this case, the directional region is filled with a color corresponding to the direction that the directional region is to represent. Regions not included in any directional region may be filled with a color not assigned to any direction.

[0018] Fig. 2 is a diagram showing an example of the spatial feature map 30. The target image 10 shown in Fig. 2 includes two persons 80. The spatial feature map 30 shown in Fig. 2 is generated for a pair of a left elbow and a left wrist. Therefore, the spatial feature map 30 includes four directional regions 32-1 to 32-4, which respectively represent the left elbow of person 80-1, the left wrist of person 80-1, the left elbow of person 80-2, and the left wrist of person 80-2.

[0019] 2, direction region 32 represents the direction from the left elbow to the left wrist of the corresponding person 80. For example, direction regions 32-1 and 32-2 corresponding to person 80-1 represent the direction from the left elbow to the left wrist of person 80-1. Because the left elbow and left wrist of person 80-1 are represented by key points 20-1 and 20-2, respectively, direction regions 32-1 and 32-2 represent the direction from key point 20-1 to key point 20-2.

[0020] After generating the spatial feature map 30, the keypoint association device 2000 divides the keypoints 20 into keypoint groups 40 based on the spatial feature map 30. A specific method for generating the keypoint groups 40 will be described later.

[0021] <Examples of effects> According to the keypoint matcher 2000, the keypoints 20 detected from the target image 10 are classified into keypoint groups 40 such that each keypoint group 40 contains only keypoints 20 belonging to the same person. To this end, the keypoint matcher 2000 generates a spatial feature map 30 for each predetermined pair of body parts. Thus, the keypoint matcher 2000 provides a novel technique for keypoint match.

[0022] Furthermore, the keypoint matching device 2000 is advantageous in the following respects. As described above, Non-Patent Document 1 generates, for each pair of body parts for each person, a feature map including a PAF that connects the two keypoints corresponding to the pair. This feature map is generated using a convolutional neural network (CNN). Since the PAF may include regions far from any of the corresponding keypoints (e.g., intermediate regions between the keypoints), a problem may occur in CNN training, in that convergence of such regions in the PAF is slow.

[0023] In this regard, the spatial feature map 30 for a pair of body parts includes a separate directional region 32 for each of the two keypoints of the pair. Therefore, regions away from the keypoints, such as the intermediate region between the keypoints, are not included in the directional region 32. Therefore, when the spatial feature map 30 is generated by a machine learning-based model, it is possible to prevent adverse effects on model learning due to delayed convergence in regions away from the keypoints.

[0024] Keypoint association device 2000 will be described in detail below.

[0025] <Example of functional configuration> 3 is a block diagram showing an example of the functional configuration of a keypoint association device 2000 according to an embodiment. The keypoint association device 2000 includes an acquisition unit 2020, a keypoint detection unit 2040, a feature map generation unit 2060, and a keypoint association unit 2080. The acquisition unit 2020 acquires a target image 10. The keypoint detection unit 2040 detects keypoints 20 from the target image 10. The feature map generation unit 2060 uses the target image 10 to generate a spatial feature map 30 for each of a predetermined pair of body parts. The keypoint association unit 2080 generates a keypoint group 40 based on the spatial feature map 30.

[0026] <Example of hardware configuration> The keypoint association apparatus 2000 may be realized by one or more computers, each of which may be a dedicated computer manufactured for implementing the keypoint association apparatus 2000, or may be a general-purpose computer such as a personal computer (PC), a server machine, or a mobile device.

[0027] Keypoint associating apparatus 2000 can be realized by installing an application on a computer. The application is implemented as a program that causes the computer to function as keypoint associating apparatus 2000. In other words, the program implements the functional components of keypoint associating apparatus 2000.

[0028] FIG. 4 is a block diagram showing an example of the hardware configuration of a computer 1000 that realizes a keypoint association device 2000 according to an embodiment. In FIG. 4, a computer 1000 includes a bus 1020, a processor 1040, a memory 1060, a storage device 1080, an input / output (I / O) interface 1100, and a network interface 1120.

[0029] The bus 1020 is a data transmission path for mutually transmitting and receiving data among the processor 1040, memory 1060, storage device 1080, input / output interface 1100, and network interface 1120. The processor 1040 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), a digital signal processor (DSP), or a field-programmable gate array (FPGA). The memory 1060 is a main storage device such as a random access memory (RAM) or a read-only memory (ROM). The storage device 1080 is an auxiliary storage device such as a hard disk, a solid state drive (SSD), or a memory card. The input / output interface 1100 is an interface between the computer 1000 and peripheral devices such as a keyboard, a mouse, and a display device. The network interface 1120 is an interface between the computer 1000 and a network. The network may be a local area network (LAN) or a wide area network (WAN).

[0030] The processor 1040 is configured to load instructions of the aforementioned programs from the storage device 1080 into the memory 1060 and execute the instructions, thereby causing the computer 1000 to operate as the keypoint association device 2000 .

[0031] The hardware configuration of computer 1000 is not limited to that shown in Fig. 4. For example, as described above, keypoint association apparatus 2000 may be realized as a combination of multiple computers, which may be connected to each other via a network.

[0032] <Processing flow> 5 is a flowchart showing an example of the flow of processing executed by the keypoint association device 2000 according to an embodiment. The acquisition unit 2020 acquires a target image 10 (S102). The keypoint detection unit 2040 detects keypoints from the target image 10 (S104). The feature map generation unit 2060 generates a spatial feature map 30 for each of a predetermined pair of body parts (S106). The keypoint association unit 2080 generates a keypoint group 40 for each person (S108).

[0033] <Acquisition of target image 10: S102> The acquisition unit 2020 acquires the target image 10 (S102). There are various methods for acquiring the target image 10. In one embodiment, the target image 10 is pre-stored in a storage device in a manner that allows it to be acquired by the keypoint association device 2000. In this case, the acquisition unit 2020 may access the storage device to acquire the target image. In another embodiment, the target image 10 may be transmitted from another computer, such as a camera that generates the target image 10. In this case, the acquisition unit 2020 may acquire the target image 10 by receiving the target image 10.

[0034] In one embodiment, the target image 10 may be one of a time series of images, such as a time series of video frames constituting a video. In this case, the keypoint matching device 2000 may acquire all or part of the time series of images as the target image 10 and perform keypoint detection and keypoint matching for each target image 10.

[0035] <Keypoint detection: S104> The key point detection unit 2040 detects key points 20 from the target image 10 (S104). There are various methods for detecting one or more positions of predetermined parts of a person's body as key points from an image, and the key point detection unit 2040 can detect key points 20 from the target image 10 using one of these methods.

[0036] In one embodiment, the keypoint detector 2040 may include a machine learning-based model (e.g., a neural network) that is pre-trained to receive an image as input and, in response to the input image, detect one or more keypoints 20 for each predetermined feature in the input image. This model will be referred to hereinafter as the "keypoint detection model."

[0037] The keypoint detection model may receive a target image 10 as input, extract features from the target image 10, detect one or more positions of each predetermined body part based on the extracted features, and output pairs of the positions and labels as keypoints. The labels of the keypoints indicate which body part the keypoints represent. In this case, the keypoint detection model may include a first model pre-trained to extract features from the target image 10 and a second model pre-trained to detect one or more positions of each predetermined body part based on the features extracted by the first model. Each of the first model and the second model may be configured as a machine learning-based model such as a neural network. Note that there are various types of machine learning models that can detect keypoints from an input image, and the keypoint detection model may be configured as one of such models.

[0038] <Generation of spatial feature map: S106> The feature map generator 2060 generates a spatial feature map 30 for each of the predetermined body part pairs (S106). To generate the spatial feature map 30, the feature map generator 2060 may include a machine learning-based model called a "feature map generation model" for each of the predetermined body part pairs. FIG. 6 is a diagram illustrating an example configuration of the feature map generator 2060. In FIG. 6, it is assumed that N body part pairs are predetermined. Therefore, the feature map generator 2060 includes a feature map generation model 70 for each of the predetermined N body part pairs.

[0039] The feature map generation model 70 for a particular body part pair is configured to receive as input image data and information about keypoints 20 detected from the image data that represent one of the body parts of the pair. The feature map generation model 70 is pre-trained to generate a spatial feature map 30 for the corresponding body part pair in response to the input data.

[0040] When the positions of the keypoints 20 are expressed in two-dimensional coordinates, as shown in Fig. 6, the direction between two keypoints 20 can be expressed as a single angle, for example, the angle between the X axis and a line connecting the two keypoints 20, so the feature map generator 2060 can generate one spatial feature map 30 for each pair of predetermined body parts. On the other hand, when the positions of the keypoints 20 are expressed in three-dimensional coordinates, the direction between two keypoints 20 can be expressed as a pair of angles, so the feature map generator 2060 can generate two spatial feature maps 30 for each pair of predetermined body parts. The case where the positions of the keypoints 20 are expressed in three-dimensional coordinates will be described in more detail below.

[0041] When the positions of the key points 20 are expressed in three-dimensional coordinates, the direction between two key points 20 may be expressed as a pair of a horizontal direction and a vertical direction. In order to express the direction between the key points 20 as a pair of a horizontal direction and a vertical direction in three-dimensional space, the feature map generator 2060 may generate a pair of a spatial feature map 30 representing the horizontal direction between the key points 20 and a spatial feature map 30 representing the vertical direction between the key points 20. Hereinafter, the spatial feature map 30 representing the horizontal direction between the keypoints 20 will be referred to as the "horizontal spatial feature map," and the spatial feature map 30 representing the vertical direction between the keypoints 20 will be referred to as the "vertical spatial feature map."

[0042] Fig. 7 is a diagram showing an example in which the directions between key points 20 in three-dimensional space are represented by a pair of a horizontal spatial feature map and a vertical spatial feature map. In Fig. 7, it is assumed that a spatial feature map 30 has been generated for a pair of a left elbow and a left wrist. It is also assumed that key point 20-1 and key point 20-2 represent the positions of a person's left elbow and left wrist, respectively.

[0043] The positions of keypoints 20-1 and 20-2 in three-dimensional space are represented by points Q1 and Q2, respectively. Therefore, the direction from keypoint 20-1 to keypoint 20-2 in three-dimensional space is represented by vector V whose start and end points are Q1 and Q2, respectively.

[0044] The horizontal direction of vector V can be represented by the angle between the vector V and the X axis when vector V is projected onto the XY plane. This angle is shown as θ in Fig. 7. Therefore, horizontal spatial feature map 50 is generated to include directional regions 32-1 and 32-2 whose pixel values ​​represent angle θ.

[0045] The vertical direction of vector V can be represented by the angle between the XY plane and vector V. This angle is shown as φ in Figure 7. Thus, vertical spatial feature map 60 is generated to include directional regions 32-3 and 32-4 whose pixel values ​​represent the angle φ.

[0046] When the positions of the keypoints 20 are expressed in three-dimensional coordinates, for each of the predetermined body part pairs, the feature map generation model 70 may include a first model for generating a horizontal spatial feature map 50 for the body part pair and a second model for generating a vertical spatial feature map 60 for the body part pair. By using these feature map generation models, the feature map generator 2060 can generate a pair of the horizontal spatial feature map 50 and the vertical spatial feature map 60 for each of the predetermined body part pairs based on the target image 10 and the keypoints 20 detected from the target image 10.

[0047] 8 is a diagram showing an example of the configuration of the feature map generation unit 2060 when the positions of the keypoints 20 are expressed in three-dimensional coordinates. Each feature map generation model 70 includes a pair of a first model 72 that generates the horizontal spatial feature map 50 and a second model 74 that generates the vertical spatial feature map 60.

[0048] <Keypoint matching: S108> The keypoint association unit 2080 performs keypoint association by generating keypoint groups 40 based on the spatial feature map 30 (S108). As described above, the keypoint groups 40 are generated to include only keypoints 20 that belong to the same person. Assume that the number of people captured in the target image 10 is N. In this case, the keypoint association unit 2080 may generate a keypoint group 40 for each of the N people. Therefore, N keypoint groups 40 may be generated.

[0049] A specific method for keypoint correspondence using the spatial feature map 30 will be described below. For simplicity, we first assume that the positions of the keypoints 20 are represented by two-dimensional coordinates. A method for keypoint correspondence when the positions of the keypoints 20 are represented by three-dimensional coordinates will be described later.

[0050] For each given pair of body parts, the keypoint associator 2080 uses the spatial feature map 30 associated with that pair to divide the keypoints 20 into keypoint groups 40. For example, using the spatial feature map 30 associated with the pair of left elbow and left wrist, the keypoint associator 2080 generates a keypoint group 40 including a pair of left elbow keypoints 20 and left wrist keypoints 20 that belong to the same person.

[0051] Theoretically, a pair of directional regions 32 in the spatial feature map 30 corresponds to a pair of two keypoints 20 belonging to the same person if the two directional regions 32 indicate the same direction. Therefore, the keypoint association unit 2080 can identify a pair of keypoints 20 belonging to the same person by identifying a pair of keypoints 20 whose directional regions 32 indicate the same direction.

[0052] However, in reality, there may be some difference in the indicated directions between direction regions 32 belonging to the same person. Therefore, in one embodiment, the keypoint matching unit 2080 identifies pairs of keypoints 20 whose direction regions 32 indicate directions that are substantially close to each other, and generates a keypoint group 40 including the identified pairs of keypoints 20.

[0053] 9 shows an example method for keypoint matching. In this example, a spatial feature map for a pair of left elbow and left wrist is used. Therefore, for each person captured in the target image 10, a keypoint group 40 is generated that includes a pair of keypoints 20 representing the left elbow and 20 representing the left wrist.

[0054] By referring to the detection results of the key points 20 executed by the key point detection unit 2040, the key point association unit 2080 identifies the key points 20 of the left elbow (key points 20-2 and 20-3) and the key points 20 of the left wrist (key points 20-1 and 20-4) on the spatial feature map 30. Next, the key point association unit 2080 identifies the direction regions 32 corresponding to each of the identified key points 20. Specifically, there are four direction regions 32-1 to 32-4 corresponding to the key points 20-1 to 20-4, respectively.

[0055] As described below, the feature map generation model 70 may be trained to generate a spatial feature map 30 in which directional regions 32 have predetermined shapes and sizes, and the positions of the directional regions 32 are defined based on the positions of the corresponding keypoints 20. Thus, the keypoint correspondence unit 2080 can identify directional regions 32 based on the predetermined shapes and sizes and the positions of the corresponding keypoints 20.

[0056] 9, it is assumed that the shape of the directional region 32 is defined as a circle, and the size of the directional region 32 is defined by a radius R. It is also assumed that the center of the directional region 32 is located at the corresponding keypoint 20. Therefore, for each keypoint 20, the keypoint association unit 2080 identifies a region whose shape is a circle, whose radius is R, and whose center is located at the keypoint 20, as the directional region 32 corresponding to the keypoint 20.

[0057] Here, if two or more directional regions 32 overlap each other, the keypoint association unit 2080 may adjust the sizes of the directional regions 32 so that they do not overlap. There are various methods for adjusting the sizes of the directional regions. For example, the keypoint association unit 2080 may reduce the sizes of the directional regions 32 by repeatedly multiplying the sizes of the directional regions 32 by an adjustment coefficient, which is a real number greater than 0 and less than 1, until they do not overlap. As another example, multiple options for the sizes of the directional regions 32 may be defined in advance. In this case, the keypoint association unit 2080 may select the option with the largest size that does not cause them to overlap.

[0058] Furthermore, as will be described later, the size of the direction regions 32 may also be adjusted when generating a training data set used to train a feature generation model. Therefore, it is desirable that the keypoint association unit 2080 adjusts the size of the direction regions 32 in a manner similar to the method of adjusting the size of the direction regions 32 when generating the training data set.

[0059] After identifying the directional regions 32 for each keypoint 20, the keypoint matcher 2080 identifies pairs of keypoints 20 for generating keypoint groups 40. To facilitate describing the operation of the keypoint matcher 2080, the pairs of body parts corresponding to the spatial feature maps 30 are referred to as a first body part and a second body part, respectively. For example, in the example shown in FIG. 9, the left elbow is referred to as the first body part and the left wrist is referred to as the second body part.

[0060] The keypoint matcher 2080 selects one of the keypoints 20 of the first body part. The keypoint matcher 2080 then evaluates the keypoints 20 of the second body part against the selected keypoints 20 of the first body part to identify which of the keypoints 20 of the second body part should be paired with the keypoint 20 of the first body part.

[0061] 9, the keypoint matcher 2080 may select keypoint 20-2 as one of the left elbow keypoints 20. Next, the keypoint matcher 2080 evaluates each of the left wrist keypoints 20 (i.e., keypoints 20-1 and 20-4) to identify which of them should be paired with keypoint 20-2.

[0062] Keypoints 20 can be evaluated using an index value called a "coefficient distance." The coefficient distance between two keypoints 20 represents the degree to which the directions indicated by their corresponding direction regions differ. For example, the coefficient distance between keypoint 20-2 and keypoint 20-1 represents the degree of difference between the direction indicated by direction region 32-2 and the direction indicated by direction region 32-1.

[0063] After selecting one of the keypoints 20 of the first body part, the keypoint matcher 2080 calculates, for each keypoint 20 of the second body part, a coefficient distance between the keypoint 20 of the second body part and the selected keypoint 20 of the first body part. The keypoint matcher 2080 then generates a pair of the selected keypoint 20 of the first body part and the keypoint 20 of the second body part that has the smallest coefficient distance.

[0064] In some embodiments, a coefficient distance threshold may be predefined, in which case the second body part keypoint 20 with the smallest coefficient distance will be paired with the selected first body part keypoint 20 only if the coefficient distance is less than the threshold.

[0065] To calculate the coefficient distance between keypoints 20, the keypoint association unit 2080 identifies a value representing a direction (hereinafter referred to as a "direction value") for each of the direction regions 32. As described above, a direction region may represent a direction by the pixel values ​​within it. Therefore, the keypoint association unit 2080 may calculate the statistical value of the pixel values ​​within a direction region 32 as the direction value of the direction region 32.

[0066] The coefficient distance between keypoints 20 can be expressed as the absolute value of the difference in the direction values ​​of their corresponding direction regions 32. This can be expressed by the following formula:

number

[0067] In one embodiment, the coefficient distance between keypoints 20 may be calculated taking into account the Euclidean distance between the keypoints 20. This is because the longer the Euclidean distance between keypoints 20, the less likely those keypoints 20 belong to the same person. When taking into account the Euclidean distance between keypoints 20, the coefficient distance between keypoints 20 can be expressed by the following formula:

number

[0068] After generating keypoint groups 40 for each of the predetermined body part pairs, the keypoint association unit 2080 may combine keypoint groups 40 that correspond to the same person. Specifically, the keypoint association unit 2080 may repeat the process of detecting two keypoint groups 40 that each contain at least one identical keypoint 20 and combining the detected two keypoint groups 40 into one keypoint group 40 until neither keypoint group 40 contains the same keypoint 20 as the other keypoint group 40.

[0069] <<When keypoint positions are expressed as 3D coordinates>> When the keypoint locations are expressed in 3D coordinates, two types of spatial feature maps 30 are generated for each given body part pair: a horizontal spatial feature map 50 and a vertical spatial feature map 60. Thus, for each given body part pair, the keypoint associator 2080 generates a keypoint group 40 using the horizontal spatial feature map 50 and the vertical spatial feature map 60 associated with that body part pair.

[0070] Keypoint correspondence when the positions of keypoints 20 are expressed in 3D coordinates differs from correspondence when the positions of keypoints 20 are expressed in 2D coordinates in that the coefficient distance is calculated based on the horizontal and vertical directions between keypoints 20. To this end, keypoint correspondence unit 2080 calculates, for each keypoint 20, a direction value for a direction region 32 in horizontal spatial feature map 50 and a direction value for a direction region 32 in vertical spatial feature map 60. The coefficient distance between keypoints 20 whose positions are expressed in 3D coordinates may be calculated as follows:

number

[0071] Furthermore, when calculating the coefficient distance between the key points 20, if the Euclidean distance between those key points 20 is taken into consideration, the coefficient distance between the key points 20 can be expressed by the following mathematical formula:

number

[0072] <Output from Keypoint Aligner 2000> The keypoint association device 2000 may be configured to output information (referred to as output information) indicating the results of the keypoint association. For example, the output information may include an identifier of the target image 10 (e.g., a frame number) and keypoint group information. For each keypoint group 40, the keypoint group information includes an identifier of the keypoint group 40 and information about each keypoint 20 included in the keypoint group 40. The information about the keypoint 20 may include an identifier of the keypoint 20, a position indicated by the keypoint 20, and an identifier of a body part indicated by the keypoint 20.

[0073] There are various ways to output the output information, and in some embodiments, the output information may be stored in a storage device, displayed on a display device, or transmitted to another computer, such as a PC or smartphone, of a user of the keypoint association apparatus 2000.

[0074] <Training feature map generation model 70> The feature map generation model 70 is trained using multiple training datasets including training input images, ground truth keypoint information, and ground truth spatial feature maps. The training input images are image data of one or more people, such as the target image 10. The ground truth keypoint information indicates, for each keypoint 20 to be detected from the target image 10, the location and body part indicated by that keypoint 20. The ground truth spatial feature map is the ideal spatial feature map 30 that should be output from the trained feature map generation model 70 when the corresponding training input images are input. The training datasets include ground truth spatial feature maps for each predetermined body part pair.

[0075] Hereinafter, the device that trains feature map generation model 70 will be referred to as the "training device." The training device may be the same device as keypoint association device 2000, or may be a different device from keypoint association device 2000. The former case means that keypoint association device 2000 also has the function of training feature map generation model 70.

[0076] For each given body part pair, the training device may train the feature map generation model 70 for that pair as follows: The training device provides input data extracted from a training dataset to the feature map generation model 70, and obtains an output spatial feature map 30 from the feature map generation model 70. The training device calculates a loss based on the obtained spatial feature map 30 and a ground truth spatial feature map, and updates the trainable parameters of the feature map generation model 70. The above process may be repeated for each of multiple training datasets.

[0077] In one embodiment, the ground truth spatial feature map may be generated in advance by an administrator of the keypoint alignment apparatus 2000. For example, the administrator operates a computer called a "dataset generation apparatus" to display training input images on a display device. The dataset generation apparatus may be the same apparatus as the keypoint alignment apparatus 2000, the same apparatus as the training apparatus, or a different apparatus from the keypoint alignment apparatus 2000 and the training apparatus. The first case means that the keypoint alignment apparatus 2000 is configured to also function as the dataset generation apparatus.

[0078] An administrator or other person operates a dataset generator to generate a training dataset. For example, the administrator or other person receives training input images from the dataset generator. Then, for each predetermined body part pair, the administrator or other person specifies keypoints for each person included in the provided training input images. The dataset generator generates a ground truth spatial feature map based on the training input images and the specified keypoints.

[0079] Assume that the training input images contain persons P1 and P2. Assume also that a ground truth spatial feature map is generated for a pair of left elbow and left wrist. In this case, an administrator or other person may specify the keypoints of person P1's left elbow and left wrist. Hereinafter, the keypoints of person P1's left elbow and left wrist are denoted as E1 and H1, respectively.

[0080] In response to the specification of E1 and H1, the dataset generator automatically generates corresponding directional regions R1 and R2. The directional regions can be generated as regions with a predetermined shape and size (e.g., a circle with a predetermined radius, a square with a predetermined side length, etc.). The directional region for a specific keypoint is positioned based on the position of the keypoint. For example, the center of the directional region is positioned at the corresponding keypoint. That is, the center of the directional region for keypoint E1 is positioned at keypoint E1.

[0081] To generate the direction regions R1 and R2, the dataset generator calculates the direction from E1 to H1 and identifies pixel values ​​corresponding to that direction, which are then set for all pixels within the direction regions R1 and R2.

[0082] Similarly, the administrator or the like specifies key points on the left elbow and left wrist of person P2, which are denoted as E2 and H2, respectively. In response to the specification of E2 and H2, the dataset generation device generates direction regions R3 and R4 for E2 and H2, respectively. Specifically, the dataset generation device calculates the direction from E2 to H2, identifies pixel values ​​corresponding to the calculated direction, and generates direction regions R3 and R4 having a predetermined shape and size and filled with the pixel values.

[0083] Here, the dataset generation device may dynamically adjust the size of the direction regions in the ground truth spatial feature map so that the direction regions do not overlap with each other. Assume that the predetermined shape and size of the direction regions are a circle with a radius R. In this case, if the distance between the direction regions R1 and R2 in the ground truth spatial feature map is less than 2*R, the direction regions R1 and R2 overlap with each other. Therefore, the dataset generation device reduces the size of the direction regions R1 and R2 so that they do not overlap. An example of reducing the size of the direction regions has already been described.

[0084] When the positions of the keypoints are expressed in 3D coordinates, the data set generator generates a horizontal spatial feature map and a vertical spatial feature map in accordance with the designation of the keypoints.

[0085] <Using key point groups> The results of the keypoint correspondence (i.e., the keypoint groups 40) can be used in various ways. For example, the keypoint groups 40 can be used for pose estimation. As a result of the pose estimation, for each keypoint group 40, the type of pose taken by the person corresponding to that keypoint group 40 can be estimated.

[0086] Furthermore, by performing pose estimation on each target image 10 of the time series data (e.g., video frames in a video), a time series of poses is obtained for each person captured in the target image 10. The time series of poses of that person can be used to identify the action or action sequence that the person is performing.

[0087] The program can be stored and provided to a computer using various types of non-transitory computer-readable media. Non-transitory computer-readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic storage media (e.g., flexible disks, magnetic tapes, hard disk drives), magneto-optical storage media (e.g., magneto-optical disks), CD-ROMs, CD-Rs, CD-R / Ws, and semiconductor memories (e.g., mask ROMs, programmable ROMs (PROMs), erasable PROMs (EPROMs), flash ROMs, and RAMs). The program may also be provided to a computer by various types of transitory computer-readable media. Examples of transitory computer-readable media include electrical signals, optical signals, and electromagnetic waves. The transitory computer-readable media can provide the program to a computer via a wired communication path such as an electric wire or optical fiber, or via a wireless communication path.

[0088] Although the present disclosure has been described with reference to the embodiments, the present disclosure is not limited to the above-described embodiments. Various changes in arrangement and detail that are apparent to those skilled in the art can be made within the scope of the present invention.

[0089] A part or all of the above-described embodiments can be described as, but not limited to, the following supplementary notes. <Additional Notes> (Appendix 1) at least one memory configured to store instructions; and at least one processor configured to execute the instructions, whereby executing the instructions acquiring a target image of one or more people; detecting key points from the target image for each of the person's body parts; generating a spatial feature map for each of the predetermined pairs of body parts using the target image, wherein the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the key points in the first direction region to the key points in the second direction region; and generating, for each of the people captured in the target image, a keypoint group including the keypoints belonging to the same person. (Appendix 2) The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting the first direction region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first direction region; For each keypoint of the second body part, detecting the second orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: calculating, for each said keypoint of said second body part, a coefficient distance between that keypoint of said first body part and that keypoint of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; 2. The keypoint matching device of claim 1, wherein a coefficient distance between the keypoint of the first body part and the keypoint of the second body part represents a degree of difference between the direction indicated by the first direction region of the keypoint of the first body part and the direction indicated by the second direction region of the keypoint of the second body part. (Appendix 3) The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating statistics of pixel values ​​within the first direction region of the key point of the first body part as a direction indicated by the first direction region; calculating statistics of pixel values ​​within the second direction region of the key point of the second body part as a direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by the first direction region and the direction indicated by the second direction region. (Appendix 4) 4. The keypoint association apparatus of claim 3, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the absolute value of the difference by a Euclidean distance between the keypoints. (Appendix 5) the locations of the keypoints are expressed in 3D coordinates; for each of the predetermined pairs of body parts, a horizontal spatial feature map and a vertical spatial feature map are generated as the spatial feature maps for that pair; In the horizontal spatial feature map, the first direction region and the second direction region belonging to the same person represent a horizontal direction from the key point in the first direction region to the key point in the second direction region; 2. The keypoint matching device of claim 1, wherein in the vertical spatial feature map, the first direction region and the second direction region belonging to the same person represent a vertical direction from the keypoint in the first direction region to the keypoint in the second direction region. (Appendix 6) The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting a first orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting a second orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: calculating, for each said keypoint of said second body part, a coefficient distance between that keypoint of said first body part and that keypoint of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the first direction region of the key point of the first body part as the direction indicated by the first direction region; calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the second direction region of the keypoint of the second body part as the direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by a first direction region of each of the horizontal direction feature map and the vertical direction feature map and the direction indicated by a second direction region of each of the horizontal direction feature map and the vertical direction feature map; and calculating a sum of the absolute values ​​of the differences. (Appendix 7) 7. The keypoint association apparatus of claim 6, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the sum of absolute values ​​of the differences by a Euclidean distance between the keypoint of the first body part and the keypoint of the second body part. (Appendix 8) acquiring a target image of one or more people; detecting key points from the target image for each of the person's body parts; generating a spatial feature map for each of the predetermined pairs of body parts using the target image, wherein the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the key points in the first direction region to the key points in the second direction region; and generating, for each person captured in the target image, a keypoint group including the keypoints that belong to the same person. (Appendix 9) The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting the first direction region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first direction region; For each keypoint of the second body part, detecting the second orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: calculating, for each said keypoint of said second body part, a coefficient distance between that keypoint of said first body part and that keypoint of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; 9. The keypoint matching method of claim 8, wherein a coefficient distance between the keypoint of the first body part and the keypoint of the second body part represents a degree of difference between the direction indicated by the first direction region of the keypoint of the first body part and the direction indicated by the second direction region of the keypoint of the second body part. (Appendix 10) The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating statistics of pixel values ​​within the first direction region of the key point of the first body part as a direction indicated by the first direction region; calculating statistics of pixel values ​​within the second direction region of the key point of the second body part as a direction indicated by the second direction region; 10. The keypoint association method of claim 9, further comprising: calculating an absolute value of a difference between the direction indicated by the first direction region and the direction indicated by the second direction region. (Appendix 11) 11. The keypoint association method of claim 10, wherein the calculating the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the absolute value of the difference by a Euclidean distance between the keypoints. (Appendix 12) the locations of the keypoints are expressed in 3D coordinates; for each of the predetermined pairs of body parts, a horizontal spatial feature map and a vertical spatial feature map are generated as the spatial feature maps for that pair; In the horizontal spatial feature map, the first direction region and the second direction region belonging to the same person represent a horizontal direction from the key point in the first direction region to the key point in the second direction region; 9. The keypoint matching method of claim 8, wherein in the vertical spatial feature map, the first direction region and the second direction region belonging to the same person represent a vertical direction from the keypoint in the first direction region to the keypoint in the second direction region. (Appendix 13) The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting a first orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting a second orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: calculating, for each said keypoint of said second body part, a coefficient distance between that keypoint of said first body part and that keypoint of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the first direction region of the key point of the first body part as the direction indicated by the first direction region; calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the second direction region of the keypoint of the second body part as the direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by a first direction region of each of the horizontal direction feature map and the vertical direction feature map and the direction indicated by a second direction region of each of the horizontal direction feature map and the vertical direction feature map; and calculating a sum of the absolute values ​​of the differences. (Appendix 14) 14. The keypoint association method of claim 13, wherein the calculating the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the sum of absolute values ​​of the differences by a Euclidean distance between the keypoint of the first body part and the keypoint of the second body part. (Appendix 15) acquiring a target image of one or more people; detecting key points from the target image for each of the person's body parts; generating a spatial feature map for each of the predetermined pairs of body parts using the target image, wherein the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the key points in the first direction region to the key points in the second direction region; and generating, for each person captured in the target image, a keypoint group including the keypoints that belong to the same person. (Appendix 16) The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting the first direction region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first direction region; For each keypoint of the second body part, detecting the second orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: calculating, for each said keypoint of said second body part, a coefficient distance between that keypoint of said first body part and that keypoint of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; 16. The storage medium of claim 15, wherein a coefficient distance between the key point of the first body part and the key point of the second body part represents a degree of difference between the direction indicated by the first direction region of the key point of the first body part and the direction indicated by the second direction region of the key point of the second body part. (Appendix 17) The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating statistics of pixel values ​​within the first direction region of the key point of the first body part as a direction indicated by the first direction region; calculating statistics of pixel values ​​within the second direction region of the key point of the second body part as a direction indicated by the second direction region; and calculating an absolute value of a difference between the direction indicated by the first direction region and the direction indicated by the second direction region. (Appendix 18) 18. The storage medium of claim 17, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the absolute value of the difference by the Euclidean distance between the keypoints. (Appendix 19) the locations of the keypoints are expressed in 3D coordinates; for each of the predetermined pairs of body parts, a horizontal spatial feature map and a vertical spatial feature map are generated as the spatial feature maps for that pair; In the horizontal spatial feature map, the first direction region and the second direction region belonging to the same person represent a horizontal direction from the key point in the first direction region to the key point in the second direction region; 16. The storage medium of claim 15, wherein in the vertical spatial feature map, the first direction region and the second direction region belonging to the same person represent a vertical direction from the key point in the first direction region to the key point in the second direction region. (Appendix 20) The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting a first orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting a second orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: calculating, for each said keypoint of said second body part, a coefficient distance between that keypoint of said first body part and that keypoint of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the first direction region of the key point of the first body part as the direction indicated by the first direction region; calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the second direction region of the keypoint of the second body part as the direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by a first direction region of each of the horizontal direction feature map and the vertical direction feature map and the direction indicated by a second direction region of each of the horizontal direction feature map and the vertical direction feature map; and calculating a sum of the absolute values ​​of the differences. (Appendix 21) 21. The storage medium of claim 20, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the sum of absolute values ​​of the differences by a Euclidean distance between the keypoint of the first body part and the keypoint of the second body part. [Explanation of symbols]

[0090] 10 Target image 20 Key Points 30 Spatial Feature Map 32 direction area 40 Key Points Group 50 Horizontal spatial feature maps 60 Vertical spatial feature maps 70 Feature Extraction Model 72 1st model 74 2nd model 80 people 1000 computers 1020 Bus 1040 processor 1060 memory 1080 storage device 1100 Input / Output Interface 1120 Network Interface 2000 Keypoint Correspondence Device 2020 Acquisition Department 2040 Keypoint Detector 2060 Feature map generation unit 2080 Keypoint Correspondence

Claims

1. at least one memory configured to store instructions; and at least one processor configured to execute the instructions, whereby executing the instructions acquiring a target image of one or more people; detecting key points from the target image for each of the person's body parts; generating a spatial feature map for each of the predetermined pairs of body parts using the target image, wherein the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the key points in the first direction region to the key points in the second direction region; and generating, for each of the people captured in the target image, a keypoint group including the keypoints belonging to the same person.

2. The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting the first orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting the second orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: for each said key point of said second body part, calculating a coefficient distance between that key point of said first body part and that key point of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; 2. The keypoint matching apparatus of claim 1, wherein a coefficient distance between the keypoint of the first body part and the keypoint of the second body part represents a degree of difference between the direction indicated by the first direction region of the keypoint of the first body part and the direction indicated by the second direction region of the keypoint of the second body part.

3. The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating statistics of pixel values ​​within the first direction region of the key point of the first body part as a direction indicated by the first direction region; calculating statistics of pixel values ​​within the second direction region of the key point of the second body part as a direction indicated by the second direction region; 3. The keypoint matching apparatus of claim 2, further comprising: calculating an absolute value of a difference between the direction indicated by the first direction region and the direction indicated by the second direction region.

4. 4. The keypoint matching apparatus of claim 3, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the absolute value of the difference by a Euclidean distance between the keypoints.

5. the locations of the keypoints are expressed in 3D coordinates; for each of the predetermined pairs of body parts, a horizontal spatial feature map and a vertical spatial feature map are generated as the spatial feature maps for that pair; In the horizontal spatial feature map, the first direction region and the second direction region belonging to the same person represent a horizontal direction from the key point in the first direction region to the key point in the second direction region; 2. The keypoint matching device of claim 1, wherein in the vertical spatial feature map, the first direction region and the second direction region belonging to the same person represent a vertical direction from the keypoint in the first direction region to the keypoint in the second direction region.

6. The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting a first orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting a second orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: for each said key point of said second body part, calculating a coefficient distance between that key point of said first body part and that key point of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the first direction region of the key point of the first body part as the direction indicated by the first direction region; calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the second direction region of the key point of the second body part as the direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by a first direction region of each of the horizontal direction feature map and the vertical direction feature map and the direction indicated by a second direction region of each of the horizontal direction feature map and the vertical direction feature map; and calculating a sum of the absolute values ​​of the differences.

7. 7. The keypoint matching apparatus of claim 6, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the sum of absolute values ​​of the differences by a Euclidean distance between the keypoint of the first body part and the keypoint of the second body part.

8. acquiring a target image of one or more people; detecting key points from the target image for each of the person's body parts; generating a spatial feature map for each of the predetermined pairs of body parts using the target image, wherein the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the key points in the first direction region to the key points in the second direction region; and generating, for each person captured in the target image, a keypoint group including the keypoints that belong to the same person.

9. The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting the first orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting the second orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: for each said key point of said second body part, calculating a coefficient distance between that key point of said first body part and that key point of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; 9. The method of claim 8, wherein a coefficient distance between the keypoint of the first body part and the keypoint of the second body part represents a degree of difference between the direction indicated by the first direction region of the keypoint of the first body part and the direction indicated by the second direction region of the keypoint of the second body part.

10. The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating statistics of pixel values ​​within the first direction region of the key point of the first body part as a direction indicated by the first direction region; calculating statistics of pixel values ​​within the second direction region of the key point of the second body part as a direction indicated by the second direction region; 10. The method of claim 9, further comprising: calculating an absolute value of a difference between the direction indicated by the first direction region and the direction indicated by the second direction region.

11. 11. The method of claim 10, wherein the calculation of the coefficient distance between the keypoints of the first body part and the keypoints of the second body part further comprises correcting the absolute value of the difference by a Euclidean distance between the keypoints.

12. the locations of the keypoints are expressed in 3D coordinates; for each of the predetermined pairs of body parts, a horizontal spatial feature map and a vertical spatial feature map are generated as the spatial feature maps for that pair; In the horizontal spatial feature map, the first direction region and the second direction region belonging to the same person represent a horizontal direction from the key point in the first direction region to the key point in the second direction region; 9. The keypoint correspondence method of claim 8, wherein in the vertical spatial feature map, the first direction region and the second direction region belonging to the same person represent a vertical direction from the keypoint in the first direction region to the keypoint in the second direction region.

13. The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting a first orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting a second orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: for each said key point of said second body part, calculating a coefficient distance between that key point of said first body part and that key point of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the first direction region of the key point of the first body part as the direction indicated by the first direction region; calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the second direction region of the key point of the second body part as the direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by a first direction region of each of the horizontal direction feature map and the vertical direction feature map and the direction indicated by a second direction region of each of the horizontal direction feature map and the vertical direction feature map; and calculating a sum of the absolute values ​​of the differences.

14. 14. The method of claim 13, wherein the calculating the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the sum of absolute values ​​of the differences by a Euclidean distance between the keypoint of the first body part and the keypoint of the second body part.

15. acquiring a target image of one or more people; detecting key points from the target image for each of the person's body parts; generating a spatial feature map for each of the predetermined pairs of body parts using the target image, wherein the spatial feature map for the pair of body parts includes a first direction region corresponding to each of the key points representing a first body part of the pair and a second direction region corresponding to each of the key points representing a second body part of the pair, the first direction region and the second direction region belonging to the same person representing directions from the key points in the first direction region to the key points in the second direction region; and generating, for each person captured in the target image, a keypoint group including the keypoints that belong to the same person.

16. The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting the first orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting the second orientation region of the keypoint from the pair of spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: for each said key point of said second body part, calculating a coefficient distance between that key point of said first body part and that key point of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; 16. The storage medium of claim 15, wherein a coefficient distance between the key point of the first body part and the key point of the second body part represents a degree of difference between the direction indicated by the first directional region of the key point of the first body part and the direction indicated by the second directional region of the key point of the second body part.

17. The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating statistics of pixel values ​​within the first direction region of the key point of the first body part as a direction indicated by the first direction region; calculating statistics of pixel values ​​within the second direction region of the key point of the second body part as a direction indicated by the second direction region; 17. The storage medium according to claim 16, further comprising: calculating an absolute value of a difference between the direction indicated by the first direction region and the direction indicated by the second direction region.

18. 20. The storage medium of claim 17, wherein the calculation of the coefficient distance between the keypoints of the first body part and the keypoints of the second body part further comprises correcting the absolute value of the difference by a Euclidean distance between the keypoints.

19. the locations of the keypoints are expressed in 3D coordinates; for each of the predetermined pairs of body parts, a horizontal spatial feature map and a vertical spatial feature map are generated as the spatial feature maps for that pair; In the horizontal spatial feature map, the first direction region and the second direction region belonging to the same person represent a horizontal direction from the key point in the first direction region to the key point in the second direction region; 16. The storage medium of claim 15, wherein in the vertical spatial feature map, the first direction region and the second direction region belonging to the same person represent a vertical direction from the key point in the first direction region to the key point in the second direction region.

20. The generation of the keypoint groups includes, for each of the predetermined pairs of body parts: For each keypoint of the first body part, detecting a first orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the first orientation region; For each keypoint of the second body part, detecting a second orientation region of the keypoint from each of the pair of horizontal and vertical spatial feature maps based on the location of the keypoint and the predetermined shape and size of the second orientation region; For each of the key points of the first body part: for each said key point of said second body part, calculating a coefficient distance between that key point of said first body part and that key point of said second body part; and including the keypoint of the first body part and the keypoint of the second body part having the smallest coefficient distance in the same keypoint group; The calculation of the coefficient distance between the key points of the first body part and the key points of the second body part comprises: calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the first direction region of the key point of the first body part as the direction indicated by the first direction region; calculating, for each of the horizontal feature map and the vertical feature map, statistics of pixel values ​​within the second direction region of the key point of the second body part as the direction indicated by the second direction region; calculating an absolute value of a difference between the direction indicated by a first direction region of each of the horizontal direction feature map and the vertical direction feature map and the direction indicated by a second direction region of each of the horizontal direction feature map and the vertical direction feature map; and calculating a sum of the absolute values ​​of the differences.

21. 21. The storage medium of claim 20, wherein the calculation of the coefficient distance between the keypoint of the first body part and the keypoint of the second body part further comprises correcting the sum of absolute values ​​of the differences by a Euclidean distance between the keypoint of the first body part and the keypoint of the second body part.

Images