JOINT POINT DETECTION DEVICE, JOINT POINT DETECTION METHOD, AND PROGRAM

Abstract

An articulation point detection device 10 comprises: an entire feature output unit 11 that outputs, for each target articulation point from image data, first features expressing each target articulation point; a partial feature output unit 12 that, for each of a plurality of specific articulation points from among the target articulation points, inputs the first features expressing articulation points other than the specific articulation points, and uses a first machine learning model that has learned positional relationships between the articulation points other than the specific articulation points to output second features expressing the articulation points other than the specific articulation points; and an integrated feature output unit 13 that, for each of the target articulation points, inputs the second features expressing the articulation points, and uses a second machine learning model that has learned the positions of the articulation points to output third features expressing the articulation points.

Description

[Technical field]

[0001] The present invention relates to a joint point detection device and a joint point detection method for detecting joint points of a living body from an image, and further relates to a program for implementing these. To Regarding. [Background technology]

[0002] In recent years, systems that estimate a person's posture from an image have been proposed. Such systems are expected to be used in fields such as video surveillance and user interfaces. For example, in an image surveillance system, if a person's posture can be estimated, it is possible to estimate what a person captured on camera is doing, improving the surveillance accuracy. In addition, in a user interface, if a person's posture can be estimated, input by gestures becomes possible.

[0003] For example, Non-Patent Document 1 discloses a system for estimating a human posture, particularly the posture of a human hand, from an image. The system disclosed in Non-Patent Document 1 first acquires image data including an image of a hand, inputs the acquired image data to a neural network that has machine-learned image features for each joint point, and outputs a heat map that represents the probability of the existence of each joint point by color and density.

[0004] Next, the system disclosed in Non-Patent Document 1 inputs the output heat map to a neural network that has learned the relationship between the joint points and the corresponding heat map by machine learning. In addition, multiple such neural networks are prepared, and the output result from one neural network is input to another neural network. As a result, the positions of the joint points on the heat map are refined.

[0005] Patent Document 1 also discloses a system for estimating a hand posture from an image. Like the system disclosed in Non-Patent Document 1, the system disclosed in Patent Document 1 also estimates the coordinates of joint points using a neural network. [Prior art documents] [Patent documents]

[0006] [Patent Document 1] JP 2017-191576 A [Non-patent literature]

[0007] [Non-Patent Document 1] Christian Zimmermann, Thomas Brox, "Learning to Estimate 3D Hand Pose from Single RGB Images", [online], University of Freiburg, [Retrieved February 8, 2021], Internet <URL:https: / / openaccess.thecvf.com / content_ICCV_2017 / papers / Zimmermann_Learning_to_Estimate_ICCV_2017_paper.pdf> Summary of the Invention [Problem to be solved by the invention]

[0008] As described above, by using the system disclosed in Non-Patent Document 1 or Patent Document 1, it is possible to estimate the coordinates of the joint points of a human hand from an image. However, these systems have a problem in that the estimation accuracy decreases as described below.

[0009] First, a living body has many joint points, and some of the joint points may not be captured in the image. In such a case, in the systems disclosed in Non-Patent Document 1 and Patent Document 1, the positions of the joint points not captured in the image may be incorrect in the heat map. As a result, when the positions of each joint point are refined by the neural network, they are dragged to the incorrect positions of the joint points not captured in the image, and even the positions of the joint points captured in the image become incorrect.

[0010] An example of an object of the present invention is to provide a joint point detection device, a joint point detection method, and program The purpose of this invention is to provide [Means for solving the problem]

[0011] In order to achieve the above object, a joint point detection device according to one aspect of the present invention comprises: a total feature output unit that outputs, for each joint point of the object, a first feature representing the joint point from the image data of the object; a partial feature output unit that receives, for each of a plurality of specific joint points among the target joint points, the first feature representing joint points other than the specific joint points, and outputs a second feature representing the joint points other than the specific joint points by using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points; an integrated feature output unit that receives, for each of the target joint points, the second feature representing the joint point, and outputs a third feature representing the joint point by using a second machine learning model that performs machine learning on a position of the joint point; The present invention is characterized in that it is provided with:

[0012] In order to achieve the above object, a joint point detection method according to one aspect of the present invention includes: a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object; a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input; an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point, with the second feature representing the joint point as an input; The present invention is characterized by having the following.

[0013] Further, in order to achieve the above object, in one aspect of the present invention, program teeth, On the computer, a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object; a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input; an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point, with the second feature representing the joint point as an input; Run 、 It is characterized by: Effect of the Invention

[0014] As described above, according to the present invention, it is possible to improve the accuracy of estimating the positions of joint points. [Brief description of the drawings]

[0015] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of a joint point detection device according to an embodiment. [Diagram 2] FIG. 2 is a diagram showing in more detail the configuration of the joint point detection device 10 according to the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating the function of the total feature output unit in the embodiment. [Figure 4] FIG. 4 is a diagram illustrating the function of the partial feature output unit in the embodiment. [Diagram 5] FIG. 5 is a diagram illustrating the function of the integrated feature output unit in the embodiment. [Figure 6]FIG. 6 is a flow diagram showing the operation of the joint point detection device in the embodiment. [Figure 7] FIG. 7 is a diagram showing a configuration of a joint point detection device 10 in a modified example of the embodiment. [Figure 8] FIG. 8 is a diagram illustrating the function of a total feature output unit in the modified example of the embodiment. [Figure 9] FIG. 9 is a diagram illustrating functions of a partial feature output unit in the modification of the embodiment. [Figure 10] FIG. 10 is a diagram illustrating the function of the integrated feature output unit in the modification of the embodiment. [Figure 11] FIG. 11 is a block diagram showing an example of a computer that realizes the joint point detection device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] (Embodiment) Hereinafter, a joint point detection device, a joint point detection method, and a program according to an embodiment will be described with reference to FIGS.

[0017] [Device configuration] First, a schematic configuration of a joint point detection device in an embodiment will be described with reference to Fig. 1. Fig. 1 is a diagram showing a schematic configuration of a joint point detection device in an embodiment.

[0018] 1, a joint point detection device 10 is a device for detecting joint points of an object, for example, a living body, a robot, etc. As shown in FIG. 1, the joint point detection device 10 includes a total feature output unit 11, a partial feature output unit 12, and an integrated feature output unit 13.

[0019] The total feature value output unit 11 outputs, for each joint point of the target, a first feature value representing the joint point from the image data of the target.

[0020] The partial feature output unit 12 receives, for each of a plurality of specific joint points among the target joint points, a first feature representing a joint point other than the specific joint point, and outputs a second feature representing the joint point other than the specific joint point by using a first machine learning model. The first machine learning model is a machine learning model that learns the positional relationships between the joint points other than the specific joint points by machine learning.

[0021] The integrated feature output unit 13 receives, for each target joint point, a second feature representing the joint point, and outputs a third feature representing the joint point by using a second machine learning model. The second machine learning model is a machine learning model that learns the positions of the joint points by machine learning.

[0022] In this manner, in the embodiment, for each specific joint point, the second feature amount is output using the first machine learning model for the other joint points. Since the first machine learning model learns the positional relationships between the joint points other than the specific joint point by machine learning, the second feature amount can appropriately indicate the positions of the other joint points when the specific joint point is not visible. And, since the third feature amount, which is the final feature amount of each joint point, is obtained from this second feature amount, according to the embodiment, it is possible to improve the estimation accuracy of the position of each joint point.

[0023] Next, the configuration and functions of the joint point detection device 10 in the embodiment will be specifically described with reference to Figs. 2 to 5. Fig. 2 is a diagram showing the configuration of the joint point detection device 10 in the embodiment more specifically. Fig. 3 is a diagram explaining the functions of the total feature output unit in the embodiment. Fig. 4 is a diagram explaining the functions of the partial feature output unit in the embodiment. Fig. 5 is a diagram explaining the functions of the integrated feature output unit in the embodiment.

[0024] As shown in FIG. 2 , in the embodiment, the joint point detection device 10 includes a joint point detection unit 14 and a memory unit 15, in addition to the total feature output unit 11, the partial feature output unit 12, and the integrated feature output unit 13 described above.

[0025] The storage unit 15 stores a first machine learning model 16 and a second machine learning model 17. In the embodiment, the first machine learning model 16 and the second machine learning model 17 are constructed by a convolutional neural network (CNN). In the embodiment, the storage unit 15 also stores a third machine learning model 18, which will be described later. The third machine learning model 18 is also constructed by a CNN. In the following, the machine learning model will also be referred to as "CNN".

[0026] Furthermore, hereinafter, a case where the target of joint point detection is a human hand will be described as an example. Note that in the embodiments, the target of joint point detection is not limited to a human hand, and may be the entire human body or other parts. Also, the target of joint point detection may be anything that has joint points, and may be something other than a human, for example, a robot. Furthermore, in the embodiments, in addition to joint points, parts other than joint points, for example, characteristic parts such as fingertips, may also be detection targets.

[0027] In addition, in the embodiment, a heat map is used as the first feature amount, the second feature amount, and the third feature amount. The heat map is a map that represents the possibility of the existence of a joint point on an image, and for example, the possibility of the existence of a joint point can be represented by a shade of color. Note that something other than a heat map may be used as the first feature amount, the second feature amount, and the third feature amount. This point will be described later in a modified example.

[0028] In the embodiment, the total feature output unit 11 first acquires target image data 20. Then, as shown in Fig. 3, the total feature output unit 11 inputs the image data 20 to the third CNN 18, and causes the third CNN 18 to output a first heat map 21 as a first feature of the joint point. In the example of Fig. 3, a plurality of first heat maps 21 are output for each joint point on the image data 20.

[0029] The third CNN 18 is a machine learning model that learns the relationship between joint points on an image and a heat map. In the machine learning of the third CNN 18, image data of the joint points and a heat map serving as a correct answer are used as training data. The machine learning of the third CNN 18 is performed by updating parameters so that the difference between the output result (heat map) of the image data serving as training data and the heat map serving as a correct answer is reduced.

[0030] In the embodiment, the partial feature output unit 12 uses a first CNN 16 constructed for each specific joint point as shown in Fig. 4. In the example of Fig. 4, the specific joint points are the first joint, the second joint, and the base of the finger, and the first CNN 16 is constructed for each of these joint points. Also, in the example of Fig. 4, the first CNN 16 is constructed for the fingertip in addition to the specific joint points.

[0031] As described above, the first CNN 16 is a machine learning model that learns the positional relationship between joint points other than the specific joint point. In the machine learning of the first CNN 16, the heat map of the joint points other than the corresponding specific joint point (or fingertip) and the correct heat map are used as training data. The machine learning of the first CNN 16 is also performed by updating parameters so that the difference between the output result (heat map) of the image data that is the training data and the correct heat map is reduced.

[0032] The partial feature output unit 12 inputs, to each of the first CNNs 16, the first heat maps 21 of joint points other than the corresponding fingertip or specific joint point, among the first heat maps 21 input from the full feature output unit 11. As a result, the second heat map 22 of the joint points other than the fingertip, the second heat map 22 of the joint points other than the first joint, the second heat map 22 of the joint points other than the second joint, and the second heat map 22 of the joint points other than the base of the finger are output as second features.

[0033] In the embodiment, the integrated feature output unit 13 uses a second CNN 17 constructed for each joint point as shown in Fig. 5. In the example of Fig. 5, the second CNN 17 is constructed for each of the 21 joint points of the hand.

[0034] As described above, the second CNN 17 is a machine learning model that machine-learns the positions of corresponding joint points. In the machine learning of the second CNN 17, the heat map of the corresponding joint points and the correct heat map are also used as training data. The machine learning of the second CNN 17 is also performed by updating parameters so that the difference between the output result (heat map) of the image data that is the training data and the correct heat map is reduced.

[0035] The integrated feature output unit 13 merges the second heat maps 22 output from each of the first CNNs 16, and inputs only the second heat maps 22 of the corresponding joint points to each of the second CNNs 17. As a result, a third heat map 23 is output for each joint point as a third feature.

[0036] Thereafter, the joint point detection unit 14 obtains a third heat map 23 for each joint point of the target hand. Then, the joint point detection unit 14 detects the coordinates of each of the target joint points using the third heat map 23 for each joint point.

[0037] Specifically, the joint point detection unit 14 identifies the location of the highest density in the third heat map 23 for each joint point, and detects the two-dimensional coordinates of the identified location on the image. In addition, when there are multiple third heat maps 23 for each joint point, the joint point detection unit 14 identifies the two-dimensional coordinates of the location of the highest density for each third heat map 23, and further calculates the average of each of the identified two-dimensional coordinates, and sets the calculated average coordinates as the final coordinates.

[0038] [Device operation] Next, the operation of the joint point detection device 10 in the embodiment will be described with reference to FIG. 6. FIG. 6 is a flow diagram showing the operation of the joint point detection device in the embodiment. In the following description, FIGS. 1 to 5 will be referred to as appropriate. Also, in the embodiment, a joint point detection method is implemented by operating the joint point detection device 10. Therefore, the description of the joint point detection method in the embodiment will be replaced with the following description of the operation of the joint point detection device 10.

[0039] As shown in FIG. 6, first, the full feature output unit 11 acquires target image data 20, inputs the acquired image data 20 to the third machine learning model 18, and outputs a first heat map 21 representing each joint point (step A1).

[0040] Next, the partial feature output unit 12 inputs, to each first CNN 16, the first heat map 21 of the corresponding fingertip or joint point other than the specific joint point from the first heat maps 21 output in step A1, and causes each first CNN 16 to output a second heat map 22 representing the fingertip or joint point other than the specific joint point (step A2).

[0041] Next, the integrated feature output unit 13 merges the second heat maps 22 output by each first CNN 16 in step A2, and inputs only the second heat maps 22 of the corresponding articulation points to each second CNN 17, causing them to output a third heat map 23 representing the corresponding articulation points (step A3).

[0042] Next, the joint point detection unit 14 detects the coordinates of each of the target joint points using the third heat map 23 for each joint point output in step A3 (step A4).

[0043] As described above, in the embodiment, for example, since the fingertip, the first joint, the second joint, and the base are likely to be invisible in the image, the second feature amount is output using the first feature amount of the other parts. Then, the third feature amount of each joint point is obtained from the second feature amount output in this way, so that the finally obtained coordinates of each joint point are accurate values. In other words, according to the embodiment, it is possible to improve the estimation accuracy of the position of each joint point.

[0044] [Variations] Next, a modified example of the embodiment will be described with reference to Figs. 7 to 10. Fig. 7 is a diagram showing the configuration of a joint point detection device 10 in the modified example of the embodiment. Fig. 8 is a diagram explaining the function of a total feature output unit in the modified example of the embodiment. Fig. 9 is a diagram explaining the function of a partial feature output unit in the modified example of the embodiment. Fig. 10 is a diagram explaining the function of an integrated feature output unit in the modified example of the embodiment.

[0045] As shown in FIG. 7, in this modification, the first machine learning model 31 and the second machine learning model 32 are constructed by a graph convolution network (GCN). A GCN is a network that receives a graph structure composed of a plurality of nodes as input and executes a convolution process using adjacent nodes. In this modification, the machine learning model is also written as "GCN". Note that the third machine learning model is the third CNN 18 shown in FIG. 2 and FIG. 3.

[0046] In addition, in this modification, the coordinate values ​​indicating the positions of the joint points on the image are used as the first feature amount and the second feature amount. Specifically, since a GCN is used, the feature amount output from the GCN has a graph structure in which each node indicates the coordinate value of the joint point.

[0047] In addition, in this modification, since coordinate values ​​are output as feature amounts, the joint point detection unit 14 shown in Fig. 2 is omitted in the joint point detection device 10. Other than the above-mentioned points, the joint point detection device 10 in this modification also has the configuration described in the embodiment.

[0048] In this modified example, as in the above example, the total feature output unit 11 inputs image data 20 to the third CNN 18 as shown in FIG. 8, and outputs a first heat map 21 as first features of the joint points.

[0049] However, in this modified example, the total feature output unit 11 calculates the coordinate values ​​(first coordinate values) 41 of each target joint point using the output first heat map 21. The calculation of the coordinate values ​​is performed in the same manner as the processing in the above-mentioned joint point detection unit 14. Then, the total feature output unit 11 outputs the first coordinate values ​​41 for each joint point as a graph structure.

[0050] In this modification, the partial feature output unit 12 uses a first GCN 31 constructed for each specific joint point as shown in Fig. 9. In the example of Fig. 9, the specific joint points are the first joint, the second joint, and the base of the finger, and a first GCN 31 is constructed for each of these joint points. Moreover, the first GCN 31 is also constructed for the fingertip in addition to the specific joint points.

[0051] The first GCN 31 is a machine learning model that performs machine learning of positional relationships between joint points other than specific joint points. In the machine learning of the first GCN 31, the coordinate values ​​of joint points other than the corresponding specific joint point (or fingertip) and the coordinate values ​​that are the correct answer are used as training data. The machine learning of the first GCN 31 is also performed by updating parameters so that the difference between the output result (coordinate values) of the image data that is the training data and the coordinate values ​​that are the correct answer is reduced.

[0052] The partial feature output unit 12 inputs, to each first GCN 31, the first coordinate values ​​41 of the corresponding joint points other than the fingertip or the specific joint point, among the first coordinate values ​​41 input from the total feature output unit 11. As a result, the second coordinate values ​​42 other than the fingertip, the second coordinate values ​​42 other than the first joint, the second coordinate values ​​42 other than the second joint, and the second coordinate values ​​42 other than the base of the finger are output as second features.

[0053] In this modification, the integrated feature output unit 13 uses a second GCN 32 constructed for each joint point as shown in Fig. 10. In the example of Fig. 10, the second GCN 32 is constructed for each of the 21 joint points of the hand.

[0054] As described above, the second GCN32 is a machine learning model that machine-learns the positions of corresponding joint points. In the machine learning of the second GCN32, the coordinate values ​​of the corresponding joint points and the coordinate values ​​that are the correct answer are training data. The machine learning of the second GCN32 is also performed by updating parameters so that the difference between the output result (coordinate values) of the image data that is the training data and the coordinate values ​​that are the correct answer is reduced.

[0055] The integrated feature output unit 13 merges the second coordinate values ​​42 output by each of the first GCNs 31, and inputs only the second coordinate values ​​42 of the corresponding joint points to each of the second GCNs 32. As a result, a third coordinate value 43 is output for each joint point as a third feature. The output third coordinate value 43 for each joint point is detected as the coordinate of each joint point.

[0056] In this way, in this modification, GCN is used as the machine learning model and coordinate values ​​are used as the feature values, but in this case too, the second feature values ​​are output using the first feature values ​​of the parts other than the parts that are likely to be invisible in the image. Therefore, in this modification too, it is possible to improve the estimation accuracy of the positions of each joint point.

[0057] [program] The program in the embodiment may be a program that causes a computer to execute steps A1 to A4 shown in Fig. 6. By installing and executing this program in a computer, the joint point detection device and the joint point detection method in the embodiment can be realized. In this case, the processor of the computer functions as a total feature output unit 11, a partial feature output unit 12, an integrated feature output unit 13, and a joint point detection unit 14, and performs processing.

[0058] In addition, in this embodiment, the memory unit 15 may be realized by storing the data files that constitute it in a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer.

[0059] In addition, examples of computers include general-purpose PCs, smartphones, and tablet terminal devices.

[0060] The program in the embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of the total feature output unit 11, the partial feature output unit 12, the integrated feature output unit 13, and the joint point detection unit 14.

[0061] [Physical configuration] Here, a computer that realizes the joint point detection device 10 by executing the program in the embodiment will be described with reference to Fig. 11. Fig. 11 is a block diagram showing an example of a computer that realizes the joint point detection device in the embodiment.

[0062] 11, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0063] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this embodiment, the GPU or FPGA can execute the programs in the embodiments.

[0064] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and is stored in the storage device 113, into the main memory 112, and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0065] Moreover, the program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed on the Internet connected via the communication interface 117.

[0066] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0067] Data reader / writer 116 mediates data transmission between CPU 111 and recording medium 120, reads programs from recording medium 120, and writes processing results in computer 110 to recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

[0068] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

[0069] The joint point detection device 10 in the embodiment can be realized by using hardware corresponding to each part, instead of a computer with a program installed. Furthermore, the joint point detection device 10 may be partially realized by a program and the remaining part by hardware.

[0070] A part or all of the above-described embodiment can be expressed by (Additional Notes 1) to (Additional Notes 12) described below, but is not limited to the following descriptions.

[0071] (Appendix 1) a total feature output unit that outputs, for each joint point of the object, a first feature representing the joint point from the image data of the object; a partial feature output unit that receives, for each of a plurality of specific joint points among the target joint points, the first feature representing joint points other than the specific joint points, and outputs a second feature representing the joint points other than the specific joint points by using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points; an integrated feature output unit that receives, for each of the target joint points, the second feature representing the joint point, and outputs a third feature representing the joint point by using a second machine learning model that performs machine learning on a position of the joint point; A joint point detection device comprising:

[0072] (Appendix 2) 2. The joint point detection device according to claim 1, The apparatus further includes a joint point detection unit that detects coordinates of each of the joint points of the object by using the third feature amount for each of the joint points of the object. A joint point detection device comprising:

[0073] (Appendix 3) 2. The joint point detection device according to claim 1, The first machine learning model and the second machine learning model are constructed by a convolutional neural network; each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility of a joint point being present on the image; A joint point detection device comprising:

[0074] (Appendix 4) 2. The joint point detection device according to claim 1, The first machine learning model and the second machine learning model are constructed by a graph convolutional network; each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image; A joint point detection device comprising:

[0075] (Appendix 5) a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object; a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input; an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point, with the second feature representing the joint point as an input; A joint point detection method comprising the steps of:

[0076] (Appendix 6) 6. The joint point detection method according to claim 5, The method further includes a joint point detection step of detecting coordinates of each of the joint points of the object by using the third feature amount for each of the joint points of the object. A joint point detection method comprising:

[0077] (Appendix 7) 6. The joint point detection method according to claim 5, The first machine learning model and the second machine learning model are constructed by a convolutional neural network; each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility of a joint point being present on the image; A joint point detection method comprising:

[0078] (Appendix 8) 6. The joint point detection method according to claim 5, The first machine learning model and the second machine learning model are constructed by a graph convolutional network; each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image; A joint point detection method comprising:

[0079] (Appendix 9) On the computer, a total feature amount output step of outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object; a partial feature output step of outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing the joint points other than the specific joint points using a first machine learning model that performs machine learning on positional relationships between the joint points other than the specific joint points, using the first feature representing the joint points other than the specific joint points as an input; an integrated feature output step of outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that performs machine learning on a position of the joint point, with the second feature representing the joint point as an input; Run R, Program Hmm.

[0080] (Appendix 10) As described in Appendix 9 program And, The computer includes: a joint point detection step of detecting coordinates of each of the joint points of the object using the third feature amount for each of the joint points of the object; Characterized by program .

[0081] (Appendix 11) As described in Appendix 9 program And, The first machine learning model and the second machine learning model are constructed by a convolutional neural network; each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility of a joint point being present on the image; Characterized by program .

[0082] (Appendix 12) As described in Appendix 9 program And, The first machine learning model and the second machine learning model are constructed by a graph convolutional network; each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image; Characterized by program .

[0083] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0084] This application claims priority based on Japanese Patent Application No. 2021-029410, filed on February 26, 2021, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]

[0085] As described above, according to the present invention, it is possible to improve the accuracy of estimating the positions of joint points. The present invention is useful in fields where posture detection of objects having joint points, such as humans and robots, is required. Specific examples of the field include video surveillance and user interfaces. [Explanation of symbols]

[0086] 10. Joint point detection device 11 All feature output section 12 Partial feature output unit 13 Integrated feature output unit 14 Joint point detection unit 15 Storage section 16 The first machine learning model (CNN) 17 Second Machine Learning Model (CNN) 18 The third machine learning model (CNN) 20 Image data 21 First Heat Map 22 Second Heat Map 23 The third heat map 31 First Machine Learning Model (GCN) 32 Second Machine Learning Model (GCN) 41 First coordinate value 42 Second coordinate value 43 Third Coordinate 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display device 120 Recording media 121 Bus

Claims

1. a total feature value output means for outputting, for each joint point of the object, a first feature value representing the joint point from the image data of the object; a partial feature output means for outputting, for each of a plurality of specific joint points among the target joint points, a second feature representing a joint point other than the specific joint point by using a first machine learning model that corresponds to the specific joint point and that performs machine learning on a positional relationship between the joint points other than the specific joint point; an integrated feature output means for outputting, for each of the target joint points, a third feature representing the joint point using a second machine learning model that corresponds to the joint point and that has machine-learned a position of the joint point; A joint point detection device comprising:

2. The joint point detection device according to claim 1, The method further includes a joint point detection means for detecting coordinates of each of the joint points of the object by using the third feature amount for each of the joint points of the object. A joint point detection device characterized by:

3. The joint point detection device according to claim 1, The first machine learning model and the second machine learning model are constructed by a convolutional neural network; each of the first feature amount, the second feature amount, and the third feature amount includes a heat map representing a possibility that a joint point exists on the image; A joint point detection device characterized by:

4. The joint point detection device according to claim 1, The first machine learning model and the second machine learning model are constructed by a graph convolutional network; each of the first feature amount, the second feature amount, and the third feature amount includes a coordinate value indicating a position of a joint point on an image; A joint point detection device characterized by:

5. outputting, for each joint point of the object, a first feature amount representative of the joint point from the image data of the object; using the first feature amount representing a joint point other than the specific joint point as an input for each of a plurality of specific joint points among the target joint points, and outputting a second feature amount representing the joint point other than the specific joint point using a first machine learning model that corresponds to the specific joint point and that has machine-learned a positional relationship between the joint points other than the specific joint point; outputting, for each of the target joint points, a third feature amount representing the joint point using a second machine learning model that corresponds to the joint point and that has machine-learned the position of the joint point, using the second feature amount representing the joint point as an input; A joint point detection method comprising:

6. On the computer, outputting, for each joint point of the object, a first feature amount representing the joint point from the image data of the object; a first machine learning model that receives, as an input, the first feature amount representing a joint point other than the specific joint point for each of a plurality of specific joint points among the target joint points, and that corresponds to the specific joint point and that has machine-learned a positional relationship between the joint points other than the specific joint point, and outputs a second feature amount representing the joint point other than the specific joint point; outputting, for each of the target joint points, a third feature amount representing the joint point using a second machine learning model that corresponds to the joint point and that has machine-learned a position of the joint point, using the second feature amount representing the joint point as an input; program.

Images