Articulation point detection device, learning model generation device, articulation point detection method, learning model generation method, and program

Abstract

A teaching model generation device 10 comprising: a training data acquisition unit 11 that acquires a first graph structure that indicates a two-dimensional feature value for a joint point and a second graph structure that indicates a three-dimensional feature value for a joint point as correct answer data; and a teaching model generation unit 12 that enters the first graph structure into a graphic convolution network, calculates the difference between the output graph structure and the correct answer data, and machine-learns parameters for the graphic convolution network so as to reduce the difference. The graphic convolution network comprises an intermediate layer and an output layer.The intermediate layer comprises: a feature extractor that executes feature extraction without changing the number of nodes in the graph structure; and a feature extractor that reduces the number of nodes in the graph structure and performs feature extraction. Each feature extractor uses the graph structure output from an upper level feature extractor as inputs therefor. The output layer outputs a graph structure, using the graph structure output by each feature extractor in an intermediate layer among the lowest layers as input therefor.

Description

[Technical field]

[0001] The present invention relates to a joint point detection device and a joint point detection method for detecting joint points of an object from an image, and further relates to a program for implementing these. To The present invention also relates to a learning model generation device and a learning model generation method for generating a learning model for detecting joint points of an object from an image, and further relates to a program for implementing these. To Regarding. [Background technology]

[0002] In recent years, systems that estimate a person's posture from an image have been proposed. Such systems are expected to be used in fields such as image surveillance and user interfaces. For example, in an image surveillance system, if a person's posture can be estimated, it is possible to estimate what a person captured on camera is doing, improving surveillance accuracy. In addition, in a user interface, if a person's posture can be estimated, input by gestures becomes possible.

[0003] For example, Non-Patent Document 1 discloses a system for estimating a human posture, particularly the posture of a human hand, from an image. The system disclosed in Non-Patent Document 1 first acquires image data including an image of a hand, inputs the acquired image data to a neural network that has machine-learned image features for each joint point, and estimates the two-dimensional coordinates of each joint point.

[0004] Next, the system disclosed in Non-Patent Document 1 inputs the estimated two-dimensional coordinates of each joint point into a graph convolution network (hereinafter also referred to as "GCN" (Graphic Convolution Network)) to estimate the three-dimensional coordinates of each joint point. The GCN is a network that receives as input a graph structure composed of multiple nodes and executes convolution processing using adjacent nodes (see, for example, Patent Document 1). In Non-Patent Document 1, in the input graph structure, each node is composed of the two-dimensional coordinates of each joint point.

[0005] In the system disclosed in Non-Patent Document 1, the GCN performs a pooling process multiple times in the first stage to reduce the number of nodes in the input graph structure, ultimately reducing the number of nodes to 1. In the second stage, the GCN performs an unpooling process to increase the number of nodes on a graph structure with 1 node the same number of times as the pooling process. In the second stage, the GCN connects the graph structure in the first stage, which has the same number of nodes as the graph structure being processed in the second stage, to perform convolution, ultimately outputting a graph structure with the same number of nodes as the input graph structure.

[0006] Furthermore, in the system disclosed in Non-Patent Document 1, a graph structure of the two-dimensional coordinates of each joint point is input to the GCN as training data, and machine learning of the GCN is performed so that the difference between the output graph structure and the graph structure of the three-dimensional coordinates of each joint point, which is the correct data, is small. [Prior art documents] [Patent documents]

[0007] [Patent Document 1] JP 2020-27399 A [Non-patent literature]

[0008] [Non-Patent Document 1] Bardia Doosti, Shujon Naha, Majid Mirbagheri, David J. Crandall, "HOPE-Net: A Graph-based Model for Hand-Object Pose Estimation", [online], IEEE Conference on Computer Vision and Pattern Recognition (CVPR), Cornell University, 31 March 2020, [Retrieved February 12, 2021], Internet<URL:https: / / arxiv.org / abs / 2004.00060> Summary of the Invention [Problem to be solved by the invention]

[0009] In the system disclosed in Non-Patent Document 1, a graph structure in the front stage is folded into a graph structure in the rear stage, so a lot of spatial information is folded. However, the number of times of folding is small in the front stage, so that feature extraction is insufficient. In addition, in the system disclosed in Non-Patent Document 1, a graph structure with one node is targeted in the rear stage, so the number of dimensions is insufficient and accurate feature extraction is not possible. For this reason, the system disclosed in Non-Patent Document 1 has a problem that the accuracy of the three-dimensional coordinates of each joint point is low.

[0010] An example of an object of the present invention is to provide a joint point detection device, a learning model generation device, a joint point detection method, a learning model generation method, and a method for detecting a joint point, which can improve detection accuracy when detecting three-dimensional coordinates of a joint point from an image. program The purpose of this invention is to provide [Means for solving the problem]

[0011] In order to achieve the above object, a joint point detection device according to one aspect of the present invention comprises: a graph structure acquisition unit that acquires a first graph structure in which two-dimensional feature values ​​of each of a plurality of joint points of a target are represented by nodes; a graph structure output unit that receives the first graph structure as an input and outputs a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolution network; Equipped with The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by:

[0012] In order to achieve the above object, a learning model generation device according to one aspect of the present invention comprises: a training data acquisition unit that acquires, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a learning model generation unit that inputs the first graph structure into a graph convolutional network, calculates a difference between the graph structure output from the graph convolutional network and the correct answer data, and generates a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; Equipped with The graph convolutional network includes a plurality of hidden layers and an output layer; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by:

[0013] In order to achieve the above object, a joint point detection method according to one aspect of the present invention includes: a graph structure acquisition step of acquiring a first graph structure indicating two-dimensional feature amounts of each of a plurality of joint points of the target; a graph structure output step of inputting the first graph structure and outputting a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolutional network; having The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by:

[0014] In order to achieve the above object, a learning model generation method according to one aspect of the present invention includes: a training data acquisition step of acquiring, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a machine learning model generation step of inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and generating a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; having The graph convolutional network includes a plurality of hidden layers and an output layer; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by:

[0015] In order to achieve the above object, the first aspect of the present invention is program teeth, On the computer, a graph structure acquisition step of acquiring a first graph structure indicating two-dimensional feature amounts of each of a plurality of joint points of the target; a graph structure output step of inputting the first graph structure and outputting a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolutional network; Run height, The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by:

[0016] In order to achieve the above object, the second aspect of the present invention is program teeth, On the computer, a training data acquisition step of acquiring, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a machine learning model generation step of inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and generating a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; Run height, The graph convolutional network includes a plurality of hidden layers and an output layer; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by:

[0017] In order to achieve the above object, a graph convolutional network according to one aspect of the present invention comprises: A plurality of intermediate layers and an output layer are provided, A relationship between a first graph structure indicating two-dimensional feature values ​​of each of a plurality of joint points of a target and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points is constructed by machine learning; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. It is characterized by: Effect of the Invention

[0018] As described above, according to the present invention, it is possible to improve the detection accuracy when detecting the three-dimensional coordinates of a joint point from an image. [Brief description of the drawings]

[0019] [Figure 1] FIG. 1 is a configuration diagram showing a schematic configuration of a learning model generating device according to the first embodiment. [Diagram 2] FIG. 2 is a configuration diagram specifically showing the configuration of the learning model generating device in the first embodiment. [Diagram 3] FIG. 3 is a configuration diagram showing the configuration of a graph convolution network in the first embodiment. [Figure 4] FIG. 4 is an explanatory diagram illustrating processing in the graph convolution network shown in FIG. [Diagram 5] FIG. 5 is a flow diagram showing the operation of the learning model generating device in the first embodiment. [Figure 6] FIG. 6 is a diagram showing a schematic configuration of a joint point detection device according to the second embodiment. [Figure 7] FIG. 7 is a diagram showing a more specific configuration of the joint point detection device in the second embodiment. [Figure 8] FIG. 8 is a flow diagram showing the operation of the joint point detection device in the second embodiment. [Figure 9] FIG. 9 is a block diagram showing an example of a computer that realizes the learning model generating device in the first embodiment and the joint point detecting device in the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0020] (Embodiment 1) In the first embodiment, a learning model generating device, a learning model generating method, and a program for generating a learning model, as well as a graph convolution network, will be described below with reference to FIGS. 1 to 5.

[0021] [Device configuration] First, a schematic configuration of the learning model generating device in the first embodiment will be described with reference to Fig. 1. Fig. 1 is a configuration diagram showing a schematic configuration of the learning model generating device in the first embodiment.

[0022] The learning model generating device 10 according to the first embodiment shown in Fig. 1 is a device that generates a machine learning model for detecting a joint point of a target. As shown in Fig. 1, the learning model generating device 10 includes a training data acquiring unit 11 and a learning model generating unit 12.

[0023] The training data acquisition unit 11 acquires, as training data, a first graph structure indicating two-dimensional features of each of a plurality of target joint points, and a second graph structure indicating three-dimensional features of each of a plurality of target joint points as ground truth data.

[0024] The learning model generation unit 12 inputs the first graph structure to a graph convolution network (GCN) and calculates the difference between the graph structure output from the graph convolution network and the correct answer data. Then, the learning model generation unit 12 generates a machine learning model constructed by the graph convolution network by machine learning the parameters in the graph convolution network so that the calculated difference becomes small.

[0025] The graph convolution network includes a plurality of intermediate layers and an output layer. All or some of the intermediate layers include a feature extractor that performs feature extraction without changing the number of nodes in the graph structure, and a feature extractor that performs feature extraction by reducing the number of nodes in the graph structure. In each of the intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in the upper layer. The output layer uses as input the graph structure output by each feature extractor in the lowest intermediate layer, and outputs the graph structure.

[0026] Thus, in the first embodiment, the graph convolution network includes a feature extractor that performs feature extraction without changing the number of nodes in the graph structure in the intermediate layer, and a feature extractor that performs feature extraction by reducing the number of nodes in the graph structure. Therefore, the graph convolution network 30 can avoid a situation in which feature extraction is insufficient due to a small number of convolutions, and a situation in which feature extraction is inaccurate due to an insufficient number of dimensions. As a result, the first embodiment improves the detection accuracy when detecting the three-dimensional coordinates of joint points from an image.

[0027] Next, the configuration of the learning model generating device 10 in the first embodiment will be described in more detail with reference to Fig. 2. Fig. 2 is a configuration diagram specifically showing the configuration of the learning model generating device in the first embodiment.

[0028] 2, in the first embodiment, the learning model generating device 10 includes a storage unit 13 in addition to the above-mentioned training data acquiring unit 11 and learning model generating unit 12. The storage unit 13 stores a graph convolution network 30 (hereinafter, referred to as "GCN 30").

[0029] In the following, a case where the target is a human hand will be described as an example. Note that in the first embodiment, the target is not limited to a human hand, and may be the entire human body or other parts. The target may be anything that has a joint point, and may be something other than a human, such as a robot.

[0030] In the first embodiment, since the target is a human hand, the nodes constituting the graph structure are represented by two-dimensional or three-dimensional feature values ​​of each joint point of the hand. An example of the feature value is a coordinate value.

[0031] In Fig. 2, 21 indicates a graph structure obtained from image data 20. In graph structure 21, node 22 represents two-dimensional coordinate values ​​of joint points of the hand as feature values. Graph structure 21 is a first graph structure. Also, 23 is a graph structure serving as correct answer data, which is a second graph structure. In second graph structure 23, node 24 represents three-dimensional coordinate values ​​of joint points of the hand as feature values.

[0032] Note that the nodes of the graph structure may represent, in addition to the joint points, two-dimensional or three-dimensional features of parts other than the joint points, for example, characteristic parts such as fingertips. In the first embodiment, the first graph structure serving as training data can be obtained by inputting target image data to a machine learning model that has machine-learned the relationship between image data of joint points and the graph structure.

[0033] In the embodiment, the training data acquiring unit 11 acquires, as training data, a first graph structure 21 and a second graph structure 23 serving as answer data. Then, the training data acquiring unit 11 inputs the acquired training data to the learning model generating unit 12.

[0034] In the embodiment, the learning model generation unit 12 first acquires the GCN 30 from the storage unit 13. Next, the learning model generation unit 12 inputs the first graph structure 21 constituting the training data to the GCN 30, and calculates the difference between the second graph structure output from the GCN 30 and the second graph structure 23 that is the correct answer data. Then, the learning model generation unit 12 updates the parameters of the GCN 30 so as to minimize the calculated difference, and stores the GCN 30 with the updated parameters in the storage unit 13. As a result, a GCN for detecting the three-dimensional coordinates of the joint point of the target is generated.

[0035] Next, the configuration and functions of the graph convolution network 30 in the first embodiment will be specifically described with reference to Fig. 3 and Fig. 4. Fig. 3 is a configuration diagram showing the configuration of the graph convolution network in the first embodiment. Fig. 4 is an explanatory diagram explaining the processing in the graph convolution network shown in Fig. 3.

[0036] As shown in Fig. 3, the GCN 30 includes an input layer 31, multiple hidden layers, and an output layer 33. The input layer 31 receives an input of the first graph structure 21. The multiple hidden layers are composed of a first hidden layer 32a and a second hidden layer 32b, and execute feature extraction of the first graph structure. The output layer 33 uses the graph structures output by the feature extractors in the lowest hidden layer as input, and outputs a second graph structure.

[0037] As shown in Fig. 3, the first hidden layer 32a only includes a first feature extractor (indicated by "○" in Fig. 3) that executes feature extraction without changing the number of nodes in the graph structure. The first feature extractor executes convolution. In addition, in each of the first hidden layers 32a, each feature extractor uses as input a graph structure output by a feature extractor in a higher layer that executes feature extraction with the same number of nodes. When there are multiple feature extractors that serve as input sources, the feature extractor in the first hidden layer 32a connects the graph structures output by each feature extractor and executes convolution.

[0038] The second hidden layer 32b includes one or both of a second feature extractor (indicated by "●" in FIG. 3) that extracts features by reducing the number of nodes in the graph structure and a third feature extractor (indicated by "○" with hatching in FIG. 3) that extracts features by increasing the number of nodes in the graph structure. The second feature extractor performs pooling, and the third feature extractor performs unpooling. The second hidden layer 32b also includes a first feature extractor (indicated by "○" in FIG. 3). In each of the second hidden layers 32b, each feature extractor uses as input a plurality of graph structures output by each feature extractor in the upper layer.

[0039] The input layer 31 also includes a first feature extractor (indicated by "○" in FIG. 3). The output layer 33 also includes a first feature extractor (indicated by "○" in FIG. 3) and a third feature extractor (indicated by hatched "○" in FIG. 3).

[0040] 4, when a first graph structure is input to the GCN 30 configured as above, convolution, pooling, and unpooling are performed in the intermediate layer. Furthermore, in the intermediate layer, information is exchanged between a feature extractor for a graph structure with a large number of nodes and a feature extractor for a graph structure with a small number of nodes.

[0041] That is, as shown in Fig. 4, in the intermediate layer, graph structures with different numbers of nodes are generated by the feature extractors, and the graph structures with different numbers of nodes are exchanged between the feature extractors. In Fig. 4, the numbers attached to the graph structures indicate the number of nodes. As a result, GCN30 suppresses a situation in which feature extraction is insufficient due to a small number of convolutions, and a situation in which feature extraction is inaccurate due to an insufficient number of dimensions.

[0042] [Device operation] Next, the operation of the learning model generation device 10 in the first embodiment will be described with reference to FIG. 5. FIG. 5 is a flow diagram showing the operation of the learning model generation device in the first embodiment. In the following description, FIGS. 1 to 4 will be referred to as appropriate. In addition, in the first embodiment, a learning model generation method is implemented by operating the learning model generation device 10. Therefore, the description of the learning model generation method in the first embodiment will be replaced with the description of the operation of the learning model generation device below.

[0043] First, as shown in FIG. 5, the training data acquisition unit 11 acquires, as training data, a first graph structure indicating two-dimensional features of each of a plurality of target joint points and a second graph structure indicating three-dimensional features of each of a plurality of joint points as ground truth data (step A1).

[0044] Next, the learning model generation unit 12 inputs the first graph structure acquired as training data in step A1 to the GCN 30 and calculates the difference between the graph structure output from the GCN and the correct answer data. Then, the learning model generation unit 12 updates the parameters of the GCN so that the calculated difference becomes smaller (step A2).

[0045] Thereafter, the learning model generation unit 12 stores the GCN whose parameters have been updated in step A2 in the storage unit 13 (step A3). As a result, a GCN capable of detecting the three-dimensional coordinates of joint points is generated.

[0046] As described above, according to the first embodiment, a GCN is constructed that can sufficiently and accurately extract features, thereby improving the detection accuracy when detecting the three-dimensional coordinates of a target joint point from an image.

[0047] [program] The program for generating the learning model in the first embodiment may be a program that causes a computer to execute steps A1 to A3 shown in Fig. 5. By installing and executing this program in a computer, the learning model generating device and the learning model generating method in the first embodiment can be realized. In this case, the processor of the computer functions as the training data acquiring unit 11 and the learning model generating unit 12 and performs the processing.

[0048] In the first embodiment, the storage unit 13 may be realized by storing data files constituting the storage unit 13 in a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0049] The program for generating the learning model in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as either the training data acquisition unit 11 or the learning model generation unit 12.

[0050] (Embodiment 2) Next, in a second embodiment, a joint point detection device, a joint point detection method, and a program for joint point detection will be described with reference to FIGS.

[0051] [Device configuration] First, a schematic configuration of a joint point detection device in the second embodiment will be described with reference to Fig. 6. Fig. 6 is a configuration diagram showing a schematic configuration of a joint point detection device in the second embodiment.

[0052] 6, a joint point detection device 40 according to the second embodiment is a device for detecting joint points of an object, for example, a living body, a robot, etc. As shown in FIG. 6, the joint point detection device 40 includes a graph structure acquisition unit 41 and a graph structure output unit 42.

[0053] The graph structure acquisition unit 41 represents the two-dimensional feature amount of each of the multiple joint points of the target as a node. The graph structure output unit 42 receives the first graph structure. Then, we use a graph convolutional network to estimate the 3D features of each joint point. The second graph structure shown is output.

[0054] The graph convolutional network includes multiple intermediate layers and an output layer, and is constructed by machine learning the relationship between a first graph structure and a second graph structure. All or a part of the multiple intermediate layers include a feature extractor that performs feature extraction without changing the number of nodes in the graph structure, and a feature extractor that performs feature extraction by reducing the number of nodes in the graph structure.

[0055] In each of the intermediate layers, each feature extractor uses as input the graph structure output by the feature extractor in the upper layer. The output layer uses as input the graph structure output by the feature extractor in the lowest intermediate layer, and outputs a graph structure.

[0056] Thus, in the second embodiment, the second graph structure is output using a graph convolution network including a feature extractor that performs feature extraction without changing the number of nodes in the graph structure in the intermediate layer, and a feature extractor that performs feature extraction by reducing the number of nodes in the graph structure. Therefore, according to the second embodiment, it is possible to avoid a situation in which feature extraction is insufficient due to a small number of convolutions, and a situation in which feature extraction is inaccurate due to an insufficient number of dimensions. As a result, according to the second embodiment, the detection accuracy when detecting the three-dimensional coordinates of joint points from an image is improved.

[0057] Next, the configuration and functions of the joint point detection device 40 in the second embodiment will be specifically described with reference to Fig. 7. Fig. 7 is a diagram showing the configuration of the joint point detection device in the second embodiment more specifically.

[0058] 7, in the second embodiment, a joint point detection device 40 includes a storage unit 43 in addition to the above-mentioned graph structure acquisition unit 41 and graph structure output unit 42. The storage unit 43 stores the GCN 30 shown in FIG. 2 in the first embodiment.

[0059] In the second embodiment as well, a case where the target is a human hand will be described as an example. Note that in the second embodiment as well, the target for joint point detection is not limited to a human hand, and may be the entire human body or other parts. Furthermore, the target for joint point detection may be anything that has a joint point, and may be something other than a human, such as a robot.

[0060] Additionally, in the second embodiment, the feature values ​​indicated by the nodes of the graph structure include two-dimensional or three-dimensional coordinate values. Furthermore, the nodes of the graph structure may represent, in addition to the joint points, two-dimensional or three-dimensional feature values ​​of parts other than the joint points, such as characteristic parts such as fingertips.

[0061] In the second embodiment, the graph structure acquisition unit 41 acquires a first graph structure 50 obtained from image data of a human hand, as shown in Fig. 7. As described in the first embodiment, the first graph structure can be obtained by inputting the image data to a machine learning model that has learned the relationship between the image data of joint points and the graph structure by machine learning.

[0062] The graph structure output unit 42 acquires the GCN 30 from the storage unit 43. Then, the graph structure output unit 42 inputs the first graph structure 50 to the GCN 30, and causes the GCN to output a second graph structure 51 indicating three-dimensional feature amounts of each of the multiple joint points.

[0063] As described in the first embodiment, the GCN 30 includes an input layer 31, a plurality of intermediate layers 32a and 32b, and an output layer 33, and is constructed by machine learning the relationship between the first graph structure and the second graph structure. Therefore, in the output second graph structure 51, the three-dimensional feature amounts (coordinate values) indicated by each node have highly accurate values.

[0064] [Device operation] Next, the operation of the joint point detection device 40 in the second embodiment will be described with reference to Fig. 8. Fig. 8 is a flow diagram showing the operation of the joint point detection device in the second embodiment. In the following description, Figs. 6 and 7 will be referred to as appropriate. Also, in the second embodiment, a joint point detection method is implemented by operating the joint point detection device 40. Therefore, the description of the joint point detection method in the second embodiment will be replaced with the following description of the operation of the joint point detection device 40.

[0065] As shown in FIG. 8, first, the graph structure acquisition unit 41 acquires a graph structure obtained from image data of a human hand. The first graph structure 50 is obtained (Step B1).

[0066] Next, the graph structure output unit 42 inputs the first graph structure to the GCN 30, and causes the GCN to output a second graph structure indicating the three-dimensional feature amounts of each of the multiple joint points (step B2).

[0067] In the second graph structure, each node represents the three-dimensional coordinates of each joint point of the target person's hand, and therefore each joint point of the person's hand is detected by step B2. As described above, in the second embodiment, GCN30 is used, and therefore the three-dimensional feature amount (coordinate value) indicated by each node in the output second graph structure is a highly accurate value.

[0068] [program] The program for joint point detection in the second embodiment may be a program that causes a computer to execute steps B1 and B2 shown in Fig. 8. By installing and executing this program in a computer, it is possible to realize the joint point detection device and the joint point detection method in the second embodiment. In this case, the processor of the computer functions as a graph structure acquisition unit 41 and a graph structure output unit 42 and performs the processing.

[0069] In the second embodiment, the storage unit 43 may be realized by storing data files constituting the storage unit 43 in a storage device such as a hard disk provided in the computer, or may be realized by a storage device of another computer. Examples of the computer include a general-purpose PC, a smartphone, and a tablet terminal device.

[0070] The joint point detection program in the second embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as either the graph structure acquisition unit 41 or the graph structure output unit 42.

[0071] [Physical configuration] Here, a computer that realizes the learning model generation device 10 by executing the program in the first embodiment, and a computer that realizes the joint point detection device 40 by executing the program in the second embodiment will be described with reference to Fig. 9. Fig. 9 is a block diagram showing an example of a computer that realizes the learning model generation device in the first embodiment and the joint point detection device in the second embodiment.

[0072] 9, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0073] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this embodiment, the GPU or FPGA can execute the programs in the embodiments.

[0074] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and is stored in the storage device 113, into the main memory 112, and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0075] Moreover, the program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed on the Internet connected via the communication interface 117.

[0076] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0077] Data reader / writer 116 mediates data transmission between CPU 111 and recording medium 120, reads programs from recording medium 120, and writes processing results in computer 110 to recording medium 120. Communication interface 117 mediates data transmission between CPU 111 and other computers.

[0078] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as a flexible disk, or optical recording media such as a CD-ROM (Compact Disk Read Only Memory).

[0079] The learning model generation device 10 and the joint point detection device 40 can be realized by using hardware corresponding to each part, instead of a computer with a program installed. Furthermore, the learning model generation device 10 and the joint point detection device 40 may be realized in part by a program and the remaining part by hardware.

[0080] A part or all of the above-described embodiment can be expressed by (Additional Notes 1) to (Additional Notes 11) described below, but is not limited to the following descriptions.

[0081] (Appendix 1) a graph structure acquisition unit that acquires a first graph structure in which two-dimensional feature values ​​of each of a plurality of joint points of a target are represented by nodes; a graph structure output unit that receives the first graph structure as an input and outputs a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolution network; Equipped with The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A joint point detection device comprising:

[0082] (Appendix 2) 2. The joint point detection device according to claim 1, In the graph convolutional network, the multiple hidden layers include a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; A joint point detection device comprising:

[0083] (Appendix 3) 3. The joint point detection device according to claim 1 or 2, the first graph structure indicates two-dimensional coordinates of each of the joint points; the second graph structure indicates three-dimensional coordinates of each of the joint points; A joint point detection device comprising:

[0084] (Appendix 4) a training data acquisition unit that acquires, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a learning model generation unit that inputs the first graph structure into a graph convolutional network, calculates a difference between the graph structure output from the graph convolutional network and the correct answer data, and generates a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; Equipped with The graph convolutional network includes a plurality of hidden layers and an output layer; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A learning model generation device comprising:

[0085] (Appendix 5) A learning model generation device according to claim 4, In the graph convolutional network, the multiple hidden layers include a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; A learning model generation device comprising:

[0086] (Appendix 6) A learning model generation device according to claim 4 or 5, the first graph structure indicates two-dimensional coordinates of each of the joint points; the second graph structure indicates three-dimensional coordinates of each of the joint points; A learning model generation device comprising:

[0087] (Appendix 7) a graph structure acquisition step of acquiring a first graph structure indicating two-dimensional feature amounts of each of a plurality of joint points of the target; a graph structure output step of inputting the first graph structure and outputting a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolutional network; having The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A joint point detection method comprising:

[0088] (Appendix 8) a training data acquisition step of acquiring, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a machine learning model generation step of inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and generating a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; having The graph convolutional network includes a plurality of hidden layers and an output layer; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A learning model generation method comprising:

[0089] (Appendix 9) On the computer, a graph structure acquisition step of acquiring a first graph structure indicating two-dimensional feature amounts of each of a plurality of joint points of the target; a graph structure output step of inputting the first graph structure and outputting a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolutional network; Run height, The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. Characterized by program .

[0090] (Appendix 10) On the computer, a training data acquisition step of acquiring, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a machine learning model generation step of inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and generating a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; Run height, The graph convolutional network includes a plurality of hidden layers and an output layer; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. Characterized by program .

[0091] (Appendix 11) A plurality of intermediate layers and an output layer are provided, A relationship between a first graph structure indicating two-dimensional feature values ​​of each of a plurality of joint points of a target and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points is constructed by machine learning; All or a part of the plurality of intermediate layers include a feature extractor that executes feature extraction without changing the number of nodes in a graph structure, and a feature extractor that executes feature extraction by reducing the number of nodes in the graph structure, and in each of the plurality of intermediate layers, each feature extractor uses as input the graph structure output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A graph convolutional network characterized by:

[0092] Although the present invention has been described above with reference to the embodiment, the present invention is not limited to the above embodiment. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention.

[0093] This application claims priority based on Japanese Patent Application No. 2021-029412, filed on February 26, 2021, the disclosure of which is incorporated herein in its entirety. [Industrial Applicability]

[0094] As described above, according to the present invention, it is possible to improve the detection accuracy when detecting the three-dimensional coordinates of joint points from an image. The present invention is useful in fields where posture detection of objects having joint points, such as humans and robots, is required. Specific fields include video surveillance and user interfaces. [Explanation of symbols]

[0095] 10 Learning model generation device 11 Training data acquisition section 12 Learning model generation unit 13 Storage section 20 Image data 21 First Graph Structure 22 nodes 23 Second Graph Structure 24 nodes 30 Graph Convolutional Networks (GCN) 31 Input Layer 32a The first middle class 32b The second middle class 33 Output layer 40 Joint point detection device 41 Graph structure acquisition unit 42 Graph structure output section 43 Storage section 50 First Graph Structure 51 Second Graph Structure 110 Computer 111 CPU 112 Main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display device 120 Recording media 121 Bus

Claims

1. a graph structure acquisition means for acquiring a first graph structure in which two-dimensional feature values ​​of each of a plurality of joint points of a target are represented by nodes; a graph structure output means for receiving the first graph structure as an input and outputting a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolution network; Equipped with The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; In the graph convolutional network, the plurality of hidden layers comprises a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A joint point detection device comprising:

2. The joint point detection device according to claim 1, the first graph structure indicates two-dimensional coordinates of each of the articulation points; the second graph structure indicating three-dimensional coordinates of each of the joint points; A joint point detection device comprising:

3. a training data acquisition means for acquiring, as training data, a first graph structure indicating two-dimensional feature values ​​of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature values ​​of each of the plurality of joint points as ground truth data; a learning model generating means for inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and generating a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; Equipped with The graph convolutional network includes a plurality of hidden layers and an output layer; In the graph convolutional network, the plurality of hidden layers comprises a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A learning model generation device comprising:

4. The learning model generating device according to claim 3, the first graph structure indicates two-dimensional coordinates of each of the articulation points; the second graph structure indicating three-dimensional coordinates of each of the joint points; A learning model generation device comprising:

5. obtaining a first graph structure indicating two-dimensional feature amounts of each of a plurality of joint points of the target; Using the first graph structure as an input, outputting a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points using a graph convolutional network; The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; In the graph convolutional network, the plurality of hidden layers comprises a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A joint point detection method comprising:

6. Acquire, as training data, a first graph structure indicating two-dimensional feature amounts of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points as ground truth data; inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and machine-learning parameters in the graph convolutional network so as to reduce the calculated difference, thereby generating a machine learning model constructed by the graph convolutional network; The graph convolutional network includes a plurality of hidden layers and an output layer; In the graph convolutional network, the plurality of hidden layers comprises a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A learning model generation method comprising:

7. On the computer, obtaining a first graph structure indicating two-dimensional feature amounts of each of a plurality of joint points of the target; outputting a second graph structure indicating three-dimensional features of each of the plurality of joint points using a graph convolutional network using the first graph structure as an input; The graph convolutional network includes a plurality of intermediate layers and an output layer, and is constructed by machine learning a relationship between the first graph structure and the second graph structure; In the graph convolutional network, the plurality of hidden layers comprises a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A program characterized by:

8. On the computer, acquiring, as training data, a first graph structure indicating two-dimensional feature amounts of each of a plurality of target joint points and a second graph structure indicating three-dimensional feature amounts of each of the plurality of joint points as ground truth data; inputting the first graph structure into a graph convolutional network, calculating a difference between the graph structure output from the graph convolutional network and the correct answer data, and generating a machine learning model constructed by the graph convolutional network by machine learning parameters in the graph convolutional network so that the calculated difference becomes small; The graph convolutional network includes a plurality of hidden layers and an output layer; In the graph convolutional network, the plurality of hidden layers comprises a first hidden layer and a second hidden layer; The first hidden layer includes only a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, and in each of the first hidden layers, each feature extractor uses as input a graph structure output by a feature extractor in an upper layer that performs feature extraction with the same number of nodes; the second intermediate layer includes one or both of a feature extractor that performs feature extraction without changing the number of nodes in a graph structure, a feature extractor that performs feature extraction by reducing the number of nodes in a graph structure, and a feature extractor that performs feature extraction by increasing the number of nodes in a graph structure, and in each of the second intermediate layers, each feature extractor uses as input a plurality of graph structures output by each feature extractor in an upper layer; The output layer uses the graph structures output by the feature extractors in the lowest intermediate layer as inputs and outputs a graph structure. A program characterized by:

Images