Learning model generation device, joint point detection device, learning model generation method, joint point detection method, and program

Abstract

To improve detection accuracy when detecting three-dimensional coordinates of joint points from an image.SOLUTION: A learning model generation device 10 includes: a data acquisition unit 11 that acquires two-dimensional joint point coordinate data from among training data including the two-dimensional joint point coordinate data capable of identifying two-dimensional coordinates of each joint point of a person in an image and three-dimensional joint point coordinate data capable of identifying three-dimensional coordinates of each joint point, and inputs the acquired two-dimensional joint point coordinate data to a machine learning model; an inter-feature-amount distance calculation unit 12 that acquires, for each piece of the two-dimensional joint point coordinate data, feature amounts calculated by the machine learning model and calculates distances between the feature amounts; a similarity degree calculation unit 13 that uses the three-dimensional joint point coordinate data to calculate similarity degrees between physiques of persons serving as the sources of the training data; a loss calculation unit 14 that calculates loss about the feature amounts using the similarity degrees and the distances between the feature amounts; and a learning model generation unit 15 that updates a parameter of the machine learning model using the loss.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] This disclosure relates to a learning model generation device and a learning model generation method for generating a learning model for detecting human joint points from images, and further relates to a program for implementing these. Furthermore, this disclosure relates to a joint point detection device and a joint point detection method for detecting human joint points from images, and further relates to a program for implementing these. [Background technology]

[0002] In recent years, technologies have been developed to estimate a person's posture by detecting the 3D coordinates of each joint from a 2D image (see, for example, Patent Document 1). Such technologies are expected to be used in fields such as image surveillance systems, sports, and games. Furthermore, in such technologies, a learning model is used to detect the 3D coordinates of each joint.

[0003] The learning model is constructed by machine learning using, for example, the 2D coordinates of joints extracted from a person in an image (hereinafter referred to as "2D joint point coordinates") and the 3D coordinates of the same person's joints (hereinafter referred to as "3D joint point coordinates") as training data (see, for example, Non-Patent Document 1). In the training data, the 3D joint point coordinates correspond to the training data.

[0004] Furthermore, machine learning is performed by inputting 2D joint point coordinates, which serve as training data, into a learning model and updating the parameters of the learning model so that the difference between the output 3D joint point coordinates and the 3D joint point coordinates, which serve as the target data, becomes smaller. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2021-47563 [Non-patent literature]

[0006] [Non-Patent Document 1] Gregory Rogez, Cordelia Schmid, “MoCap-guided Data Augmentation for 3D Pose Estimation in the Wild”, arXiv:1607.02046v2 [cs.CV], 28 Oct 2016, [Retrieved November 1, 2021], Internet<URL:http: / / https: / / arxiv.org / pdf / 1607.02046.pdf> [Overview of the project] [Problems that the invention aims to solve]

[0007] Incidentally, in order to improve the accuracy of detecting 3D joint point coordinates using a learning model, a large amount of training data is required. However, machine learning in a learning model proceeds in a way that outputs values ​​for the average body type, even if the training data is obtained from people with diverse body types. As a result, a problem arises where the detection accuracy decreases when the height of the person being targeted for posture estimation is higher or lower than the height of the person from whom the training data was obtained.

[0008] One example of the purpose of this disclosure is to improve the detection accuracy when detecting the three-dimensional coordinates of joint points from an image. [Means for solving the problem]

[0009] To achieve the above objective, the learning model generation device in one aspect of this disclosure is: A data acquisition unit acquires the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of the person, and inputs the acquired 2D joint point coordinate data into a machine learning model. A feature interval distance calculation unit obtains the feature quantities calculated in the machine learning model for each of the two-dimensional joint point coordinate data, and further calculates the distance between the feature quantities using the obtained feature quantities. A similarity calculation unit calculates the similarity between the body types of the people that formed the basis of the training data, using the three-dimensional joint point coordinate data which is the training data. A loss calculation unit calculates the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features. A learning model generation unit updates the parameters of the machine learning model using the calculated loss, It is characterized by having the following features.

[0010] To achieve the above objective, the joint point detection device in one aspect of this disclosure is: A data acquisition unit obtains 2D joint point coordinate data that can identify the 2D coordinates of each of the multiple joint points of a person in an image. The joint point detection unit applies the acquired 2D joint point coordinate data to a machine learning model that has learned the relationship between the 2D and 3D coordinates of a person's joint points, and detects the 3D coordinates of each of the multiple joint points of the person. Equipped with, The parameters of the aforementioned machine learning model are: In machine learning using 2D joint point coordinate data and 3D joint point coordinate data that can identify the 3D coordinates of multiple joint points in a person, The feature loss for the machine learning model, obtained from the distance between features calculated by the machine learning model and the similarity between the body types of the people that formed the basis of the training data, is used to update the feature loss for the machine learning model. It is characterized by the following:

[0011] Furthermore, in order to achieve the above objective, the learning model generation method in one aspect of this disclosure is: A data acquisition step involves acquiring the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of the person, and inputting the acquired 2D joint point coordinate data into a machine learning model. For each of the two-dimensional joint point coordinate data, the feature quantities calculated in the machine learning model are obtained, and further, the distance between the feature quantities is calculated using the obtained feature quantities in the feature quantity distance calculation step. A similarity calculation step, which involves using the three-dimensional joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data. A loss calculation step, which involves calculating the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features, A learning model generation step in which the parameters of the machine learning model are updated using the calculated loss, It is characterized by having the following:

[0012] Furthermore, in order to achieve the above objectives, the program in one aspect of this disclosure is On the computer, A data acquisition step involves acquiring the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of the person, and inputting the acquired 2D joint point coordinate data into a machine learning model. For each of the two-dimensional joint point coordinate data, the feature quantities calculated in the machine learning model are obtained, and further, the distance between the feature quantities is calculated using the obtained feature quantities in the feature quantity distance calculation step. A similarity calculation step, which involves using the three-dimensional joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data. A loss calculation step, which involves calculating the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features, A learning model generation step in which the parameters of the machine learning model are updated using the calculated loss, It is characterized by causing the execution of [the specified action]. [Effects of the Invention]

[0013] As described above, this disclosure makes it possible to improve the detection accuracy when detecting the three-dimensional coordinates of joint points from an image. [Brief explanation of the drawing]

[0014] [Figure 1] Figure 1 is a schematic diagram showing the general configuration of the learning model generation device in Embodiment 1. [Figure 2] Figure 2 is a diagram specifically showing the configuration of the learning model generation device in Embodiment 1. [Figure 3] Figure 3 shows an example of the bone length vector and the bone length ratio vector used in Embodiment 1. [Figure 4] Figure 4 illustrates the generation of a machine learning model in Embodiment 1. [Figure 5] Figure 5 is a flowchart showing the operation of the learning model generation device in Embodiment 1. [Figure 6] Figure 6 is a configuration diagram showing the configuration of the learning model generation device in Embodiment 2. [Figure 7] Figure 7 is a flowchart showing the operation of the learning model generation device in Embodiment 2. [Figure 8] Figure 8 shows the configuration of the learning model generation device in Embodiment 3. [Figure 9] Figure 9 shows an example of 3D pose data used in Embodiment 3. [Figure 10] Figure 10 is an explanatory diagram illustrating the manipulation process of a 3D pose dataset in Embodiment 3. [Figure 11] Figure 11 is an explanatory diagram illustrating the similarity calculation process in Embodiment 3. [Figure 12] Figure 12 schematically shows a new two-dimensional image created in Embodiment 3. [Figure 13] Figure 13 is a configuration diagram showing the configuration of the joint point detection device in Embodiment 4. [Figure 14] Figure 14 is a flowchart showing the operation of the joint point detection device in Embodiment 4. [Figure 15] Figure 15 is a block diagram showing an example of a computer that implements the learning model generation device and joint point detection device in Embodiments 1 to 4. [Modes for carrying out the invention]

[0015] (Embodiment 1) The learning model generation apparatus, learning model generation method, and program in Embodiment 1 will be described below with reference to Figures 1 to 5.

[0016] [Device configuration] First, the schematic configuration of the learning model generation device in Embodiment 1 will be described using Figure 1. Figure 1 is a configuration diagram showing the schematic configuration of the learning model generation device in Embodiment 1.

[0017] The learning model generation device 10 in Embodiment 1, shown in Figure 1, is a device for generating a machine learning model to estimate the 3D coordinates of joint points of a person from the 2D coordinates of joint points in a 2D image of a person.

[0018] As shown in Figure 1, the learning model generation device 10 includes a data acquisition unit 11, a feature distance calculation unit 12, a similarity calculation unit 13, a loss calculation unit 14, and a learning model generation unit 15.

[0019] The data acquisition unit 11 acquires 2D joint point coordinate data from training data that includes 2D joint point coordinate data and 3D joint point coordinate data of a person in an image, and inputs the acquired 2D joint point coordinate data into a machine learning model. The machine learning model is a model that, upon receiving 2D joint point coordinate data, outputs 3D joint point coordinate data according to the input 2D joint point coordinate data.

[0020] Two-dimensional joint point coordinate data is data that can identify the two-dimensional coordinates of each of the multiple joint points of a person in an image. Three-dimensional joint point coordinate data is data that can identify the three-dimensional coordinates of each of the multiple joint points of a person and serves as training data.

[0021] The feature distance calculation unit 12 obtains the features calculated by the machine learning model for each 2D joint point coordinate data, and further calculates the distance between features using the obtained features for each 2D joint point coordinate data. The similarity calculation unit 13 uses the 3D joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data.

[0022] The loss calculation unit 14 calculates the loss for the features in the machine learning model using the similarity calculated by the similarity calculation unit 13 and the distance between features calculated by the feature distance calculation unit 12. The learning model generation unit 15 updates the parameters of the machine learning model 20 using the loss calculated by the loss calculation unit 14.

[0023] Thus, in Embodiment 1, a loss is calculated that reflects the difference in the feature space of the person from whom the training data was obtained and the variation in body size, and the parameters of the machine learning model are updated based on this loss. Therefore, using the machine learning model obtained in Embodiment 1, the problem that "detection accuracy decreases when the height of the person whose posture is to be estimated is higher or lower than the height of the person from whom the training data was obtained" is resolved. In other words, according to this embodiment, the detection accuracy when detecting the 3D coordinates of joint points from an image is improved.

[0024] Next, the configuration and functions of the learning model generation device 10 in Embodiment 1 will be specifically described using Figures 2 to 4. Figure 2 is a configuration diagram specifically showing the configuration of the learning model generation device in Embodiment 1.

[0025] As shown in Figure 2, the learning model generation device 10 includes, in addition to the data acquisition unit 11, feature distance calculation unit 12, similarity calculation unit 13, loss calculation unit 14, and learning model generation unit 15 described above, a ground truth data acquisition unit 16 and a machine learning model 20.

[0026] In Embodiment 1, the machine learning model 20 is a neural network, specifically a DNN (Deep Neural Network). The machine learning model 20 has an input layer, a hidden layer (intermediate layer), and an output layer. The machine learning model 20 is actually implemented by a machine learning program that runs on a computer. Alternatively, the machine learning model 20 may be implemented on a device (computer) separate from the learning model generation device 10.

[0027] In Embodiment 1, the 2D and 3D joint point coordinate data, which serve as training data for the machine learning model 20, are stored in the database 30. The 3D joint point coordinate data is the training data.

[0028] Here, the 2D joint point coordinate data can be an image of a person and a set of 2D coordinates for each joint point on the image. Alternatively, instead of a set of 2D coordinates for each joint point, a map that represents the probability of each joint point existing, such as a heatmap, may be used.

[0029] The 3D joint point coordinate data serves as training data. Each 3D joint point coordinate data corresponds to one 2D joint point coordinate data. The 3D joint point coordinate data can be defined as the set of 3D coordinates of each joint point of a person in the corresponding 2D joint point coordinate data.

[0030] The data acquisition unit 11 acquires each of the two-dimensional joint point coordinate data prepared as training data from the database 30, and sequentially inputs each of the acquired two-dimensional joint point coordinate data into the machine learning model 20.

[0031] Each time the data acquisition unit 11 inputs 2D joint point coordinate data to the machine learning model 20, the feature distance calculation unit 12 acquires the features calculated in the machine learning model 20, specifically the output values ​​of the hidden layers of the machine learning model 20 (hereinafter also referred to as "intermediate features").

[0032] The feature distance calculation unit 12 then sets combinations of two intermediate features so that each acquired intermediate feature is examined exhaustively, and calculates the feature distance for each combination of two intermediate features. In other words, the combination of two intermediate features is the combination of people that formed the basis of the training data input into the machine learning model 20. Therefore, the feature distance calculation unit 12 calculates the feature distance for each combination of people that formed the basis of the training data input into the machine learning model 20. Hereafter, "combinations of people that formed the basis of the training data input into the machine learning model 20" will also be simply referred to as "combinations of people".

[0033] Specifically, the feature distance calculation unit 12 calculates the difference between two intermediate features as the feature distance. Here, for example, if the intermediate features of person A are fea A , the intermediate features of person B are fea B Therefore, the distance between features is "L2_norm( fea A - fea B It is expressed as ")".

[0034] The ground truth data acquisition unit 16 acquires 3D joint point coordinate data, which is both training data and teacher data, from the database 30. The ground truth data acquisition unit 16 then passes each acquired 3D joint point coordinate data to the similarity calculation unit 13.

[0035] The similarity calculation unit 13 calculates the similarity between body builds for each combination of persons that were the source of the training data input to the machine learning model 20, using the received three-dimensional joint point coordinate data. Specifically, the similarity calculation unit 13 first calculates, for each person, a bone length vector using the three-dimensional joint point coordinate data, and further calculates a "bone length ratio vector" from the calculated bone length vector.

[0036] FIG. 3 is a diagram showing an example of the bone length vector and the bone length ratio vector used in Embodiment 1. As shown in FIG. 3, the bone length vector is composed of "the length from the right shoulder to the right elbow", "the length from the right elbow to the right wrist", "the length from the right hip to the right ankle", "the length from the left hip to the left ankle", etc. Also, as shown in FIG. 3, each length is calculated from the difference in coordinate values between joint points. The bone length ratio vector is calculated by dividing each length constituting the bone length vector by a reference length.

[0037] Subsequently, the similarity calculation unit 13 calculates an average vector for the bone length ratio vectors of all persons targeted by the training data. Here, the average vector is denoted as "phy mean ". Further, the similarity calculation unit 13 calculates, for each person, a vector representing their body build (hereinafter denoted as "body build vector"). For example, if the body build vector of person A is "phy A " and the body build vector of person B is "phy B ", the body build vector is calculated by the following Equation 1.

[0038]

Equation

[0039] Subsequently, the similarity calculation unit calculates, for each combination of persons that were the source of the training data input to the machine learning model 20, a similarity, for example, the cosine similarity (cos_sim(phy A , phy B)) is calculated. Note that in Embodiment 1, the similarity is not limited to cosine similarity, and for example, Euclidean distance may be used.

[0040] In Embodiment 1, the loss calculation unit 14 calculates the cosine similarity (cos_sim(phy) calculated by the similarity calculation unit 13. A phy B )) and the feature distance (L2_norm( fea A - fea B Using ), the loss for features in machine learning model 20 m Calculate.

[0041] Specifically, the loss calculation unit 14 uses, for example, the following equation 2 to calculate the loss m The following equation 2 calculates the loss. In equation 2 below, i and j are indices indicating the people from whom the training data input to the machine learning model 20 originated. (i, j) indicates the combination of people from whom the training data input to the machine learning model 20 originated. Note that (i, j) and (j, i) overlap, so in equation 2 below, only one of them is calculated. The loss calculation unit 14 uses a different formula to calculate the loss. m It is also possible to calculate this.

[0042]

number

[0043] The learning model generation unit 15 processes each loss calculated by the loss calculation unit 14. pThe parameters of the DNN, which is the machine learning model 20, are updated so that the ratio becomes smaller. As a result, as shown in Figure 4, the distance between features in the feature space of the DNN becomes a distance proportional to the size of the person. Figure 4 is a diagram illustrating the generation of the machine learning model in Embodiment 1. As a result of updating the parameters in this way, a machine learning model 20 is generated that can accurately estimate the 3D coordinates of joint points without being affected by the size of the person on which the training data is based.

[0044] [Device operation] Next, the operation of the learning model generation device 10 in Embodiment 1 will be explained using Figure 5. Figure 5 is a flowchart showing the operation of the learning model generation device in Embodiment 1. In the following explanation, Figures 1 to 4 will be referred to as appropriate. In Embodiment 1, the learning model generation method is carried out by operating the learning model generation device 10. Therefore, the explanation of the learning model generation method in Embodiment 1 will be replaced by the following explanation of the operation of the learning model generation device 10.

[0045] As shown in Figure 5, first, the data acquisition unit 11 acquires 2D joint point coordinate data for each person, which is prepared as training data (step A1). Next, the data acquisition unit 11 inputs the 2D joint point coordinate data acquired in step A1 into the machine learning model 20 (step A2). Steps A1 and A2 may be performed for all of the prepared training data, or for only a set number of training data.

[0046] Next, in step A2, when the data acquisition unit 11 inputs the 2D joint point coordinate data to the machine learning model 20, the feature distance calculation unit 12 acquires the intermediate features calculated by the machine learning model 20 (step A3). Steps A2 and A3 are repeated for the number of 2D joint point coordinate data acquired in step A1.

[0047] Next, the feature distance calculation unit 12 calculates the feature distance for each combination of two intermediate features once intermediate features have been obtained for all the 2D joint point coordinate data acquired in step A1 (step A4).

[0048] Next, the correct answer data acquisition unit 16 acquires 3D joint point coordinate data from the database 30 and passes each acquired 3D joint point coordinate data to the similarity calculation unit 13 (step A5).

[0049] Next, the similarity calculation unit 13 uses the received 3D joint point coordinate data to calculate the similarity between body types for each combination of people that formed the basis of the training data input to the machine learning model 20 (step A6).

[0050] Next, the loss calculation unit 14 uses the similarity calculated by the similarity calculation unit 13 and the distance between features calculated by the feature distance calculation unit 12, and uses the above equation 2 to calculate the loss for features in the machine learning model. m Calculate (Step A7).

[0051] Next, the learning model generation unit 15 uses the loss calculated in step A7. m Update the parameters of the machine learning model 20 so that the value becomes smaller (Step A8).

[0052] As described above, according to Embodiment 1, a loss is calculated that reflects the difference in the feature space of the person from whom the training data was obtained and the variation in body size, and the parameters of the machine learning model are updated based on this loss. As a result, a machine learning model 20 capable of accurately estimating the 3D coordinates of joint points is generated. According to Embodiment 1, the problem that "detection accuracy decreases when the height of the person whose posture is to be estimated is higher or lower than the height of the person from whom the training data was obtained" is resolved, and the detection accuracy when detecting the 3D coordinates of joint points from an image is improved.

[0053] [Differentiation] Next, a modification of Embodiment 1 will be described. In this modification, an SMPL (Skinned Multi-Person Linear) model is created for the person used as training data. The SMPL model is a model that can identify the surface shape of a person's body and has a shape vector β that indicates whether the person is overweight or not. The shape vector β consists of 10 parameters that indicate the shape of the body. In this modification, the 3D joint point coordinate data includes the shape vector β.

[0054] Therefore, in this modified example, the correct data acquisition unit 16 acquires a shape vector β as the 3D joint point coordinate data of a person, and passes the acquired shape vector β to the similarity calculation unit 13. In this case, the similarity calculation unit 13 uses the shape vector β for each person to calculate the similarity between body types.

[0055] Furthermore, the SMPL model includes parameters that indicate the 3D coordinates of the joint points, in addition to the shape vector β. Therefore, if an SMPL model has been created for the person to be used as training data, the ground truth data acquisition unit 16 acquires the 3D joint point coordinate data from the SMPL model for each person and passes each acquired 3D joint point coordinate data to the similarity calculation unit 13.

[0056] [program] The program in Embodiment 1 can be any program that causes a computer to execute steps A1 to A8 shown in Figure 5. By installing and running this program on a computer, the learning model generation device 10 and the learning model generation method in Embodiment 1 can be realized. In this case, the computer's processor functions as a data acquisition unit 11, a feature distance calculation unit 12, a similarity calculation unit 13, a loss calculation unit 14, a learning model generation unit 15, and a ground truth data acquisition unit 16, and performs processing. Examples of computers include general-purpose PCs, smartphones, and tablet devices. The computer's processor also constructs the machine learning model 20.

[0057] Furthermore, the program in Embodiment 1 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: a data acquisition unit 11, a feature distance calculation unit 12, a similarity calculation unit 13, a loss calculation unit 14, a learning model generation unit 15, and a ground truth data acquisition unit 16.

[0058] (Embodiment 2) Next, the learning model generation apparatus, learning model generation method, and program in Embodiment 2 will be described with reference to Figures 6 and 7.

[0059] [Device configuration] First, the configuration of the learning model generation device in Embodiment 2 will be explained using Figure 6. Figure 6 is a configuration diagram showing the configuration of the learning model generation device in Embodiment 2.

[0060] The learning model generation device 40 in Embodiment 2, shown in Figure 2, is a device for generating a machine learning model 20, similar to the learning model generation device 10 shown in Figure 2 in Embodiment 1.

[0061] Furthermore, as shown in Figure 2, the learning model generation device 40, like the learning model generation device 10, includes a data acquisition unit 11, a feature distance calculation unit 12, a similarity calculation unit 13, a loss calculation unit 14, a learning model generation unit 15, a ground truth data acquisition unit 16, and a machine learning model 20.

[0062] However, as shown in Figure 2, unlike the learning model generation device 10, the learning model generation device 40 includes, in addition to the above, a loss integration unit 41 and a second loss calculation unit 42. The differences from Embodiment 1 will be explained below.

[0063] The second loss calculation unit 42 acquires the 3D joint point coordinate data output by the machine learning model 20 in response to the input of 2D joint point coordinate data (training data) from the data acquisition unit 11. Then, the second loss calculation unit 42 uses the 3D joint point coordinate data output by the machine learning model 20 and the 3D joint point coordinate data which is the training data to be input to the machine learning model 20 to calculate the loss for the output of the machine learning model 20 for each person from whom the training data was based. After that, the second loss calculation unit 42 sums up the losses for each person and calculates the loss loss. p Calculate.

[0064] loss loss p The calculation process is represented by the following equation 3. Below, 3D_data m This is the 3D joint point coordinate data output by the machine learning model 20, and is 3D_data t is the 3D joint point coordinate data, which is the training data. i is an index indicating the person from whom the training data was input to the machine learning model 20.

[0065]

number

[0066] The loss integration unit 41 calculates the loss for the feature quantities calculated by the loss calculation unit 14. m and the loss for the output calculated by the second loss calculation unit 42 p The and are integrated. Specifically, the loss integration unit 41 uses the following equation 4 to calculate the loss for the feature quantities. m and loss about the output p The two values ​​are combined by calculating a weighted average of the two to obtain the final loss. In equation 4 below, λ represents the weighting coefficient in the weighted average. The value of the weighting coefficient λ is set as appropriate.

[0067]

number

[0068] In Embodiment 2, the learning model generation unit 15 uses the loss obtained through integration to update the parameters of the DNN, which is the machine learning model 20, so that the loss is reduced.

[0069] [Device operation] Next, the operation of the learning model generation device 40 in Embodiment 2 will be explained using Figure 7. Figure 7 is a flowchart showing the operation of the learning model generation device in Embodiment 2. In the following explanation, Figure 6 will be referred to as appropriate. In Embodiment 2, the learning model generation method is carried out by operating the learning model generation device 40. Therefore, the explanation of the learning model generation method in Embodiment 2 will be replaced by the following explanation of the operation of the learning model generation device 40.

[0070] As shown in Figure 7, first, the data acquisition unit 11 acquires 2D joint point coordinate data for each person, which is prepared as training data (step B1). Next, the data acquisition unit 11 inputs the 2D joint point coordinate data acquired in step B1 into the machine learning model 20 (step B2).

[0071] Next, in step B2, when the data acquisition unit 11 inputs the 2D joint point coordinate data to the machine learning model 20, the feature distance calculation unit 12 acquires the intermediate features calculated by the machine learning model 20 (step B3). Steps B2 and B3 are repeated for the number of 2D joint point coordinate data acquired in step B1.

[0072] Next, the feature distance calculation unit 12 calculates the feature distance for each combination of two intermediate features once intermediate features have been obtained for all the 2D joint point coordinate data acquired in step B1 (step B4).

[0073] Next, the correct data acquisition unit 16 acquires 3D joint point coordinate data from the database 30 and passes each acquired 3D joint point coordinate data to the similarity calculation unit 13 (step B5).

[0074] Next, the similarity calculation unit 13 uses the received 3D joint point coordinate data to calculate the similarity between body types for each combination of people that formed the basis of the training data input to the machine learning model 20 (step B6).

[0075] Next, the loss calculation unit 14 uses the similarity calculated by the similarity calculation unit 13 and the distance between features calculated by the feature distance calculation unit 12, and uses the above equation 2 to calculate the loss for features in the machine learning model. m Calculate (Step B7). Note that in Embodiment 2, Steps B1 to B7 are the same as Steps A1 to A7 in Embodiment 1.

[0076] Next, the second loss calculation unit 42 acquires the 3D joint point coordinate data output by the output layer of the machine learning model 20 in response to the input in step B3. Furthermore, the second loss calculation unit 42 uses the above equation 3 to calculate the loss for the output of the machine learning model 20 for each person, using the acquired 3D joint point coordinate data and the training data 3D joint point coordinate data. Then, the second loss calculation unit 42 sums up the losses for each person and calculates the loss loss p Calculate (Step B8).

[0077] Next, the loss integration unit 41 uses the above number 4 to calculate the loss for the feature calculated in step B7. p and the loss for the output calculated in step B8 p Combine these to calculate the final loss (Step B9).

[0078] Subsequently, the learning model generation unit 15 updates the parameters of the machine learning model 20 so that the loss calculated in step B9 becomes smaller (step B10).

[0079] As described above, in Embodiment 2 as well, a loss is calculated that reflects the difference in the feature space of the person from whom the training data was obtained and the variation in body size, and the parameters of the machine learning model are updated based on this loss. For this reason, in Embodiment 2 as well, the problem that "detection accuracy decreases when the height of the person whose posture is to be estimated is higher or lower than the height of the person from whom the training data was obtained" is resolved, and the detection accuracy when detecting the 3D coordinates of joint points from images is improved. Furthermore, according to Embodiment 2, the loss due to the output of the machine learning model 20 is also taken into consideration, so the detection accuracy is further improved.

[0080] [program] The program in Embodiment 2 can be any program that causes a computer to execute steps B1 to B10 shown in Figure 7. By installing and running this program on a computer, the learning model generation device 10 and the learning model generation method in Embodiment 2 can be realized. In this case, the computer's processor functions as a data acquisition unit 11, a feature distance calculation unit 12, a similarity calculation unit 13, a loss calculation unit 14, a learning model generation unit 15, a ground truth data acquisition unit 16, a loss integration unit 41, and a second loss calculation unit 42, and performs processing. Examples of computers include general-purpose PCs, smartphones, and tablet devices. The computer's processor also constructs the machine learning model 20.

[0081] Furthermore, the program in Embodiment 2 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as one of the following: data acquisition unit 11, feature distance calculation unit 12, similarity calculation unit 13, loss calculation unit 14, learning model generation unit 15, ground truth data acquisition unit 16, loss integration unit 41, and second loss calculation unit 42.

[0082] (Embodiment 3) Next, the learning model generation apparatus, learning model generation method, and program in Embodiment 3 will be described with reference to Figures 8 to 11.

[0083] Figure 8 shows the configuration of the learning model generation device in Embodiment 3. As shown in Figure 8, in Embodiment 3, the learning model generation device 40 is combined with the data augmentation device 50. The learning model generation device 40 is the same as the one shown in Figure 6 in Embodiment 2. In Embodiment 3, the learning model generation device 10 may be used instead of the learning model generation device 40. The data augmentation device 50 will be described in detail below.

[0084] As shown in Figure 8, the data expansion device 50 includes a data acquisition unit 51, a projection processing unit 52, a data search unit 53, an image generation unit 54, and a database 70.

[0085] The data acquisition unit 11 acquires a 3D pose dataset 60 as shown in Figure 9. Figure 9 is a diagram showing an example of 3D pose data used in Embodiment 3. As shown in Figure 9, the 3D pose dataset 60 consists of a set of 3D coordinates for each joint point 61 of a person. The 3D pose dataset is also accompanied by identification data (right wrist, left wrist, neck, etc.) to identify each joint point 61.

[0086] Furthermore, in the example in Figure 9, the 3D coordinates of each joint point 61 are expressed in the camera coordinate system, but the coordinate system is not particularly limited. The 3D coordinates of each joint point 61 may also be in the world coordinate system. The camera coordinate system is a coordinate system with the camera's position as the origin. In the camera coordinate system, the horizontal direction of the camera is set as the x-axis, the vertical direction as the y-axis, and the optical axis direction as the z-axis. The z-coordinate represents the distance from the camera. The world coordinate system is a coordinate system arbitrarily set in real space, with the origin set at the ground beneath the camera's feet. In the world coordinate system, the vertical direction is set as the Z-axis.

[0087] The projection processing unit 52 projects each of the joint points 61 (see Figure 9) included in all or specific parts of the 3D pose dataset 60 onto a 2D plane, i.e., the image coordinate system, and generates the projected coordinates (2D coordinates) of each joint point 61 in the image coordinate system. The image coordinate system is a coordinate system on a 2D image, and is usually set as the origin at the top left pixel.

[0088] Database 70 has pre-registered multiple data sets 71. Each data set 71 is a set of data that associates a 3D pose dataset of a person with 2D image data of a person in the same pose as the 3D pose dataset, and the corresponding camera parameters.

[0089] For camera parameters, if the 3D coordinates of the joint points are expressed in the camera coordinate system, intrinsic parameters are used. If the 3D coordinates of the joint points are expressed in the world coordinate system, both intrinsic and extrinsic parameters are used. Intrinsic parameters are represented by a matrix connecting the camera coordinate system and the image coordinate system, focal length, optical axis misalignment, etc. Extrinsic parameters are represented by a matrix connecting the world coordinate system and the camera coordinate system, the camera's position relative to the world coordinates, and the camera's tilt.

[0090] The data search unit 53 performs the following processing for each set of data. A set of data is data that associates a set of 3D coordinates for each joint point of a person, a 2D image of that person, and camera parameters with each other.

[0091] The data search unit 53 first uses internal parameters to identify the corresponding 2D coordinates in the image coordinate system for each 3D coordinate of each joint point included in all or specific parts of the 3D pose dataset of the data set.

[0092] Next, the data search unit 53 manipulates the 3D pose dataset of the target data for each data set so that the set of projected coordinates generated from the 3D pose dataset overlaps with the set of identified 2D coordinates. Then, the data search unit 13 calculates the similarity between the manipulated 3D pose dataset and the 3D pose dataset of the data set for each data set. Furthermore, if projected coordinates and 2D coordinates are obtained for a specific part, the data search unit 13 calculates the similarity using the 3D pose dataset of that specific part.

[0093] Specifically, first, as a condition for overlap, it is set that, for each set of data, two or more joint points included in the generated set of projected coordinates coincide with two or more joint points included in the identified set of 2D coordinates. The data search unit 53 performs one or a combination of translation, rotation, scaling, and shrinking on the 3D pose dataset (set of 3D coordinates) of the target data or set of data as an operation to satisfy the condition.

[0094] Then, when the conditions are met, the data search unit 53 finds a unit vector pointing from a specific joint point to another joint point in the 3D coordinates after the operation, and a unit vector pointing from a specific joint point to another joint point in the 3D coordinates of the data set. The data search unit 53 then calculates the similarity based on the two unit vectors it has found. Subsequently, the data search unit 53 identifies the data set corresponding to the initially acquired 3D pose dataset based on the similarity calculated for each data set.

[0095] The manipulation process of the 3D pose dataset and the similarity calculation process by the data search unit 53 will be explained in more detail using Figures 10 and 11. Figure 10 is an explanatory diagram illustrating the manipulation process of the 3D pose dataset in Embodiment 3. Figure 11 is an explanatory diagram illustrating the similarity calculation process in Embodiment 3.

[0096] First, the 3D pose dataset of the target data is p(={p1,p2,···p n}), the 3D pose dataset of the set data in database 70 is q(={q1,q2,···q n}) Let p n and q n These indicate joint points.

[0097] As shown in Figure 10, in the target data, two articular points p of the same person j and p i Assume the following: Articular point p j and articular point p j The set of joint points connected by bones is p AD,j Let's assume that articular point p j and p i In a 3D pose dataset, p c j and p c i This is how it is written. This joint point p c j and p c i The joint points obtained by projecting onto the image coordinate system are p l j and p l i It is written as p l i ∈p l AD,j is, p l j Let it be the joint point furthest from the center.

[0098] Furthermore, in the set data, the two corresponding joint points q j and q i Assuming this, the set of joint points connected to these by bone is q AD,j Let's assume the joint point is q. j and q i In a 3D pose dataset, q c j and q c i This is how it is written. Articular point q j and q i The joint point in the image coordinate system corresponding to q l j and q l iis denoted as. Also, q l i ∈q l AD,j is the joint point farthest from q l j in q

[0099] As shown in FIG. 9, the data search unit 53, in the image coordinate system, for the joint points p l j and p l i and the joint points q l j and q l i in the camera coordinate system, performs any one or a combination of translation, rotation, enlargement, and reduction so that they match. Also, by this operation, the joint points in the image coordinate system including q c will also be operated. After the operation, q l j and q l i are represented by q l j and q c j respectively (see FIG. 5). l’ j and q c’ j

[0100] Also, in the example of FIG. 9, the rotation is performed only within the xy plane of the camera coordinate system. The enlargement and reduction are performed at the same magnification on any of the x-axis, y-axis, and z-axis of the camera coordinate system. Also, according to the operation by the data search unit 13, any one or a combination of translation, rotation, enlargement, and reduction is performed on the two-dimensional image I constituting the combined data. Let the two-dimensional image after the operation be I'.

[0101] After the operation, as shown in FIG. 11, the data search unit 53, in the target data, for the joint points p c j from p c k ∈p C AD,j the unit vector t jk ​To find the joint point q in the set data, c’ j From q c’ k ∈q C’ AD,j A unit vector s pointing towards it jk Next, the data search unit 13 uses the following equation 5 to find the joint point p. c j The structure centered around the joint point q c’ j The structure centered around and the similarity D j Calculate k, where p c k ∈p c AD,j This is the index of the articular points that satisfy the condition.

[0102]

number

[0103] Note that in the above number 5, cosine similarity is used as the measure of similarity. The similarity is p c k ∈p C AD,j and q c’ k ∈q C’ AD,j The Euclidean distance between the two points may also be used.

[0104] The data search unit 53 calculates the similarity D for all the paired data stored in the database 70. j We calculate the similarity D of these. j The data set that maximizes this value is identified. If projected coordinates and 2D coordinates are available for a specific part, the data set that is similar only for that specific part will be identified.

[0105] The image generation unit 54 generates a patch image by extracting a specific part (e.g., left foot, right foot, right arm, etc.) from the 2D image I' after the above-described operation, if a set of data with similar specific parts has been identified. The image generation unit 14 also generates a patch image by extracting the joint point q in the coordinate system of the manipulated image. l’j And the joint points q of the 3D pose dataset after manipulation c’ j Using this, a portion of the corresponding 3D pose dataset is assigned to the generated patch image. The image generation unit 14 then combines the generated patch image with another 2D image (such as an image of a person with a specific part obscured) to generate a new 2D image. The new 2D image obtained in this way is stored in the database 30 as training data for the machine learning model 20.

[0106] Furthermore, the data search unit 53 can identify the data set with the highest similarity for each different body part. In this case, the image generation unit 54 generates patch images for each body part, and then pastes each patch image onto a background image to generate a new image of a single person (a new 2D image). At this time, the image generation unit 14 also synthesizes a 3D pose dataset corresponding to each patch image. The new 2D image obtained in this way, as well as the synthesized 3D pose dataset, become training data for the machine learning model 20.

[0107] Figure 12 schematically shows a new two-dimensional image created in Embodiment 3. In the example in Figure 12, the new two-dimensional image is created by combining patch images 62, 63, 64, 65, and a background image 66, each representing a different area.

[0108] Thus, in Embodiment 3, new training data is created using the data expansion device 50. Therefore, problems caused by insufficient training data, such as a decrease in detection accuracy, are suppressed.

[0109] (Embodiment 4) Next, in Embodiment 4, the joint point detection device, joint point detection method, and program will be described with reference to the drawings.

[0110] [Device configuration] First, the configuration of the joint point detection device in Embodiment 4 will be explained using Figure 13. Figure 13 is a configuration diagram showing the configuration of the joint point detection device in Embodiment 4.

[0111] As shown in Figure 13, the joint point detection device 80 comprises a data acquisition unit 81 and a joint point detection unit 82. The joint point detection device 80 also includes a machine learning model 20.

[0112] The data acquisition unit 81 acquires 2D joint point coordinate data that can identify the 2D coordinates of each of the multiple joint points of a person in the image. The 2D joint point coordinate data acquired by the data acquisition unit 81 is 2D joint point coordinate data of a person for which the detection of the 3D coordinates of each joint point is required. The 2D joint point coordinate data is input from an external device or the like.

[0113] The joint point detection unit 82 applies the two-dimensional joint point coordinate data acquired by the data acquisition unit 81 to the machine learning model 20 to detect the three-dimensional coordinates of each of the multiple joint points of the person.

[0114] The machine learning model 20 is a machine model that learns the relationship between the 2D and 3D coordinates of human joint points. In Embodiment 4, the machine learning model 20 is a machine learning model created according to Embodiments 1 to 3.

[0115] In other words, in Embodiment 4, the machine learning model 20 is created by machine learning using 2D joint point coordinate data and 3D joint point coordinate data, which serve as training data. The parameters of the machine learning model 20 are updated, as in Embodiment 1, using the distance between features obtained from the features calculated by the machine learning model 20 and the loss on features obtained from the similarity between the body sizes of the people that formed the basis of the training data.

[0116] Furthermore, the parameters 40 of the machine learning model 20 may be updated, similar to Embodiment 2, not only using the loss on features, but also using the loss obtained from the 3D joint point coordinate data output by the machine learning model 20 and the 3D joint point coordinate data used for training.

[0117] In Embodiment 4, the machine learning model 20 is implemented by a machine learning program executed on a computer. Alternatively, the machine learning model 20 may be implemented on a separate device (computer) from the joint point detection device 80.

[0118] [Device operation] Next, the operation of the joint point detection device 80 in Embodiment 4 will be described using Figure 14. Figure 14 is a flowchart showing the operation of the joint point detection device 80 in Embodiment 4. In the following description, Figure 13 will be referred to as appropriate. In Embodiment 4, the joint point detection method is performed by operating the joint point detection device 80. Therefore, the explanation of the joint point detection method in Embodiment 4 will be replaced by the following explanation of the operation of the joint point detection device 80.

[0119] As shown in Figure 14, first, the data acquisition unit 81 acquires two-dimensional joint point coordinate data for the person whose joint points are to be detected (step C1).

[0120] Next, the joint point detection unit 82 applies the 2D joint point coordinate data acquired by the data acquisition unit 81 in step C1 to the machine learning model 20 to detect the 3D coordinates of each joint point of the person to be detected (step C2).

[0121] Specifically, the joint point detection unit 82 inputs the 2D joint point coordinate data acquired by the data acquisition unit 81 in step C1 into the machine learning model 20. As a result, the machine learning model 20 outputs 3D joint point coordinate data, and the joint point detection unit 82 acquires the outputted 3D joint point coordinate data.

[0122] Thus, according to Embodiment 4, the machine learning model 20 can be used to detect the three-dimensional coordinates of each joint point of a person.

[0123] [program] The program in Embodiment 4 can be any program that causes a computer to execute steps C1 to C2 shown in Figure 14. By installing and running this program on a computer, the joint point detection device 80 and joint point detection method in Embodiment 4 can be realized. In this case, the computer's processor functions as a data acquisition unit 81 and a joint point detection unit 82, and performs processing. Examples of computers include general-purpose PCs, smartphones, and tablet devices. The computer's processor also constructs a machine learning model 20.

[0124] Furthermore, the program in Embodiment 4 may be executed by a computer system constructed by multiple computers. In this case, for example, each computer may function as either the data acquisition unit 81 or the joint point detection unit 82.

[0125] [Physical configuration] Here, a computer that implements a learning model generation device and a joint point detection device by executing the programs in Embodiments 1 to 4 will be described with reference to Figure 15. Figure 15 is a block diagram showing an example of a computer that implements a learning model generation device and a joint point detection device in Embodiments 1 to 4.

[0126] As shown in Figure 15, the computer 110 comprises a CPU (Central Processing Unit) 111, main memory 112, storage device 113, input interface 114, display controller 115, data reader / writer 116, and communication interface 117. Each of these components is connected to the others via a bus 121, enabling data communication.

[0127] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to, or instead of, the CPU 111. In this embodiment, the GPU or FPGA can execute the program in the embodiment.

[0128] The CPU 111 loads the program in the embodiment, which consists of a set of codes stored in the storage device 113, into the main memory 112, and performs various calculations by executing each code in a predetermined order. The main memory 112 is typically a volatile storage device such as DRAM (Dynamic Random Access Memory).

[0129] Furthermore, the program in this embodiment is provided stored on a computer-readable recording medium 120. The program in this embodiment may also be distributed over the internet via a communication interface 117.

[0130] Specific examples of the storage device 113 include hard disk drives and semiconductor storage devices such as flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and mouse. The display controller 115 is connected to the display device 119 and controls the display on the display device 119.

[0131] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0132] Furthermore, specific examples of the recording medium 120 include general-purpose semiconductor memory devices such as CF (Compact Flash®) and SD (Secure Digital), magnetic recording media such as Flexible Disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0133] Furthermore, the learning model generation device and joint point detection device in the embodiment can be implemented not by a computer with a program installed, but by using hardware corresponding to each part, such as electronic circuits. Moreover, the learning model generation device and joint point detection device may be partially implemented by a program and the remaining part by hardware. In the embodiment, the computer is not limited to the computer shown in Figure 15.

[0134] Some or all of the embodiments described above can be expressed by (Appendix 1) to (Appendix 15) described below, but are not limited to the following descriptions.

[0135] (Note 1) A data acquisition unit acquires the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of the person, and inputs the acquired 2D joint point coordinate data into a machine learning model. A feature interval distance calculation unit obtains the feature quantities calculated in the machine learning model for each of the two-dimensional joint point coordinate data, and further calculates the distance between the feature quantities using the obtained feature quantities. A similarity calculation unit calculates the similarity between the body types of the people that formed the basis of the training data, using the three-dimensional joint point coordinate data which is the training data. A loss calculation unit calculates the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features. A learning model generation unit updates the parameters of the machine learning model using the calculated loss, A learning model generation device characterized by having the following features.

[0136] (Note 2) A second loss calculation unit calculates the loss for the output of the machine learning model using the 3D joint point coordinate data output by the machine learning model and the 3D joint point coordinate data which is the training data, in response to the input of the 2D joint point coordinate data by the data acquisition unit. A loss integration unit that integrates the loss for the aforementioned feature quantity and the loss for the aforementioned output, Equipped with, The learning model generation unit updates the parameters of the machine learning model using the loss obtained through integration. The learning model generation device described in Appendix 1.

[0137] (Note 3) The loss integration unit integrates the loss for the feature and the loss for the output by calculating a weighted average of the two. The learning model generation device described in Appendix 2.

[0138] (Note 4) The aforementioned machine learning model is a neural network, The feature distance calculation unit obtains the feature from the intermediate layer of the neural network. The learning model generation device described in Appendix 1.

[0139] (Note 5) A data acquisition unit obtains 2D joint point coordinate data that can identify the 2D coordinates of each of the multiple joint points of a person in an image. The joint point detection unit applies the acquired 2D joint point coordinate data to a machine learning model that has learned the relationship between the 2D and 3D coordinates of a person's joint points, and detects the 3D coordinates of each of the multiple joint points of the person. Equipped with, The parameters of the aforementioned machine learning model are: In machine learning using 2D joint point coordinate data and 3D joint point coordinate data that can identify the 3D coordinates of multiple joint points in a person, The feature loss for the machine learning model, obtained from the distance between features calculated by the machine learning model and the similarity between the body types of the people that formed the basis of the training data, is used to update the feature loss for the machine learning model. A joint point detection device characterized by the following features.

[0140] (Note 6) The parameters of the aforementioned machine learning model are, in the machine learning, In addition to the loss for the aforementioned features, The 3D joint point coordinate data output by the machine learning model and the 3D joint point coordinate data used as training data are further updated using the loss on the output of the machine learning model. The joint point detection device described in Appendix 5.

[0141] (Note 7) A data acquisition step involves acquiring the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of the person, and inputting the acquired 2D joint point coordinate data into a machine learning model. For each of the two-dimensional joint point coordinate data, the feature quantities calculated in the machine learning model are obtained, and further, the distance between the feature quantities is calculated using the obtained feature quantities in the feature quantity distance calculation step. A similarity calculation step, which involves using the three-dimensional joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data. A loss calculation step, which involves calculating the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features, A learning model generation step in which the parameters of the machine learning model are updated using the calculated loss, A method for generating a learning model, characterized by having [a certain feature].

[0142] (Note 8) A second loss calculation step is performed to calculate the loss for the output of the machine learning model using the 3D joint point coordinate data output by the machine learning model in response to the input of the 2D joint point coordinate data in the data acquisition step, and the 3D joint point coordinate data which is the training data. A loss integration step that integrates the loss for the aforementioned feature and the loss for the aforementioned output, It further possesses, In the learning model generation step, the parameters of the machine learning model are updated using the loss obtained by integration. The learning model generation method described in Appendix 7.

[0143] (Note 9) In the loss consolidation step, the two are consolidated by calculating a weighted average of the loss for the feature and the loss for the output. The learning model generation method described in Appendix 8.

[0144] (Note 10) The aforementioned machine learning model is a neural network, In the step of calculating the distance between features, the features are obtained from the intermediate layer of the neural network. The learning model generation method described in Appendix 7.

[0145] (Note 11) On the computer, A data acquisition step involves acquiring the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of the person, and inputting the acquired 2D joint point coordinate data into a machine learning model. For each of the two-dimensional joint point coordinate data, the feature quantities calculated in the machine learning model are obtained, and further, the distance between the feature quantities is calculated using the obtained feature quantities in the feature quantity distance calculation step. A similarity calculation step, which involves using the three-dimensional joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data. A loss calculation step, which involves calculating the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features, A learning model generation step in which the parameters of the machine learning model are updated using the calculated loss, A program that executes something.

[0146] (Note 12) On the computer, A second loss calculation step is performed to calculate the loss for the output of the machine learning model using the 3D joint point coordinate data output by the machine learning model in response to the input of the 2D joint point coordinate data in the data acquisition step, and the 3D joint point coordinate data which is the training data. A loss integration step that integrates the loss for the aforementioned feature and the loss for the aforementioned output, Let's execute this further, In the learning model generation step, the parameters of the machine learning model are updated using the loss obtained by integration. The program described in Appendix 11.

[0147] (Note 13) In the loss consolidation step, the two are consolidated by calculating a weighted average of the loss for the feature and the loss for the output. The program described in Appendix 12.

[0148] (Note 14) The aforementioned machine learning model is a neural network, In the step of calculating the distance between features, the features are obtained from the intermediate layer of the neural network. The program described in Appendix 11. [Industrial applicability]

[0149] As described above, this disclosure makes it possible to improve the detection accuracy when detecting the three-dimensional coordinates of joint points from an image. The present invention is useful for various systems that estimate a person's posture from an image. [Explanation of Symbols]

[0150] 10. Learning Model Generation Device (Embodiment 1) 11 Data Acquisition Unit 12. Feature Distance Calculation Unit 13 Similarity calculation unit 14 Loss calculation section 15. Learning Model Generation Unit 16 Correct Answer Data Acquisition Unit 20 Machine Learning Models 30 databases 40 Learning Model Generation Device (Embodiment 2) 41 Loss integration section 42 Second Loss Calculation Unit 50 Data Expansion Devices 51 Data acquisition unit 51 52 Projection Processing Unit 53 Data Exploration Unit 54 Image generation unit 60 3D pose datasets 61 joint points 70 Databases 80 Joint Point Detection Device 81 Data Acquisition Unit 82 Joint point detection unit 110 Computer 111 CPU 112 Main Memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Readers / Writers 117 Communication Interface 118 Input devices 119 Display device 120 recording media 121 Bus

Claims

1. A data acquisition unit acquires the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of a person, and inputs the acquired 2D joint point coordinate data into a machine learning model that is learning the relationship between the 2D and 3D coordinates of a person's joint points. A feature interval distance calculation unit obtains the feature quantities calculated in the machine learning model for each of the two-dimensional joint point coordinate data, and further calculates the distance between the feature quantities using the obtained feature quantities. A similarity calculation unit calculates the similarity between the body types of the people that formed the basis of the training data, using the three-dimensional joint point coordinate data which is the training data. A loss calculation unit calculates the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features. A learning model generation unit updates the parameters of the machine learning model using the calculated loss, A learning model generation device characterized by having the following features.

2. A second loss calculation unit calculates the loss for the output of the machine learning model using the three-dimensional joint point coordinate data output by the machine learning model and the three-dimensional joint point coordinate data which is the training data, in response to the input of the two-dimensional joint point coordinate data by the data acquisition unit. A loss integration unit that integrates the loss for the aforementioned feature quantity and the loss for the aforementioned output, Equipped with, The learning model generation unit updates the parameters of the machine learning model using the loss obtained through integration. A learning model generation device according to claim 1.

3. The loss integration unit integrates the loss for the feature and the loss for the output by calculating a weighted average of the two. The learning model generation apparatus according to claim 2.

4. The aforementioned machine learning model is a neural network, The feature distance calculation unit obtains the feature from the intermediate layer of the neural network. A learning model generation device according to claim 1.

5. A data acquisition unit obtains 2D joint point coordinate data that can identify the 2D coordinates of each of the multiple joint points of a person in an image. The joint point detection unit applies the acquired two-dimensional joint point coordinate data to a machine learning model that has learned the relationship between the two-dimensional and three-dimensional coordinates of a person's joint points, thereby detecting the three-dimensional coordinates of each of the multiple joint points of the person. Equipped with, The parameters of the aforementioned machine learning model are: In machine learning using two-dimensional joint point coordinate data and three-dimensional joint point coordinate data that allows for the identification of the three-dimensional coordinates of multiple joint points in a person, The feature loss for the machine learning model, obtained from the distance between features calculated by the machine learning model and the similarity between the body types of the people that formed the basis of the training data, is used to update the feature loss for the machine learning model. A joint point detection device characterized by the following features.

6. The parameters of the aforementioned machine learning model are, in the machine learning, In addition to the loss for the aforementioned features, The 3D joint point coordinate data output by the machine learning model and the 3D joint point coordinate data used as training data are further updated using the loss on the output of the machine learning model. The joint point detection device according to claim 5.

7. A data acquisition step involves acquiring the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of a person, and inputting the acquired 2D joint point coordinate data into a machine learning model that is learning the relationship between the 2D and 3D coordinates of a person's joint points. For each of the two-dimensional joint point coordinate data, the feature quantities calculated in the machine learning model are obtained, and further, the distance between the feature quantities is calculated using the obtained feature quantities in the feature quantity distance calculation step. A similarity calculation step, which involves using the three-dimensional joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data. A loss calculation step, which involves calculating the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features, A learning model generation step in which the parameters of the machine learning model are updated using the calculated loss, A method for generating a learning model, characterized by having [a certain feature].

8. A second loss calculation step is performed to calculate the loss for the output of the machine learning model using the 3D joint point coordinate data output by the machine learning model in response to the input of the 2D joint point coordinate data in the data acquisition step, and the 3D joint point coordinate data which is the training data. A loss integration step that integrates the loss for the aforementioned feature and the loss for the aforementioned output, It further possesses, In the learning model generation step, the parameters of the machine learning model are updated using the loss obtained by integration. The method for generating a learning model according to claim 7.

9. In the loss consolidation step, the two are consolidated by calculating a weighted average of the loss for the feature and the loss for the output. The method for generating a learning model according to claim 8.

10. The aforementioned machine learning model is a neural network, In the step of calculating the distance between features, the features are obtained from the intermediate layer of the neural network. The method for generating a learning model according to claim 7.

11. On the computer, A data acquisition step involves acquiring the 2D joint point coordinate data from training data that includes 2D joint point coordinate data capable of identifying the 2D coordinates of each of several joint points of a person in an image, and 3D joint point coordinate data capable of identifying the 3D coordinates of each of the said multiple joint points of a person, and inputting the acquired 2D joint point coordinate data into a machine learning model that is learning the relationship between the 2D and 3D coordinates of a person's joint points. For each of the two-dimensional joint point coordinate data, the feature quantities calculated in the machine learning model are obtained, and further, the distance between the feature quantities is calculated using the obtained feature quantities in the feature quantity distance calculation step. A similarity calculation step, which involves using the three-dimensional joint point coordinate data, which is the training data, to calculate the similarity between the body types of the people that formed the basis of the training data. A loss calculation step, which involves calculating the loss for the features in the machine learning model using the calculated similarity and the calculated distance between the features, A learning model generation step in which the parameters of the machine learning model are updated using the calculated loss, A program that executes something.

12. On the computer, A second loss calculation step is performed to calculate the loss for the output of the machine learning model using the 3D joint point coordinate data output by the machine learning model in response to the input of the 2D joint point coordinate data in the data acquisition step, and the 3D joint point coordinate data which is the training data. A loss integration step that integrates the loss for the aforementioned feature and the loss for the aforementioned output, Let's execute this further, In the learning model generation step, the parameters of the machine learning model are updated using the loss obtained by integration. The program according to claim 11.

13. In the loss consolidation step, the two are consolidated by calculating a weighted average of the loss for the feature and the loss for the output. The program according to claim 12.

14. The aforementioned machine learning model is a neural network, In the step of calculating the distance between features, the features are obtained from the intermediate layer of the neural network. The program according to claim 11.

Images