Data generation method, data generation program, and data generation device

Abstract

This data generation device acquires a reasoning result for skeleton information for each piece of teacher data when inputting a plurality of pieces of teacher data into a machine learning model, the reasoning result including an error for each part of a skeleton. The data generation device specifies, from the plurality of pieces of teacher data, first teacher data in which the error of the first part is greater than the error of the first part of the other pieces of teacher data, on the basis of the reasoning result. The data generation device determines, from the plurality of pieces of teacher data, second teacher data in which the error of the second part is greater than the error of the second part of the other pieces of teacher data, on the basis of the reasoning result. The data generation device generates third teacher data by replacing the information related to the second part included in the first teacher data with the information related to the second part included in the second teacher data.

Description

Technical Field

[0001] The present invention relates to a data generation method and the like. Background Art

[0002] A technology has been established for detecting human skeleton information using image data of a person captured by a camera. Skeleton information is information indicating the coordinates of each joint of a person. In the above technology, a plurality of teacher data are prepared, and teacher learning is performed on a machine learning model such as a deep learning network. By inputting a time series of human image data into the machine learning model that has been learned, it is possible to detect the skeleton information of a time series of people.

[0003] Here, the existing teacher data is processed and the processed teacher data is added as new teacher data, so that the general performance and reliability of the machine learning model can be improved. For example, there is a technology that applies geometric transformation, noise or moves the image patch of the body part to the teacher data to generate new teacher data.

[0004] In addition, there is a technology for expanding teacher data to generate new teacher data. In this data expansion technology, the gradient information of the inference error of the machine learning model under learning is fed back, and at each learning stage, teacher data that maximizes the performance of the machine learning model is generated. For example, in the data expansion technology, a three-dimensional model of a person is projected onto a two-dimensional plane to generate image data of the teacher data. In the data expansion technology, the three-dimensional model is rotated, translated, etc. based on the gradient information to generate image data in which the inference error of the machine learning model becomes larger. In the data expansion technology, there is a recognizer that excludes the above-mentioned image data when the posture of the person in the image data is an undesirable posture for a person. The process of generating image data based on a three-dimensional model using gradient information is called differentiable data expansion.

[0005] For example, the above-mentioned data extension techniques include data extension techniques (1) to (3) described below.

[0006] Data expansion technology (1) is described. In data expansion technology (1), Neural Radiance Fields (NeRF) is used to generate image data that improves the inference error of the machine learning model (object detection model). For example, in data expansion technology (1), the definition domain (360°) of the photographic angle, which will become the control parameter of image generation, is divided into k=8 bins, and the selection probability (classification distribution) of the bin is formulated using Gambel-softmax. In data expansion technology (1), the machine learning model is learned, and the selection probability of the bin is simultaneously learned in the direction of maximizing the inference error.

[0007] The data extension technology (2) is described. In the data extension technology (2), regarding the machine learning model that transforms 2D skeleton information into 3D skeleton information, paired data of 2D skeleton information and 3D skeleton information that improves the inference error of the machine learning model is generated. For example, in the data extension technology (2), a learnable extension operation for the existing 3D skeleton information is performed through a multilayer perceptron. The extension operation includes perturbations of joint angles, perturbations of bone lengths, and perturbations of rotation and translation. In the data extension technology (2), learning of the machine learning model is performed, and learning of the extension operation is performed in the direction that maximizes the inference error.

[0008] The data expansion technology (3) is described. In the data expansion technology (3), a teacher data set is increased by generating and adding a new 3D skeleton group based on an existing 3D skeleton group. For example, in the data expansion technology (3), a process of replacing a part of the skeleton of two 3D skeleton information and a process of perturbing the joint angle are performed.

[0009] Patent document 1: Japanese Patent Application Publication No. 2019-212106.

[0010] Non-patent literature 1: Y.Ge et al. “Neural-Sim: Learning to Generate Training Data with NeRF,” ECCV 2022

[0011] Non-patent literature 2: Gong et al., “PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation,” CVPR 2021

[0012] Non-patent literature 3: S.Li et al., “Cascaded Deep Monocular 3D Human PoseEstimation with Evolutionary Training Data,” CVPR 2020

[0013] However, in the above-mentioned conventional technology, there is a problem that new teacher data tends to stay near the existing teacher data, and new teacher data separated from the distribution of the existing teacher data cannot be generated.

[0014] For example, suppose that new teacher data in other fields are generated based on teacher data in the field of gymnastics. Other fields are fields other than gymnastics, such as sports and rehabilitation. The teacher data in the field of gymnastics is limited to certain 3D skeleton information that a person can obtain in gymnastics. Therefore, when a machine learning model based on teacher data in the field of gymnastics is used in other fields, sometimes the inference accuracy is reduced for skeleton information not included in the teacher data.

[0015] In order to suppress the reduction of inference accuracy in other fields, it is preferred to retrain the machine learning model with new teacher data separated from the distribution of existing teacher data. In addition, here, the teacher data in the field of gymnastics is used for illustration, but it is not limited to this. Summary of the invention

[0016] In one aspect, an object of the present invention is to provide a data generation method, a data generation program, and a data generation device capable of generating new teacher data separated from the distribution of existing teacher data.

[0017] In the first case, a computer is caused to perform the following processing. The computer obtains an inference result of skeleton information for each teacher data when multiple teacher data are input into a machine learning model, and the inference result includes errors in each part of the skeleton. Based on the inference result, the computer determines first teacher data from multiple teacher data in which the error of the first part is larger than the error of the first part of other teacher data. Based on the inference result, the computer determines second teacher data from multiple teacher data in which the error of the second part is larger than the error of the second part of other teacher data. The computer replaces the information related to the second part contained in the first teacher data with the information related to the second part contained in the second teacher data, thereby generating third teacher data.

[0018] Ability to generate new teacher data separated from the distribution of existing teacher data. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 This is a diagram showing an example of a human body model.

[0020] Figure 2 This is a diagram showing an example of joint names.

[0021] Figure 3 It is a diagram for explaining the processing of the data generating device involved in this embodiment.

[0022] Figure 4A It is a diagram used to illustrate the properties.

[0023] Figure 4B This is a diagram showing an example of attributes and extended data.

[0024] Figure 5 It is a diagram for explaining the body part p.

[0025] Figure 6 FIG. (1) is a diagram for explaining the processing of the general section.

[0026] Figure 7 FIG. (2) is a diagram for explaining the processing of the general section.

[0027] Figure 8 This is a diagram showing an example of extended data generated from attributes and vulnerability attributes.

[0028] Fig. 9 It is a functional block diagram showing the structure of the data generating device involved in this embodiment.

[0029] Fig.10 This is a flowchart showing the processing steps of the data generating device involved in this embodiment.

[0030] Fig.11 It is a flowchart showing the processing steps of the integrated processing.

[0031] Fig.12 It is a diagram for explaining the effect of the data generating device involved in this embodiment.

[0032] Fig.13 This is a diagram showing an example of the hardware configuration of a computer that realizes the same function as the data generating device of the embodiment. DETAILED DESCRIPTION

[0033] Hereinafter, embodiments of the data generation method, data generation program, and data generation device disclosed in the present application will be described in detail based on the drawings. In addition, the present invention is not limited to these embodiments.

[0034] Example

[0035] Before describing this embodiment, an example of a human body model will be described. Figure 1 is a diagram showing an example of a human body model. Figure 1 As shown, the human body model is defined by 21 joints ar0 to ar20. In the skeleton information used in this embodiment, two-dimensional or three-dimensional coordinates are set for each joint ar0 to ar20 defined by the human body model.

[0036] Figure 1 The relationship between the joints ar0 to ar20 and the joint names is shown in Figure 2 shown. Figure 2 is a diagram showing an example of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to a20 are Figure 2 As shown, description is omitted.

[0037] Next, the processing of the data generating device according to this embodiment will be described. Figure 3 The data generating device uses the teacher data 30 .

[0038] The teacher data 30 is existing data. For example, the teacher data includes image data and attributes of a person. The attributes include skeleton information, camera parameters, and appearance. The skeleton information is Figure 1 The coordinates of the joints described in the figure represent the coordinates of each joint of the person contained in the image data. The coordinates of each joint are two-dimensional coordinates or three-dimensional coordinates. The camera parameters represent the viewpoint position of the camera that captured the image data. Appearance is information related to the appearance and background of the person contained in the image data.

[0039] Figure 4A is a diagram for explaining attributes. For example, the attribute A1 of a certain teacher data includes skeleton information a1-1, camera parameters a1-2, appearance a1-3, a1-4. Figure 4A In FIG. 1 , the camera parameter a1-2 is shown as a conceptual illustration, but actually includes information on the viewpoint position of the camera that captured the image data. Appearance a1-3 sets the appearance of the person (body shape, color of uniform, etc.). Appearance a1-4 sets the background information of the person.

[0040] Return to Figure 3 The data generating device inputs the teacher data 30 into the data expansion unit 151.

[0041] The data expansion unit 151 is set with a parameter θ1, and based on the above parameter θ1, the attribute of the teacher data 30 is expanded. The data expansion unit 151 outputs the information of the expanded attribute to the image generation unit 152. The image generation unit 152 described later generates the expanded data 40 based on the attribute expanded by the parameter θ1, and the above expanded data 40 is input to the training object model 50 to calculate the inference error.

[0042] The data expansion unit 151 trains the parameter θ1 in a direction in which the inference error increases when the expanded data 40 is input to the training model 50 , based on the gradient information of the inference error fed back from the training model 50 .

[0043] For example, the data expansion unit 151 changes the joint angles and the lengths of the bones between the joints of the skeleton information included in the attributes in the direction where the inference error increases according to the parameter θ1. The data expansion unit 151 can also perform data expansion by changing the camera parameters and the appearance in the direction where the inference error increases according to the parameter θ1.

[0044] The data expansion unit 151 ensures the rationality of the data when expanding the attributes. For example, the data expansion unit 151 changes the joint angle within the operable range of the joint of the skeleton information. When changing the length of the bone, the data expansion unit 151 changes the length of the bone within a specified range.

[0045] The image generation unit 152 generates the extended data 40 based on the information of the attribute extended by the data extension unit 151 or the integration unit 153. The image generation unit 152 is a differentiable image generator such as NeRF. In the following description related to the image generation unit 152, "extended attribute" is simply expressed as "attribute".

[0046] The image generation unit 152 generates a human model and a background model based on the skeleton information and appearance included in the attributes. The image generation unit 152 generates image data (extended data 40) from a viewpoint based on the camera parameters of the attribute information for the model synthesized from the human model and the background model.

[0047] Figure 4B Attribute A1 includes skeleton information a1-1, camera parameters a1-2, and appearance a1-3 and a1-4. The image generation unit 152 generates the extended data 40-1 based on the attribute A1.

[0048] The attribute A2 includes skeleton information a2-1, camera parameters a2-2, and appearance a2-3 and a2-4. The image generation unit 152 generates the extended data 40-2 based on the attribute A2.

[0049] The attribute A3 includes skeleton information a3-1, camera parameters a3-2, and appearance a3-3 and a2-4. The image generation unit 152 generates the extended data 40-3 based on the attribute A3.

[0050] The data generation device performs machine learning of the training target model 50 based on the extended data 40 and the skeleton information (skeleton information of the extended attribute) used when generating the extended data 40. For example, the skeleton information used when generating the extended data 40 is used as a correct answer label. The training target model 50 is NN (Neural Network) or the like. The parameter θ2 is set in the training target model 50.

[0051] The data generation device inputs the extended data 40 into the training object model 50, and obtains the inference result output from the training object model 50. The data generation device updates the parameter θ2 of the training object model 50 so that the inference error between the inference result and the correct answer label becomes smaller. The data generation device feeds back the gradient information of the inference error to the data expansion unit 151.

[0052] The data generating device establishes a correspondence between a group of inference result information indicating the relationship between the inference result and the true value (correct answer label) and attribute information (expanded attributes) used when generating extended data 40 for each joint of the skeleton information, and outputs the correspondence to the integrating unit 153 .

[0053] The synthesis unit 153 performs the following processing based on the inference result information and the information of the expanded attribute. In the following description related to the synthesis unit 153, "expanded attribute" is simply expressed as "attribute". For example, the synthesis unit 153 waits for the training object model 50 times until the parameter θ2 is updated, and obtains a plurality of inference result information and attribute information sets.

[0054] The integration unit 153 determines the inference error of each body part p based on the inference result information. Figure 5 : is a diagram for explaining body part p. In this embodiment, in the joint hierarchy, the joint group from the end joint to the branch point joint is defined as body part p. For example, as body part p, there are "head", "armL", "armR", "legL", "legR". That is, it is defined as p∈{head, armL, armR, legL, legR}.

[0055] The body part "head" corresponds to joints ar3 and ar18. The body part "armL" corresponds to joints ar4, ar5, ar6, and ar19. The body part "armR" corresponds to joints ar7, ar8, ar9, and ar20. The body part "legL" corresponds to joints ar10, ar11, ar12, and ar13. The body part "legR" corresponds to joints ar14, ar15, ar16, and ar17.

[0056] The integration unit 153 specifies the inference error of each body part p for each inference result information. That is, the integration unit 153 specifies the inference errors of the body parts "head", "armL", "armR", "legL", and "legR" based on one inference result information.

[0057] For example, the inference error of the body part “head” is the MSE (Mean Squared Error) between the inference result of the joints ar3 and ar18 and the true value. The inference error of the body part “armL” is the MSE between the inference result of the joints ar4, ar5, ar6, and ar19 and the true value. The inference error of the body part “armR” is the MSE between the inference result of the joints ar7, ar8, ar9, and ar20 and the true value. The inference error of the body part “legL” is the MSE between the inference result of the joints ar10, ar11, ar12, and ar13 and the true value. The inference error of the body part “legR” is the MSE between the inference result of the joints ar14, ar15, ar16, and ar17 and the true value.

[0058] The integration unit 153 compares the inference errors of each body part p calculated based on each inference result information, and determines the maximum value of the inference error of each body part p and the attribute corresponding to the inference result information with the maximum inference error.

[0059] Figure 6 Figure (1) is used to illustrate the processing of the general department. Figure 6 In the example shown, the inference result information obtained by inputting the extended data generated according to the attribute A1 into the training object model 50 is set to the inference result information R1. The inference error of the body part "head" obtained according to the inference result R1 is set to the inference error E1-1. The inference error of the body part "armL" obtained according to the inference result R1 is set to the inference error E1-2. The inference error of the body part "armR" obtained according to the inference result R1 is set to the inference error E1-3. The inference error of the body part "legR" obtained according to the inference result R1 is set to the inference error E1-4. The inference error of the body part "legL" obtained according to the inference result R1 is set to the inference error E1-5.

[0060] The inference result information obtained by inputting the extended data generated according to the attribute A2 into the training object model 50 is set as the inference result information R2. The inference error of the body part "head" obtained according to the inference result R2 is set as the inference error E2-1. The inference error of the body part "armL" obtained according to the inference result R2 is set as the inference error E2-2. The inference error of the body part "armR" obtained according to the inference result R2 is set as the inference error E2-3. The inference error of the body part "legR" obtained according to the inference result R2 is set as the inference error E2-4. The inference error of the body part "legL" obtained according to the inference result R2 is set as the inference error E2-5.

[0061] The inference result information obtained by inputting the extended data generated according to the attribute An into the training object model 50 is set as the inference result information Rn. n is a natural number greater than 3. The inference error of the body part "head" obtained according to the inference result Rn is set as the inference error En-1. The inference error of the body part "armL" obtained according to the inference result Rn is set as the inference error En-2. The inference error of the body part "armR" obtained according to the inference result Rn is set as the inference error En-3. The inference error of the body part "legR" obtained according to the inference result Rn is set as the inference error En-4. The inference error of the body part "legL" obtained according to the inference result Rn is set as the inference error En-5.

[0062] The synthesis unit 153 compares the inference errors E1-1 to En-1 of the body part "head" and determines the inference error that takes the maximum value. In this embodiment, as an example, the inference error of the inference error E1-1 among the inference errors E1-1 to En-1 is set to the maximum value. The attribute corresponding to the inference error E1-1 is attribute A1. In this case, the synthesis unit 153 determines the weakness attribute of the body part "head" as the weakness attribute A1.

[0063] The synthesis unit 153 compares the inference errors E1-2 to En-2 of the body part "armL" and determines the inference error that takes the maximum value. In this embodiment, as an example, the inference error of the inference error E2-2 among the inference errors E1-2 to En-2 is set to the maximum value. The attribute corresponding to the inference error E2-2 is attribute A2. In this case, the synthesis unit 153 determines the weakness attribute of the body part "armL" as the weakness attribute A2.

[0064] The synthesis unit 153 compares the inference errors E1-3 to En-3 of the body part "armR" and determines the inference error with the maximum value. In this embodiment, as an example, the inference error E3-3 among the inference errors E1-3 to En-3 is set to the maximum value. The attribute corresponding to the inference error E3-3 is attribute A3. In this case, the synthesis unit 153 determines the weakness attribute of the body part "armR" as the weakness attribute A3. Figure 6 In the figure, the illustration of attribute A3 is omitted.

[0065] The synthesis unit 153 compares the inference errors E1-4 to En-4 of the body part "legL" and determines the inference error with the maximum value. In this embodiment, as an example, the inference error of the inference error E4-4 among the inference errors E1-4 to En-4 is set to the maximum value. The attribute corresponding to the inference error E4-4 is attribute A4. In this case, the synthesis unit 153 determines the weakness attribute of the body part "legL" as the weakness attribute A4. Figure 6 In the figure, the illustration of attribute A4 is omitted.

[0066] The synthesis unit 153 compares the inference errors E1-5 to En-5 of the body part "legR" and determines the inference error with the maximum value. In this embodiment, as an example, the inference error of the inference error E5-5 among the inference errors E1-5 to En-5 is set to the maximum value. The attribute corresponding to the inference error E5-5 is attribute A5. In this case, the synthesis unit 153 determines the weakness attribute of the body part "legR" as the weakness attribute A5. Figure 6 In the figure, the illustration of attribute A5 is omitted.

[0067] General Department 153 by execution Figure 6 The processing shown in the figure determines the weakness attribute of each body part p respectively. In the following description, as an example, the weakness attribute of the body part "head" is set to weakness attribute A1. The weakness attribute of the body part "armL" is set to weakness attribute A2. The weakness attribute of the body part "armR" is set to weakness attribute A3. The weakness attribute of the body part "legL" is set to weakness attribute A4. The weakness attribute of the body part "legR" is set to weakness attribute A5.

[0068] Figure 7 This is a figure (2) used to illustrate the processing of the comprehensive unit. The comprehensive unit 153 combines the weakness attributes A1 to A5 based on the weakness attribute A1 to generate the weakness attribute A'1. In the weakness attribute A'1, as with the attribute, skeleton information, camera parameters, and appearance are included. The skeleton information of the weakness attribute A'1 is the skeleton information formed by combining the joint coordinates of the body part "head" of the weakness attribute A1, the joint coordinates of the body part "armL" of the weakness attribute A2, the joint coordinates of the body part "armR" of the weakness attribute A3, the joint coordinates of the body part "legL" of the weakness attribute A4, and the joint coordinates of the body part "legR" of the weakness attribute A5. The camera parameters and appearance of the weakness attribute A'1 are based on the camera parameters and appearance of the weakness attribute A1.

[0069] The synthesis unit 153 synthesizes the weakness attributes A1 to A5 based on the weakness attribute A2 to generate the weakness attribute A'2. In the weakness attribute A'2, similar to the attribute, skeleton information, camera parameters, and appearance are included. The skeleton information of the weakness attribute A'2 is the skeleton information formed by combining the joint coordinates of the body part "head" of the weakness attribute A1, the joint coordinates of the body part "armL" of the weakness attribute A2, the joint coordinates of the body part "armR" of the weakness attribute A3, the joint coordinates of the body part "legL" of the weakness attribute A4, and the joint coordinates of the body part "legR" of the weakness attribute A5. The camera parameters and appearance of the weakness attribute A'2 are based on the camera parameters and appearance of the weakness attribute A2.

[0070] The synthesis unit 153 synthesizes the weakness attributes A1 to A5 based on the weakness attribute A3 to generate the weakness attribute A'3. In the weakness attribute A'3, similar to the attribute, skeleton information, camera parameters, and appearance are included. The skeleton information of the weakness attribute A'3 is the skeleton information formed by combining the joint coordinates of the body part "head" of the weakness attribute A1, the joint coordinates of the body part "armL" of the weakness attribute A2, the joint coordinates of the body part "armR" of the weakness attribute A3, the joint coordinates of the body part "legL" of the weakness attribute A4, and the joint coordinates of the body part "legR" of the weakness attribute A5. The camera parameters and appearance of the weakness attribute A'3 are based on the camera parameters and appearance of the weakness attribute A3.

[0071] The synthesis unit 153 generates a weakness attribute A'4 by synthesizing the weakness attributes A1 to A5 based on the weakness attribute A4. The weakness attribute A'4 includes skeleton information, camera parameters, and appearance, just like the attributes. The skeleton information of the weakness attribute A'4 is the skeleton information formed by combining the joint coordinates of the body part "head" of the weakness attribute A1, the joint coordinates of the body part "armL" of the weakness attribute A2, the joint coordinates of the body part "armR" of the weakness attribute A3, the joint coordinates of the body part "legL" of the weakness attribute A4, and the joint coordinates of the body part "legR" of the weakness attribute A5. The camera parameters and appearance of the weakness attribute A'4 are based on the camera parameters and appearance of the weakness attribute A4.

[0072] The synthesis unit 153 synthesizes the weakness attributes A1 to A5 based on the weakness attribute A5, thereby generating the weakness attribute A'5. In the weakness attribute A'5, similar to the attribute, skeleton information, camera parameters, and appearance are included. The skeleton information of the weakness attribute A'5 is the skeleton information formed by combining the joint coordinates of the body part "head" of the weakness attribute A1, the joint coordinates of the body part "armL" of the weakness attribute A2, the joint coordinates of the body part "armR" of the weakness attribute A3, the joint coordinates of the body part "legL" of the weakness attribute A4, and the joint coordinates of the body part "legR" of the weakness attribute A5. The camera parameters and appearance of the weakness attribute A'5 are based on the camera parameters and appearance of the weakness attribute A5.

[0073] General Department 153 by execution Figure 7 The processing described in , generates the weakness attributes A'1 to A'5. The integration unit 153 inputs the weakness attributes A'1 to A'5 to the image generation unit 152, and generates extended data corresponding to the weakness attributes A'1 to A'5, respectively.

[0074] Figure 8 : is a diagram showing an example of extended data generated by attributes and weakness attributes. For example, the extended data Im10 is set to the image data obtained by inputting the attribute A2 into the image generation unit 152. If the extended data Im10 is input into the training object model 50, the inference error of the body part "armL" takes the maximum value compared with other inference errors. The extended data Im11 is the image data obtained by inputting the weakness attribute A'2 into the image generation unit 152. If the extended data Im11 is input into the training object model 50, the inference error of each body part "head", "armL", "armR", "legL", "legR" takes the maximum value. That is, new teacher data separated from the distribution of existing teacher data can be generated.

[0075] The data generating device generates a set of extended data obtained by inputting the weakness attribute to the image generating unit 152 and skeleton information included in the weakness attribute as teacher data, and uses it for machine learning of the training object model 50 .

[0076] As described above, the data generation device involved in this embodiment determines the inference error of each body part p based on the inference result information obtained by inputting the extended data 40 into the training object model 50. The data generation device compares the inference errors of each body part p calculated based on each inference result information, and determines the attribute (weakness attribute) of the inference result information where the inference error becomes the maximum value. The data generation device integrates the weakness attributes of each body part p, and generates extended data based on the integrated weakness attributes. In this way, new teacher data separated from the distribution of existing teacher data can be generated.

[0077] In addition, in the above description, the data generation device determines and integrates the weakness attribute based on the inference result information obtained by inputting the extended data 40 into the training object model 50, but it is not limited to this. For example, the teacher data 30 can also be directly input into the training object model 50, and the weakness attribute can be determined and integrated based on the inference result information obtained by inputting into the training object model 50.

[0078] Next, a configuration example of a data generating device that executes the above-mentioned processing will be described. Fig. 9 is a functional block diagram showing the structure of the data generating device involved in this embodiment. Fig. 9 As shown, the data generating device 100 includes a communication unit 110 , an input unit 120 , a display unit 130 , a storage unit 140 , and a control unit 150 .

[0079] The communication unit 110 performs data communication with external devices and the like via a network. The communication unit 110 is a NIC (Network Interface Card) and the like. The control unit 150 described later exchanges data with external devices via the communication unit 110 .

[0080] The input unit 120 is an input device for inputting various information to the control unit 150 of the data generating device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0081] The display unit 130 is a display device that displays information output from the control unit 150 .

[0082] The storage unit 140 has a training target model 50 and a teacher data set 141. The storage unit 140 is a storage device such as a memory.

[0083] The training target model 50 is a machine learning model that takes image data (extended data) as input and outputs the inference result of skeleton information. The training target model 50 is NN or the like.

[0084] The teacher data set 141 has a plurality of teacher data. The teacher data includes image data and attributes of a person. The attributes include skeleton information, camera parameters, and appearance.

[0085] The control unit 150 includes a data expansion unit 151, an image generation unit 152, an integration unit 153, and a learning unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0086] The data expansion unit 151 is set with a parameter θ1, and expands the attribute of the teacher data based on the above parameter θ1. The data expansion unit 151 outputs the information of the expanded attribute to the image generation unit 152. The data expansion unit 151 trains the parameter θ1 in the direction in which the inference error increases when the expanded data is input to the training object model 50 based on the gradient information of the inference error fed back from the training object model 50. Other descriptions related to the data expansion unit 151 are as follows Figure 3 The description related to the data expansion unit 151 is the same as described in .

[0087] The image generation unit 152 generates extended data based on the attribute information expanded by the data expansion unit 151. In addition, the image generation unit 152 generates extended data based on the weakness attribute information generated by the integration unit 153. The image generation unit 152 may also add a set of weakness attribute information and extended data to the teacher data set 141 as new teacher data.

[0088] The synthesis unit 153 determines the inference error of each body part p for each inference result information. The synthesis unit 153 compares the inference errors of each body part p calculated based on each inference result information, and determines the maximum value of the inference error of each body part p and the attribute (weakness attribute) corresponding to the inference result information with the maximum inference error. The synthesis unit 153 generates the weakness attribute by synthesizing the weakness attributes of each body part p. Other descriptions related to the synthesis unit 153 are as follows: Figure 3 , Figure 5 to Figure 7 The description related to the comprehensive unit 153 is the same as that described in .

[0089] The learning unit 154 performs machine learning of the training object model 50 based on the teacher data set 141. For example, the learning unit 154 updates the parameter θ2 of the training object model 50 based on the error back propagation method so that when the image data is input to the training object model 50, the error between the inference result output from the training object model 50 and the correct answer label becomes smaller.

[0090] In the learning unit 154 , the combination of image data and correct answer labels input to the training model 50 is a combination of first image data and first correct answer labels or a combination of second image data and second correct answer labels described below.

[0091] The first correct answer label is the skeleton information when the attribute of the training data is expanded by the data expansion unit 151. The first image data is the expanded data generated by the image generation unit 152 based on the attribute of the training data expanded by the data expansion unit 151.

[0092] The second correct answer label is the skeleton information of the weakness attribute integrated by the integration unit 153. The second image data is extended data generated by the image generation unit 152 based on the weakness attribute.

[0093] The learning unit 154 feeds back the gradient information of the inference error to the data expansion unit 151. The learning unit 154 outputs a set of inference result information and attribute information indicating the relationship between the inference result and the true value (correct answer label) for each joint of the skeleton information to the integration unit 153.

[0094] Next, an example of the processing procedure of the data generating device 100 according to the present embodiment will be described. Fig.10 1 is a flowchart showing the processing steps of the data generating device involved in this embodiment. Fig.10 As shown, the data expansion unit 151 of the data generating device 100 obtains the teacher data from the teacher data set 141 (step S101). The data expansion unit 151 expands the attributes of the teacher data in a direction where the inference error based on the training object model 50 increases based on the parameter θ1 (step S102).

[0095] The image generation unit 152 of the data generation device 100 generates extended data based on the extended attributes (step S103). The learning unit 154 of the data generation device 100 performs machine learning on the training target model 50 based on the extended data and the correct answer label (step S104).

[0096] The integration unit 153 of the data generating device 100 executes integration processing (step S105 ).

[0097] The data expansion unit 151 of the data generating device 100 receives feedback of the gradient information of the inference result and updates the parameter θ1 (step S106 ).

[0098] When the data generating device 100 continues the process (step S107: Yes), the process proceeds to step S101. On the other hand, when the data generating device 100 does not continue the process (step S107: No), the process ends.

[0099] Next, Fig.10 An example of the integrated processing shown in step S105 is described. Fig.11 is a flowchart showing the processing steps of the integrated processing. Fig.11 As shown, the integration unit 153 of the data generating device 100 determines the inference error of each body part p for each inference result information (step S201).

[0100] The integration unit 153 compares the inference errors of each body part p, and determines the weakness attribute of each body part p (step S202). The integration unit 153 integrates the weakness attributes of each body part p (step S203).

[0101] The image generation unit 152 of the data generation device 100 generates extended data based on the integrated weakness attributes (step S204).

[0102] The learning unit 154 of the data generating device 100 performs machine learning on the training target model 50 based on the extended data and the correct answer label (skeleton information of the weakness attribute) (step S205 ).

[0103] Next, the effect of the data generating device 100 involved in this embodiment is described. The data generating device 100 determines the inference error of each body part p based on the inference result information obtained by inputting the extended data into the training object model. The data generating device 100 compares the inference errors of each body part p calculated based on each inference result information, and determines the attribute (weakness attribute) of the inference result information where the inference error becomes the maximum value. The data generating device 100 integrates the weakness attributes of each body part p, and generates extended data based on the integrated weakness attributes. In this way, new teacher data separated from the distribution of existing teacher data can be generated.

[0104] Fig.12 It is a diagram for explaining the effect of the data generating device involved in this embodiment. Fig.12 The image data Im20 is the extended data generated by the image generation unit 152 according to the integrated weakness attribute. If the learning unit 154 inputs the image data Im20 to the training object model 50, it outputs the inference result 60. In the inference result 60, the inference of the joints ar9, ar20 and the joints ar5, ar6, and ar7 fails. That is, by performing machine learning using the image data Im20 that becomes new teacher data, that is, new image data (teacher data) separated from the distribution of the existing teacher data, the inference accuracy of the training object model 50 can be improved.

[0105] For example, by training the training object model 50 for machine learning using new image data (teacher data) separated from the distribution of existing teacher data, it is possible to obtain inference results 60a and 60b. In the inference result 60a, inference fails in joint ar19, but inference accuracy improves in joints ar5 and ar6. In the inference result 60b, inference accuracy improves in joints ar9, ar20, joints ar5, ar6, ar19, and joint ar11.

[0106] In addition, the processing of the data generation device 100 involved in this embodiment is not limited to the above content. For example, the data generation device 100 can perform a body detection task and a body region extraction (segmentation) task from image data. The data generation device 100 performs the body detection task and the body region extraction, and refers to the body part with a large inference error, so as to determine the weakness attribute of each body part and apply the above processing.

[0107] The processing of the data generating device 100 can also be applied to tasks that are not human bodies, but are more general multi-jointed bodies such as four-legged animals. The processing of the data generating device 100 can be applied to both 2D body skeleton inference tasks and 3D body skeleton inference tasks. The processing of the data generating device 100 can also be applied to 2D-to-3D skeleton inference tasks that do not require image data.

[0108] The data generating device 100 may also use a structure for evaluating the rationality of the skeleton information obtained by integrating the weak point attributes, and discard or correct the data synthesized into unreasonable skeleton information. The data generating device 100 uses VPoser for evaluating the distance in the latent space of the posture generator, Penetration loss for evaluating the penetration of the body model, and hyper-bending loss for evaluating the bending of the elbow and knee in opposite directions as a structure for evaluating the rationality of the skeleton information. For example, the technology related to VPoser is the technology described in the document "G. Pavlakos et al., "Expressive Body Capture: 3D Hands, Face, and Body from a Single Image," CVPR 201".

[0109] The data generating device 100 may also directly discard the unreasonable skeleton information. Alternatively, the data generating device 100 may project the unreasonable skeleton information to a diversity of reasonable skeletons (e.g., the latent space of VPoser) and correct it to reasonable skeleton information. The data generating device 100 may also select only a part of the parts within the range of reasonable skeletons through combined optimization such as greedy algorithms to maximize the inference error of all parts in total.

[0110] When the image generation unit 152 obtains the information of the weakness attribute (the weakness attribute after integration) generated by the integration unit 153, it determines whether the skeleton information of the weakness attribute after integration is reasonable skeleton information. For example, the image generation unit 152 maintains the information of the operation range of each joint, and when each joint of the skeleton information is within the range of the operation range, it is determined to be reasonable skeleton information.

[0111] The data generating device 100 can determine and use multiple weakness attributes Nw (>1) for each body part in the order of inference error from large to small. The data generating device 100 can also generate new weakness attributes according to all combinations Nw^Np (Np is the number of parts). The data generating device 100 can also select the best combination of weakness attributes from all combinations Nw^Np by optimizing the combination with the rationality of the above-mentioned skeleton information as a constraint.

[0112] Next, an example of a hardware configuration of a computer that realizes the same function as the above-described data generating device 100 will be described. Fig.13 This is a diagram showing an example of the hardware configuration of a computer that realizes the same function as the data generating device of the embodiment.

[0113] like Fig.13 As shown, the computer 200 has a CPU 201 that performs various calculations, an input device 202 that receives data input from a user, and a display 203. In addition, the computer 200 has a communication device 204 that transmits and receives data with the camera 15, an external device, etc. via a wired or wireless network, and an interface device 205. In addition, the computer 200 has a RAM 206 that temporarily stores various information and a hard disk device 207. Moreover, each device 201 to 207 is connected to a bus 208.

[0114] The hard disk device 207 has a data expansion program 207a, an image generation program 207b, an integration program 207c, and a learning program 207d. The CPU 201 reads out each of the programs 207a to 207d and expands them in the RAM 206.

[0115] The data expansion program 207a functions as the data expansion process 206a. The image generation program 207b functions as the image generation process 206b. The integration program 207c functions as the integration process 206c. The learning program 207d functions as the learning process 206d.

[0116] The processing of the data expansion process 206a corresponds to the processing of the data expansion unit 151. The processing of the image generation process 206b corresponds to the processing of the image generation unit 152. The processing of the integration process 206c corresponds to the processing of the integration unit 153. The processing of the learning process 206d corresponds to the processing of the learning unit 154.

[0117] In addition, each program 207a to 207d may not necessarily be stored in the hard disk device 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a floppy disk (FD), CD-ROM, DVD, optical magnetic disk, IC card, etc. inserted into the computer 200. Furthermore, the computer 200 may read each program 207a to 207d and execute it.

[0118] Description of Reference Numerals

[0119] 50…training object model; 100…data generating device; 110…communication unit; 120…input unit; 130…display unit; 140…storage unit; 141…teacher data set; 150…control unit; 151…data expansion unit; 152…image generating unit; 153…integration unit; 154…learning unit.

Claims

1. A data generation program, characterized in that: The computer is made to execute the following processing: Obtaining the inference results of the skeleton information for each teacher data when multiple teacher data are input into the machine learning model, wherein the inference results include errors of each part of the skeleton, According to the inference result, first teacher data whose first part error is larger than the first part error of other teacher data is determined from the plurality of teacher data, According to the inference result, second teacher data whose error in the second part is larger than the error in the second part of other teacher data is determined from the plurality of teacher data, The third teaching data is generated by replacing the information related to the second part included in the first teaching data with the information related to the second part included in the second teaching data.

2. The data generation program according to claim 1, characterized in that The computer is further caused to execute a process of replacing the information related to the first part included in the second teaching data with the information related to the first part included in the first teaching data, thereby generating fourth teaching data.

3. The data generation program according to claim 1, characterized in that The computer is also caused to perform the following processing: performing machine learning of the machine learning model based on the third teacher data.

4. The data generation program according to claim 3, characterized in that The computer is also made to perform the following processing: determining whether the skeleton formed by the first part and the second part contained in the third teacher data is reasonable; if the skeleton formed by the first part and the second part is reasonable, executing the machine learning processing and executing the machine learning of the machine learning model based on the third teacher data.

5. A data generation method, characterized in that: The following processing is performed by a computer: Obtaining the inference results of the skeleton information for each teacher data when multiple teacher data are input into the machine learning model, wherein the inference results include errors of each part of the skeleton, According to the inference result, first teacher data whose first part error is larger than the first part error of other teacher data is determined from the plurality of teacher data, According to the inference result, second teacher data whose error in the second part is larger than the error in the second part of other teacher data is determined from the plurality of teacher data, The third teaching data is generated by replacing the information related to the second part included in the first teaching data with the information related to the second part included in the second teaching data.

6. The data generation method according to claim 5, characterized in that: The computer is further caused to execute a process of replacing the information related to the first part included in the second teaching data with the information related to the first part included in the first teaching data, thereby generating fourth teaching data.

7. The data generation method according to claim 5, characterized in that: The following processing is also performed by the computer: based on the third teacher data, machine learning of the machine learning model is performed.

8. The data generation method according to claim 7, characterized in that: The computer also performs the following processing: determining whether the skeleton formed by the first part and the second part contained in the third teacher data is reasonable; if the skeleton formed by the first part and the second part is reasonable, executing the machine learning processing; and executing the machine learning of the machine learning model based on the third teacher data.

9. A data generating device, comprising a control unit, wherein the control unit performs the following processing: Obtaining the inference results of the skeleton information for each teacher data when multiple teacher data are input into the machine learning model, wherein the inference results include errors of each part of the skeleton, According to the inference result, first teacher data whose first part error is larger than the first part error of other teacher data is determined from the plurality of teacher data, According to the inference result, second teacher data whose error in the second part is larger than the error in the second part of other teacher data is determined from the plurality of teacher data, The third teaching data is generated by replacing the information related to the second part included in the first teaching data with the information related to the second part included in the second teaching data.

10. The data generating device according to claim 9, characterized in that: The control unit further executes a process of replacing the information related to the first part included in the second teaching data with the information related to the first part included in the first teaching data, thereby generating fourth teaching data.

11. The data generating device according to claim 9, characterized in that: The control unit also performs the following processing: performing machine learning of the machine learning model based on the third teacher data.

12. The data generating device according to claim 11, characterized in that: The control unit also performs the following processing: determining whether the skeleton formed by the first part and the second part contained in the third teacher data is reasonable, and if the skeleton formed by the first part and the second part is reasonable, executing the machine learning processing based on the third teacher data, and executing machine learning of the machine learning model.

Images