Data-generating method, data-generating program, and data-generating device

Abstract

The present invention provides a data-generating device that obtains skeletal information inference results for each piece of teaching data when a plurality of teaching data is input to a machine learning model, the inference results including an error for each part of a skeleton. The data-generating device uses the inference results as a basis for identifying, from the plurality of teaching data, first teaching data in which the error of a first part is greater than the error of the first part in other teaching data. The data-generating device uses the inference results as a basis for identifying, from the plurality of teaching data, second teaching data in which the error of a second part is greater than the error of the second part in other teaching data. The data-generating device generates third teaching data by replacing information about the second part included in the first teaching data with information about the second part included in the second teaching data.

Description

Data generation method, data generation program, and data generation device

[0001] The present invention relates to a data generation method and the like.

[0002] A technology has been established for detecting a person's skeletal information using image data of the person captured by a camera. Skeletal information is information that indicates the coordinates of each of the person's joints. In this technology, multiple training data sets are prepared, and supervised learning is performed on a machine learning model such as a deep learning network. By inputting time-series image data of the person into the trained machine learning model, it is possible to detect time-series skeletal information of the person.

[0003] Here, by processing existing training data and adding the processed training data as new training data, it is possible to improve the general-purpose performance and robustness of the machine learning model. For example, there is a technology that generates new training data by adding geometric transformations or noise to the training data, or by moving image patches of body parts.

[0004] There is also a technique for generating new training data by expanding training data. This data augmentation technique feeds back gradient information on the inference error of a machine learning model during training, and generates training data at each training stage that maximizes the performance of the machine learning model. For example, in data augmentation, a 3D model of a person is projected onto a 2D plane to generate image data of training data. In data augmentation, the 3D model is rotated, translated, etc. based on the gradient information to generate image data that increases the inference error of the machine learning model. In data augmentation, a classifier is provided that excludes image data if the pose of the person in the image data is an impossible pose for a person. The process of generating image data from a 3D model using gradient information is called differentiable data augmentation.

[0005] For example, the above-mentioned data extension techniques include the following data extension techniques (1) to (3).

[0006] This section explains data augmentation technique (1). Data augmentation technique (1) uses Neural Radiance Fields (NeRF) to generate image data that improves the inference error of a machine learning model (object detection model). For example, data augmentation technique (1) divides the domain (360°) of camera angles, which are the control parameters for image generation, into k = 8 bins, and formulates the bin selection probability (categorical distribution) using Gambel-softmax. Data augmentation technique (1) simultaneously trains the machine learning model while simultaneously learning the bin selection probability in the direction that maximizes the inference error.

[0007] Data augmentation technique (2) will now be described. Data augmentation technique (2) relates to a machine learning model that converts 2D skeletal information into 3D skeletal information, and generates paired data of 2D and 3D skeletal information that improves the inference error of the machine learning model. For example, data augmentation technique (2) uses a multilayer perceptron to express learnable augmentation operations for existing 3D skeletal information. The augmentation operations include joint angle perturbations, bone length perturbations, and rotational / translational perturbations. Data augmentation technique (2) performs learning of the augmentation operations in a direction that maximizes the inference error while training the machine learning model.

[0008] We now describe data augmentation technique (3). In this technique, new 3D skeletons are generated from existing 3D skeletons and added to increase the training dataset. For example, data augmentation technique (3) performs a process of exchanging partial skeletons of two 3D skeletons and a process of perturbing joint angles.

[0009] Japanese Patent Application Laid-Open No. 2019-212106

[0010] Y. Ge et al. “Neural-Sim: Learning to Generate Training Data with NeRF,” ECCV 2022Gong et al., “PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation,” CVPR 2021S. Li et al., “Cascaded Deep Monocular 3D Human Pose Estimation with Evolutionary Training Data,” CVPR 2020

[0011] However, the above-mentioned conventional techniques have the problem that new training data tends to remain in the vicinity of existing training data, and it is not possible to generate new training data that deviates from the distribution of existing training data.

[0012] For example, consider a case where new training data for another field is generated based on training data for the field of gymnastics. The other fields include sports other than gymnastics, rehabilitation, and the like. The training data for the field of gymnastics is limited to certain 3D skeletal information that a person can assume during gymnastics. Therefore, when a machine learning model based on training data for the field of gymnastics is used in another field, the inference accuracy may decrease for skeletal information that is not included in the training data.

[0013] To prevent a decline in inference accuracy in other fields, it is desirable to retrain the machine learning model using new training data that is far from the distribution of the existing training data. Note that although the training data in the field of gymnastics has been used in this explanation, the present invention is not limited to this.

[0014] In one aspect, the present invention aims to provide a data generation method, a data generation program, and a data generation device that can generate new training data that deviates from the distribution of existing training data.

[0015] In a first proposal, a computer is caused to execute the following process. The computer acquires an inference result of skeletal information for each piece of training data when multiple pieces of training data are input into a machine learning model, the inference result including an error for each part of the skeleton. Based on the inference result, the computer identifies, from the multiple pieces of training data, first training data in which the error for a first part is larger than the error for the first part of the other pieces of training data. Based on the inference result, the computer identifies, from the multiple pieces of training data, second training data in which the error for a second part is larger than the error for the second part of the other pieces of training data. The computer generates third training data by replacing information about the second part included in the first training data with information about the second part included in the second training data.

[0016] It is possible to generate new training data that deviates from the distribution of existing training data.

[0017] FIG. 1 is a diagram illustrating an example of a human body model. FIG. 2 is a diagram illustrating an example of joint names. FIG. 3 is a diagram illustrating processing by a data generation device according to this embodiment. FIG. 4A is a diagram illustrating attributes. FIG. 4B is a diagram illustrating an example of attributes and extended data. FIG. 5 is a diagram illustrating a body part p. FIG. 6 is a diagram (1) illustrating processing by an integration unit. FIG. 7 is a diagram (2) illustrating processing by the integration unit. FIG. 8 is a diagram illustrating an example of extended data generated based on attributes and weak point attributes. FIG. 9 is a functional block diagram illustrating a configuration of a data generation device according to this embodiment. FIG. 10 is a flowchart illustrating a processing procedure of the data generation device according to this embodiment. FIG. 11 is a flowchart illustrating a processing procedure of integration processing. FIG. 12 is a diagram illustrating the effects of the data generation device according to this embodiment. FIG. 13 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the data generation device of the embodiment.

[0018] Hereinafter, embodiments of the data generating method, the data generating program, and the data generating device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments.

[0019] Before describing this embodiment, an example of a human body model will be described. FIG. 1 is a diagram showing an example of a human body model. As shown in FIG. 1, the human body model is defined by 21 joints ar0 to ar20. In the skeleton information used in this embodiment, two-dimensional or three-dimensional coordinates are set for each of the joints ar0 to ar20 defined in the human body model.

[0020] The relationship between the joint names and the joints ar0 to ar20 shown in Fig. 1 is as shown in Fig. 2. Fig. 2 is a diagram showing examples of joint names. For example, the joint name of joint ar0 is "SPINE_BASE". The joint names of joints ar1 to ar20 are as shown in Fig. 2, and explanations thereof will be omitted.

[0021] Next, the processing of the data generating device according to this embodiment will be described. Fig. 3 is a diagram for explaining the processing of the data generating device according to this embodiment. The data generating device uses training data 30.

[0022] The training data 30 is existing data. For example, the training data includes image data and attributes of a person. The attributes include skeletal information, camera parameters, and appearance. The skeletal information is the joint coordinates described in FIG. 1 and indicates the coordinates of each joint of the person included in the image data. The coordinates of each joint are two-dimensional or three-dimensional coordinates. The camera parameters indicate the viewpoint position of the camera that captured the image data. The appearance is information about the appearance of the person included in the image data and the background of the person.

[0023] 4A is a diagram for explaining attributes. For example, attribute A of certain training data １ The parameters a1-1, a1-2, and appearances a1-3 and a1-4 are included in the parameter a1-1. In FIG. 4A, the camera parameters a1-2 are shown as a conceptual illustration, but in reality, the parameter a1-2 includes information on the viewpoint position of the camera that captured the image data. The appearance a1-3 specifies the person's appearance (physique, uniform color, etc.). The appearance a1-4 specifies the person's background information.

[0024] Returning to the explanation of Fig. 3, the data generating device inputs the teacher data 30 to the data extension unit 151.

[0025] The data extension unit 151 uses the parameter θ １ is set, and the parameter θ １ The data extension unit 151 outputs the information of the extended attributes to the image generation unit 152. The image generation unit 152, which will be described later, uses the parameter θ １ Based on the attributes extended by the method, the extended data 40 is generated, and the extended data 40 is input to the training target model 50 to calculate the inference error.

[0026] The data augmentation unit 151 adjusts the parameter θ in a direction that increases the inference error when the augmented data 40 is input to the training object model 50, based on the gradient information of the inference error fed back from the training object model 50. １ Train.

[0027] For example, the data extension unit 151 uses the parameter θ １ Based on the parameter θ, the joint angles and lengths of bones between joints in the skeletal information included in the attribute are changed in a direction that increases the inference error. １ Based on this, data augmentation may be performed by modifying the camera parameters and appearance in a direction that increases the inference error.

[0028] When extending attributes, the data extension unit 151 ensures the likelihood of the data. For example, the data extension unit 151 changes the joint angles within the operable range of the joints in the skeletal information. When changing the length of a bone, the data extension unit 151 changes the length of the bone within a predetermined range.

[0029] The image generation unit 152 generates extended data 40 based on the information of the attributes extended by the data extension unit 151 or the integration unit 153. The image generation unit 152 is a differentiable image generator such as NeRF. In the following description of the image generation unit 152, the "extended attributes" will be referred to simply as "attributes."

[0030] The image generation unit 152 generates a person model and a background model based on the skeletal information and appearance included in the attributes. The image generation unit 152 generates image data (augmented data 40) from a viewpoint based on the camera parameters in the attribute information for a model obtained by combining the person model and the background model.

[0031] 4B is a diagram showing an example of attributes and extended data. １ The image generator 152 generates the attribute A. １ Based on this, the extended data 40-1 is generated.

[0032] Attribute A ２ The image generator 152 generates the attribute A. ２ Based on this, the extended data 40-2 is generated.

[0033] Attribute A ３ The image generator 152 generates the attribute A. ３ Based on this, the extended data 40-3 is generated.

[0034] The data generating device performs machine learning of the training object model 50 based on the augmented data 40 and skeletal information (skeletal information of the extended attributes) used when generating the augmented data 40. For example, the skeletal information used when generating the augmented data 40 is used as a correct label. The training object model 50 is a neural network (NN) or the like. The training object model 50 includes a parameter θ ２ is set.

[0035] The data generator inputs the augmented data 40 into the training object model 50 and obtains the inference result output from the training object model 50. The data generator adjusts the parameter θ of the training object model 50 so as to reduce the inference error between the inference result and the correct label. ２ The data generator feeds back the gradient information of the inference error to the data extension unit 151.

[0036] The data generation device associates, for each joint in the skeletal information, a pair of inference result information indicating the relationship between the inference result and the true value (correct label) with information on the attribute (extended attribute) used when generating the extended data 40, and outputs the result to the integration unit 153.

[0037] The integrating unit 153 executes the following process based on the inference result information and the information on the extended attributes. In the following description of the integrating unit 153, the "extended attributes" will be simply referred to as "attributes." For example, the integrating unit 153 performs the following process on the training target model 50: ２ is updated, and multiple pairs of inference result information and attribute information are obtained.

[0038] The integration unit 153 identifies the inference error for each body part p based on the inference result information. FIG. 5 is a diagram for explaining body part p. In this embodiment, in the hierarchical structure of joints, a group of joints from the terminal joint to just before the bifurcation point joint is defined as body part p. For example, body parts p may be "head," "armL," "armR," "legL," and "legR." In other words, p∈{head, armL, armR, legL, legR} is defined.

[0039] The body part "head" corresponds to joints ar3 and ar18. The body part "armL" corresponds to joints ar4, ar5, ar6, and ar19. The body part "armR" corresponds to joints ar7, ar8, ar9, and ar20. The body part "legL" corresponds to joints ar10, ar11, ar12, and ar13. The body part "legR" corresponds to joints ar14, ar15, ar16, and ar17.

[0040] The integration unit 153 identifies an inference error for each body part p for each piece of inference result information. That is, from one piece of inference result information, the inference errors for the body parts “head,” “armL,” “armR,” “legL,” and “legR” are identified.

[0041] For example, the inference error for the body part "head" is the MSE (Mean Squared Error) between the inference results and the true values ​​for joints ar3 and ar18. The inference error for the body part "armL" is the MSE between the inference results and the true values ​​for joints ar4, ar5, ar6, and ar19. The inference error for the body part "armR" is the MSE between the inference results and the true values ​​for joints ar7, ar8, ar9, and ar20. The inference error for the body part "legL" is the MSE between the inference results and the true values ​​for joints ar10, ar11, ar12, and ar13. The inference error for the body part "legR" is the MSE between the inference results and the true values ​​for joints ar14, ar15, ar16, and ar17.

[0042] The integration unit 153 compares the inference errors for each body part p calculated from each inference result information, and identifies the maximum inference error for each body part p and the attribute corresponding to the inference result information with the maximum inference error.

[0043] 6 is a diagram (1) for explaining the processing of the integration unit. In the example shown in FIG. 6, attribute A １ The inference result information obtained by inputting the augmented data generated based on the １ Let the inference result R １ The inference error for the body part "head" obtained based on the inference result R is assumed to be inference error E1-1. １ The inference error for the body part "armL" obtained based on the inference result R is assumed to be inference error E1-2. １ The inference error for the body part "armR" obtained based on the inference result R is assumed to be inference error E1-3. １ The inference error for the body part "legR" obtained based on the inference result R is assumed to be inference error E1-4. １ The inference error for the body part "legL" obtained based on the above is defined as inference error E1-5.

[0044] Attribute A ２ The inference result information obtained by inputting the augmented data generated based on the ２ Let the inference result R ２The inference error for the body part "head" obtained based on the inference result R is assumed to be inference error E2-1. ２ The inference error for the body part "armL" obtained based on the inference result R is assumed to be inference error E2-2. ２ The inference error for the body part "armR" obtained based on the inference result R is assumed to be inference error E2-3. ２ The inference error for the body part "legR" obtained based on the inference result R is assumed to be inference error E2-4. ２ The inference error for the body part "legL" obtained based on the above is assumed to be inference error E2-5.

[0045] Attribute A ｎ The inference result information obtained by inputting the augmented data generated based on the ｎ n is a natural number greater than or equal to 3. Inference result R ｎ The inference error for the body part "head" obtained based on the inference result R is assumed to be the inference error En-1. ｎ The inference error for the body part "armL" obtained based on the inference result R is assumed to be the inference error En-2. ｎ The inference error for the body part "armR" obtained based on the inference result R is assumed to be the inference error En-3. ｎ The inference error for the body part "legR" obtained based on the inference result R is assumed to be the inference error En-4. ｎ The inference error for the body part "legL" obtained based on the above is assumed to be inference error En-5.

[0046] The integration unit 153 compares the inference errors E1-1 to En-1 of the body part "head" and identifies the inference error with the maximum value. In this embodiment, as an example, among the inference errors E1-1 to En-1, the inference error E1-1 is set to be the maximum value. The attribute corresponding to the inference error E1-1 is attribute A. １ In this case, the integration unit 153 sets the weak point attribute of the body part “head” as weak point attribute A １ Identify as:

[0047] The integration unit 153 compares the inference errors E1-2 to En-2 of the body part "armL" and identifies the inference error with the maximum value. In this embodiment, as an example, among the inference errors E1-2 to En-2, the inference error E2-2 is set to be the maximum value. The attribute corresponding to the inference error E2-2 is attribute A. ２ In this case, the integration unit 153 sets the weak point attribute of the body part “armL” as weak point attribute A ２ Identify as:

[0048] The integration unit 153 compares the inference errors E1-3 to En-3 of the body part "armR" and identifies the inference error with the maximum value. In this embodiment, as an example, among the inference errors E1-3 to En-3, the inference error E3-3 is set to be the maximum value. The attribute corresponding to the inference error E3-3 is attribute A. ３ In this case, the integration unit 153 sets the weak point attribute of the body part “armR” as weak point attribute A ３ In FIG. 6, attribute A ３ The illustration of is omitted.

[0049] The integration unit 153 compares the inference errors E1-4 to En-4 for the body part "legL" and identifies the inference error with the maximum value. In this embodiment, as an example, among the inference errors E1-4 to En-4, the inference error E4-4 is set to be the maximum value. The attribute corresponding to the inference error E4-4 is attribute A. ４ In this case, the integration unit 153 sets the weak point attribute of the body part “legL” as weak point attribute A ４ In FIG. 6, attribute A ４ The illustration of is omitted.

[0050] The integration unit 153 compares the inference errors E1-5 to En-5 for the body part "legR" and identifies the inference error with the maximum value. In this embodiment, as an example, among the inference errors E1-5 to En-5, the inference error E5-5 is set to be the maximum value. The attribute corresponding to the inference error E5-5 is attribute A. ５ In this case, the integration unit 153 sets the weak point attribute of the body part “legR” as weak point attribute A ５ In FIG. 6, attribute A ５ The illustration of is omitted.

[0051] The integration unit 153 executes the process shown in FIG. 6 to identify the weak point attribute of each body part p. In the following description, as an example, the weak point attribute of the body part “head” is set as weak point attribute A １ The weak attribute of the body part "armL" is set to weak attribute A. ２ The weak attribute of the body part "armR" is set to weak attribute A. ３ The weak attribute of the body part "legL" is set to weak attribute A. ４ The weak attribute of the body part "legR" is set to weak attribute A. ５ Let's say.

[0052] 7 is a diagram (2) for explaining the processing of the integration unit. １ Based on, weak attribute A １ ~Weakness attribute A ５ By integrating these, the weak attribute A' １ Generates weak attribute A'. １ Similar to the attributes, the weak point attribute A' includes skeletal information, camera parameters, and appearance. １ The skeleton information is weak attribute A １ The joint coordinates of the body part "head", weak attribute A ２ The coordinates of each joint of the body part "armL", weak point attribute A ３ The coordinates of each joint of the body part "armR", weak point attribute A ４ The joint coordinates of the body part "legL" and weak point attribute A ５ This is skeletal information that combines the coordinates of each joint of the body part "legR". １ The camera parameters and appearance of the base weak point attribute A １ The camera parameters and appearance of the original will be reused.

[0053] The integration unit 153 is a weak point attribute A ２ Based on, weak attribute A １ ~Weakness attribute A ５ By integrating these, the weak attribute A' ２ Generates weak attribute A'. ２ Similar to the attributes, the weak point attribute A' includes skeletal information, camera parameters, and appearance. ２ The skeleton information is weak attribute A１ The joint coordinates of the body part "head", weak attribute A ２ The coordinates of each joint of the body part "armL", weak point attribute A ３ The coordinates of each joint of the body part "armR", weak point attribute A ４ The joint coordinates of the body part "legL" and weak point attribute A ５ This is skeletal information that combines the coordinates of each joint of the body part "legR". ２ The camera parameters and appearance of the base weak point attribute A ２ The camera parameters and appearance of the original will be reused.

[0054] The integration unit 153 is a weak point attribute A ３ Based on, weak attribute A １ ~Weakness attribute A ５ By integrating these, the weak attribute A' ３ Generates weak attribute A'. ３ Similar to the attributes, the weak point attribute A' includes skeletal information, camera parameters, and appearance. ３ The skeleton information is weak attribute A １ The joint coordinates of the body part "head", weak attribute A ２ The coordinates of each joint of the body part "armL", weak point attribute A ３ The coordinates of each joint of the body part "armR", weak point attribute A ４ The joint coordinates of the body part "legL" and weak point attribute A ５ This is skeletal information that combines the coordinates of each joint of the body part "legR". ３ The camera parameters and appearance of the base weak point attribute A ３ The camera parameters and appearance of the original will be reused.

[0055] The integration unit 153 is a weak point attribute A ４ Based on, weak attribute A １ ~Weakness attribute A ５ By integrating these, the weak attribute A' ４ Generates weak attribute A'. ４ Similar to the attributes, the weak point attribute A' includes skeletal information, camera parameters, and appearance. ４ The skeleton information is weak attribute A １The joint coordinates of the body part "head", weak attribute A ２ The coordinates of each joint of the body part "armL", weak point attribute A ３ The coordinates of each joint of the body part "armR", weak point attribute A ４ The joint coordinates of the body part "legL" and weak point attribute A ５ This is skeletal information that combines the coordinates of each joint of the body part "legR". ４ The camera parameters and appearance of the base weak point attribute A ４ The camera parameters and appearance of the original will be reused.

[0056] The integration unit 153 is a weak point attribute A ５ Based on, weak attribute A １ ~Weakness attribute A ５ By integrating these, the weak attribute A' ５ Generates weak attribute A'. ５ Similar to the attributes, the weak point attribute A' includes skeletal information, camera parameters, and appearance. ５ The skeleton information is weak attribute A １ The joint coordinates of the body part "head", weak attribute A ２ The coordinates of each joint of the body part "armL", weak point attribute A ３ The coordinates of each joint of the body part "armR", weak point attribute A ４ The joint coordinates of the body part "legL" and weak point attribute A ５ This is skeletal information that combines the coordinates of each joint of the body part "legR". ５ The camera parameters and appearance of the base weak point attribute A ５ The camera parameters and appearance of the original will be reused.

[0057] The integration unit 153 executes the process described with reference to FIG. 7 to obtain the weak point attribute A′. １ ~A' ５ The integration unit 153 generates the weak point attribute A′. １ ~A' ５ is input to the image generating unit 152, the weak point attribute A' is obtained. １ ~A' ５ and generate corresponding extension data for each of the

[0058] 8 is a diagram showing an example of extended data generated by attributes and weak point attributes. For example, extended data Im10 is generated by adding attribute A ２ is input to the image generation unit 152. When the extended data Im10 is input to the training target model 50, the inference error for the body part "armL" is the maximum value compared to the other inference errors. The extended data Im11 is input to the training target model 50, and the weak point attribute A' is input to the training target model 50. ２ This is image data obtained by inputting the augmented data Im11 to the image generation unit 152. When the augmented data Im11 is input to the training target model 50, the inference errors for the body parts "head", "armL", "armR", "legL", and "legR" reach their maximum values. In other words, it is possible to generate new training data that is far from the distribution of existing training data.

[0059] The data generation device generates a set of extended data obtained by inputting the weak point attributes into the image generation unit 152 and skeletal information included in the weak point attributes as training data, and uses this data for machine learning of the training target model 50.

[0060] As described above, the data generation device according to this embodiment identifies an inference error for each body part p based on the inference result information obtained by inputting the augmented data 40 into the training target model 50. The data generation device compares the inference errors for each body part p calculated from each piece of inference result information, and identifies the attribute (weak attribute) of the inference result information for which the inference error is maximum. The data generation device integrates the weak attribute for each body part p, and generates augmented data based on the integrated weak attribute. This makes it possible to generate new training data that deviates from the distribution of existing training data.

[0061] In the above description, the data generation device identifies and integrates weak attributes based on inference result information obtained as a result of inputting the augmented data 40 into the training object model 50, but the present invention is not limited to this. For example, the teacher data 30 may be directly input into the training object model 50, and weak attributes may be identified and integrated based on inference result information obtained as a result of inputting the teacher data 30 into the training object model 50.

[0062] Next, an example of the configuration of a data generating device that executes the above-mentioned processing will be described. Fig. 9 is a functional block diagram showing the configuration of a data generating device according to this embodiment. As shown in Fig. 9, this data generating device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0063] The communication unit 110 executes data communication with external devices etc. via a network. The communication unit 110 is a network interface card (NIC) etc. The control unit 150 (described later) exchanges data with external devices via the communication unit 110.

[0064] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the data generating device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0065] The display unit 130 is a display device that displays information output from the control unit 150 .

[0066] The storage unit 140 includes a training target model 50 and a teacher dataset 141. The storage unit 140 is a storage device such as a memory.

[0067] The training target model 50 is a machine learning model that receives image data (augmented data) as input and outputs an inference result of skeletal information. The training target model 50 is, for example, a neural network.

[0068] The training data set 141 includes a plurality of training data. The training data includes image data of a person and attributes. The attributes include skeletal information, camera parameters, and appearance.

[0069] The control unit 150 includes a data expansion unit 151, an image generation unit 152, an integration unit 153, and a learning unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0070] The data extension unit 151 uses the parameter θ １ is set, and the parameter θ １The data augmentation unit 151 outputs information on the augmented attributes to the image generation unit 152. Based on gradient information on the inference error fed back from the training object model 50, the data augmentation unit 151 adjusts the parameter θ in the direction in which the inference error increases when the augmented data is input to the training object model 50. １ The rest of the description regarding the data augmentation unit 151 is the same as the description regarding the data augmentation unit 151 described with reference to FIG.

[0071] The image generation unit 152 generates extended data based on the information on the attributes extended by the data extension unit 151. The image generation unit 152 also generates extended data based on the information on the weak point attributes generated by the integration unit 153. The image generation unit 152 may add a set of the information on the weak point attributes and the extended data to the teacher dataset 141 as new teacher data.

[0072] The integrating unit 153 identifies an inference error for each body part p for each piece of inference result information. The integrating unit 153 compares the inference errors for each body part p calculated from each piece of inference result information, and identifies the maximum value of the inference error for each body part p and the attribute (weak point attribute) corresponding to the inference result information with the maximum inference error. The integrating unit 153 generates a weak point attribute by integrating the weak point attributes for each body part p. Other explanations regarding the integrating unit 153 are the same as those regarding the integrating unit 153 described in Figures 3 and 5 to 7.

[0073] The learning unit 154 performs machine learning of the training object model 50 based on the teacher data set 141. For example, when image data is input to the training object model 50, the learning unit 154 uses the backpropagation method to adjust the parameter θ of the training object model 50 so that the error between the inference result output from the training object model 50 and the correct label is reduced. ２ Update.

[0074] The pair of image data and correct label that the learning unit 154 inputs into the training target model 50 will be a pair of first image data and first correct label, or a pair of second image data and second correct label, as described below.

[0075] The first correct label is skeleton information obtained when the attributes of the training data are extended by the data extension unit 151. The first image data is extended data generated by the image generation unit 152 based on the attributes of the training data extended by the data extension unit 151.

[0076] The second correct label is skeleton information of the weak point attribute integrated by the integration unit 153. The second image data is extended data generated by the image generation unit 152 based on the weak point attribute.

[0077] The learning unit 154 feeds back gradient information of the inference error to the data expansion unit 151. The learning unit 154 outputs to the integration unit 153 a set of inference result information indicating the relationship between the inference result and the true value (correct label) and attribute information for each joint of the skeletal information.

[0078] Next, an example of the processing procedure of the data generating device 100 according to this embodiment will be described. FIG. 10 is a flowchart showing the processing procedure of the data generating device according to this embodiment. As shown in FIG. 10, the data extension unit 151 of the data generating device 100 acquires training data from the training data set 141 (step S101). The data extension unit 151 acquires training data from the training data set 141 by using a parameter θ １ Based on this, the attributes of the training data are expanded in a direction that increases the inference error of the training target model 50 (step S102).

[0079] The image generation unit 152 of the data generation device 100 generates augmented data based on the augmented attributes (step S103). The learning unit 154 of the data generation device 100 performs machine learning of the training target model 50 based on the augmented data and the correct labels (step S104).

[0080] The integration unit 153 of the data generating device 100 executes integration processing (step S105).

[0081] The data expansion unit 151 of the data generating device 100 receives feedback of gradient information of the inference result and calculates the parameter θ １ is updated (step S106).

[0082] If the data generating device 100 determines to continue the process (Yes at step S107), the data generating device 100 proceeds to step S101. On the other hand, if the data generating device 100 determines not to continue the process (No at step S107), the data generating device 100 ends the process.

[0083] Next, an example of the integration process shown in step S105 in Fig. 10 will be described. Fig. 11 is a flowchart showing the processing procedure of the integration process. As shown in Fig. 11, the integration unit 153 of the data generating device 100 identifies an inference error for each body part p for each piece of inference result information (step S201).

[0084] The integrating unit 153 compares the inference errors for each body part p and identifies a weak point attribute for each body part p (step S202).The integrating unit 153 integrates the weak point attributes for each body part p (step S203).

[0085] The image generating unit 152 of the data generating device 100 generates extended data based on the integrated weak point attributes (step S204).

[0086] The learning unit 154 of the data generating device 100 performs machine learning of the training target model 50 based on the augmented data and the correct label (skeleton information of the weak attribute) (step S205).

[0087] Next, the effects of the data generating device 100 according to this embodiment will be described. The data generating device 100 identifies an inference error for each body part p based on inference result information obtained by inputting augmented data into a training target model. The data generating device 100 compares the inference errors for each body part p calculated from each piece of inference result information, and identifies the attribute (weak attribute) of the inference result information that maximizes the inference error. The data generating device 100 integrates the weak attribute for each body part p, and generates augmented data based on the integrated weak attribute. This makes it possible to generate new training data that deviates from the distribution of existing training data.

[0088] FIG. 12 is a diagram illustrating the effect of the data generation device according to this embodiment. Image data Im20 in FIG. 12 is augmented data generated by the image generation unit 152 based on the integrated weak point attributes. When the learning unit 154 inputs the image data Im20 into the training target model 50, an inference result 60 is output. In the inference result 60, inference between joints ar9 and ar20 and joints ar5, ar6, and ar7 fails. In other words, by performing machine learning using image data Im20 as new training data, which is new image data (training data) that is far from the distribution of existing training data, the inference accuracy of the training target model 50 can be improved.

[0089] For example, inference results 60a and 60b are obtained by training the training target model 50 when machine learning is performed using new image data (teaching data) that is different from the distribution of existing teaching data. In the inference result 60a, inference fails at joint ar19, but the inference accuracy is improved at joints ar5 and ar6. In the inference result 60b, the inference accuracy is improved at joints ar9 and ar20, joints ar5, ar6, ar19, and joint ar11.

[0090] The processing of the data generating device 100 according to this embodiment is not limited to the above. For example, the data generating device 100 can execute a body detection task or a body region extraction (segmentation) task from image data. The data generating device 100 executes the body detection task or body region extraction, and by referencing body parts with large inference errors, can identify weak point attributes for each body part and apply the above-described processing.

[0091] The processing of the data generating device 100 can be applied to tasks that target more general articulated bodies, such as quadrupedal animals, rather than human bodies. The processing of the data generating device 100 can be applied to both 2D and 3D body skeleton estimation tasks. The processing of the data generating device 100 can also be applied to 2D-to-3D skeleton estimation tasks that do not require image data.

[0092] The data generating device 100 may use a mechanism for evaluating the likelihood of skeletal information obtained by integrating weak attribute values ​​to reject or correct data that has been combined with implausible skeletal information. As a mechanism for evaluating the likelihood of skeletal information, the data generating device 100 uses a VPoser that evaluates the distance in the latent space of the pose generator, a penetration loss that evaluates the penetration of the body model, and a hyper-bending loss that evaluates the backward bending of elbows and knees. For example, technology related to VPoser is described in the literature "G. Pavlakos et al., "Expressive Body Capture: 3D Hands, Face, and Body from a Single Image," CVPR 201."

[0093] The data generating device 100 may immediately discard implausible skeleton information. Alternatively, the data generating device 100 may project the implausible skeleton information onto a manifold of plausible skeletons (e.g., the latent space of a VPoser) and correct it to plausible skeleton information. The data generating device 100 may select only some parts within the range of plausible skeletons by combinatorial optimization such as a greedy algorithm so as to maximize the total inference error for all parts.

[0094] When the image generation unit 152 acquires information on the weak point attribute (integrated weak point attribute) generated by the integration unit 153, the image generation unit 152 determines whether the skeletal information of the integrated weak point attribute is plausible. For example, the image generation unit 152 holds information on the working area of ​​each joint, and determines that the skeletal information is plausible when each joint in the skeletal information is within the range of the working area.

[0095] The data generating device 100 may identify and use Nw (>1) weak attribute combinations for each body part in descending order of inference error. The data generating device 100 may generate new weak attribute combinations for all combinations Nw^Np (Np is the number of body parts). The data generating device 100 may select the optimal combination of weak attribute combinations from among all combinations Nw^Np by combinatorial optimization using the likelihood of the skeletal information as a constraint.

[0096] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the above-described data generating device 100. Fig. 13 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the data generating device of the embodiment.

[0097] 13, the computer 200 has a CPU 201 that executes various types of arithmetic processing, an input device 202 that accepts data input from a user, and a display 203. The computer 200 also has a communication device 204 that exchanges data with the camera 15, external devices, etc. via a wired or wireless network, and an interface device 205. The computer 200 also has a RAM 206 that temporarily stores various types of information, and a hard disk drive 207. The devices 201 to 207 are connected to a bus 208.

[0098] The hard disk drive 207 stores a data expansion program 207a, an image generation program 207b, an integration program 207c, and a learning program 207d. The CPU 201 reads out each of the programs 207a to 207d and loads them into the RAM 206.

[0099] The data extension program 207a functions as the data extension process 206a. The image generation program 207b functions as the image generation process 206b. The integration program 207c functions as the integration process 206c. The learning program 207d functions as the learning process 206d.

[0100] The processing of the data extension process 206a corresponds to the processing of the data extension unit 151. The processing of the image generation process 206b corresponds to the processing of the image generation unit 152. The processing of the integration process 206c corresponds to the processing of the integration unit 153. The processing of the learning process 206d corresponds to the processing of the learning unit 154.

[0101] It should be noted that each of the programs 207a to 207d does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each program may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d.

[0102] 50 Training target model 100 Data generation device 110 Communication unit 120 Input unit 130 Display unit 140 Storage unit 141 Teacher dataset 150 Control unit 151 Data expansion unit 152 Image generation unit 153 Integration unit 154 Learning unit

Claims

1. Obtaining an inference result of skeletal information for each training data when multiple training data are input into a machine learning model, the inference result including an error for each part of the skeleton; Based on the inference result, a first teacher data piece having a larger error in a first portion than the errors in the first portions of other teacher data pieces is identified from the plurality of teacher data pieces; Identifying second teacher data having a larger error in a second portion than the errors in the second portions of other teacher data from the plurality of teacher data based on the inference result; The information on the second part included in the first teacher data is replaced with the information on the second part included in the second teacher data, thereby generating third teacher data. A data generation program that causes a computer to execute a process.

2. The data generation program according to claim 1, further comprising causing the computer to execute a process of generating fourth teacher data by replacing information regarding the first part contained in the second teacher data with information regarding the first part contained in the first teacher data.

3. The data generation program according to claim 1 , further comprising a step of causing a computer to execute a process of executing machine learning of the machine learning model based on the third teacher data.

4. The data generation program of claim 3, further comprising a process for causing a computer to execute a process for determining whether a skeleton consisting of the first part and the second part included in the third teacher data is plausible or not, and the process for executing the machine learning comprises executing machine learning of the machine learning model based on the third teacher data when the skeleton consisting of the first part and the second part is plausible.

5. Obtaining an inference result of skeletal information for each training data when multiple training data are input into a machine learning model, the inference result including an error for each part of the skeleton; Based on the inference result, a first teacher data piece having a larger error in a first portion than the errors in the first portions of other teacher data pieces is identified from the plurality of teacher data pieces; Identifying second teacher data having a larger error in a second portion than the errors in the second portions of other teacher data from the plurality of teacher data based on the inference result; The information on the second part included in the first teacher data is replaced with the information on the second part included in the second teacher data, thereby generating third teacher data. A data generating method, characterized in that processing is executed by a computer.

6. Obtaining an inference result of skeletal information for each training data when multiple training data are input into a machine learning model, the inference result including an error for each part of the skeleton; Based on the inference result, a first teacher data piece having a larger error in a first portion than the errors in the first portions of other teacher data pieces is identified from the plurality of teacher data pieces; Identifying second teacher data having a larger error in a second portion than the errors in the second portions of other teacher data from the plurality of teacher data based on the inference result; The information on the second part included in the first teacher data is replaced with the information on the second part included in the second teacher data, thereby generating third teacher data. A data generating device having a control unit that executes processing.