Data generation method, data generation program, and data generation device

Abstract

The present invention provides a data-generating device that obtains skeletal information inference results for each piece of teaching data when a plurality of teaching data is input to a machine learning model, the inference results including an error for each part of a skeleton. The data-generating device uses the inference results as a basis for identifying, from the plurality of teaching data, first teaching data in which the error of a first part is greater than the error of the first part in other teaching data. The data-generating device uses the inference results as a basis for identifying, from the plurality of teaching data, second teaching data in which the error of a second part is greater than the error of the second part in other teaching data. The data-generating device generates third teaching data by replacing information about the second part included in the first teaching data with information about the second part included in the second teaching data.

Description

[Technical Field]

[0001] The present invention relates to a data generation method and the like. [Background technology]

[0002] A technology has been established for detecting a person's skeletal information using image data of the person captured by a camera. Skeletal information is information that indicates the coordinates of each of the person's joints. In this technology, multiple training data sets are prepared, and supervised learning is performed on a machine learning model such as a deep learning network. By inputting time-series image data of the person into the trained machine learning model, it is possible to detect time-series skeletal information of the person.

[0003] Here, by processing existing training data and adding the processed training data as new training data, it is possible to improve the general-purpose performance and robustness of the machine learning model. For example, there is a technology that generates new training data by adding geometric transformations or noise to the training data, or by moving image patches of body parts.

[0004] There is also a technology that generates new training data by expanding training data. This data augmentation technology feeds back gradient information on the inference error of a machine learning model during training, and generates training data at each training stage that maximizes the performance of the machine learning model. For example, data augmentation technology projects a 3D model of a person onto a 2D plane to generate image data as training data. Data augmentation technology rotates, translates, etc. the 3D model based on the gradient information to generate image data that increases the inference error of the machine learning model. Data augmentation technology is equipped with a classifier that excludes image data if the pose of the person in the image data is an impossible pose for a person. The process of generating image data from a 3D model using gradient information is called differentiable data augmentation.

[0005] For example, the above-mentioned data extension techniques include the following data extension techniques (1) to (3).

[0006] This section explains data augmentation technique (1). Data augmentation technique (1) uses Neural Radiance Fields (NeRF) to generate image data that improves the inference error of a machine learning model (object detection model). For example, data augmentation technique (1) divides the domain (360°) of camera angles, which is the control parameter for image generation, into k = 8 bins, and formulates the bin selection probability (categorical distribution) using Gambel-softmax. Data augmentation technique (1) simultaneously trains the machine learning model while simultaneously learning the bin selection probability in the direction that maximizes the inference error.

[0007] This section explains data augmentation technology (2). Data augmentation technology (2) is related to a machine learning model that converts 2D skeletal information into 3D skeletal information, and generates paired data of 2D and 3D skeletal information that improves the inference error of the machine learning model. For example, data augmentation technology (2) uses a multilayer perceptron to express learnable augmentation operations for existing 3D skeletal information. The augmentation operations include perturbations of joint angles, bone lengths, and rotational / translational perturbations. Data augmentation technology (2) performs learning of the augmentation operations in a direction that maximizes the inference error while training the machine learning model.

[0008] We now explain data augmentation technique (3). In data augmentation technique (3), new 3D skeletons are generated from existing 3D skeletons and added to increase the training dataset. For example, data augmentation technique (3) performs processes such as exchanging partial skeletons of two 3D skeletons and perturbing joint angles. [Prior art documents] [Patent documents]

[0009] [Patent Document 1] Japanese Patent Application Publication No. 2019-212106 [Non-patent literature]

[0010] [Non-Patent Document 1] Y. Ge et al. “Neural-Sim: Learning to Generate Training Data with NeRF,” ECCV 2022 [Non-patent document 2] Gong et al., “PoseAug: A Differentiable Pose Augmentation Framework for 3D Human Pose Estimation,”CVPR 2021 [Non-patent document 3] S. Li et al., “Cascaded Deep Monocular 3D Human Pose Estimation with Evolutionary Training Data,” CVPR 2020 Summary of the Invention [Problem to be solved by the invention]

[0011] However, the above-mentioned conventional techniques have the problem that new training data tends to remain in the vicinity of existing training data, and it is not possible to generate new training data that deviates from the distribution of existing training data.

[0012] For example, consider the case where new training data for another field is generated based on training data for the field of gymnastics. The other fields could be sports other than gymnastics or rehabilitation. Training data for the field of gymnastics is limited to certain 3D skeletal information that a person can assume during gymnastics. For this reason, if a machine learning model based on training data for the field of gymnastics is used in another field, the inference accuracy may decrease for skeletal information that is not included in the training data.

[0013] To prevent a decline in inference accuracy in other fields, it is desirable to retrain the machine learning model using new training data that is far from the distribution of the existing training data. Note that although the training data in the field of gymnastics has been used in this explanation, the present invention is not limited to this.

[0014] In one aspect, the present invention aims to provide a data generation method, a data generation program, and a data generation device that can generate new training data that deviates from the distribution of existing training data. [Means for solving the problem]

[0015] In the first proposal, a computer is caused to execute the following process. The computer acquires an inference result that is an inference result of skeletal information for each piece of training data when multiple pieces of training data are input into a machine learning model, the inference result including an error for each part of the skeleton. Based on the inference result, the computer identifies, from the multiple pieces of training data, first training data in which the error for a first part is larger than the error for the first part in the other pieces of training data. Based on the inference result, the computer identifies, from the multiple pieces of training data, second training data in which the error for a second part is larger than the error for the second part in the other pieces of training data. The computer generates third training data by replacing information about the second part included in the first training data with information about the second part included in the second training data. [Effects of the Invention]

[0016] It is possible to generate new training data that deviates from the distribution of existing training data. [Brief explanation of the drawings]

[0017] [Figure 1] FIG. 1 is a diagram showing an example of a human body model. [Figure 2] FIG. 2 is a diagram showing an example of joint names. [Figure 3] FIG. 3 is a diagram for explaining the processing of the data generating device according to the present embodiment. [Figure 4A]FIG. 4A is a diagram for explaining attributes. [Figure 4B] FIG. 4B is a diagram showing an example of attributes and extended data. [Figure 5] FIG. 5 is a diagram for explaining a body part p. [Figure 6] FIG. 6 is a diagram (1) for explaining the processing of the integration unit. [Figure 7] FIG. 7 is a diagram (2) for explaining the processing of the integration unit. [Figure 8] FIG. 8 is a diagram illustrating an example of extended data generated based on attributes and weak point attributes. [Figure 9] FIG. 9 is a functional block diagram illustrating a configuration of a data generating device according to this embodiment. [Figure 10] FIG. 10 is a flowchart illustrating a processing procedure of the data generating device according to the present embodiment. [Figure 11] FIG. 11 is a flowchart showing the processing procedure of the integration process. [Figure 12] FIG. 12 is a diagram for explaining the effect of the data generating device according to the present embodiment. [Figure 13] FIG. 13 is a diagram illustrating an example of a hardware configuration of a computer that realizes the same functions as the data generating device of the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the data generating method, the data generating program, and the data generating device disclosed in the present application will be described in detail with reference to the accompanying drawings. However, the present invention is not limited to these embodiments. [Example]

[0019] Before describing this embodiment, an example of a human body model will be described. Fig. 1 is a diagram showing an example of a human body model. As shown in Fig. 1, the human body model is defined by 21 joints ar0 to ar20. In the skeleton information used in this embodiment, two-dimensional or three-dimensional coordinates are set for each of the joints ar0 to ar20 defined in the human body model.

[0020] The relationship between the joints ar0 to ar20 shown in FIG. 1 and the joint names is as shown in FIG. 2. FIG. 2 is a diagram showing examples of joint names. For example, the joint name of joint ar0 is "SPINE_BASE." The joint names of joints ar1 to ar20 are as shown in FIG. 2, and explanations thereof will be omitted.

[0021] Next, the processing of the data generating device according to this embodiment will be described. Fig. 3 is a diagram for explaining the processing of the data generating device according to this embodiment. The data generating device uses teacher data 30.

[0022] The training data 30 is existing data. For example, the training data includes image data and attributes of a person. The attributes include skeletal information, camera parameters, and appearance. The skeletal information is the joint coordinates described in FIG. 1, and indicates the coordinates of each joint of the person included in the image data. The coordinates of each joint are two-dimensional or three-dimensional coordinates. The camera parameters indicate the viewpoint position of the camera that captured the image data. The appearance is information about the appearance of the person included in the image data and the background of the person.

[0023] FIG. 4A is a diagram for explaining attributes. For example, attribute A1 of certain training data includes skeletal information a1-1, camera parameters a1-2, and appearances a1-3 and a1-4. In FIG. 4A, camera parameters a1-2 are shown as a conceptual illustration, but in reality, information on the viewpoint position of the camera that captured the image data is included. Appearance a1-3 sets the person's appearance (physique, uniform color, etc.). Appearance a1-4 sets the person's background information.

[0024] Returning to the explanation of Fig. 3, the data generating device inputs the teacher data 30 to the data extension unit 151.

[0025] A parameter θ1 is set in the data augmentation unit 151, and the data augmentation unit 151 augments the attributes of the training data 30 based on the parameter θ1. The data augmentation unit 151 outputs information on the augmented attributes to the image generation unit 152. The image generation unit 152, which will be described later, generates augmented data 40 based on the attributes augmented by the parameter θ1, and the augmented data 40 is input to the training target model 50, which calculates the inference error.

[0026] The data augmentation unit 151 trains the parameter θ1 based on gradient information of the inference error fed back from the training object model 50 in a direction that increases the inference error when the augmented data 40 is input to the training object model 50.

[0027] For example, the data augmentation unit 151 may change the joint angles and inter-joint bone lengths of the skeletal information included in the attributes based on the parameter θ1 in a direction that increases the inference error. The data augmentation unit 151 may also perform data augmentation by changing the camera parameters and appearance based on the parameter θ1 in a direction that increases the inference error.

[0028] When extending attributes, the data extension unit 151 ensures the likelihood of data. For example, the data extension unit 151 changes the joint angle within the movable range of the joint in the skeletal information. When changing the length of a bone, the data extension unit 151 changes the length of the bone within a predetermined range.

[0029] The image generation unit 152 generates extended data 40 based on the information of the attributes extended by the data extension unit 151 or the integration unit 153. The image generation unit 152 is a differentiable image generator such as NeRF. In the following description of the image generation unit 152, the "extended attributes" will be referred to simply as "attributes."

[0030] The image generation unit 152 generates a person model and a background model based on the skeletal information and appearance included in the attributes. The image generation unit 152 generates image data (augmented data 40) from a viewpoint based on the camera parameters of the attribute information for a model obtained by combining the person model and the background model.

[0031] 4B is a diagram showing an example of attributes and extended data. The attribute A1 includes skeletal information a1-1, camera parameters a1-2, and appearances a1-3 and a1-4. The image generation unit 152 generates extended data 40-1 based on the attribute A1.

[0032] The attribute A2 includes skeletal information a2-1, camera parameters a2-2, and appearances a2-3 and a2-4. The image generation unit 152 generates extended data 40-2 based on the attribute A2.

[0033] The attribute A3 includes skeletal information a3-1, camera parameters a3-2, and appearances a3-3 and a2-4. The image generation unit 152 generates extended data 40-3 based on the attribute A3.

[0034] The data generating device performs machine learning of a training object model 50 based on the augmented data 40 and skeletal information (skeletal information of the extended attributes) used when generating the augmented data 40. For example, the skeletal information used when generating the augmented data 40 is used as a correct label. The training object model 50 is a neural network (NN) or the like. A parameter θ2 is set in the training object model 50.

[0035] The data generating device inputs the augmented data 40 to the training object model 50 and obtains the inference result output from the training object model 50. The data generating device updates the parameter θ2 of the training object model 50 so as to reduce the inference error between the inference result and the correct label. The data generating device feeds back gradient information of the inference error to the data augmentation unit 151.

[0036] The data generation device associates, for each joint of the skeletal information, a pair of inference result information indicating the relationship between the inference result and the true value (correct label) with information on the attribute (extended attribute) used when generating the extended data 40, and outputs the pair to the integration unit 153.

[0037] The integrating unit 153 executes the following process based on the inference result information and the information on the extended attributes. In the following description of the integrating unit 153, the "extended attributes" will be simply referred to as "attributes." For example, the integrating unit 153 waits for the parameter θ2 to be updated multiple times for the training target model 50, and acquires multiple pairs of inference result information and information on the attributes.

[0038] The integration unit 153 identifies the inference error for each body part p based on the inference result information. FIG. 5 is a diagram for explaining body part p. In this embodiment, in the hierarchical structure of joints, a group of joints from the terminal joint to just before the branch point joint is defined as body part p. For example, body parts p may be "head", "armL", "armR", "legL", and "legR". In other words, p∈{head, armL, armR, legL, legR} is defined.

[0039] The body part "head" corresponds to joints ar3 and ar18. The body part "armL" corresponds to joints ar4, ar5, ar6, and ar19. The body part "armR" corresponds to joints ar7, ar8, ar9, and ar20. The body part "legL" corresponds to joints ar10, ar11, ar12, and ar13. The body part "legR" corresponds to joints ar14, ar15, ar16, and ar17.

[0040] The integration unit 153 identifies an inference error for each body part p for each piece of inference result information. That is, from one piece of inference result information, the inference errors for the body parts “head”, “armL”, “armR”, “legL”, and “legR” are identified.

[0041] For example, the inference error for the body part "head" is the MSE (Mean Squared Error) between the inference results and the true values ​​for joints ar3 and ar18. The inference error for the body part "armL" is the MSE between the inference results and the true values ​​for joints ar4, ar5, ar6, and ar19. The inference error for the body part "armR" is the MSE between the inference results and the true values ​​for joints ar7, ar8, ar9, and ar20. The inference error for the body part "legL" is the MSE between the inference results and the true values ​​for joints ar10, ar11, ar12, and ar13. The inference error for the body part "legR" is the MSE between the inference results and the true values ​​for joints ar14, ar15, ar16, and ar17.

[0042] The integration unit 153 compares the inference errors for each body part p calculated from each inference result information, and identifies the maximum inference error for each body part p and the attribute corresponding to the inference result information with the maximum inference error.

[0043] FIG. 6 is a diagram (1) for explaining the processing of the integration unit. In the example shown in FIG. 6, inference result information obtained by inputting augmented data generated based on attribute A1 into the training target model 50 is referred to as inference result information R1. The inference error for the body part "head" obtained based on the inference result R1 is referred to as inference error E1-1. The inference error for the body part "armL" obtained based on the inference result R1 is referred to as inference error E1-2. The inference error for the body part "armR" obtained based on the inference result R1 is referred to as inference error E1-3. The inference error for the body part "legR" obtained based on the inference result R1 is referred to as inference error E1-4. The inference error for the body part "legL" obtained based on the inference result R1 is referred to as inference error E1-5.

[0044] The inference result information obtained by inputting the augmented data generated based on attribute A2 into the training target model 50 is defined as inference result information R2. The inference error for body part "head" obtained based on inference result R2 is defined as inference error E2-1. The inference error for body part "armL" obtained based on inference result R2 is defined as inference error E2-2. The inference error for body part "armR" obtained based on inference result R2 is defined as inference error E2-3. The inference error for body part "legR" obtained based on inference result R2 is defined as inference error E2-4. The inference error for body part "legL" obtained based on inference result R2 is defined as inference error E2-5.

[0045] Attribute A n The inference result information obtained by inputting the augmented data generated based on the n Let n be a natural number greater than or equal to 3. The inference result R n The inference error for the body part "head" obtained based on the inference result R is assumed to be the inference error En-1. n The inference error for the body part "armL" obtained based on the inference result R is assumed to be the inference error En-2. n The inference error for the body part "armR" obtained based on the inference result R is assumed to be the inference error En-3. n The inference error for the body part "legR" obtained based on the inference result R is assumed to be inference error En-4. n The inference error for the body part “legL” obtained based on the above is defined as inference error En-5.

[0046] The integrating unit 153 compares the inference errors E1-1 to En-1 of the body part "head" and identifies the inference error with the maximum value. In this embodiment, as an example, of the inference errors E1-1 to En-1, the inference error of E1-1 is set to be the maximum value. The attribute corresponding to the inference error E1-1 is attribute A1. In this case, the integrating unit 153 identifies the weak point attribute of the body part "head" as weak point attribute A1.

[0047] The integrating unit 153 compares the inference errors E1-2 to En-2 of the body part "armL" and identifies the inference error with the maximum value. In this embodiment, as an example, of the inference errors E1-2 to En-2, the inference error E2-2 is set to be the maximum value. The attribute corresponding to the inference error E2-2 is attribute A2. In this case, the integrating unit 153 identifies the weak point attribute of the body part "armL" as weak point attribute A2.

[0048] The integration unit 153 compares the inference errors E1-3 to En-3 of the body part "armR" and identifies the inference error with the maximum value. In this embodiment, as an example, of the inference errors E1-3 to En-3, the inference error of the inference error E3-3 is set to be the maximum value. The attribute corresponding to the inference error E3-3 is the attribute A3. In this case, the integration unit 153 identifies the weak point attribute of the body part "armR" as the weak point attribute A3. In FIG. 6, the attribute A3 is not shown.

[0049] The integration unit 153 compares the inference errors E1-4 to En-4 for the body part "legL" and identifies the inference error with the maximum value. In this embodiment, as an example, of the inference errors E1-4 to En-4, the inference error E4-4 is identified as the maximum value. The attribute corresponding to the inference error E4-4 is attribute A4. In this case, the integration unit 153 identifies the weak point attribute of the body part "legL" as weak point attribute A4. Attribute A4 is not shown in FIG. 6.

[0050] The integration unit 153 compares the inference errors E1-5 to En-5 of the body part "legR" and identifies the inference error with the maximum value. In this embodiment, as an example, of the inference errors E1-5 to En-5, the inference error of E5-5 is set to be the maximum value. The attribute corresponding to the inference error E5-5 is attribute A5. In this case, the integration unit 153 identifies the weak point attribute of the body part "legR" as weak point attribute A5. Attribute A5 is not shown in FIG. 6.

[0051] The integration unit 153 executes the process shown in FIG. 6 to identify the weak point attribute of each body part p. In the following description, as an example, the weak point attribute of the body part "head" is defined as weak point attribute A1. The weak point attribute of the body part "armL" is defined as weak point attribute A2. The weak point attribute of the body part "armR" is defined as weak point attribute A3. The weak point attribute of the body part "legL" is defined as weak point attribute A4. The weak point attribute of the body part "legR" is defined as weak point attribute A5.

[0052] FIG. 7 is a diagram (2) for explaining the processing of the integrating unit. The integrating unit 153 generates a weak spot attribute A'1 by integrating the weak spot attributes A1 to A5 using the weak spot attribute A1 as a base. The weak spot attribute A'1 includes skeletal information, camera parameters, and appearance, just like the attributes. The skeletal information of the weak spot attribute A'1 is skeletal information that combines the joint coordinates of the body part "head" of the weak spot attribute A1, the joint coordinates of the body part "armL" of the weak spot attribute A2, the joint coordinates of the body part "armR" of the weak spot attribute A3, the joint coordinates of the body part "legL" of the weak spot attribute A4, and the joint coordinates of the body part "legR" of the weak spot attribute A5. The camera parameters and appearance of the weak spot attribute A'1 are the same as those of the base weak spot attribute A1.

[0053] The integration unit 153 generates a weak spot attribute A'2 by integrating the weak spot attributes A1 to A5 using the weak spot attribute A2 as a base. The weak spot attribute A'2 includes skeletal information, camera parameters, and appearance, just like the attributes. The skeletal information of the weak spot attribute A'2 is skeletal information that combines the joint coordinates of the body part "head" of the weak spot attribute A1, the joint coordinates of the body part "armL" of the weak spot attribute A2, the joint coordinates of the body part "armR" of the weak spot attribute A3, the joint coordinates of the body part "legL" of the weak spot attribute A4, and the joint coordinates of the body part "legR" of the weak spot attribute A5. The camera parameters and appearance of the weak spot attribute A'2 reuse the camera parameters and appearance of the base weak spot attribute A2.

[0054] The integration unit 153 generates a weak spot attribute A'3 by integrating the weak spot attributes A1 to A5 using the weak spot attribute A3 as a base. The weak spot attribute A'3 includes skeletal information, camera parameters, and appearance, just like the attributes. The skeletal information of the weak spot attribute A'3 is skeletal information that combines the joint coordinates of the body part "head" of the weak spot attribute A1, the joint coordinates of the body part "armL" of the weak spot attribute A2, the joint coordinates of the body part "armR" of the weak spot attribute A3, the joint coordinates of the body part "legL" of the weak spot attribute A4, and the joint coordinates of the body part "legR" of the weak spot attribute A5. The camera parameters and appearance of the weak spot attribute A'3 reuse the camera parameters and appearance of the base weak spot attribute A3.

[0055] The integration unit 153 generates a weak spot attribute A'4 by integrating the weak spot attributes A1 to A5 using the weak spot attribute A4 as a base. The weak spot attribute A'4 includes skeletal information, camera parameters, and appearance, just like the attributes. The skeletal information of the weak spot attribute A'4 is skeletal information that combines the joint coordinates of the body part "head" of the weak spot attribute A1, the joint coordinates of the body part "armL" of the weak spot attribute A2, the joint coordinates of the body part "armR" of the weak spot attribute A3, the joint coordinates of the body part "legL" of the weak spot attribute A4, and the joint coordinates of the body part "legR" of the weak spot attribute A5. The camera parameters and appearance of the weak spot attribute A'4 reuse the camera parameters and appearance of the base weak spot attribute A4.

[0056] The integration unit 153 generates a weak spot attribute A'5 by integrating the weak spot attributes A1 to A5 using the weak spot attribute A5 as a base. The weak spot attribute A'5 includes skeletal information, camera parameters, and appearance, just like the attributes. The skeletal information of the weak spot attribute A'5 is skeletal information that combines the joint coordinates of the body part "head" of the weak spot attribute A1, the joint coordinates of the body part "armL" of the weak spot attribute A2, the joint coordinates of the body part "armR" of the weak spot attribute A3, the joint coordinates of the body part "legL" of the weak spot attribute A4, and the joint coordinates of the body part "legR" of the weak spot attribute A5. The camera parameters and appearance of the weak spot attribute A'5 reuse the camera parameters and appearance of the base weak spot attribute A5.

[0057] The integration unit 153 generates weak point attributes A'1 to A'5 by executing the process described in Fig. 7. The integration unit 153 inputs the weak point attributes A'1 to A'5 to the image generation unit 152, thereby generating extended data corresponding to the weak point attributes A'1 to A'5, respectively.

[0058] FIG. 8 is a diagram illustrating an example of augmented data generated based on attributes and weak attribute values. For example, augmented data Im10 is image data obtained by inputting attribute A2 to image generation unit 152. When augmented data Im10 is input to training target model 50, the inference error for body part "armL" is the largest compared to the other inference errors. Augmented data Im11 is image data obtained by inputting weak attribute A'2 to image generation unit 152. When augmented data Im11 is input to training target model 50, the inference errors for each of body parts "head," "armL," "armR," "legL," and "legR" are the largest. In other words, new training data that deviates from the distribution of existing training data can be generated.

[0059] The data generating device generates a set of extended data obtained by inputting the weak point attributes into the image generating unit 152 and skeletal information included in the weak point attributes as training data, and uses this data for machine learning of the training target model 50.

[0060] As described above, the data generating device according to this embodiment identifies an inference error for each body part p based on the inference result information obtained by inputting the extended data 40 into the training target model 50. The data generating device compares the inference errors for each body part p calculated from each piece of inference result information, and identifies the attribute (weak attribute) of the inference result information for which the inference error is maximum. The data generating device integrates the weak attribute for each body part p, and generates extended data based on the integrated weak attribute. This makes it possible to generate new training data that is different from the distribution of existing training data.

[0061] In the above description, the data generation device identifies and integrates weak attributes based on inference result information obtained as a result of inputting the augmented data 40 into the training object model 50, but the present invention is not limited to this. For example, the teacher data 30 may be directly input into the training object model 50, and weak attributes may be identified and integrated based on inference result information obtained as a result of inputting the teacher data 30 into the training object model 50.

[0062] Next, an example of the configuration of a data generating device that executes the above-mentioned processing will be described. Fig. 9 is a functional block diagram showing the configuration of a data generating device according to this embodiment. As shown in Fig. 9, this data generating device 100 has a communication unit 110, an input unit 120, a display unit 130, a storage unit 140, and a control unit 150.

[0063] The communication unit 110 executes data communication with an external device or the like via a network. The communication unit 110 is a NIC (Network Interface Card) or the like. The control unit 150, which will be described later, exchanges data with the external device via the communication unit 110.

[0064] The input unit 120 is an input device that inputs various types of information to the control unit 150 of the data generating device 100. For example, the input unit 120 corresponds to a keyboard, a mouse, a touch panel, or the like.

[0065] The display unit 130 is a display device that displays information output from the control unit 150.

[0066] The storage unit 140 includes a training target model 50 and a teacher dataset 141. The storage unit 140 is a storage device such as a memory.

[0067] The training target model 50 is a machine learning model that receives image data (augmented data) as input and outputs an inference result of skeletal information. The training target model 50 is a neural network (NN) or the like.

[0068] The training data set 141 includes a plurality of training data. The training data includes image data of a person and attributes. The attributes include skeletal information, camera parameters, and appearance.

[0069] The control unit 150 includes a data extension unit 151, an image generation unit 152, an integration unit 153, and a learning unit 154. The control unit 150 is a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0070] A parameter θ1 is set in the data augmentation unit 151, and the data augmentation unit 151 augments the attributes of the training data based on this parameter θ1. The data augmentation unit 151 outputs information on the augmented attributes to the image generation unit 152. Based on gradient information of the inference error fed back from the training target model 50, the data augmentation unit 151 trains the parameter θ1 in a direction that increases the inference error when the augmented data is input to the training target model 50. Other explanations regarding the data augmentation unit 151 are the same as those regarding the data augmentation unit 151 described in FIG. 3.

[0071] The image generation unit 152 generates extended data based on the information on the attributes extended by the data extension unit 151. The image generation unit 152 also generates extended data based on the information on the weak point attributes generated by the integration unit 153. The image generation unit 152 may add a pair of the information on the weak point attributes and the extended data to the teacher dataset 141 as new teacher data.

[0072] The integrating unit 153 identifies an inference error for each body part p for each piece of inference result information. The integrating unit 153 compares the inference errors for each body part p calculated from each piece of inference result information, and identifies the maximum value of the inference error for each body part p and the attribute (weak point attribute) corresponding to the inference result information for which the inference error is maximum. The integrating unit 153 generates a weak point attribute by integrating the weak point attributes for each body part p. Other explanations regarding the integrating unit 153 are the same as those regarding the integrating unit 153 described in Figures 3 and 5 to 7.

[0073] The learning unit 154 performs machine learning of the training object model 50 based on the teacher dataset 141. For example, when image data is input to the training object model 50, the learning unit 154 updates the parameter θ2 of the training object model 50 based on the backpropagation method so as to reduce the error between the inference result output from the training object model 50 and the correct label.

[0074] The pair of image data and correct label that the learning unit 154 inputs to the training target model 50 will be a pair of first image data and first correct label, or a pair of second image data and second correct label, which will be described next.

[0075] The first correct label is skeleton information when the attributes of the training data are extended by the data extension unit 151. The first image data is extended data generated by the image generation unit 152 based on the attributes of the training data extended by the data extension unit 151.

[0076] The second correct label is skeletal information of the weak point attributes integrated by the integration unit 153. The second image data is extended data generated by the image generation unit 152 based on the weak point attributes.

[0077] The learning unit 154 feeds back the gradient information of the inference error to the data expansion unit 151. The learning unit 154 outputs to the integration unit 153 a set of inference result information indicating the relationship between the inference result and the true value (correct label) and attribute information for each joint of the skeletal information.

[0078] Next, an example of a processing procedure of the data generating device 100 according to this embodiment will be described. Fig. 10 is a flowchart showing the processing procedure of the data generating device according to this embodiment. As shown in Fig. 10, the data extension unit 151 of the data generating device 100 acquires teacher data from the teacher dataset 141 (step S101). The data extension unit 151 extends the attributes of the teacher data based on the parameter θ1 in a direction that increases the inference error of the training target model 50 (step S102).

[0079] The image generation unit 152 of the data generation device 100 generates augmented data based on the augmented attributes (step S103). The learning unit 154 of the data generation device 100 performs machine learning of the training target model 50 based on the augmented data and the correct label (step S104).

[0080] The integration unit 153 of the data generating device 100 executes the integration process (step S105).

[0081] The data extension unit 151 of the data generating device 100 receives feedback of the gradient information of the inference result and updates the parameter θ1 (step S106).

[0082] If the data generating device 100 continues the processing (step S107, Yes), the data generating device 100 proceeds to step S101. On the other hand, if the data generating device 100 does not continue the processing (step S107, No), the data generating device 100 ends the processing.

[0083] Next, an example of the integration process shown in step S105 of Fig. 10 will be described. Fig. 11 is a flowchart showing the processing procedure of the integration process. As shown in Fig. 11, the integration unit 153 of the data generating device 100 identifies an inference error for each body part p for each piece of inference result information (step S201).

[0084] The integrating unit 153 compares the inference errors for each body part p and identifies a weak point attribute for each body part p (step S202). The integrating unit 153 integrates the weak point attributes for each body part p (step S203).

[0085] The image generating unit 152 of the data generating device 100 generates extended data based on the integrated weak point attributes (step S204).

[0086] The learning unit 154 of the data generating device 100 performs machine learning of the training target model 50 based on the extended data and the correct label (skeleton information of the weak attribute) (step S205).

[0087] Next, the effects of the data generating device 100 according to this embodiment will be described. The data generating device 100 identifies an inference error for each body part p based on inference result information obtained by inputting augmented data into a training target model. The data generating device 100 compares the inference errors for each body part p calculated from each piece of inference result information, and identifies the attribute (weak attribute) of the inference result information that maximizes the inference error. The data generating device 100 integrates the weak attribute for each body part p, and generates augmented data based on the integrated weak attribute. This makes it possible to generate new training data that is different from the distribution of existing training data.

[0088] FIG. 12 is a diagram illustrating the effect of the data generation device according to the present embodiment. Image data Im20 in FIG. 12 is augmented data generated by the image generation unit 152 based on the integrated weak point attributes. When the learning unit 154 inputs the image data Im20 to the training target model 50, an inference result 60 is output. In the inference result 60, inference between joints ar9 and ar20 and joints ar5, ar6, and ar7 fails. In other words, by performing machine learning using new image data (training data) that is image data Im20 and that is new training data and deviates from the distribution of existing training data, the inference accuracy of the training target model 50 can be improved.

[0089] For example, inference results 60a and 60b are obtained by training a training target model 50 using new image data (teaching data) that deviates from the distribution of existing teaching data through machine learning. In inference result 60a, inference fails at joint ar19, but the inference accuracy improves at joints ar5 and ar6. In inference result 60b, the inference accuracy improves at joints ar9 and ar20, joints ar5, ar6, ar19, and joint ar11.

[0090] The processing of the data generating device 100 according to this embodiment is not limited to the above. For example, the data generating device 100 can execute a body detection task or a body region extraction (segmentation) task from image data. The data generating device 100 executes the body detection task or body region extraction, and by referencing body parts with large inference errors, can identify weak point attributes for each body part and apply the above-described processing.

[0091] The processing of the data generating device 100 can be applied to tasks that target more general articulated bodies, such as quadrupedal animals, rather than human bodies. The processing of the data generating device 100 can be applied to both 2D and 3D body skeleton estimation tasks. The processing of the data generating device 100 can also be applied to 2D-to-3D skeleton estimation tasks that do not require image data.

[0092] The data generating device 100 may use a mechanism for evaluating the likelihood of skeletal information obtained by integrating weak attribute values ​​to reject or correct data that has been combined with implausible skeletal information. To evaluate the likelihood of skeletal information, the data generating device 100 uses a VPoser that evaluates the distance in the latent space of the pose generator, a penetration loss that evaluates the penetration of the body model, and a hyper-bending loss that evaluates the backward bending of elbows and knees. For example, a technique related to VPoser is described in the literature "G. Pavlakos et al., "Expressive Body Capture: 3D Hands, Face, and Body from a Single Image," CVPR 201."

[0093] The data generating device 100 may immediately discard implausible skeleton information. Alternatively, the data generating device 100 may project the implausible skeleton information onto a manifold of plausible skeletons (e.g., the latent space of a VPoser) and correct it to plausible skeleton information. The data generating device 100 may select only some parts within the range of plausible skeletons by combinatorial optimization such as a greedy algorithm so as to maximize the total inference error for all parts.

[0094] When the image generating unit 152 acquires information on the weak point attribute (integrated weak point attribute) generated by the integrating unit 153, the image generating unit 152 determines whether the skeletal information of the integrated weak point attribute is plausible skeletal information. For example, the image generating unit 152 holds information on the working area of ​​each joint, and determines that the skeletal information is plausible when each joint of the skeletal information is within the range of the working area.

[0095] The data generating device 100 may identify and use Nw (>1) weak point attributes for each body part in descending order of inference error. The data generating device 100 may generate new weak point attributes for all combinations Nw^Np (Np is the number of body parts). The data generating device 100 may select the optimal combination of weak point attributes from all combinations Nw^Np by combinatorial optimization using the above-mentioned plausibility of skeletal information as a constraint.

[0096] Next, a description will be given of an example of a hardware configuration of a computer that realizes the same functions as the above-described data generating device 100. Fig. 13 is a diagram showing an example of a hardware configuration of a computer that realizes the same functions as the data generating device of the embodiment.

[0097] 13, computer 200 has CPU 201 that executes various types of arithmetic processing, input device 202 that accepts data input from a user, and display 203. Computer 200 also has communication device 204 that exchanges data with camera 15, external devices, etc. via a wired or wireless network, and interface device 205. Computer 200 also has RAM 206 that temporarily stores various types of information, and hard disk device 207. Each of devices 201 to 207 is connected to bus 208.

[0098] The hard disk drive 207 stores a data extension program 207a, an image generation program 207b, an integration program 207c, and a learning program 207d. The CPU 201 reads out each of the programs 207a to 207d and expands them in the RAM 206.

[0099] The data augmentation program 207a functions as the data augmentation process 206a. The image generation program 207b functions as the image generation process 206b. The integration program 207c functions as the integration process 206c. The learning program 207d functions as the learning process 206d.

[0100] The processing of the data extension process 206a corresponds to the processing of the data extension unit 151. The processing of the image generation process 206b corresponds to the processing of the image generation unit 152. The processing of the integration process 206c corresponds to the processing of the integration unit 153. The processing of the learning process 206d corresponds to the processing of the learning unit 154.

[0101] It should be noted that each of the programs 207a to 207d does not necessarily have to be stored in the hard disk drive 207 from the beginning. For example, each of the programs may be stored in a "portable physical medium" such as a flexible disk (FD), CD-ROM, DVD, magneto-optical disk, or IC card that is inserted into the computer 200. Then, the computer 200 may read and execute each of the programs 207a to 207d. [Explanation of symbols]

[0102] 50 training models 100 Data generation device 110 Communications Department 120 Input section 130 Display section 140 Storage section 141 training dataset 150 control section 151 Data Extension 152 Image generation unit 153 Integration Department 154 Learning Department

Claims

1. acquiring an inference result of skeletal information for each training data when a plurality of training data are input into a machine learning model, the inference result including an error for each part of the skeleton; Based on the inference result, first teacher data having a larger error in a first portion than the errors in the first portions of other teacher data is identified from the plurality of teacher data; Based on the inference result, second teacher data having a larger error in a second portion than the errors in the second portions of other teacher data is identified from the plurality of teacher data; Third teacher data is generated by replacing information about the second region included in the first teacher data with information about the second region included in the second teacher data. A data generation program that causes a computer to execute a process.

2. The data generation program described in claim 1 further causes the computer to execute a process of generating fourth teacher data by replacing information regarding the first part contained in the second teacher data with information regarding the first part contained in the first teacher data.

3. The data generation program according to claim 1, further comprising causing a computer to execute a process of performing machine learning of the machine learning model based on the third teacher data.

4. The data generation program of claim 3 further causes a computer to execute a process for determining whether a skeleton consisting of the first part and the second part included in the third training data is plausible, and the process for executing the machine learning executes machine learning of the machine learning model based on the third training data when the skeleton consisting of the first part and the second part is plausible.

5. acquiring an inference result of skeletal information for each piece of training data when multiple pieces of training data are input into a machine learning model, the inference result including errors for each part of the skeleton; Based on the inference result, first teacher data having a larger error in a first portion than the errors in the first portions of other teacher data is identified from the plurality of teacher data; Based on the inference result, second teacher data having a larger error in a second portion than the errors in the second portions of other teacher data is identified from the plurality of teacher data; Third teacher data is generated by replacing information about the second region included in the first teacher data with information about the second region included in the second teacher data. A data generation method characterized in that processing is executed by a computer.

6. acquiring an inference result of skeletal information for each training data when a plurality of training data are input into a machine learning model, the inference result including an error for each part of the skeleton; Based on the inference result, first teacher data having a larger error in a first portion than the errors in the first portions of other teacher data is identified from the plurality of teacher data; Based on the inference result, second teacher data having a larger error in a second portion than the errors in the second portions of other teacher data is identified from the plurality of teacher data; Third teacher data is generated by replacing information about the second region included in the first teacher data with information about the second region included in the second teacher data. A data generating device having a control unit that executes processing.

Images