Information processing method, program, and information processing apparatus

Abstract

To provide an information processing method etc. capable of suitably supporting structuring of a machine learning model which estimates three-dimensional attitude information from a two-dimensional image.SOLUTION: An information processing method causes a computer to execute processing to acquire three-dimensional attitude information on an object and input the acquired three-dimensional attitude information to an image generation model 30 to generate a two-dimensional image so as to generate training data including the generated two-dimensional image and the acquired three-dimensional attitude information. It is preferred to generate the model which is to output the three-dimensional attitude information based upon the training data when inputting the two-dimensional image.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present invention relates to an information processing method, a program, and an information processing apparatus.

Background Art

[0002] There is a technology for estimating three-dimensional pose information of a subject from a two-dimensional image of the subject. For example, in Patent Document 1, two-dimensional joint point coordinate data capable of specifying the two-dimensional coordinates of each joint point of a person is acquired from an image, and the acquired two-dimensional joint point coordinate data is applied to a machine learning model that has learned the relationship between the two-dimensional coordinates and the three-dimensional coordinates of the joint points of a person, and a joint point detection device that detects the three-dimensional coordinates of each joint point of a person is disclosed.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0004] In one aspect, an object is to provide an information processing method or the like that can suitably support the construction of a machine learning model for estimating three-dimensional pose information from a two-dimensional image.

Means for Solving the Problems

[0005] In one aspect, the information processing method includes: Performing a specific action acquiring three-dimensional pose information of an object, and inputting the acquired three-dimensional pose information The prompt instructs the generation of a two-dimensional image including an object performing the aforementioned specific action, and the prompt specifies the name of the action being performed by the object. into an image generation model to generate a two-dimensional image, and the computer executes a process of creating training data from the generated two-dimensional image against and the acquired three-dimensional pose information. Annotated as the correct value.

Effects of the Invention

[0006] In one respect, it can be effectively used to support the construction of machine learning models that estimate 3D pose information from 2D images. [Brief explanation of the drawing]

[0007] [Figure 1] This is an explanatory diagram showing an example configuration of a posture estimation system. [Figure 2] This is a block diagram showing an example server configuration. [Figure 3] This is a diagram illustrating the overview of Embodiment 1. [Figure 4] This flowchart shows an example of the processing steps performed by the server. [Figure 5] This flowchart shows an example of a processing procedure executed by the server according to Embodiment 2. [Modes for carrying out the invention]

[0008] The present invention will be described in detail below with reference to the drawings illustrating its embodiments. (Embodiment 1) Figure 1 is an explanatory diagram showing an example configuration of a posture estimation system. In this embodiment, we describe a posture estimation system that generates a posture estimation model 40 (see Figure 3) that estimates 3D posture information from 2D images by training 2D images generated using an image generation model 30 as training data. The posture estimation system includes an information processing device 1, a terminal 2, and a generation server 3. Each device is connected via a network N such as the Internet.

[0009] The information processing device 1 is an information processing device capable of various information processing and information transmission and reception, such as a server computer or a personal computer. In this embodiment, the information processing device 1 is assumed to be a server computer, and for simplicity, it will be referred to as Server 1 below. Server 1 generates a posture estimation model 40 that outputs 3D posture information when a 2D image is input, by learning training data including a 2D image of an object and 3D posture information of the object corresponding to the 2D image using machine learning. Server 1 estimates the 3D posture information by inputting the 2D image of the object to be analyzed into the posture estimation model 40.

[0010] The "object" depicted in the 2D image is, for example, a person. In this embodiment, the primary purpose is to analyze the posture of a person performing a sport. By inputting a 2D image of the person performing the sport into the posture estimation model 40, the person's 3D posture information is estimated.

[0011] Furthermore, the "object" is not limited to a person; it may also be an animal other than a human, or an object other than an animal (for example, a robot).

[0012] "3D posture information" is data that includes at least the 3D position information of the subject's joints. In this embodiment, the "3D posture information" is an SMPL model (Skinned Multi-Person Linear Model) that includes not only joint position information but also body shape information and joint angle information. The SMPL model is a model that can identify the surface shape of a person's body, and is a model in which a 3D human model represented by 6890 vertices can be controlled by a 72-dimensional posture parameter θ and a 10-dimensional body shape parameter β. By estimating the parameters related to the SMPL model as 3D posture information, it is possible to analyze not only joint positions but also body shape and joint rotation angles.

[0013] Note that the SMPL model is just one example of 3D pose information; 3D pose information may be in other data formats as well.

[0014] As described above, Server 1 generates a pose estimation model 40 by learning training data that includes 2D images and 3D pose information. However, in order to generate (learn) the pose estimation model 40, it is necessary to prepare training data in which the correct 3D pose information is annotated on a large number of 2D images. In particular, in this embodiment, the SMPL model is used as the 3D pose information, but currently there is not enough training data available to learn the SMPL model (data pairing 2D images with parameters related to the SMPL model), so it is necessary to manually annotate the parameters related to the SMPL model on the 2D images. However, it is difficult to perform this annotation accurately by hand.

[0015] Therefore, in this embodiment, instead of annotating 3D pose information to 2D images, training data is prepared by generating 2D images based on 3D pose information set to plausible values. Specifically, 2D images are generated by inputting 3D pose information into the image generation model 30, and the generated 2D images and the 3D pose information input into the image generation model 30 are used as training data to learn and generate a pose estimation model 40.

[0016] Terminal 2 is a client terminal connected to Server 1, such as a personal computer. During training, Server 1 receives input such as 3D posture information from Terminal 2, generates a 2D image from this 3D posture information to use as training data, and generates a posture estimation model 40. During estimation, Server 1 obtains a 2D image of the target for analysis from Terminal 2 and inputs it into the posture estimation model 40 to estimate 3D posture information.

[0017] The generation server 3 is a server computer that generates images upon receiving requests from clients. For example, server 1 requests the generation server 3 to generate an image through a predetermined API (Application Programmable Interface). The generation server 3 generates an image by inputting data including a prompt instructing the generation of the image into the image generation model 30.

[0018] The image generation model 30 is a machine learning model that has learned a large number of images and is, for example, a diffusion model such as Stable Diffusion. Note that the image generation model 30 is not limited to a diffusion model and may be any model that can generate a 2D image from 3D pose information or the like, as will be described later.

[0019] Figure 2 is a block diagram showing a configuration example of server 1. Server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary storage unit 14. The control unit 11 has arithmetic processing devices such as one or more CPUs (Central Processing Unit), MPUs (Micro-Processing Unit), GPUs (Graphics Processing Unit), etc., and performs various information processing, control processing, etc. by reading and executing the program P stored in the auxiliary storage unit 14. The main memory unit 12 is a temporary storage area such as SRAM (Static Random Access Memory), DRAM (Dynamic Random Access Memory), etc., and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing and transmits and receives information to and from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory, a hard disk, etc., and stores the program P (program product) and other data necessary for the control unit 11 to execute processing.

[0020] The auxiliary storage unit 14 may be an external storage device connected to the server 1. Furthermore, the server 1 may be a multi-computer system consisting of multiple computers, or it may be a virtual machine virtually constructed by software.

[0021] Furthermore, in this embodiment, the server 1 is not limited to the above configuration and may include, for example, an input unit for receiving operation input, a display unit for displaying images, etc. The server 1 may also include a read unit for reading portable storage media 1a such as a CD (Compact Disk)-ROM or DVD (Digital Versatile Disc)-ROM, and may read and execute a program P from the portable storage media 1a.

[0022] Figure 3 is a diagram illustrating the overview of Embodiment 1. In Figure 3, a two-dimensional image is generated by inputting data including three-dimensional pose information into the image generation model 30, and a pose estimation model 40 is generated by training the generated two-dimensional image and the three-dimensional pose information input into the image generation model 30 as training data. An overview of this embodiment will be explained based on Figure 3.

[0023] As described above, Server 1 uses the image generation model 30 to create training data for generating the pose estimation model 40. Specifically, Server 1 generates a 2D image by inputting the subject's 3D pose information, a reference image, and prompts into the image generation model 30.

[0024] As described above, the 3D posture information consists of 82-dimensional (72-dimensional + 10-dimensional) parameters related to the SMPL model, and includes data such as joint position information, body shape information, and joint angle information. Server 1 acquires 3D posture information set to appropriate values ​​from terminal 2 and inputs it into the image generation model 30. There are no particular restrictions on the method of acquiring (creating) the 3D posture information, but for example, Server 1 may acquire 3D posture information by manually modifying the output of an existing 3D posture estimation model, or it may acquire 3D posture information created based on data measured by attaching motion sensors to the subject (athlete).

[0025] The reference image is a two-dimensional image that the image generation model 30 should refer to in order to generate a two-dimensional image. In this embodiment, the reference image is an image of a person performing a sport, and the image shows the person performing a specific action related to that sport. In addition to the person, the reference image includes the background and the equipment the person is using. For example, if the sport is figure skating, the reference image would be an image that shows a skating rink in the background and the person wearing sports equipment such as skates. By providing the image generation model 30 with a reference image, the image generation model 30 can generate a two-dimensional image with similar features to the reference image. In particular, in this embodiment, by providing a reference image that includes a background and equipment appropriate to the sport the person is playing, it is possible to generate a two-dimensional image of a person with a similar background and wearing similar equipment.

[0026] In this embodiment, the accuracy of the 2D image is improved by providing a reference image that includes the equipment (sports equipment). However, in addition to the reference image that includes the equipment, data indicating the 3D shape of the equipment may also be input to the image generation model 30. For example, server 1 inputs data indicating the 3D shape of the equipment as part of the 3D posture information (SMPL model) to the image generation model 30. Even in this case, a suitable 2D image of a person wearing the equipment can be generated.

[0027] A prompt is an instruction that tells the image generation model 30 to generate an image. For example, server 1 creates a prompt that specifies a pre-configured sport name and the quality of the 2D image (such as "High Quality"), and inputs it to the image generation model 30. The user may also set (change) the prompt each time a 2D image is generated.

[0028] Server 1 generates a 2D image by inputting 3D pose information, a reference image, and a prompt into the image generation model 30. When inputting 3D pose information into the image generation model 30, Server 1 inputs data (2D image) that visualizes the 3D pose information on a 2D plane into the image generation model 30, but the 3D pose information may also be used directly as input to the image generation model 30. Server 1 learns the pairs of the 2D image generated by the image generation model 30 and the 3D pose information input into the image generation model 30 as training data, and generates a pose estimation model 40 that outputs 3D pose information when a 2D image is input.

[0029] The pose estimation model 40 is a neural network such as a CNN (Convolutional Neural Network), but it may also be a machine learning model other than a neural network. Server 1 estimates 3D pose information by inputting a 2D training image into the pose estimation model 40, compares the estimated 3D pose information with the ground truth 3D pose information, and optimizes parameters such as the weights between neurons so that the two approximate each other. In this way, Server 1 generates the pose estimation model 40. Server 1 uses this pose estimation model 40 to estimate 3D pose information.

[0030] As described above, according to this embodiment, training data is created by generating a 2D image from 3D posture information using the image generation model 30. This makes it possible to suitably generate a posture estimation model 40 that estimates 3D posture information from a 2D image.

[0031] Figure 4 is a flowchart showing an example of the processing procedure performed by Server 1. Based on Figure 4, the processing details for generating (training) the pose estimation model 40 will be explained. The control unit 11 of server 1 acquires 3D pose information of the object from terminal 2 (step S11). The 3D pose information includes parameters related to the SMPL model, such as the position information of human joints, body shape information, and joint angle information.

[0032] The control unit 11 acquires a reference image that the image generation model 30 should refer to for generating a two-dimensional image (step S12). The reference image is, for example, a two-dimensional image of a person playing sports. The reference image is an image of a subject performing a specific action and includes, in addition to the subject, the background and the equipment (sports equipment) the subject is using.

[0033] The control unit 11 creates a prompt to instruct the image generation model 30 to generate a 2D image (step S13). The control unit 11 then inputs the 3D pose information acquired in step S11, the reference information acquired in step S12, and the prompt created in step S13 to the image generation model 30 to generate a 2D image for training (step S14).

[0034] The control unit 11 performs machine learning to learn training data that includes the 2D image generated in step S14 and the 3D pose information input to the image generation model 30, and generates a pose estimation model 40 that outputs 3D pose information when a 2D image is input (step S15). The control unit 11 then completes the series of processes.

[0035] Based on the above, this embodiment 1 can suitably support the construction of a posture estimation model 40 that estimates three-dimensional posture information from a two-dimensional image.

[0036] (Embodiment 2) This embodiment describes a method for improving the accuracy of 2D images by performing additional image learning (fine tuning) on ​​the image generation model 30. Note that components that overlap with those in Embodiment 1 are denoted by the same reference numerals and their descriptions are omitted.

[0037] Embodiment 1 described a method for improving the accuracy of a 2D image by inputting a reference image along with 3D pose information into the image generation model 30. On the other hand, it is also possible to improve the accuracy of a 2D image without providing a reference image by training the image generation model 30 with a desired image in advance.

[0038] For example, Server 1 acquires a small number of additional training images (e.g., a few to several dozen) from Terminal 2 and trains the image generation model 30 with these images to tune the model to be suitable for generating specific two-dimensional images (in this embodiment, images of a person performing a specific sport). The additional training images are two-dimensional images of a person performing a sport, similar to the "reference images" in Embodiment 1, and depict the subject performing a specific action. The additional training images include the subject, the background, and the equipment (sports equipment) the subject is using.

[0039] Server 1 performs additional training on the image generation model 30 based on additional training images. In this embodiment, fine tuning is performed using the LoRA (Low-Rank Adaptation) method. LoRA is a method that modifies parameters by introducing a low-rank matrix instead of directly changing the parameters (weights) of the original model. By adopting LoRA, the computational amount required for training can be significantly reduced.

[0040] Server 1 generates a 2D image by inputting 3D pose information and prompts to the newly trained image generation model 30. Similar to Embodiment 1, Server 1 generates a pose estimation model 40 based on training data that includes the generated 2D image and the 3D pose information input to the image generation model 30.

[0041] Figure 5 is a flowchart showing an example of the processing procedure executed by Server 1 according to Embodiment 2. Based on Figure 5, the contents of the pose estimation model 40 generation process according to this embodiment will be explained. The control unit 11 of server 1 acquires additional training images for training the image generation model 30 from terminal 2 (step S201). The additional training images are, for example, two-dimensional images of a person playing sports, and the images show the subject performing a specific action. For example, the additional training images include the subject, the background, and the equipment (sports equipment) the subject is using. Based on the acquired additional training images, the control unit 11 performs additional training (fine tuning) of the image generation model 30 (step S202). In this embodiment, fine tuning is performed using the LoRA method.

[0042] The control unit 11 acquires the subject's 3D pose information from the terminal 2 (step S203). The control unit 11 also creates a prompt to instruct the image generation model 30 to generate a 2D image (step S204). The control unit 11 generates a 2D image by inputting the 3D pose information and the prompt to the image generation model 30, which was further trained in step S202 (step S205).

[0043] The control unit 11 learns from training data that includes the generated 2D image and the 3D pose information input to the image generation model 30, thereby generating a pose estimation model 40 that outputs 3D pose information when a 2D image is input (step S206). The control unit 11 then completes the series of processes.

[0044] Based on the above, according to this embodiment 2, by performing additional training on the image generation model 30, a suitable two-dimensional image can be generated without inputting a reference image.

[0045] The embodiments disclosed herein should be considered in all respects to be illustrative and not restrictive. The scope of the present invention is indicated by the claims, not in the sense described above, and all modifications within the sense and scope equivalent to the claims are intended.

[0046] The matters described in each embodiment can be combined with each other. Furthermore, the independent and dependent claims described in the claims can be combined with each other in any combination, regardless of the form of reference. In addition, the claims use a form in which claims referencing two or more other claims (multi-claim form), but are not limited to this. A form in which multi-claims referencing at least one multi-claim (multi-multi-claim) may also be used. [Explanation of Symbols]

[0047] 1. Server (Information Processing Device) 11 Control Unit 12 Main memory 13 Communications Department 14 Auxiliary storage P Program 2 terminals 3. Generation Server 30 Image Generation Models 40. Pose Estimation Models

Claims

1. Obtain 3D pose information of an object performing a specific action, By inputting the acquired 3D pose information and a prompt instructing the generation of a 2D image including the object performing the specific action, and specifying the name of the action being performed by the object, into the image generation model, a 2D image is generated, and training data is created in which the acquired 3D pose information is annotated as the ground truth value on the generated 2D image. An information processing method in which a computer performs the processing.

2. Based on the aforementioned training data, a model is generated that outputs 3D pose information when a 2D image is input. The information processing method according to claim 1.

3. The aforementioned object is a person, an animal, or a robot. The three-dimensional posture information includes the position information of the joints, body shape information, and joint angle information of the object. The information processing method according to claim 1.

4. The image generation model generates the two-dimensional image by inputting a reference image, which it should refer to for generating the two-dimensional image, along with the three-dimensional pose information. The information processing method according to claim 1.

5. The aforementioned object is a person, an animal, or a robot. The aforementioned reference image is an image of the object performing a specific action, an image including the object and a background, or an image of the person using a specific tool. The information processing method according to claim 4.

6. The aforementioned object is a person, The two-dimensional image is generated by inputting the three-dimensional posture information, which includes data showing the three-dimensional shape of the tools used by the person, into the image generation model. The information processing method according to claim 1.

7. Additional training images are obtained by capturing the aforementioned object. Based on the aforementioned additional training images, the image generation model is further trained. The 2D image is generated by inputting the 3D pose information into the further trained image generation model. The information processing method according to claim 1.

8. Obtain three-dimensional pose information of an object performing a specific action, By inputting the acquired 3D pose information and a prompt instructing the generation of a 2D image including the object performing the specific action, and specifying the name of the action being performed by the object, into the image generation model, a 2D image is generated, and training data is created in which the acquired 3D pose information is annotated as the ground truth value on the generated 2D image. A program that instructs a computer to perform a process.

9. An information processing device comprising a control unit, The control unit, By acquiring the 3D pose information of an object performing a specific action, By inputting the acquired 3D pose information and a prompt instructing the generation of a 2D image including the object performing the specific action, and specifying the name of the action being performed by the object, into the image generation model, a 2D image is generated, and training data is created in which the acquired 3D pose information is annotated as the ground truth value on the generated 2D image. Information processing device.

10. Obtain a two-dimensional image of an object performing a specific action, The 3D pose information of the object is output by inputting three-dimensional pose information and a prompt instructing the image generation model to generate a two-dimensional image including the object performing the specific action, and a prompt specifying the name of the action the object is performing. The acquired two-dimensional image is then input into a model trained on training data annotated with the three-dimensional pose information as the correct value, thereby outputting the three-dimensional pose information of the object. A program that instructs a computer to perform a process.

Images