Information processing method, program and information processing unit

Abstract

To provide an information processing method etc. capable of suitably supporting structuring of a machine learning model which estimates three-dimensional attitude information from a two-dimensional image.SOLUTION: An information processing method causes a computer to execute processing to acquire three-dimensional attitude information on an object and input the acquired three-dimensional attitude information to an image generation model 30 to generate a two-dimensional image so as to generate training data including the generated two-dimensional image and the acquired three-dimensional attitude information. It is preferred to generate the model which is to output the three-dimensional attitude information based upon the training data when inputting the two-dimensional image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present invention relates to an information processing method, a program, and an information processing device. [Background technology]

[0002] There is a technology for estimating three-dimensional posture information of a subject from a two-dimensional image of the subject. For example, Patent Document 1 discloses a joint point detection device that acquires two-dimensional joint point coordinate data that can identify the two-dimensional coordinates of each joint point of a person from an image, and applies the acquired two-dimensional joint point coordinate data to a machine learning model that learns the relationship between the two-dimensional coordinates and three-dimensional coordinates of the joint points of the person, thereby detecting the three-dimensional coordinates of each joint point of the person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2023-167320 Summary of the Invention [Problem to be solved by the invention]

[0004] In one aspect, an object of the present invention is to provide an information processing method and the like that can favorably support the construction of a machine learning model that estimates three-dimensional posture information from two-dimensional images. [Means for solving the problem]

[0005] In one aspect, the information processing method includes a process in which a computer acquires three-dimensional posture information of an object, inputs the acquired three-dimensional posture information into an image generation model to generate a two-dimensional image, and creates training data including the generated two-dimensional image and the acquired three-dimensional posture information. [Effects of the Invention]

[0006] In one aspect, the present invention can effectively support the construction of a machine learning model that estimates three-dimensional pose information from two-dimensional images. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is an explanatory diagram illustrating an example of the configuration of a posture estimation system. [Figure 2] FIG. 2 is a block diagram illustrating an example of the configuration of a server. [Figure 3] FIG. 1 is a diagram showing an overview of a first embodiment. [Figure 4] 10 is a flowchart illustrating an example of a processing procedure executed by a server. [Figure 5] 10 is a flowchart illustrating an example of a processing procedure executed by a server according to the second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0008] The present invention will be described in detail below with reference to the drawings showing embodiments thereof. (Embodiment 1) Fig. 1 is an explanatory diagram showing an example of the configuration of a posture estimation system. In this embodiment, a posture estimation system will be described that generates a posture estimation model 40 (see Fig. 3) that estimates three-dimensional posture information from a two-dimensional image by learning from a two-dimensional image generated using an image generation model 30 as training data. The posture estimation system includes an information processing device 1, a terminal 2, and a generation server 3. Each device is communicatively connected via a network N such as the Internet.

[0009] Information processing device 1 is an information processing device capable of various information processing and information transmission and reception, such as a server computer or a personal computer. In this embodiment, information processing device 1 is assumed to be a server computer, and for simplicity, will be referred to as server 1 below. Server 1 uses machine learning to learn training data including a two-dimensional image of an object and three-dimensional posture information of the object corresponding to the two-dimensional image, thereby generating a posture estimation model 40 that outputs three-dimensional posture information when a two-dimensional image is input. Server 1 estimates the three-dimensional posture information by inputting the two-dimensional image of the object to be analyzed into posture estimation model 40.

[0010] The "object" shown in the two-dimensional image is, for example, a person. In this embodiment, it is assumed that the posture of a subject playing a sport is analyzed, and three-dimensional posture information of the subject is estimated by inputting a two-dimensional image of the subject playing a sport into posture estimation model 40.

[0011] It should be noted that the "target object" is not limited to a person, but may be a non-human animal or a non-animal object (for example, a robot).

[0012] "Three-dimensional posture information" is data including at least three-dimensional position information of the joints of a subject. In this embodiment, the "three-dimensional posture information" is an SMPL model (Skinned Multi-Person Linear Model), which includes body shape information and joint angle information in addition to joint position information. The SMPL model is a model that can identify the surface shape of a person's body, and is a model that can control a three-dimensional human model represented by 6,890 vertices using a 72-dimensional posture parameter θ and a 10-dimensional body shape parameter β. By estimating parameters related to the SMPL model as three-dimensional posture information, it is possible to analyze body shape and joint rotation angles in addition to joint positions.

[0013] Note that the SMPL model is an example of three-dimensional posture information, and the three-dimensional posture information may be data in other formats.

[0014] As described above, server 1 generates posture estimation model 40 by learning training data including two-dimensional images and three-dimensional pose information. However, in order to generate (learn) posture estimation model 40, it is necessary to prepare training data in which correct three-dimensional pose information is annotated for a large number of two-dimensional images. In particular, in this embodiment, an SMPL model is used as the three-dimensional pose information. However, sufficient training data (data pairing two-dimensional images with parameters related to the SMPL model) for learning the SMPL model is not currently available, and therefore it is necessary to manually annotate two-dimensional images with parameters related to the SMPL model. However, it is difficult to perform this annotation accurately by hand.

[0015] Therefore, in this embodiment, instead of annotating 3D pose information to 2D images, training data is prepared by generating 2D images based on 3D pose information set to plausible values. Specifically, 2D images are generated by inputting the 3D pose information into image generation model 30, and pairs of the generated 2D images and the 3D pose information input to image generation model 30 are learned as training data to generate pose estimation model 40.

[0016] Terminal 2 is a client terminal connected to server 1, such as a personal computer. During learning, server 1 accepts input of 3D posture information and the like from terminal 2, generates 2D images from the 3D posture information to serve as training data, and generates posture estimation model 40. During estimation, server 1 acquires 2D images of the analysis target from terminal 2 and inputs them into posture estimation model 40 to estimate 3D posture information.

[0017] The generation server 3 is a server computer that generates an image upon receiving a request from a client. For example, the server 1 requests the generation server 3 to generate an image through a predetermined API (Application Programmable Interface). The generation server 3 generates an image by inputting data including a prompt instructing image generation into the image generation model 30.

[0018] The image generation model 30 is a machine learning model that has been trained on a large number of images, and is, for example, a diffusion model such as stable diffusion. Note that the image generation model 30 is not limited to a diffusion model, and may be any model that can generate a two-dimensional image from three-dimensional posture information, etc., as will be described later.

[0019] 2 is a block diagram showing an example of the configuration of the server 1. The server 1 includes a control unit 11, a main memory unit 12, a communication unit 13, and an auxiliary memory unit . The control unit 11 has one or more arithmetic processing devices such as a central processing unit (CPU), a micro-processing unit (MPU), a graphics processing unit (GPU), etc., and performs various information processing, control processing, etc. by reading and executing a program P stored in the auxiliary storage unit 14. The main storage unit 12 is a temporary storage area such as a static random access memory (SRAM) or a dynamic random access memory (DRAM), and temporarily stores data necessary for the control unit 11 to execute arithmetic processing. The communication unit 13 is a communication module for performing communication-related processing, and transmits and receives information to and from the outside. The auxiliary storage unit 14 is a non-volatile storage area such as a large-capacity memory or a hard disk, and stores the program P (program product) and other data necessary for the control unit 11 to execute processing.

[0020] The auxiliary storage unit 14 may be an external storage device connected to the server 1. The server 1 may be a multi-computer consisting of multiple computers, or may be a virtual machine virtually constructed by software.

[0021] Furthermore, in this embodiment, the server 1 is not limited to the above configuration, and may include, for example, an input unit that accepts operation input, a display unit that displays images, etc. Furthermore, the server 1 may be provided with a reading unit that reads a portable storage medium 1a such as a CD (Compact Disk)-ROM or a DVD (Digital Versatile Disc)-ROM, and may read and execute the program P from the portable storage medium 1a.

[0022] Fig. 3 is a diagram showing an overview of the first embodiment. Fig. 3 conceptually illustrates how a two-dimensional image is generated by inputting data including three-dimensional posture information into image generation model 30, and posture estimation model 40 is generated by learning the generated two-dimensional image and the three-dimensional posture information input into image generation model 30 as training data. An overview of this embodiment will be described with reference to Fig. 3.

[0023] As described above, the server 1 uses the image generation model 30 to create training data for generating the posture estimation model 40. Specifically, the server 1 inputs the subject's three-dimensional posture information, a reference image, and a prompt into the image generation model 30 to generate a two-dimensional image.

[0024] As described above, the 3D posture information is 82-dimensional (72+10) parameters related to the SMPL model, and is data including joint position information, body shape information, and joint angle information. The server 1 acquires the 3D posture information set to likely values ​​from the terminal 2 and inputs it to the image generation model 30. There is no particular restriction on the method for acquiring (creating) the 3D posture information. For example, the server 1 may acquire 3D posture information obtained by manually correcting the output of an existing 3D posture estimation model, or may acquire 3D posture information created based on data measured by attaching a motion sensor to the subject (athlete).

[0025] The reference image is a two-dimensional image that the image generation model 30 uses as a reference for generating a two-dimensional image. In this embodiment, the reference image is an image of a person playing a sport, in which the subject is performing a specific movement related to the sport. The reference image includes not only the subject but also the background and the equipment used by the subject. For example, if the sport is figure skating, an image with a skating rink as the background and the subject wearing sports equipment such as skates is used as the reference image. By providing the reference image to the image generation model 30, the image generation model 30 can generate a two-dimensional image with features similar to the reference image. In particular, in this embodiment, by providing a reference image including a background, equipment, etc. corresponding to the sport played by the subject, a two-dimensional image of a person wearing similar equipment with a similar background can be generated.

[0026] In this embodiment, the accuracy of the 2D image is improved by providing a reference image including the tool (sports equipment), but in addition to the reference image including the tool, data indicating the 3D shape of the tool may also be input to the image generation model 30. For example, the server 1 inputs data indicating the 3D shape of the tool to the image generation model 30 as part of the 3D pose information (SMPL model). Even in this case, a 2D image of a person wearing the tool can be suitably generated.

[0027] A prompt is an instruction statement that instructs the image generation model 30 to generate an image. For example, the server 1 creates an instruction statement specifying a preset sport name or 2D image quality (such as "High Quality") as a prompt and inputs it to the image generation model 30. Note that the user may set (change) the prompt each time a 2D image is generated.

[0028] The server 1 generates a two-dimensional image by inputting three-dimensional posture information, a reference image, and a prompt to the image generation model 30. When inputting three-dimensional posture information to the image generation model 30, the server 1 inputs data (two-dimensional image) in which the three-dimensional posture information is visualized on a two-dimensional plane to the image generation model 30, but the three-dimensional posture information may be used as is as input to the image generation model 30. The server 1 learns pairs of the two-dimensional image generated by the image generation model 30 and the three-dimensional posture information input to the image generation model 30 as training data, and generates a posture estimation model 40 that outputs three-dimensional posture information when a two-dimensional image is input.

[0029] Pose estimation model 40 is, for example, a neural network such as a convolutional neural network (CNN), but may also be a machine learning model other than a neural network. Server 1 estimates 3D posture information by inputting training 2D images into posture estimation model 40, compares the estimated 3D posture information with correct 3D posture information, and optimizes parameters such as weights between neurons so that the two approximate each other. In this way, server 1 generates posture estimation model 40. Server 1 estimates 3D posture information using posture estimation model 40.

[0030] As described above, according to this embodiment, training data is created by generating two-dimensional images from three-dimensional posture information using image generation model 30. This makes it possible to suitably generate posture estimation model 40 that estimates three-dimensional posture information from two-dimensional images.

[0031] 4 is a flowchart showing an example of a processing procedure executed by server 1. The processing contents when generating (learning) posture estimation model 40 will be described with reference to FIG. The control unit 11 of the server 1 acquires three-dimensional posture information of the object from the terminal 2 (step S11). The three-dimensional posture information is parameters related to the SMPL model, including, for example, position information of the joints of a human, body type information, and joint angle information.

[0032] The control unit 11 acquires a reference image to be used by the image generation model 30 to generate a two-dimensional image (step S12). The reference image is, for example, a two-dimensional image of a person playing a sport. The reference image is an image of the subject performing a specific action, and includes the subject, the background, and the equipment (sports goods) used by the subject.

[0033] The control unit 11 generates a prompt that instructs the image generation model 30 to generate a two-dimensional image (step S13). Then, the control unit 11 inputs the three-dimensional posture information acquired in step S11, the reference information acquired in step S12, and the prompt created in step S13 into the image generation model 30, thereby generating a two-dimensional image for training (step S14).

[0034] Control unit 11 performs machine learning to learn training data including the two-dimensional image generated in step S14 and the three-dimensional posture information input to image generation model 30, and generates posture estimation model 40 that outputs three-dimensional posture information when a two-dimensional image is input (step S15). Control unit 11 ends the series of processes.

[0035] As described above, according to the first embodiment, it is possible to suitably support the construction of posture estimation model 40 that estimates three-dimensional posture information from a two-dimensional image.

[0036] (Embodiment 2) In this embodiment, we will describe an embodiment in which the accuracy of a two-dimensional image is improved by performing additional image learning (fine tuning) on ​​the image generation model 30. Note that the same reference numerals are used to designate the same parts as in the first embodiment, and the description thereof will be omitted.

[0037] In the first embodiment, a configuration has been described in which the accuracy of a two-dimensional image is improved by inputting a reference image together with three-dimensional posture information to image generation model 30. On the other hand, by having image generation model 30 learn a desired image in advance, the accuracy of a two-dimensional image can also be improved without providing a reference image.

[0038] For example, the server 1 acquires a small number of additional training images (e.g., several to several tens) from the terminal 2 and trains the image generation model 30 on the images, thereby tuning the model to be suitable for generating a specific two-dimensional image (in this embodiment, an image of a person playing a specific sport). Similar to the "reference image" in embodiment 1, the additional training image is a two-dimensional image of a person playing a sport, and is an image of the subject performing a specific action. The additional training image includes the subject, a background, and equipment (sports goods) used by the subject.

[0039] The server 1 performs additional training of the image generation model 30 based on the additional training images. In this embodiment, fine tuning is performed using the LoRA (Low-Rank Adaptation) method. LoRA is a method of changing parameters by introducing a low-rank matrix instead of directly changing the parameters (weights) of the original model. By employing LoRA, the amount of calculation required for training can be significantly reduced.

[0040] Server 1 generates a two-dimensional image by inputting three-dimensional posture information and a prompt to the trained image generation model 30. As in the first embodiment, server 1 generates a posture estimation model 40 based on training data including the generated two-dimensional image and the three-dimensional posture information input to image generation model 30.

[0041] 5 is a flowchart showing an example of a processing procedure executed by server 1 according to embodiment 2. The content of the processing for generating posture estimation model 40 according to this embodiment will be described with reference to FIG. The control unit 11 of the server 1 acquires additional training images from the terminal 2 to be used by the image generation model 30 for training (step S201). The additional training images are, for example, two-dimensional images of a person playing a sport, and are images of the subject performing a specific movement. For example, the additional training images include the subject, a background, and equipment (sports goods) used by the subject. The control unit 11 performs additional training (fine tuning) of the image generation model 30 based on the acquired additional training images (step S202). In this embodiment, fine tuning is performed using the LoRA method.

[0042] The control unit 11 acquires three-dimensional posture information of the subject from the terminal 2 (step S203). The control unit 11 also creates a prompt that instructs the image generation model 30 to generate a two-dimensional image (step S204). The control unit 11 generates a two-dimensional image by inputting the three-dimensional posture information and the prompt into the image generation model 30 that has undergone additional learning in step S202 (step S205).

[0043] Control unit 11 learns training data including the generated two-dimensional image and the three-dimensional posture information input to image generation model 30, thereby generating posture estimation model 40 that outputs three-dimensional posture information when a two-dimensional image is input (step S206). Control unit 11 then ends the series of processes.

[0044] As described above, according to the second embodiment, by performing additional learning on the image generation model 30, it is possible to generate a suitable two-dimensional image without inputting a reference image.

[0045] The embodiments disclosed herein are to be considered as illustrative in all respects and not restrictive. The scope of the present invention is defined by the claims, not by the above meaning, and is intended to include all modifications within the meaning and scope of the claims.

[0046] The matters described in each embodiment can be combined with each other. Furthermore, the independent claims and dependent claims described in the claims can be combined with each other in any combination, regardless of the reference format. Furthermore, although the claims use a format in which a claim references two or more other claims (multi-claim format), this is not limited to this format. A format in which multiple claims (multi-multi-claims) that reference at least one other multiple claim may also be used. [Explanation of symbols]

[0047] 1. Server (information processing device) 11 Control section 12 Main memory 13 Communications Department 14 Auxiliary storage P Program 2. Terminal 3 Generation Server 30 Image generation model 40 Pose estimation model

Claims

1. Acquire three-dimensional posture information of the object, The acquired three-dimensional posture information is input into an image generation model to generate a two-dimensional image, thereby creating training data including the generated two-dimensional image and the acquired three-dimensional posture information. An information processing method in which processing is performed by a computer.

2. Based on the training data, a model is generated that outputs three-dimensional posture information when a two-dimensional image is input. The information processing method according to claim 1 .

3. the object is a person, The three-dimensional posture information includes joint position information, body shape information, and joint angle information of the subject. The information processing method according to claim 1 .

4. A prompt instructing the generation of the two-dimensional image is input to the image generation model together with the three-dimensional posture information, thereby generating the two-dimensional image. The information processing method according to claim 1 .

5. The two-dimensional image is generated by inputting a reference image to be referred to by the image generation model for generating the two-dimensional image together with the three-dimensional posture information into the image generation model. The information processing method according to claim 1 .

6. the object is a person, The reference image is an image of the subject performing a specific action, an image including the subject and a background, or an image of the subject using a specific tool. The information processing method according to claim 5 .

7. the object is a person, The two-dimensional image is generated by inputting the three-dimensional posture information, which includes data indicating the three-dimensional shape of the tool used by the subject, into the image generation model. The information processing method according to claim 1 .

8. acquiring an additional learning image of the object; additional learning of the image generation model based on the additional learning images; The two-dimensional image is generated by inputting the three-dimensional posture information into the additionally trained image generation model. The information processing method according to claim 1 .

9. Acquire three-dimensional posture information of the object, The acquired three-dimensional posture information is input into an image generation model to generate a two-dimensional image, thereby creating training data including the generated two-dimensional image and the acquired three-dimensional posture information. A program that causes a computer to perform a process.

10. An information processing device including a control unit, The control unit Acquire three-dimensional posture information of the object, The acquired three-dimensional posture information is input into an image generation model to generate a two-dimensional image, thereby creating training data including the generated two-dimensional image and the acquired three-dimensional posture information. Information processing device.

11. Acquire a two-dimensional image of the object; The acquired two-dimensional image is input to a model trained on the basis of training data including a two-dimensional image generated by an image generation model based on the three-dimensional posture information, and the three-dimensional posture information of the object is output. A program that causes a computer to perform a process.

Images