Image generation device, image generation method, and program

Abstract

To designate the posture of a person or the like in an image when generating the image.SOLUTION: The image generation device 10 includes a feature amount extraction unit 11 that extracts a skeleton feature amount of a skeleton from skeleton information that specifies a position of each joint constituting the skeleton, and an image generation unit 12 that inputs the extracted skeleton feature amount and a noise image to a machine learning model that estimates noise added to an image, and generates an image corresponding to the skeleton by removing the noise from the image using an output result from the machine learning model.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image generation device and an image generation method for performing image generation, and further to a program for realizing these. [Background technology]

[0002] In recent years, image generation using image generation AI (Artificial Intelligence) has been proposed. Image generation AI can generate images based on input text, so it is used in various fields such as web design, game design, and advertising.

[0003] For example, Patent Document 1 and Non-Patent Document 1 disclose examples of image generation AI. When text is input, the image generation AI disclosed in Patent Document 1 and Non-Patent Document 1 uses a machine learning model to generate an image corresponding to the input text. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2024-060907 [Non-patent literature]

[0005] [Non-Patent Document 1] Robin Rombach, Andreas Blattmann, Dominik Lorenz, Patrick Esser, Bjorn Ommer, “High-Resolution Image Synthesis with Latent Diffusion Models”, [online], arXiv:2112.10752v2 [cs.CV], 13 Apr 2022, [Retrieved May 1, 2024], Internet<URL:https: / / arxiv.org / abs / 2112.10752> Summary of the Invention [Problem to be solved by the invention]

[0006] However, the image generation AI disclosed in the above-mentioned Patent Document 1 and Non-Patent Document 1 has the problem that when generating an image of a person, it only accepts text input, making it impossible to specify the posture of the person in the image in detail.

[0007] An example of an objective of the present disclosure is to enable the specification of the posture of a person or the like in an image when generating the image. [Means for solving the problem]

[0008] In order to achieve the above object, an image generating device according to one aspect of the present disclosure includes: a feature extraction unit that extracts skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint that constitutes the skeleton; an image generation unit that inputs the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and generates an image corresponding to the skeleton by removing noise from the image using an output result from the machine learning model; The present invention is characterized by the following features.

[0009] In order to achieve the above object, an image generating method according to one aspect of the present disclosure includes: a feature extraction step of extracting skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint constituting the skeleton; an image generation step of inputting the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and removing noise from the image using an output result from the machine learning model, thereby generating an image corresponding to the skeleton; The present invention is characterized in that it is provided with:

[0010] Furthermore, in order to achieve the above object, a program according to one aspect of the present disclosure includes: On the computer, a feature extraction step of extracting skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint constituting the skeleton; an image generation step of inputting the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and removing noise from the image using an output result from the machine learning model, thereby generating an image corresponding to the skeleton; The method is characterized in that: [Effects of the Invention]

[0011] As described above, according to the present disclosure, the posture of a person or the like in an image can be specified when generating the image. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram showing a schematic configuration of an example of an image generating device. [Figure 2] FIG. 2 is a diagram specifically illustrating the configuration of an example of an image generating device. [Figure 3] FIG. 3 is a block diagram showing in detail the configuration of the noise estimation unit of the image generating device shown in FIG. [Figure 4] FIG. 4 is a flow diagram showing an example of the overall operation of the image generating device. [Figure 5] FIG. 5 is a flow diagram showing in detail the image generation steps shown in FIG. [Figure 6] FIG. 6 is a diagram illustrating a configuration of an example of a learning model generating device. [Figure 7] FIG. 7 is a diagram illustrating an example of training data used to generate a learning model. [Figure 8] FIG. 8 is a flow diagram showing an example of the operation of the learning model generating device. [Figure 9] FIG. 9 is a configuration diagram showing an example of the configuration of a learning model generation device for performing machine learning of the second machine learning model. [Figure 10] FIG. 10 is a diagram showing an example of a bone length vector and a bone length ratio vector. [Figure 11] FIG. 11 is a diagram showing an example of an angle vector obtained by concatenating a camera vector and a physique vector. [Figure 12] FIG. 12 is a diagram illustrating an example of a parameter update process in the learning model generation device illustrated in FIG. [Figure 13] FIG. 13 is a flowchart showing an example of the operation of the learning model generating device shown in FIG. [Figure 14] FIG. 14 is a block diagram showing an example of a computer that realizes the image generation device and the learning model generation device. DETAILED DESCRIPTION OF THE INVENTION

[0013] (Embodiment 1) Hereinafter, in a first embodiment, an image generating device, an image generating method, and a program will be described with reference to FIGS.

[0014] [Device configuration] First, the schematic configuration of an example of an image generating device will be described with reference to Fig. 1. Fig. 1 is a diagram showing the schematic configuration of an example of an image generating device.

[0015] 1 is a device for generating an image of a person or the like according to a specified pose. As shown in FIG. 1, the image generating device 10 includes a feature extraction unit 11 and an image generating unit 12.

[0016] The feature extraction unit 11 extracts skeletal features of the skeleton from skeletal information that identifies the positions of each joint that constitutes the skeleton. The image generation unit 12 inputs the extracted skeletal features and a noise image to a machine learning model that estimates noise added to the image. The image generation unit 12 then generates an image corresponding to the skeleton by removing noise from the image using the output result from the machine learning model.

[0017] In this way, the image generating device 10 can generate an image using skeletal features extracted from skeletal information, rather than text. Therefore, the image generating device 10 allows the user to specify a posture when generating an image.

[0018] Next, the configuration and functions of the image generating device 10 will be specifically described with reference to Fig. 2 and Fig. 3. Fig. 2 is a block diagram specifically showing the configuration of an example of the image generating device. Fig. 3 is a block diagram specifically showing the configuration of the noise estimation unit of the image generating device shown in Fig. 2.

[0019] 2, the image generating device 10 includes a skeletal information acquiring unit 13 in addition to the feature extraction unit 11 and image generating unit 12 described above. The skeletal information acquiring unit 13 acquires skeletal information and inputs the acquired skeletal information to the feature extraction unit 11. The skeletal information is input by a user requesting image generation via the user's terminal device.

[0020] In the example of FIG. 2, the skeletal information is shown as a skeleton, but in reality, it is composed of, for example, coordinates indicating the position of each joint point of a person in the image data. The origin of the coordinates of each joint point is set, for example, based on the camera. The skeletal information may further include information indicating body type (for example, parameters indicating the degree of obesity, etc.), and information indicating the surface of the body that forms the skeleton.

[0021] In the first embodiment, the feature extraction unit 11 extracts skeletal features from skeletal information using a second machine learning model. The second machine learning model is a machine learning model that has previously learned the relationship between skeletal information and its features. An example of the second machine learning model is a neural network.

[0022] The second machine learning model can be implemented by a machine learning program executed on a computer. The second machine learning model may be implemented in a device (computer) other than the image generating device 10. The machine learning of the second machine learning model will be described in the third embodiment.

[0023] 2, in the first embodiment, the image generation unit 12 includes a time information generation unit 14, a noise-added data output unit 15, a noise estimation unit 16, and a noise subtraction unit 17. The image generation unit 12 removes noise from the noisy image in stages, and ultimately generates a clear image with almost no noise.

[0024] The time information generating unit 14 decreases the value of time t from T (T: arbitrary value) to 0 at set intervals, for example, and outputs the value of time t as time information. The output time information is input to the noise-added data output unit 15 and the noise estimation unit 16.

[0025] When time information is input, the noise-added data output unit 15 generates image data of an image (noise image) to which noise has been added according to the value of time t, and outputs the generated image data. Specifically, when time t=T, an initial noise image formed entirely of random noise is generated, and the image data is output to the noise estimation unit 16. On the other hand, when time t is smaller than T, the noise-added data output unit 15 acquires the latest noise image from which noise has been removed by the noise subtraction unit 17, which will be described later, and outputs the acquired noise image to the noise estimation unit 16.

[0026] The value of time t varies between T and 0 because, as will be explained in the second embodiment below, in the training data image used for learning the machine learning model 20 described below, the noise is set to be 0 when time t=0 and the entire image is noise when time t=T.

[0027] As shown in Fig. 3, the noise estimation unit 16 includes a machine learning model 20 and a DNN (Deep Neural Network) 21. Of these, the DNN 21 performs machine learning to learn the relationship between time t and feature amounts, and when time information is input, outputs a feature amount corresponding to the time information (hereinafter referred to as a "time feature amount"). The output time feature amount is input to the machine learning model 20. Note that the noise estimation unit 16 may also include a machine learning model other than the DNN 21. In this case, it is sufficient that the machine learning model performs machine learning to learn the relationship between time t and feature amounts.

[0028] The machine learning model 20 is, for example, a model that performs a dediffusion process on a noise image. In the example of Fig. 3, the machine learning model 20 is a neural network called U-Net. The machine learning model 20 is configured by connecting multiple ResBlocks and multiple AttnBlocks in a U-shape.

[0029] As shown in FIG. 3, a time feature is input to each ResBlock, and a skeleton feature p is input to each AttnBlock. The first ResBlock contains a noise image Z at time t. t is entered.

[0030] Specifically, ResBlock is a module that contains a convolutional layer and generates a noise image Z t AttnBlock is a module that includes an attention layer and is used to condition the noise image Z t By alternately processing ResBlock and AttnBlock, the final AttnBlock will produce the predicted noise e t In this way, the machine learning model 20 calculates the noise e at time t from the noise image, the skeleton feature, and the time feature. t Predict the predicted noise e t Output.

[0031] Details of U-Net are disclosed, for example, at https: / / github.com / CompVis / stable-diffusion.

[0032] The noise subtractor 17 subtracts the noise e output from the noise estimator 16. t , and from the noise image at time t, the received noise e t As a result, a noise image at time (t+1) is generated. The noise-added data output unit 15 acquires the noise image generated by the noise subtraction unit 17 as described above, and outputs the acquired noise image to the noise estimation unit 16.

[0033] The series of processes by the time information generating unit 14, noise-added data output unit 15, noise estimating unit 16, and noise subtracting unit 17 described above are repeatedly executed until the value of time t changes from 0 to T. As a result, noise is gradually removed from the noisy image at time t=0, and finally a clear image with almost no noise (hereinafter referred to as the "generated image") is generated.

[0034] [Device operation] Next, an example of the operation of the image generation device will be described with reference to Figures 4 and 5. In the following description, Figures 1 to 3 will be referenced as appropriate. In addition, in the first embodiment, an image generation method is implemented by operating the image generation device 10. Therefore, in the first embodiment, the description of the image generation method will be replaced by the following description of the operation of the image generation device 10.

[0035] First, the overall operation of the image generating device 10 will be described with reference to Fig. 4. Fig. 4 is a flow chart showing an example of the overall operation of the image generating device.

[0036] 4, first, when a user requesting image generation inputs skeletal information via a terminal device, the skeletal information acquisition unit 13 acquires the input skeletal information (step A1). The skeletal information acquisition unit 13 then inputs the acquired skeletal information to the feature extraction unit 11.

[0037] Next, the feature extraction unit 11 extracts skeletal features from the skeletal information using the second machine learning model (step A2). The feature extraction unit 11 also inputs the extracted skeletal features to the image generation unit 12.

[0038] Next, the image generation unit 12 inputs the skeletal feature amount and the noise image extracted in step A2 into the machine learning model 20 (see FIG. 3), and removes noise from the image using the output result from the machine learning model 20, thereby generating an image corresponding to the skeleton of the skeletal information (step A3). The generated image (generated image) is transmitted to the user's terminal device.

[0039] Next, the image generation steps shown in Fig. 4 will be described in detail with reference to Fig. 5. Fig. 5 is a flow chart showing the image generation steps shown in Fig. 4 in detail.

[0040] 5, at the timing when the skeletal feature is input in step A2, the time information generating unit 14 sets the value of time t to 0 and generates time information (step A31). Then, the time information generating unit 14 inputs the generated time information to the noise-added data output unit 15 and the noise estimation unit 16.

[0041] Next, the noise-added data output unit 15 determines whether the value of time t in the time information is 0 (step A32). If the result of the determination in step A32 is that the value of time t is 0, the noise-added data output unit 15 generates an initial noise image generated only from random noise, and outputs the image data to the noise estimation unit 16 (step A33).

[0042] On the other hand, if the result of the judgment in step A32 is that the value of time t is greater than 0, the noise-added data output unit 15 acquires the latest noise image from which noise has been removed by the noise subtraction unit 17, and outputs the acquired noise image to the noise estimation unit 16 (step A34).

[0043] Next, the noise estimation unit 16 inputs the time information generated in step A31 to the DNN 21, and further inputs the skeletal features extracted in step A2 and the noise image output in step A33 or A34 to the machine learning model 20 to predict noise (step A35).

[0044] Next, the noise subtraction unit 17 subtracts the noise predicted in step A35 from the noise image output in step A33 or A34 (step A36), thereby generating the latest noise image.

[0045] Next, the noise subtractor 17 determines whether the value of the time t has reached a preset value T (step A37).

[0046] If it is determined in step A37 that the value of time t has not reached the preset value T, the noise subtractor 17 increments the value of time t by 1 (step A38).

[0047] Next, when step A38 is executed, the noise-added data output section 15 executes step A32 again, whereby step A32 and subsequent steps are executed again.

[0048] On the other hand, if the result of the determination in step A37 is that the value of time t has reached the preset value T, the noise subtractor 17 transmits the latest noise image from which noise has been subtracted in step A36 as a generated image to the user's terminal device (step A39), thereby completing step A3.

[0049] In this way, when steps A32 to A38 are executed a predetermined number of times, noise is gradually removed from the initial noisy image, and finally a clear image with almost no noise (hereinafter referred to as a "generated image") is generated.

[0050] As described above, according to the first embodiment, an image of a person or the like can be generated based on skeletal information, rather than on text as in the past. In the first embodiment, in generating an image, a user can specify the posture of a person or the like based on skeletal information.

[0051] Furthermore, as will be described in a second embodiment below, it is assumed that image data serving as training data is reduced in resolution by a VAE (Variational Autoencoder) encoder during learning of the machine learning model 20. In this case, the image generation unit 12 may further include a VAE decoder, and after executing step A39, performs decoding by the VAE decoder.

[0052] [program] In the first embodiment, an example of the program may be a program that causes a computer to execute steps A1 to A3 shown in Fig. 4. By installing and executing this program on a computer, the image generating device 10 and the image generating method can be realized. In this case, the processor of the computer functions as the feature extraction unit 11, the image generating unit 12, and the skeletal information acquisition unit 13 and performs the processing. Furthermore, examples of the computer include a general-purpose PC, a server computer, a smartphone, and a tablet terminal device.

[0053] Furthermore, the program in the first embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the feature extraction unit 11, the image generation unit 12, and the skeletal information acquisition unit 13.

[0054] (Embodiment 2) Next, in the second embodiment, a learning model generation device, a learning model generation method, and a program for performing machine learning of the machine learning model 20 will be described.

[0055] [Device configuration] First, the configuration of an example of a learning model generation device will be described with reference to Fig. 6. Fig. 6 is a configuration diagram showing the configuration of an example of a learning model generation device.

[0056] The learning model generation device 30 shown in Fig. 6 is a device for performing machine learning of the machine learning model 20 shown in Fig. 3. As shown in Fig. 6, the learning model generation device 30 includes a skeletal information acquisition unit 31, a feature extraction unit 32, an image acquisition unit 33, a time information generation unit 34, a noise generation unit 35, a noise estimation unit 36, an addition unit 37, a loss calculation unit 38, and a parameter update unit 39.

[0057] The skeletal information acquisition unit 31 has the same functions as the skeletal information acquisition unit 13 shown in Fig. 2. The skeletal information acquisition unit 31 acquires skeletal information that serves as training data, and inputs the acquired skeletal information to the feature extraction unit 32. The skeletal information is input by a user who requests image generation via the user's terminal device. The skeletal information is the same information as the skeletal information described in the first embodiment.

[0058] The feature extraction unit 32 has the same functions as the feature extraction unit 11 shown in Fig. 2. In the second embodiment, the feature extraction unit 32 also extracts skeletal features from skeletal information using a second machine learning model. The feature extraction unit 32 inputs the extracted skeletal features to the noise estimation unit 36. As in the first embodiment, the second machine learning model is a machine learning model that has learned in advance the relationship between skeletal information and its features.

[0059] The image acquisition unit 33 acquires image data that will be used as training data, and inputs the acquired image data to the addition unit 37. The image data is data obtained by photographing a person or the like with a camera, and is output from the camera. The image data may be image data of a still image, or image data of each frame that constitutes a moving image.

[0060] Fig. 7 is a diagram showing an example of training data used to generate a learning model. As shown in Fig. 7, the training data is composed of pairs of image data and skeletal information that correspond to each other. In the example of Fig. 7, arrows indicate corresponding image data and skeletal information. Of the training data, the image data is acquired by the image acquisition unit 33, and the skeletal information is acquired by the skeletal information acquisition unit 31.

[0061] The time information generating unit 34 randomly sets the value of time t between 0 and T (T: arbitrary value) and outputs the set value of time t as time information. The output time information is input to the noise generating unit 35 and the noise estimating unit 36.

[0062] When time information is input, the noise generation unit 35 generates noise according to the value of time t. Specifically, the noise generation unit 35 generates noise according to the value of time t so that the noise is 0 when time t=0 and the entire image data is noise when time t=T. The noise generation unit 35 also inputs the generated noise to the addition unit 37 and loss calculation unit 38.

[0063] The adder 37 adds the noise generated by the noise generator 35 to the image data input from the image acquirer 33. This generates image data of a noise image to which noise has been added according to the value of time t. The adder 37 inputs the generated noise image to the noise estimation unit 36.

[0064] The noise estimation unit 36 ​​has the same configuration and functions as the noise estimation unit 16 shown in Fig. 2 and Fig. 3. Similar to the example in Fig. 3, the noise estimation unit 36 ​​also includes a machine learning model 20 and a DNN (Deep Neural Network) 21. Therefore, when skeletal features, time information, and a noise image are input to the noise estimation unit 36, the noise estimation unit 36 ​​predicts noise at time t from the noise image, skeletal features, and time features, similar to the noise estimation unit 16. Furthermore, the noise estimation unit 36 ​​inputs the predicted noise to the loss calculation unit 38.

[0065] When the noise generated by the noise generating unit 35 and the predicted noise by the noise estimating unit 36 ​​are input, the loss calculating unit 38 calculates the difference between the two. The calculated difference becomes the loss. The loss calculating unit 38 also inputs the calculated difference (loss) to the parameter updating unit 39.

[0066] When the loss is input, the parameter update unit 39 updates the parameters of the machine learning model 20 so that the loss becomes 0 or a value close to 0. At this time, the parameter update unit 39 can also update the parameters of the DNN 21 and further the parameters of the second machine learning model of the feature extraction unit 32 based on the loss.

[0067] The series of processes by the skeletal information acquisition unit 31, feature extraction unit 32, image acquisition unit 33, time information generation unit 34, noise generation unit 35, noise estimation unit 36, addition unit 37, loss calculation unit 38, and parameter update unit 39 described above are executed for each set of training data by setting the value of time t between 0 and T. As a result, when the noise estimation unit 36, whose parameters have been estimated, is used in an image generation device, it can generate an image from skeletal information with high accuracy.

[0068] [Device operation] Next, an example of the operation of the learning model generation device will be described with reference to FIG. 8. FIG. 8 is a flow diagram showing an example of the operation of the learning model generation device. In the following description, reference will be made to FIGS. 3, 6, and 7 as appropriate. In addition, in the second embodiment, the learning model generation method is implemented by operating the learning model generation device 30. Therefore, in the second embodiment, the description of the learning model generation method will be replaced with the following description of the operation of the learning model generation device 30.

[0069] First, it is assumed that training data, each pair of corresponding skeletal information and image data, is prepared in advance in a database, and the learning model generation device 30 is connected to the database so as to be able to communicate data with it.

[0070] 8, first, the skeletal information acquisition unit 31 acquires skeletal information to be used as training data (step B1). In step B1, the skeletal information acquisition unit 31 further inputs the acquired skeletal information to the feature extraction unit 32.

[0071] Next, the feature extraction unit 32 uses the second machine learning model to extract skeletal features from the skeletal information acquired in step B1 (step B2). In step B2, the feature extraction unit 32 inputs the extracted skeletal features to the noise estimation unit 36.

[0072] Next, the image acquisition unit 33 acquires image data that will be used as training data (step B3). In step B3, the image acquisition unit 33 inputs the acquired image data to the adder 37. Note that step B3 may be executed before step B1, or may be executed simultaneously with step B1. Furthermore, the image acquisition unit 33 may also use a VAE encoder to reduce the resolution of the acquired image data in order to reduce the processing load.

[0073] Next, the time information generating unit 14 generates time information according to the value of the time t (step B4). At this time, if the processes from step B4 onwards have not yet been executed, the time information generating unit 14 sets the value of the time t to 0 and generates time information. In step B4, the time information generating unit 14 inputs the time information to the noise generating unit 35 and the noise estimating unit 36.

[0074] Next, the noise generator 35 generates noise according to the value of the time t indicated by the time information generated in step B4 (step B5). In step B5, the noise generator 35 inputs the generated noise to the adder 37 and the loss calculator 38.

[0075] Next, the adder 37 adds the noise generated in step B5 to the image data acquired in step B3 to generate a noise image (step B6). In step B6, the adder 37 inputs the generated noise image to the noise estimation unit 36.

[0076] Next, the noise estimation unit 36 ​​predicts noise at time t from the noise image, the skeletal feature, and the time feature (step B7). In step B7, the noise estimation unit 36 ​​inputs the predicted noise to the loss calculation unit 38.

[0077] Next, the loss calculation unit 38 calculates the difference (loss) between the noise generated in step B5 and the noise predicted in step B7 (step B8). In step B8, the loss calculation unit 38 inputs the calculated difference (loss) to the parameter update unit 39.

[0078] Next, the parameter update unit 39 updates the parameters of the machine learning model 20 so that the loss calculated in step B8 becomes 0 or a value close to 0 (step B9). In step B9, the parameter update unit 39 can also update the parameters of the DNN 21 based on the loss.

[0079] After step B9 is executed, if there is training data that has not yet been used to update the parameters, step B1 is executed again.

[0080] In this way, steps B1 to B9 are executed for each set of training data, thereby training the machine learning model 20 in the noise estimation unit 36. As described above, according to the second embodiment, the machine learning model 20 used in the image generating device 10 can be trained by machine learning.

[0081] [program] In the second embodiment, an example of the program may be a program that causes a computer to execute steps B1 to B9 shown in FIG. 8. By installing and executing this program on a computer, the learning model generation device 30 and the learning model generation method can be realized. In this case, the processor of the computer functions as a skeleton information acquisition unit 31, a feature extraction unit 32, an image acquisition unit 33, a time information generation unit 34, a noise generation unit 35, a noise estimation unit 36, an addition unit 37, a loss calculation unit 38, and a parameter update unit 39, and performs processing. Furthermore, examples of the computer include a general-purpose PC, a server computer, a smartphone, and a tablet terminal device.

[0082] Furthermore, the program in the second embodiment may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as any one of a skeleton information acquisition unit 31, a feature extraction unit 32, an image acquisition unit 33, a time information generation unit 34, a noise generation unit 35, a noise estimation unit 36, an addition unit 37, a loss calculation unit 38, and a parameter update unit 39.

[0083] (Embodiment 3) In the third embodiment, a learning model generation device, a learning model generation method, and a program for performing machine learning of a second machine learning model used in a feature extraction unit will be described with reference to FIGS. 9 to 13.

[0084] [Device configuration] First, the schematic configuration of an example of a learning model generation device for performing machine learning of a second machine learning model will be described with reference to Figures 9 to 12. Figure 9 is a configuration diagram showing the configuration of an example of a learning model generation device for performing machine learning of a second machine learning model.

[0085] 9 is a device for performing machine learning of a second machine learning model 50. As shown in FIG. 9, the learning model generation device 40 includes a training data acquisition unit 41, a feature extraction unit 42, an image feature extraction unit 43, a similarity calculation unit 44, a corresponding skeleton information acquisition unit 45, a skeleton similarity calculation unit 46, and a parameter update unit 47.

[0086] The training data acquisition unit 41 acquires training data. The training data is the same as the training data shown in the second embodiment, and is made up of pairs of image data and skeletal information corresponding to each other (see FIG. 7).

[0087] The image data is data obtained by photographing a person or the like with a camera and is output from the camera, as in embodiment 2. The image data may be image data of a still image or image data of each frame constituting a moving image.

[0088] As in the first and second embodiments, the skeletal information is composed of, for example, coordinates indicating the position of each joint point of the person in the image data. The origin of the coordinates of each joint point is set, for example, based on the camera. The skeletal information may further include information indicating the body shape and information indicating the surface of the body that forms the skeleton.

[0089] Furthermore, the corresponding image data and skeletal information are stored in a linked state, for example, in a database, etc. The linking is performed by the meta information of the image data and the skeletal information stored together in the database, etc. Specifically, the meta information of the image data includes an identifier of the corresponding skeletal information, and the meta information of the skeletal information includes an identifier of the corresponding image data.

[0090] The feature extraction unit 42 has the same functions as the feature extraction unit 11 shown in FIG. 2, and extracts skeletal features from skeletal information using a second machine learning model 50. The second machine learning model learns the relationship between the skeletal information and its features through the process described below. An example of the second machine learning model 50 is a neural network, as described in the first and second embodiments.

[0091] The image feature extraction unit 43 extracts image data features (hereinafter referred to as "image features") from the acquired image data using a third machine learning model 51. The third machine learning model 51 learns the relationship between the image data and its features through the processing described below. Another example of the third machine learning model 51 is a neural network.

[0092] Hereinafter, the skeleton feature will be referred to as "T i " and the image feature is "I j ". Both i and j are integers from 1 to N (i, j = 1, ..., N), and are numbers assigned to the image data and skeletal information that will become training data. The value of N matches the number of image data and skeletal information that will become training data. Furthermore, image data and skeletal information that are assigned the same number correspond to each other. In other words, for example, the skeleton represented by the first skeletal information corresponds to the skeleton of the person appearing in the first image data.

[0093] The similarity calculation unit 44 sets a combination of image data and skeletal information. Then, for each set combination, the similarity calculation unit 44 calculates a skeletal feature amount T i and image feature I j Similarity sim(T i ,I j Specifically, the similarity calculation unit 44 calculates the skeletal feature T i and image feature I j Using this, the similarity sim(T i ,I j ) to calculate the cosine similarity or Euclidean distance.

[0094] The corresponding skeleton information acquisition unit 45 first acquires meta information for each image data and meta information for each skeleton information. Then, the corresponding skeleton information acquisition unit 45 calculates the similarity sim(T i ,I j ) is calculated, the skeleton information (corresponding skeleton information) corresponding to the image data is identified, and the identified corresponding skeleton information is acquired.

[0095] The skeleton similarity calculation unit 46 calculates the skeleton similarity S between the i-th corresponding skeleton information and the j-th skeleton information for each combination set by the similarity calculation unit 44. i,j Specifically, the skeleton similarity calculation unit 46 calculates the skeleton similarity S i,j For example, the cosine similarity between the coordinate values ​​of the joint points or the cosine similarity between the angle vectors is calculated as follows. Note that the angle vector will be described later.

[0096] Here, the skeletal similarity S i,j The calculation method of skeletal similarity S i,j When the cosine similarity between the coordinate values ​​of the joint points is calculated as follows, the skeletal similarity calculation unit 46 calculates the skeletal similarity S i,j In the following formula 1, P i indicates the coordinate vector obtained from the coordinate values ​​of each joint point included in the i-th corresponding skeleton information. j indicates a coordinate value vector obtained from the coordinate values ​​of each joint point included in the j-th skeleton information.

[0097]

number

[0098] In addition, the skeletal similarity S i,jWhen the cosine similarity between angle vectors is calculated as follows, for example, the following process is executed. First, the skeletal similarity calculation unit 46 calculates a camera posture vector for each piece of image data that has become training data. Examples of camera posture vectors include a vector indicating the angle between the optical axis direction of the camera and the vertical direction, and a vector indicating the angle between the optical axis of the camera and parts of a person. Note that the camera posture vector may be calculated in advance for each piece of image data.

[0099] Next, the skeleton similarity calculation unit 46 calculates the average vector of the camera posture vectors for each image data. mean Furthermore, the skeleton similarity calculation unit 46 calculates the vector of the camera used to take the image (hereinafter referred to as "camera vector") for each piece of image data. For example, the camera vector of person A is expressed as "cam A ”, and the camera vector of person B is “cam B ", the camera vector is calculated by the following equation 2.

[0100]

number

[0101] Next, the skeletal similarity calculation unit 46 calculates a bone length vector for each piece of skeletal information using the coordinates of each joint contained in it, and further calculates a "bone length ratio vector" from the calculated bone length vector.

[0102] FIG. 10 is a diagram showing an example of a bone length vector and a bone length ratio vector. As shown in FIG. 10, the bone length vector is composed of "the length from the right shoulder to the right elbow," "the length from the right elbow to the right wrist," "the length from the right hip to the right ankle," "the length from the left hip to the left ankle," etc. Each length is calculated from the difference in coordinate values ​​(three-dimensional coordinates) between joints. The bone length ratio vector is calculated by dividing each length constituting the bone length vector by a reference length.

[0103] The skeletal similarity calculation unit 46 also calculates an average vector of the ratio vectors of the bone lengths of all the people who are the subject of the skeletal information. mean Furthermore, the skeleton similarity calculation unit 46 calculates a physique vector representing the physique of the target person using the average vector for each skeleton information. For example, the physique vector of person A is expressed as "phy A ”, and the physique vector of person B is “phy B ", the physique vector is calculated by the following equation 3.

[0104]

number

[0105] Next, the skeleton similarity calculation unit 46 concatenates the camera vector and the physique vector for each piece of skeleton information, as shown in Fig. 11. The vector obtained by concatenation is the above-mentioned "angle vector." Fig. 11 is a diagram showing an example of an angle vector obtained by concatenating the camera vector and the physique vector.

[0106] Thereafter, for each combination set by the similarity calculation unit 44, the corresponding skeleton information I i The angle vector obtained for and the skeleton information I j The similarity between the angle vector obtained for and is calculated. The similarity is calculated by using the cosine similarity (cos_sim(cam i +phy i , cam j +phy j The calculated similarity is called the skeletal similarity S i,j This becomes:

[0107] Furthermore, the angle vector may be a vector indicating the angle between the bone connecting the joints and the optical axis of the camera. In this case, the skeletal similarity calculation unit 46 first calculates the vector b k,l Specifically, the vector b k,lis expressed as the difference in three-dimensional coordinate values ​​for each combination of two joint points (k, l), as shown in the following equation 4.

[0108]

number

[0109] The combination of joint points (k, l) is predetermined. The combination of joint points (k, l) may be determined as a natural combination that indicates the skeleton of a person, or may be a combination of joint points selected randomly.

[0110] Next, the skeletal similarity calculation unit 46 acquires a vector C indicating the optical axis direction of the camera. It is assumed that the vector C has been measured in advance. Note that when the coordinate value vector P of the joint point is expressed in the camera coordinate system, it is expressed as C=(0,0,1).

[0111] Furthermore, the skeletal similarity calculation unit 46 calculates a vector C indicating the optical axis direction of the camera and a vector b representing the bones connecting the joint points using the following equation 5. k,l The angle θ k,l Calculate.

[0112]

number

[0113] The skeletal similarity calculation unit 46 calculates the above-mentioned vector b k,l Calculation of the vector C indicating the optical axis direction of the camera, and angle θ k,l Then, the skeleton similarity calculation unit 46 calculates the obtained angle θ k,l Arrange them in order to create the vector Θ.

[0114]

number

[0115] Furthermore, the skeletal similarity calculation unit 46 performs the above-described process on all the training data to create the vector Θ.

[0116] Next, the similarity calculation unit 44 calculates the similarity between the vectors Θ for each combination of image data and skeletal information as a skeletal similarity S i,j It is calculated as follows.

[0117]

number

[0118] The parameter update unit 47 calculates the similarity sim(T i ,I j ) and the skeletal similarity S i,j Then, the parameter update unit 47 updates the parameters of the second machine learning model 50 and the third machine learning model 51 so that the calculated difference becomes 0 or close to 0. Furthermore, before calculating the difference, the parameter update unit 47 updates the parameters of the second machine learning model 50 and the third machine learning model 51 so that the calculated difference becomes 0 or close to 0. i,j is normalized and the range of its values ​​is expressed as the similarity sim(T i ,I j ) range of values.

[0119] Parameter update will be specifically described with reference to Fig. 12. Fig. 12 is a diagram showing an example of parameter update processing in the learning model generation device shown in Fig. 9. In Fig. 12, the numerical values ​​shown in the matrix are the skeletal similarity S calculated by the skeletal similarity calculation unit 46. i,j The parameter update unit 47 updates the skeletal feature T i and image feature I j Similarity sim(T i ,I j The parameters of the first machine learning model 21 and the second machine learning model 22 are updated so that the values ​​of the first machine learning model 21 and the second machine learning model 22 are the corresponding values ​​on the matrix.

[0120] [Device operation] Next, the operation of the learning model generation device 40 will be described with reference to Fig. 13. Fig. 13 is a flow diagram showing an example of the operation of the learning model generation device shown in Fig. 9. In the following description, Figs. 9 to 12 will be referred to as appropriate. In addition, in the third embodiment, a learning model generation method is implemented by operating the learning model generation device 40. Therefore, the description of the learning model generation method in the third embodiment will be replaced by the following description of the operation of the learning model generation device 40.

[0121] 13, first, the training data acquisition unit 41 acquires training data from a database (step C1). Then, the training data acquisition unit 41 inputs the image data acquired as the training data to the image feature extraction unit 43, and inputs the skeletal information acquired as the training data to the feature extraction unit 42.

[0122] Next, when the skeleton information is input, the feature extraction unit 42 extracts skeleton features using the second machine learning model 22 (step C2). Furthermore, the feature extraction unit 42 outputs the extracted skeleton features to the similarity calculation unit 44.

[0123] Next, when the image data is input, the image feature extraction unit 43 extracts image features using the first machine learning model 21 (step C3). Furthermore, the image feature extraction unit 43 outputs the extracted image features to the similarity calculation unit 44.

[0124] Next, the similarity calculation unit 44 calculates the skeletal feature T extracted in step C2 for each combination of image data and skeletal information. i and the image feature I extracted in step C3 j Similarity sim(T i ,I j ) is calculated (step C4).

[0125] Next, the corresponding skeletal information acquisition unit 45 acquires, for each of the image data acquired in step C1, skeletal information (corresponding skeletal information) corresponding to the person in the image data (step C5).

[0126] Specifically, in step C5, the corresponding skeleton information acquisition unit 45 acquires meta information for each image data and meta information for each skeleton information. Then, the corresponding skeleton information acquisition unit 45 calculates the similarity sim(T i ,I j ) is calculated, the skeleton information (corresponding skeleton information) corresponding to the image data is identified, and the identified corresponding skeleton information is acquired.

[0127] Next, the skeletal similarity calculation unit 46 calculates the skeletal similarity S between the corresponding skeletal information and the skeletal information for each combination of image data and skeletal information. i,j is calculated (step C6).

[0128] Next, the parameter update unit 47 calculates the similarity sim(T i ,I j ) and the skeletal similarity S i,j (Step C7) Then, parameter update unit 47 uses the difference calculated in Step C7 to update the parameters of second machine learning model 50 and third machine learning model 51 (Step C8).

[0129] As described above, according to the third embodiment, image data and skeletal information can be used as training data to perform machine learning, i.e., parameter updating, for each of the second machine learning model 50 and the third machine learning model 51. Furthermore, by using the second machine learning model 51 and the second machine learning model 51, it becomes possible to determine a person's posture from image data.

[0130] [program] In the third embodiment, an example of the program is a program that causes a computer to execute steps C1 to C8 shown in Fig. 13. By installing and executing this program on a computer, a learning model generation device 40 and a learning model generation method can be realized. In this case, the processor of the computer functions as a training data acquisition unit 41, an image feature extraction unit 43, a feature extraction unit 42, a similarity calculation unit 44, a corresponding skeleton information acquisition unit 45, a skeleton similarity calculation unit 46, and a parameter update unit 47, and performs processing. Furthermore, examples of the computer include a general-purpose PC, a server computer, a smartphone, and a tablet terminal device.

[0131] In the third embodiment, the program may be executed by a computer system constructed by a plurality of computers. In this case, for example, each computer may function as one of the training data acquisition unit 41, the image feature extraction unit 43, the feature extraction unit 42, the similarity calculation unit 44, the corresponding skeleton information acquisition unit 45, the skeleton similarity calculation unit 46, and the parameter update unit 47.

[0132] [Physical configuration] Here, a computer that realizes the image generation device and the learning model generation device by executing the programs in each of the above-described embodiments will be described with reference to Fig. 14. Fig. 14 is a block diagram showing an example of a computer that realizes the image generation device and the learning model generation device.

[0133] 14, a computer 110 includes a CPU (Central Processing Unit) 111, a main memory 112, a storage device 113, an input interface 114, a display controller 115, a data reader / writer 116, and a communication interface 117. These components are connected to each other via a bus 121 so as to be able to communicate data with each other.

[0134] Furthermore, the computer 110 may include a GPU (Graphics Processing Unit) or an FPGA (Field-Programmable Gate Array) in addition to or instead of the CPU 111. In this aspect, the GPU or FPGA can execute the programs in the embodiments.

[0135] The CPU 111 loads a program in the embodiment, which is composed of a group of codes and stored in the storage device 113, into the main memory 112 and executes each code in a predetermined order to perform various calculations. The main memory 112 is typically a volatile storage device such as a DRAM (Dynamic Random Access Memory).

[0136] The program in the embodiment is provided in a state stored in a computer-readable recording medium 120. The program in the embodiment may be distributed over the Internet connected via the communication interface 117.

[0137] Specific examples of the storage device 113 include a hard disk drive and a semiconductor storage device such as a flash memory. The input interface 114 mediates data transmission between the CPU 111 and input devices 118 such as a keyboard and a mouse. The display controller 115 is connected to a display device 119 and controls the display on the display device 119.

[0138] The data reader / writer 116 mediates data transmission between the CPU 111 and the recording medium 120, reads programs from the recording medium 120, and writes processing results from the computer 110 to the recording medium 120. The communication interface 117 mediates data transmission between the CPU 111 and other computers.

[0139] Specific examples of the recording medium 120 include general-purpose semiconductor storage devices such as CF (Compact Flash (registered trademark)) and SD (Secure Digital), magnetic recording media such as flexible disks, or optical recording media such as CD-ROMs (Compact Disk Read Only Memory).

[0140] The image generating device and the learning model generating device can be realized not by a computer with a program installed, but by hardware corresponding to each part, for example, an electronic circuit. Furthermore, the image generating device and the learning model generating device may be realized in part by a program and in the remaining part by hardware. In the embodiments, the computer is not limited to the computer shown in FIG. 14.

[0141] Some or all of the above-described embodiments can be expressed by (Supplementary Note 1) to (Supplementary Note 15) described below, but are not limited to the following descriptions.

[0142] (Appendix 1) a feature extraction unit that extracts skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint that constitutes the skeleton; an image generation unit that inputs the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and generates an image corresponding to the skeleton by removing noise from the image using an output result from the machine learning model; An image generating device comprising:

[0143] (Appendix 2) The skeleton information includes at least one of information specifying coordinates indicating positions of joint points, information indicating a body shape, and information indicating a surface of a body that is the basis of the skeleton. 2. The image generating device of claim 1.

[0144] (Appendix 3) the feature extraction unit extracts skeletal features of the skeleton using a second machine learning model that has learned the relationship between skeletal information and the skeletal features through machine learning; 2. The image generating device of claim 1.

[0145] (Appendix 4) The parameters of the second machine learning model are: For each combination of sample skeletal information and corresponding image data, calculate a similarity between a feature of the corresponding image data and a feature extracted from the sample skeletal information; Furthermore, for each of the combinations, a similarity between the corresponding skeletal information corresponding to the person in the corresponding image data and the skeletal information is calculated as a skeletal similarity; Furthermore, for each of the combinations, a difference between the calculated similarity and the skeletal similarity is calculated, and the calculated difference is used to Has been updated, 4. The image generation device of claim 3.

[0146] (Appendix 5) a feature extraction step of extracting skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint constituting the skeleton; an image generation step of inputting the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and removing noise from the image using an output result from the machine learning model, thereby generating an image corresponding to the skeleton; An image generating method comprising:

[0147] (Appendix 6) The skeleton information includes at least one of information specifying coordinates indicating positions of joint points, information indicating a body shape, and information indicating a surface of a body that is the basis of the skeleton. 6. The image generation method of claim 5.

[0148] (Appendix 7) In the feature extraction step, a skeletal feature of the skeleton is extracted using a second machine learning model that has learned the relationship between skeletal information and the skeletal feature. 6. The image generation method of claim 5.

[0149] (Appendix 8) The parameters of the second machine learning model are: For each combination of sample skeletal information and corresponding image data, calculate a similarity between a feature of the corresponding image data and a feature extracted from the sample skeletal information; Furthermore, for each of the combinations, a similarity between the corresponding skeletal information corresponding to the person in the corresponding image data and the skeletal information is calculated as a skeletal similarity; Furthermore, for each of the combinations, a difference between the calculated similarity and the skeletal similarity is calculated, and the calculated difference is used to Has been updated, 8. The image generation method of claim 7.

[0150] (Appendix 9) On the computer, a feature extraction step of extracting skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint constituting the skeleton; an image generation step of inputting the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and removing noise from the image using an output result from the machine learning model, thereby generating an image corresponding to the skeleton; A program that executes.

[0151] (Appendix 10) The skeleton information includes at least one of information specifying coordinates indicating positions of joint points, information indicating a body shape, and information indicating a surface of a body that is the basis of the skeleton. 10. The program described in Appendix 9.

[0152] (Appendix 11) In the feature extraction step, a skeletal feature of the skeleton is extracted using a second machine learning model that has learned the relationship between skeletal information and the skeletal feature. 10. The program described in Appendix 9.

[0153] (Appendix 12) The parameters of the second machine learning model are: For each combination of sample skeletal information and corresponding image data, calculate a similarity between a feature of the corresponding image data and a feature extracted from the sample skeletal information; Furthermore, for each of the combinations, a similarity between the corresponding skeletal information corresponding to the person in the corresponding image data and the skeletal information is calculated as a skeletal similarity; Furthermore, for each of the combinations, a difference between the calculated similarity and the skeletal similarity is calculated, and the calculated difference is used to Has been updated, 11. The program described in Appendix 11. [Industrial Applicability]

[0154] As described above, according to the present disclosure, it is possible to specify a posture when generating an image. The present disclosure is useful for various systems that perform image generation. [Explanation of symbols]

[0155] 10 Image generation device (first embodiment) 11 Feature extraction unit 12 Image generation unit 13 Skeleton information acquisition unit 14 Time information generation section 15 Noise-added data output section 16 Noise estimation section 17 Noise subtraction section 20 Machine Learning Models 21 DNN 30 Learning model generation device (Embodiment 2) 31 Skeletal information acquisition unit 32 Feature extraction unit 33 Image acquisition unit 34 Time information generation section 35 Noise generation section 36 Noise estimation unit 37 Addition section 38 Loss calculation section 39 Parameter Update Section 40 Learning Model Generation Device (Third Embodiment) 41 Training data acquisition section 42 Feature extraction unit 43 Image feature extraction unit 44 Similarity calculation unit 45 Corresponding skeleton information acquisition unit 46 Skeleton similarity calculation unit 47 Parameter Update Section 110 Computer 111 CPU 112 main memory 113 Storage device 114 Input Interface 115 Display Controller 116 Data Reader / Writer 117 Communication Interface 118 Input Devices 119 Display Device 120 Recording Media 121 Bus

Claims

1. a feature extraction unit that extracts skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint that constitutes the skeleton; an image generation unit that inputs the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and generates an image corresponding to the skeleton by removing noise from the image using an output result from the machine learning model; An image generating device comprising:

2. The skeleton information includes at least one of information specifying coordinates indicating positions of joint points, information indicating a body shape, and information indicating a surface of a body that is the basis of the skeleton. The image generating device of claim 1 .

3. the feature extraction unit extracts skeletal features of the skeleton using a second machine learning model that has learned by machine learning the relationship between skeletal information and the skeletal features; The image generating device of claim 1 .

4. The parameters of the second machine learning model are: For each combination of sample skeletal information and corresponding image data, calculate a similarity between a feature of the corresponding image data and a feature extracted from the sample skeletal information; Furthermore, for each of the combinations, a similarity between the corresponding skeletal information corresponding to the person in the corresponding image data and the skeletal information is calculated as a skeletal similarity; Furthermore, for each of the combinations, a difference between the calculated similarity and the skeletal similarity is calculated, and the calculated difference is used to Has been updated, The image generating device according to claim 3 .

5. a feature extraction step of extracting skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint constituting the skeleton; an image generation step of inputting the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and removing noise from the image using an output result from the machine learning model, thereby generating an image corresponding to the skeleton; An image generating method comprising:

6. The skeleton information includes at least one of information specifying coordinates indicating positions of joint points, information indicating a body shape, and information indicating a surface of a body that is the basis of the skeleton. The image generating method according to claim 5 .

7. In the feature extraction step, a skeleton feature of the skeleton is extracted using a second machine learning model that has machine-learned the relationship between skeleton information and the skeleton feature. The image generating method according to claim 5 .

8. The parameters of the second machine learning model are: For each combination of sample skeletal information and corresponding image data, calculate a similarity between a feature of the corresponding image data and a feature extracted from the sample skeletal information; Furthermore, for each of the combinations, a similarity between the corresponding skeletal information corresponding to the person in the corresponding image data and the skeletal information is calculated as a skeletal similarity; Furthermore, for each of the combinations, a difference between the calculated similarity and the skeletal similarity is calculated, and the calculated difference is used to Has been updated, The image generating method according to claim 7.

9. On the computer, a feature extraction step of extracting skeletal feature values ​​of the skeleton from skeletal information that identifies the positions of each joint constituting the skeleton; an image generation step of inputting the extracted skeletal feature amount and a noise image into a machine learning model that estimates noise added to an image, and removing noise from the image using an output result from the machine learning model, thereby generating an image corresponding to the skeleton; A program that executes.

10. The skeleton information includes at least one of information specifying coordinates indicating positions of joint points, information indicating a body shape, and information indicating a surface of a body that is the basis of the skeleton. The program according to claim 9.

11. In the feature extraction step, a skeleton feature of the skeleton is extracted using a second machine learning model that has machine-learned the relationship between skeleton information and the skeleton feature. The program according to claim 9.

12. The parameters of the second machine learning model are: For each combination of sample skeletal information and corresponding image data, calculate a similarity between a feature of the corresponding image data and a feature extracted from the sample skeletal information; Furthermore, for each of the combinations, a similarity between the corresponding skeletal information corresponding to the person in the corresponding image data and the skeletal information is calculated as a skeletal similarity; Furthermore, for each of the combinations, a difference between the calculated similarity and the skeletal similarity is calculated, and the calculated difference is used to Has been updated, The program according to claim 11.

Images