Posture data generation device and behavior recognition device

Abstract

To provide a posture data generation device and an action recognition device for generating posture data which satisfies an arbitrarily set restriction condition and regenerates a natural posture of a human.SOLUTION: A posture data generation device 1 includes: a decoder 10 as a posture generation unit being a posture generation model having learned by using posture data on postures that a human can take, as a learning model for generating posture data on a human from a potential variable; error calculation means 14 for calculating an error between a position of a specific site of regeneration posture data generated by inputting a potential variable to decoder 10 and a target value showing a target position of the specific site; and potential variable update means 16 for updating the value of a potential variable on the basis of the error. The decoder 10 generates target posture data by inputting the potential variable updated by the potential variable update menas 16.SELECTED DRAWING: Figure 2

Description

[[Technical Field]]

[0001] The present invention relates to a posture data generation device and a behavior recognition device that recognizes the behavior of a person. [[Background Art]]

[0002] Patent Document 1 below describes a behavior recognition method for recognizing a person's behavior based on a posture detected from a captured image of the person. [[Prior Art Documents]] [[Patent Documents]]

[0003] [[Patent Document 1]] Japanese Unexamined Patent Application Publication No. 2022-189456 [[Summary of the Invention]] [[Problem to be Solved by the Invention]]

[0004] There are countless patterns for the posture of a person when performing a certain behavior. For example, when recognizing the "posture of taking a product from a shelf", not only an upright posture but also a posture of taking the product by squatting or bending over can be considered. Therefore, in order to recognize a person's behavior based on the person's posture, numerous posture patterns are required, and it is necessary to prepare a large amount of training data. Additionally, when generating posture data of a person from a captured image of the person, an unnatural posture that is impossible from a skeletal perspective may sometimes be estimated. In such cases, correction to natural posture data that a human can adopt is required. The present invention has been made in view of these problems, and an object of the present invention is to generate posture data that reproduces the natural posture of a human while satisfying arbitrarily set constraint conditions. [[Means for Solving the Problem]]

[0005] A posture data generation device according to one embodiment of the present invention comprises: a posture generation unit, which is a posture generation model trained using posture data of postures that a person can take as a learning model for generating posture data of a person from latent variables; an error calculation unit, which calculates the error between the position of a specific part of the reproduced posture data generated by inputting latent variables to the posture generation unit and a target value indicating the target position of the specific part; and an update unit, which updates the value of the latent variables based on the error. The posture generation unit generates posture data by inputting the latent variables updated by the update unit. [Effects of the Invention]

[0006] According to the present invention, posture data that reproduces a natural human posture can be generated while satisfying arbitrarily set constraints. [Brief explanation of the drawing]

[0007] [Figure 1] This is a schematic diagram showing an example of the hardware configuration of a posture data generation device or an action recognition device according to an embodiment of the present invention. [Figure 2] This is a block diagram of an example of the functional configuration of the posture data generation device of the first embodiment. [Figure 3] This is a schematic diagram of an example of posture data. [Figure 4] This is a block diagram showing an example of the functional configuration of a variational autoencoder (VAE). [Figure 5] This is a block diagram showing an example of VAEGAN's functional configuration. [Figure 6] (a) is a schematic diagram of the latent variables in the latent variable space, (b) is a schematic diagram of the regenerated pose data and target values ​​generated from the initial values ​​of the latent variables, and (c) is a schematic diagram of the regenerated pose data generated from the updated latent variables. [Figure 7] (a) to (c) are schematic diagrams of posture data for various positions when opening a door. [Figure 8] This is a schematic diagram of an example of the functional configuration of the behavior recognition device according to the first embodiment. [Figure 9] This is a flowchart of an example of a posture data generation method according to the first embodiment. [Figure 10] This is a block diagram of an example of the functional configuration of the posture data generation device of the second embodiment. [Modes for carrying out the invention]

[0008] Embodiments of the present invention will be described below with reference to the drawings. The embodiments of the present invention described below are illustrative examples of devices and methods for realizing the technical concept of the present invention, and the technical concept of the present invention is not limited to the structure, arrangement, etc., of the components described below. Various modifications can be made to the technical concept of the present invention within the technical scope defined by the claims described in the patent claims.

[0009] (First Embodiment) Figure 1 is a schematic diagram showing an example of a hardware configuration for realizing a posture data generation device 1 and an action recognition device 30 according to an embodiment of the present invention. The posture data generation device 1 and the action recognition device 30 consist of an operation input unit 2, a file input / output unit 3, a storage unit 4, a control unit 5, and an output unit 6. Of these, the file input / output unit 3, the storage unit 4, and the control unit 5 can be implemented using a computer, while the operation input unit 2 and the output unit 6 can be implemented as peripheral devices for that computer.

[0010] The operation input unit 2 is a user interface such as a keyboard or mouse, and is used by the user to input data, etc. The operation input unit 2 is connected to the control unit 5 and converts the user's operations into operation signals and outputs them to the control unit 5.

[0011] The file input / output unit 3 is a DVD (Digital Versatile Disc) drive, a USB (Universal Serial Bus) interface, a network interface, etc., with one end connected to an external device, recording media, network, etc. (not shown), and the other end connected to the control unit 5. It inputs data as a file to the control unit 5 and outputs data as a file from the control unit 5.

[0012] The storage unit 4 is a memory device such as ROM (Read Only Memory), RAM (Random Access Memory), and stores various programs and various data. The storage unit 4 is connected to the control unit 5, and inputs and outputs such information to and from the control unit 5.

[0013] The control unit 5 is constituted by an arithmetic device such as a CPU (Central Processing Unit), a DSP (Digital Signal Processor), an MCU (Micro Control Unit) or the like. The control unit 5 is connected to the storage unit 4, reads and executes a program from the storage unit 4 to operate as various processing units, and causes the storage unit 4 to store and read various data. The control unit 5 is also connected to the operation input unit 2, the file input / output unit 3 and the output unit 6, acquires data input by a user operating the operation input unit 2, acquires and outputs data as a file from the outside via the file input / output unit 3, outputs a data file of calculation results based on the acquired data from the file input / output unit 3, and / or outputs data of the calculation results from the output unit 6.

[0014] The output unit 6 is a display device such as a liquid crystal display or a CRT (Cathode Ray Tube) display that outputs calculation results obtained by the control unit 5, a projector, a printer, or the like. Furthermore, the behavior recognition device 30 and the posture data generation device 1 according to a second embodiment described later may include an image input unit 7 that acquires a captured image obtained by capturing a person and inputs the captured image to the control unit 5. For example, the image input unit 7 may be an imaging device (camera). Alternatively, the behavior recognition device 30 and the posture data generation device 1 according to the second embodiment described later may acquire an image data file of a captured image of a person from the file input / output unit 3.

[0015] Figure 2 is a block diagram showing an example of the functional configuration of the posture data generation device 1 according to the first embodiment. The posture data generation device 1 includes a decoder 10, an initial value setting means 11, an input switching means 12, a target value setting means 13, an error calculation means 14, a differential value calculation means 15, and a latent variable updating means 16. The control unit 5 in Fig. 1 functions as an initial value setting means 11, an input switching means 12, an error calculation means 14, a differential value calculation means 15, and a latent variable updating means 16. The operation input unit 2 and the file input / output unit 3 function as a target value setting means 13.

[0016] The posture data generation device 1 includes a decoder 10 as a learning model that generates human posture data from latent variables, and as a posture generation model trained using posture data of postures that a human can take. For example, the decoder 10 may be a decoder configured by causing a Variational AutoEncoder (VAE) to learn posture data of postures that a human can take. The decoder 10 is an example of the "posture generation unit" recited in the claims. Fig. 3 is a schematic diagram of an example of posture data x. For example, the posture data x includes joint points (key points) J, which are feature points of a human posture, and links L, which are connection information connecting the joint points J to each other.

[0017] A variational autoencoder (hereinafter referred to as "VAE") is a generative model that generates new data having features of input data by reducing the number of dimensions of the input data and reproducing original data from the reduced number of dimensions. Fig. 4 is a block diagram of an example of the functional configuration of a VAE 20A trained on posture data of postures that a human can take. The VAE 20A includes a posture data input means 20, an encoder 21, a latent variable setting means 22, and the decoder 10.

[0018] The posture data input means 20 acquires posture data x generated by estimating a human posture from a captured image of a human and inputs the posture data x to the encoder 21. In the following description, the posture data x input to the encoder 21 may be referred to as "input posture data x". For example, the posture data input means 20 may include an imaging device (camera) that photographs a person and generates an image, and a posture estimation device that generates input posture data x from the image. Alternatively, the posture data input means 20 may obtain input posture data x from an existing database.

[0019] The encoder 21 processes the input posture data x into a latent variable z with a probability distribution N(μ,σ) of a dimension smaller than the spatial dimension of the input posture data x. 2 Convert to ). In the following explanation, the space that the latent variable z can take is sometimes referred to as the "latent variable space". The latent variable setting means 22 sets the probability distribution N(μ,σ) estimated by the encoder 21. 2 The latent variable z is sampled according to ). Decoder 10 reconstructs (restores) attitude data x* from the latent variable z. In the following description, the attitude data x* reconstructed from the latent variable z by decoder 10 may be referred to as "reconstructed attitude data x*".

[0020] When training VAE20A, the reconstruction error calculation means 23 calculates the reconstruction error L1 based on the error between the input posture data x and the regenerated posture data x*. Furthermore, the KL divergence loss calculation means 24 calculates the probability distribution N(μ,σ) estimated by the encoder 21. 2 Calculate the Kullback-Leibler (KL) divergence loss L2 between the ) and the normal distribution.

[0021] The first learning means 25 trains the encoder 21 and decoder 10 so that the loss function L = L1 + L2, which is the sum of the reconstruction error L1 and the KL divergence loss L2, is minimized. For example, the parameters of the neural networks that make up the encoder 21 and decoder 10 are adjusted so that the loss function L is minimized.

[0022] The posture data generation device 1 may use a VAEGAN decoder instead of a VAE 20A decoder. VAEGAN is a generative model that combines VAE and Generative Adversarial Networks (GANs). That is, the decoder 10 may be a decoder constructed by training a VAEGAN with posture data x of possible postures that a person can take.

[0023] Figure 5 is a block diagram of an example of the functional configuration of VAEGAN20B. In addition to the configuration of the VAE shown in Figure 4, VAEGAN20B includes a pseudo-variable setting means 26 and an identification means 27. The pseudo-variable setting means 26 samples a pseudo-latent variable zp from the latent variable space according to a normal distribution N(0,I). The decoder 10 reconstructs (restores) the pose data xp from the pseudo-latent variable zp. In the following description, the pose data xp reconstructed from the pseudo-latent variable zp may be referred to as "pseudo-pose data xp".

[0024] The identification means 27 is a classifier that has been trained to distinguish between input posture data x and pseudo-posture data xp. When training the discrimination means 27, a first loss function L3, which is the discriminator loss when the discrimination means 27 distinguishes between input posture data x and pseudo-posture data xp, is calculated. The second learning means 28 adjusts the parameters of the discrimination means 27 so that the first loss function L3 is minimized, thereby training the discrimination means 27 to be able to distinguish between input posture data x and pseudo-posture data xp.

[0025] VAEGAN20B trains the decoder 10 so that the identification means 27 can no longer distinguish between the input posture data x and the reproduced posture data x*. When training the decoder 10, the identification means 27 distinguishes between input posture data x and reproduced posture data x*. The second loss function L4, which is the classifier loss when the input posture data x and reproduced posture data x* are distinguished, is then calculated. The second learning means 28 adjusts the parameters of the decoder 10 so that the second loss function L4 is maximized, thereby training the decoder 10 so that the identification means 27 can no longer distinguish between input posture data x and reproduced posture data x*.

[0026] Furthermore, the posture data generation device 1 may be equipped with a Generative Adversarial Network (GAN) generator as a posture model that generates posture data xp from a pseudo-latent variable zp, instead of the decoder 10 of VAE20A in Figure 4 and the decoder 10 of VAEGAN20B in Figure 5. The GAN generator is a generator in which a discriminator that distinguishes between reference posture data generated as posture data of postures that a person can take and reproduced posture data generated by the generator from latent variables is trained so that it cannot distinguish between reference posture data and reproduced posture data.

[0027] Refer to Figure 2. The initial value setting means 11 sets the initial value z0 of a latent variable for generating the starting point playback posture data x*. Figure 6(a) schematically represents the initial value z0 of a latent variable set in the latent variable space. For example, the initial value setting means 11 may set a latent variable randomly selected in the latent variable space as the initial value z0.

[0028] Refer to Figure 2. The input switching means 12 switches the latent variable z input to the decoder 10 between the initial value z0 of the latent variable set by the initial value setting means 11 and the latent variable z updated by the latent variable update means 16. Specifically, when the initial value setting means 11 sets the initial value z0 of the latent variable and then generates the first playback posture data x*, the input switching means 12 inputs the initial value z0 of the latent variable set by the initial value setting means 11 to the decoder 10. After generating the first playback posture data x*, when the latent variable update means 16 updates the latent variable z as described later, the input switching means 12 inputs the latent variable z updated by the latent variable update means 16 to the decoder 10.

[0029] The decoder 10 generates regenerated attitude data x* from the latent variable z input from the input switching means 12. The target value setting means 13 sets the target value of the posture data to be generated by the posture data generation device 1. For example, the target value setting means 13 may set the target positions of some or all of the joint points J of the posture data x shown in Figure 3 as the target value. Figure 6(b) is a schematic diagram of the regenerated posture data x* generated from the initial value z0 of the latent variable and the target value. In the example in Figure 6(b), the target position Jt of the joint point J of the right hand in the regenerated posture data x* is set as the target value.

[0030] The error calculation means 14 calculates the error e between the target value set by the target value setting means 13 and the playback posture data x*. For example, the error calculation means 14 may calculate the error e according to the difference between the target position of one or more joint points set as the target value and the position of the joint point in the playback posture data x*. For example, if the target position of a single joint point is set as the target value, the difference between the target position and the position of the joint point in the playback posture data x* may be calculated as the error e. If the target positions of multiple joint points are set as the target value, the sum of the differences between the target positions and the positions of the joint points in the playback posture data x* may be calculated as the error e.

[0031] Refer to Figure 2. The differential value calculation means 15 calculates the differential value in the latent variable space using the backpropagation method based on the difference calculated by the error calculation means 14. The latent variable update means 16 updates the latent variable z with the derivative value calculated by the derivative value calculation means 15. Subsequently, the input switching means 12 inputs the latent variable z updated by the latent variable update means 16 to the decoder 10. The decoder 10 generates regenerated pose data x* from the latent variable z updated by the latent variable update means 16.

[0032] Similarly, by repeating the generation of the regenerated posture data x*, the calculation of the error e, the calculation of the derivative value, and the updating of the latent variable z, the latent variable z is sequentially updated from the initial value z0 to z1, z2, and so on, as shown in Figure 6(a). As a result, the regenerated posture data x* generated by the decoder 10 approaches the target value, as shown in Figure 6(c). In the example in Figure 6(c), the position of the joint point J of the right hand in the regenerated posture data x* can be brought closer to the target position Jt set as the target value.

[0033] This invention uses a decoder 10 of VAE20A or VAEGAN20B that has been trained on posture data of postures that a person can take. Therefore, the decoder 10 can generate reconstructed posture data x* that reproduces a natural human posture. By setting arbitrary target values ​​as constraints for the reconstructed posture data x*, posture data can be generated that reproduces a natural human posture while satisfying arbitrary constraints.

[0034] The posture data generation device 1 terminates the update of the latent variable z and outputs the regenerated posture data x* when it satisfies predetermined calculation termination conditions. For example, the posture data generation device 1 may determine that the predetermined termination conditions are satisfied when the error e calculated by the error calculation means 14 falls below a predetermined threshold. Alternatively, the posture data generation device 1 may determine that the predetermined termination conditions are satisfied when the magnitude of the derivative calculated by the derivative value calculation means 15 (for example, the absolute value of the derivative or the length of the derivative vector) falls below a predetermined threshold. In other words, the predetermined termination conditions may be determined to be satisfied when the magnitude of the difference between the latent variable z before and after the update, or the difference between the regenerated posture data x* generated from the latent variable z before the update and the regenerated posture data x* generated from the latent variable z after the update falls below a predetermined threshold.

[0035] As described above, the posture data generation device 1 can generate regenerated posture data x* that reproduces a natural human posture while satisfying arbitrarily set constraints. Therefore, by setting the same target value and generating playback posture data x* from different initial values ​​z0 of latent variables (for example, the initial value setting means 11 obtains different initial values ​​z0 of latent variables by repeatedly executing a process of randomly generating and setting the initial values ​​z0 of latent variables), it is possible to generate multiple playback posture data x* of people who take different natural postures while satisfying the same constraints.

[0036] According to the posture data generation device 1 of this embodiment, it is possible to generate posture data for various patterns of a person when they are performing the same action, for example. Figures 7(a) to 7(c) show examples of multiple different reconstructed posture data x for opening door D. There are countless patterns for the posture of opening door D. Therefore, by fixing the target position of the joint point J of the hand opening door D as the target value and generating reconstructed posture data x* from different initial values ​​z0 of latent variables, it is possible to prepare many multiple reconstructed posture data x* of a person who takes a natural posture while allowing freedom in all other aspects except the position of the hand opening door D.

[0037] By preparing numerous patterns of posture data of individuals performing the same action as training data, and training a classifier to increase the similarity of posture data of individuals performing the same action, the recognition accuracy of an action recognition device that recognizes human actions can be improved.

[0038] Figure 8 is a schematic diagram of an example of the functional configuration of the behavior recognition device 30 of the first embodiment. The behavior recognition device 30 comprises a person image acquisition means 31, a posture data generation means 32, a feature extraction means 33, and a recognition means 34. The image input unit 7 or file input / output unit 3 in Figure 1 functions as a person image acquisition means 31. The control unit 5 functions as a posture data generation means 32, a feature extraction means 33, and a recognition means 34.

[0039] The person image acquisition means 31 acquires an image of a person. For example, the person image acquisition means 31 may have an imaging device that generates an image of a person. Alternatively, for example, the person image acquisition means 31 may acquire an image data file of the image of the person. The posture data generation means 32 generates posture data by estimating the posture of a person based on the captured image acquired by the person image acquisition means 31.

[0040] The feature extraction means 33 extracts features from the pose data generated by the pose data generation means 32. For example, the feature extraction means 33 may extract features from the pose data by inputting the pose data into a feature extraction model modeled on a CNN composed of multiple layers of a network used in deep learning.

[0041] The recognition means 34 recognizes the actions of a person in an image captured by the person image acquisition means 31 based on the features extracted by the feature extraction means 33. For example, the feature extraction means 33 may recognize the actions of a person in an image based on the similarity between the template features of each action (class) that have been trained in advance and the features extracted by the feature extraction means 33. The recognition means 34 may be modeled by a CNN composed of multiple layers of a network used in deep learning, for example. The feature extraction means 33 and the recognition means 34 form an action recognition model that recognizes a person's actions based on posture data.

[0042] When training the behavior recognition device 30, the posture data generation device 1 generates posture data for multiple patterns of people performing the same action, and stores this data as training data in the training data storage means 35. The third learning means 36 trains the feature extraction means 33 and the recognition means 34 by using the training posture data stored in the learning data storage means 35 as training data.

[0043] For example, the template features of the feature extraction means 33 and the recognition means 34 may be trained so that the similarity of the features of multiple different posture data of a person performing the same action, generated by the posture data generation device 1, is increased. This allows, for example, multiple different posture data sets in which the position of a specific body part is approximately the same (for example, multiple different posture data sets in which the position of a specific body part is fixed) to be used as training data, and the action recognition device 30 to learn a specific action in which the position of a specific body part is approximately the same (for example, a specific action characterized by the position of a specific body part). In the examples in Figures 7(a) to 7(c), the action recognition device 30 can be taught an action characterized by the position of the joint points of the right hand.

[0044] Figure 9 is a flowchart of an example of the posture data generation method of the first embodiment. In step S1, the target value setting means 13 sets the target value of the posture data to be generated by the posture data generation device 1. In step S2, the initial value setting means 11 sets the initial values ​​of the latent variables for generating the starting point playback posture data x*.

[0045] In step S3, the decoder 10 generates regenerated pose data x* from the latent variables. In step S4, the error calculation means 14 calculates the error e between the target value set by the target value setting means 13 and the playback posture data x*. In step S5, the differential value calculation means 15 calculates the differential value in the latent variable space using the backpropagation method based on the difference calculated by the error calculation means 14.

[0046] In step S6, the latent variable update means 16 updates the latent variable z with the derivative value calculated by the derivative value calculation means 15. In step S7, the posture data generation device 1 determines whether or not a predetermined calculation termination condition is met. If the calculation termination condition is not met (step S7:N), the process returns to step S3. If the predetermined calculation termination condition is met (step S7:Y), the process terminates.

[0047] (Second Embodiment) The posture data generation device 1 of the second embodiment generates posture data of a person from an image of the person captured by an imaging device such as a camera. When generating posture data from captured images of a person, the generated posture data may result in an unnatural posture that is skeletally impossible due to factors such as joint point detection errors and imaging device calibration errors.

[0048] Therefore, the posture data generation device 1 of the second embodiment uses the posture data generated from the captured image as a target value, and corrects the posture data generated from the captured image by generating the reproduced posture data x* using a latent variable z that is updated based on the error between the reproduced posture data x* of the decoder 10 and the target value. This makes it possible to correct the posture data generated from the captured image of a person to a natural posture.

[0049] The posture data generation device 1 of the second embodiment includes, in addition to the configuration of the posture data generation device 1 of the first embodiment, a person image acquisition means 40, a posture data generation means 41, and a user input receiving means 42. The image input unit 7 or file input / output unit 3 in Figure 1 functions as the person image acquisition means 40. The control unit 5 functions as the posture data generation means 41. The operation input unit 2 and the output unit 6 function as the user input receiving means 42.

[0050] The person image acquisition means 40 acquires captured images of a person. For example, the person image acquisition means 40 may have an imaging device that generates captured images of a person. Alternatively, for example, the person image acquisition means 40 may acquire an image data file of the captured images of a person. For example, the person image acquisition means 40 may acquire multiple captured images of the same person taken simultaneously by multiple imaging devices from different viewpoints.

[0051] The posture data generation means 41 generates posture data by estimating the posture of a person based on the captured images acquired by the person image acquisition means 40. When the person image acquisition means 40 acquires multiple captured images of the same person simultaneously, the posture data generation means 41 estimates the person's three-dimensional posture so that the joint points detected from each captured image are consistent. In the following description, the posture data generated by the posture data generation means 41 may be referred to as "reference posture data xr".

[0052] The target value setting means 13 sets the reference attitude data xr set by the attitude data generation means 41 as the target value for the regenerated attitude data x* generated by the decoder 10. For example, the position of one or more joint points in the reference posture data xr is set as the target position of each joint point in the regenerated posture data x* that corresponds to these joint points. For example, the target value setting means 13 may set the positions of all joint points in the reference posture data xr as the target position of each joint point in the regenerated posture data x* that corresponds to these joint points.

[0053] Alternatively, the target value setting means 13 may set some of the joint points of the reference posture data xr as target values ​​for the regenerated posture data x*. For example, the positions of some of the joint points of the reference posture data xr may be set as target positions for the joint points of the regenerated posture data x* that correspond to these joint points. For example, the posture data generation means 41 may set only the joint points of the parts visible in the captured image as target values ​​from the posture data estimated from the captured image. For example, the target value setting means 13 may estimate the parts visible in the captured image and the parts not visible in the captured image (for example, parts hidden by the person's own body or obstacles) based on the captured image acquired by the person image acquisition means 40, and set only the parts visible in the captured image as target values. For example, only the parts visible in the captured image may be selected, and the position of the joint points of the selected parts may be set as the target position of the joint points of the regenerated posture data x* corresponding to the joint points.

[0054] Alternatively, the posture data generation means 41 may generate reference posture data xr and calculate a score for each part (e.g., reference point) of the reference posture data xr. For example, the posture data generation means 41 may use existing posture estimation software such as OpenPose to calculate the accuracy (confidence level) of position detection of parts of the reference posture data xr as a score. The target value setting means 13 may set only the parts of the reference posture data xr whose scores are equal to or greater than a threshold as target values. For example, only the parts whose scores are equal to or greater than a threshold may be selected, and the position of the joint point of the selected part may be set as the target position of the joint point of the regenerated posture data x* corresponding to the joint point.

[0055] Alternatively, for example, the user input receiving means 42 may receive user input to select one or more parts in the reference posture data xr generated by the posture data generation means 41. For example, the user input receiving means 42 may display the reference posture data xr on the display device of the output unit 6 and receive input signals from the operation input unit 2 (e.g., keyboard, mouse, etc.) to select one or more parts (e.g., joint points) of the displayed reference posture data xr. The target value setting means 13 may set only the parts selected by the user via the user input receiving means 42 as target values. For example, the position of the joint point of the selected part may be set as the target position of the joint point in the regenerated posture data x* corresponding to the joint point.

[0056] (Effects of the embodiment) (1) The posture data generation device 1 comprises a decoder 10, which is a posture generation model trained using posture data of postures that a person can take, as a learning model for generating a person's posture data from latent variables; an error calculation means 14 that calculates the error between the position of a specific part of the reproduced posture data generated by inputting latent variables to the decoder 10 and a target value indicating the target position of that specific part; and a latent variable update means 16 that updates the value of the latent variables based on the error. The decoder 10 generates target posture data by inputting the latent variables updated by the latent variable update means 16. For example, the latent variable update means 16 may update the value of the latent variable according to the backpropagation method.

[0057] Traditionally, when training behavior recognition models that recognize human actions based on human posture, it was necessary to manually create training data using posture data obtained from motion capture of people. This required people to act out as many patterns as possible for the action to be recognized, resulting in high costs for generating training data, and there was a risk of insufficient training data due to the limited number of patterns that people could come up with.

[0058] This invention enables the automatic generation of a large amount of posture data of a person performing a specific action, thereby allowing for the efficient generation of training data for an action recognition model. Furthermore, when generating posture data from captured images of a person, it is possible to estimate unnatural postures that are skeletal and impossible. According to the present invention, posture data generated from captured images can be corrected to result in a natural posture.

[0059] (2) The posture data generation device 1 may include a posture data generation means 41 that generates a person's posture data as reference posture data from an image of the person captured by a camera. The error calculation means 14 may calculate the error between the reproduced posture data and the reference posture data using the position of one or more specific body parts in the reference posture data as the target value. This allows the posture data generated from captured images to be corrected to result in a more natural posture.

[0060] (3) The posture data generation device 1 may include a target value setting means 13 that identifies a reference part and its position from the camera's captured image of a person's body partially photographed by the camera, and sets the reference part and its position as a target value. The error calculation means 14 may calculate the difference between the position of the part corresponding to the reference part in the reproduced posture data and the position of the reference part as the error. This allows for the restoration of parts of the body that are not visible in the captured image, restoring them to a natural posture.

[0061] (4) The posture data generation device 1 may include a posture data generation means 41 that generates a person's posture data as reference posture data from an image of the person captured by a camera and calculates a score representing the accuracy of position detection of each part of the reference posture data, and a target value setting means 13 that sets the reference part and its position as target values ​​for which the score is equal to or greater than a threshold. The error calculation means 14 may calculate the difference between the position of the part corresponding to the reference part in the reproduced posture data and the position of the reference part as the error. This allows for correction of posture data to achieve a more natural posture by adjusting the position of areas with low scores (e.g., areas with low reliability).

[0062] (5) The posture data generation device 1 may include a posture data generation means 41 that generates a person's posture data as reference posture data from an image of the person captured by a camera, and a user input receiving means 42 that receives user input to select one or more body parts in the reference posture data as target values. This allows posture data to be corrected to ensure a natural posture even for body parts that the user has not selected.

[0063] (6) The decoder 10 may be a variational autoencoder decoder that has been trained on pose data of possible postures that a person can take. This makes it possible to generate posture data for natural postures that a person can adopt.

[0064] (7) The decoder 10 may be a decoder in which a classifier trained to distinguish between first attitude data input to the encoder of a variational autoencoder and second attitude data generated by inputting latent variables sampled from a predetermined normal distribution to the decoder 10 cannot distinguish between third attitude data generated by inputting the output of the encoder to which the first attitude data was input to the decoder 10 and the first attitude data. This makes it possible to generate posture data for natural postures that a person can adopt.

[0065] (8) Instead of the decoder 10, a generator of a generative adversarial network trained with pose data of possible postures a person can take may be used as a pose generation model trained with pose data of possible postures a person can take, in order to generate pose data of a person from latent variables. This makes it possible to generate posture data for natural postures that a person can adopt.

[0066] (9) The posture data generation device 1 may repeatedly generate the reproduced posture data with the decoder 10, calculate the error with the error calculation means 14, and update the value of the latent variable with the latent variable update means 16 until the magnitude of the error between the position of a specific part of the reproduced posture data obtained from the updated latent variable and the target value is within a predetermined value. The reproduced posture data obtained when the latent error between the position of a specific part of the reproduced posture data and the target value is within the predetermined value may be acquired as the target posture data. This makes it possible to generate posture data that reproduces a natural human posture while ensuring that the position of specific body parts is sufficiently close to the target value.

[0067] (10) The behavior recognition device 30 includes a behavior recognition model (feature extraction means 33 and recognition means 34) that recognizes a person's behavior based on posture data. The behavior recognition model may be a learning model that has learned a specific behavior of a person whose specific body part is in substantially the same position, using a plurality of different posture data generated by the posture data generation device 1 based on a plurality of different latent variables and target values. This allows for the generation of numerous posture data patterns of individuals performing specific actions as training data, enabling the action recognition model of the action recognition device 30 to be trained. [Explanation of Symbols]

[0068] 1...Attitude data generation device, 2...Operation input unit, 3...File input / output unit, 4...Storage unit, 5...Control unit, 6...Output unit, 7...Image input unit, 10...Decoder, 11...Initial value setting means, 12...Input switching means, 13...Target value setting means, 14...Error calculation means, 15...Differential value calculation means, 16...Latent variable update means, 20...Attitude data input means, 20A...Variational autoencoder, 20B...VAEGAN, 21...Encoder, 22...Latent variable setting means 23... Reconstruction error calculation means, 24... KL divergence loss calculation means, 25... First learning means, 26... Pseudo-variable setting means, 27... Identification means, 28... Second learning means, 30... Action recognition device, 31... Person image acquisition means, 32... Posture data generation means, 33... Feature extraction means, 34... Recognition means, 35... Learning data storage means, 36... Third learning means, 40... Person image acquisition means, 41... Posture data generation means, 42... User input reception means

Claims

1. As a learning model that generates human posture data from latent variables, the posture generation unit is a posture generation model that has been trained using posture data of postures that a person can take, An error calculation unit calculates the error between the position of a specific part of the reproduced posture data generated by inputting latent variables into the posture generation unit and a target value indicating the target position of the specific part. An update unit that updates the value of the latent variable based on the aforementioned error, Equipped with, The posture data generation device is characterized in that the posture generation unit generates target posture data by inputting the latent variables updated by the update unit.

2. The attitude data generation device according to claim 1, characterized in that the update unit updates the value of the latent variable according to the backpropagation method.

3. The system includes a posture data generation unit that generates posture data of a person from an image of the person captured by a camera, as reference posture data. The error calculation unit calculates the error between the regenerated posture data and the reference posture data, using the position of one or more of the specified parts in the reference posture data as the target value. The posture data generation device according to feature 1 or 2.

4. The system includes a target value setting unit that identifies a reference part and its position from the image captured by the camera of a person's body partially photographed by the camera, and sets the reference part and its position as the target value. The error calculation unit calculates the error as the difference between the position of the part corresponding to the reference part in the playback posture data and the position of the reference part. The posture data generation device according to feature 1 or 2.

5. A posture data generation unit generates posture data of a person from an image of the person captured by a camera as reference posture data, and calculates a score representing the accuracy of position detection for each part of the reference posture data. The system includes a target value setting unit that sets the reference part and its position, where the score is equal to or greater than a threshold, as the target value. The error calculation unit calculates the error as the difference between the position of the part corresponding to the reference part in the playback posture data and the position of the reference part. The posture data generation device according to feature 1 or 2.

6. The system includes a posture data generation unit that generates posture data of a person from an image of the person captured by a camera, as reference posture data. An operation reception unit that receives user input to select one or more of the specified body parts in the reference posture data as the target value, The posture data generation device according to claim 1 or 2, characterized by comprising the following:

7. The posture data generation device according to claim 1 or 2, characterized in that the posture generation unit is a decoder of a variational autoencoder that has been trained with posture data of postures that a person can take.

8. The attitude data generation device according to claim 7, characterized in that the decoder is a decoder in which a classifier trained to distinguish between first attitude data input to the encoder of the variational autoencoder and second attitude data generated by inputting latent variables sampled from a predetermined normal distribution to the decoder is trained not to distinguish between third attitude data generated by inputting the output of the encoder into which the first attitude data has been input to the decoder and the first attitude data.

9. The posture data generation device according to claim 1 or 2, characterized in that the posture generation unit is a generator of a generative adversarial network that has been trained on posture data of postures that a person can take.

10. The generation of the reproduced posture data by the posture generation unit, the calculation of the error by the error calculation unit, and the updating of the value of the latent variable by the update unit are repeated until the magnitude of the error between the position of a specific part of the reproduced posture data obtained from the updated latent variable and the target value falls within a predetermined value. When the error between the position of a specific part of the reproduced posture data and the target value falls within the predetermined value, the reproduced posture data is acquired as the target posture data. The posture data generation device according to feature 1 or 2.

11. An action recognition device equipped with an action recognition model that recognizes a person's actions based on posture data, The behavior recognition device is characterized in that the behavior recognition model is a learning model that learns a specific behavior of a person whose specific body part is substantially the same, using a plurality of different posture data generated by the posture data generation device according to claim 1 or 2 based on a plurality of different latent variables and the target value.

Images