Monocular pose prediction method and device and non-transitory computer readable medium

Abstract

Computer animation involving monocular pose prediction is disclosed. A plurality of candidate pose sequences of a three-dimensional model of an animated character are generated such that each candidate pose of each sequence has a segmentation map that matches a segmentation map of the corresponding character derived from a corresponding frame of a video. The distance between the candidate poses at each time step is maximized. An optimal pose sequence is determined and used to generate a corresponding sequence of animation frames.

Description

Technical Field

[0001] The present application relates to computer animation, and more particularly, to determining poses when generating animation from video. Background Art

[0002] Computer animation generally refers to the process for generating animated images digitally. Modern computer animation generally uses 3D computer graphics to generate images. In order to produce the illusion of movement, a computer-generated image is displayed and repeatedly replaced (for example, at a rate of 24, 25 or 30 frames per second) with a new image that is similar to it but slightly ahead of time. Computer animation techniques can achieve the same illusion of movement that is traditionally associated with television and movies. Computer animation can be considered as a digital stop-motion animation technique that uses computers to generate frames of moving images. The use of digital computing allows the use of 3D models to achieve more detailed and realistic animations. Compared with more traditional physics-based processes (such as using miniatures for special effects shots, or hiring extras for crowd scenes), computer animation provides greater control and flexibility. Modern computer animation systems and software can create images that cannot be achieved using any other technology.

[0003] In most 3D computer animation systems, animators create a simplified representation of the segments that make up the character's anatomy. This simplified representation can be thought of as a skeletal model. The segments are arranged into default positions, sometimes called a bind pose, or a T-pose for the character. The position of each segment is defined by animation variables, which together define the character's pose. In human and animal characters, many parts of the skeletal model may correspond to actual bones, but skeletal animation can also be used to animate other things, including anthropomorphic objects, such as humanoid robots, or other inanimate objects that animators wish to portray as if they were alive. The computer uses this model to calculate the exact position and orientation of a particular character, which is ultimately rendered into an image. Thus, by changing the values ​​of the animation variables over time, animators create motion by causing a character to move from one frame to another.

[0004] There are several ways to generate animation variable values ​​to achieve realistic motion. Traditionally, animators manipulate these values ​​directly. While this can be done by setting values ​​for each frame, it is more common to set these values ​​at strategic points (frames) in time, and the computer interpolates or "tweens" between them in a process called keyframing.

[0005] A newer method called motion capture uses live-action footage. When computer animation is driven by motion capture, real performers act out scenes as if they were the characters to be animated. The performers' movements are recorded to a computer using a camera and markers at key points (e.g., joints and limbs), and the performance is then applied to an animated character. Recently, techniques have been developed to use motion capture data as a reference in reinforcement learning (RL) to train neural networks to control humanoid robots or create realistic animations at a lower cost.

[0006] It is against this background that aspects of the present disclosure are presented. BRIEF DESCRIPTION OF THE DRAWINGS

[0007] The teachings of the present disclosure may be readily understood by considering the following detailed description in conjunction with the accompanying drawings, in which:

[0008] Figure 1A and Figure 1B is a diagram illustrating the pose ambiguity problem in images acquired with a monocular camera.

[0009] Figure 2A is a schematic diagram illustrating a pose disambiguation method in computer animation according to aspects of the present disclosure.

[0010] Figure 2B is a flow chart illustrating a method of pose disambiguation in computer animation according to aspects of the present disclosure.

[0011] Figure 3 is a schematic diagram illustrating an apparatus for computer animation involving gesture disambiguation in accordance with aspects of the present disclosure.

[0012] Figure 4A is a flow chart illustrating monocular pose prediction in video-derived computer animation in accordance with aspects of the present disclosure.

[0013] Figure 4B is a flow chart illustrating monocular pose prediction in computer animation control of a robot according to aspects of the present disclosure.

[0014] Figure 5 is a schematic diagram illustrating an apparatus for computer animation involving monocular pose prediction in computer animation in accordance with aspects of the present disclosure. DETAILED DESCRIPTION

[0015] Although the following detailed description contains many specific details for illustrative purposes, it will be appreciated by any person skilled in the art that many variations and modifications to the following details are within the scope of the present invention. Therefore, the exemplary embodiments of the present invention described below are set forth without loss of generality and without implying limitations to the claimed invention.

[0016] introduction

[0017] Although motion capture works well on human performers, it is somewhat more difficult on animals, especially wild animals. Specifically, there are challenges in fitting animals with motion capture markers, and animals are unlikely to act naturally with motion capture markers attached. It would be more advantageous to create animations directly from videos of animals, especially wild animals. Frames of such videos can be analyzed, for example, with image analysis software to determine the pose of the animal at each frame.

[0018] According to aspects of the present disclosure, animations of characters such as animals can be derived from video frames. Specifically, a segmentation mask of the animal can be generated from the video frames of the animal and the 3D model of the animal. The more poses the real animal and the 3D animal model have, the more different their segmentation masks are. For example, a quantitative representation of the difference can be an intersection-over-union ratio.

[0019] As is well known, Intersection over Union is an evaluation metric used to measure the accuracy of an object detector on a specific dataset. Intersection over Union is used to evaluate the performance of object detectors and convolutional neural network detectors (R-CNN, Faster R-CNN, YOLO, etc.), regardless of the algorithm used to generate the predictions. Any algorithm that provides as output a predicted bounding box or segmentation mask for an object or character in an image can be evaluated using Intersection over Union (IoU). Applying Intersection over Union to evaluate (arbitrary) object detectors typically requires (1) ground truth bounding boxes (e.g., manually labeled bounding boxes from a test set that specify the location of the object in the image) and (2) predicted bounding boxes from the model. With these two sets of bounding boxes, the Intersection over Union (IoU) can be determined as

[0020] IoU = Overlap Area / Union Area.

[0021] The closer the value is to 1, the better the prediction.

[0022] like Figure 1AAs shown, in order to generate a computer animation frame (target frame) of a source character SC from an input video frame (referred to herein as a source frame), the animation program generates a segmentation mask of the character C in the video image. Image segmentation creates a pixel-level source mask 102 for the character in the video image. The animation program models the source character SC using the source mask 102 and generates a corresponding current animation frame with a corresponding current character CC in a certain predicted initial pose. A current segmentation mask 104 is then generated from the current animation frame. Computer animation can model the current character CC using three-dimensional data representing the position and orientation of the joints and limbs of the current character. The combination of the position and orientation of the joints and limbs of the character is generally referred to as the pose of the character. The current pose can be optimized by iteratively comparing the current segmentation mask with the source segmentation mask 102 (e.g., by calculating IoU) and adjusting the pose to generate an updated current animation frame and current segmentation mask. In order to determine the correct pose of the character C, a target segmentation mask 106 is generated for the corresponding target character TC in a known pose. The target segmentation mask 106 can be generated from a corresponding three-dimensional target data set representing the position and orientation of the joints and limbs of the target character when the target character is in a known posture. The correct posture can be determined by determining the IoU between the current segmentation mask 104 and one or more different target segmentation masks 106. According to some implementations, one or more of the current segmentation mask and / or the target segmentation mask may be an edge mask that only shows the posture outline of the character. The benefit of using the edge mask for the segmentation mask is that the edge mask can provide more accurate posture matching. Using the edge mask as a segmentation mask can avoid the situation where the virtual character is far away and therefore has a different scale. In this case, the virtual character can fit inside the target segmentation mask and be blocked by the target mask.

[0023] The above process may be repeated for subsequent frames. In addition, the physics-based simulation may simulate the interaction between the character and its surroundings to evaluate the feasibility of a given candidate pose determined from the current segmentation mask 104 and / or the target segmentation mask 106. Examples of infeasible poses include, for example, poses that would cause the character to fall. The pose feasibility evaluation process may be iteratively repeated before generating the target segmentation mask 106, so that the segmentation mask generation is limited to feasible poses.

[0024] The segmentation mask is a 2D projection of all body points onto the image plane. Therefore, it does not carry complete information about the original 3D pose. Therefore, the pose of an object in a monocular image may be ambiguous. Figure 1B An example of this ambiguity is shown in Figure 1B In , an image of a wolf W from a video frame has been analyzed to generate a segmented image 100 as input for computer animation. In FIG1 , there is ambiguity as to which of the wolf's front legs F1, F2 or hind legs H1, H2 is closest to the camera. In order to better match the pose in 3D, some techniques can be used.

[0025] Posture disambiguation

[0026] According to aspects of the present disclosure, a computer animation method may use a target segmentation mask for multiple camera views of a character to resolve ambiguities in poses. This may be accomplished by minimizing the difference between a current segmentation mask and different target segmentation masks for different poses to obtain the correct pose (e.g., intersection-over-union).

[0027] Figure 2A and Figure 2B Possible implementations of computer animation methods according to various aspects of the present disclosure are depicted. Figure 2A As shown, the video frame 201 is analyzed to generate corresponding current segmentation masks 203 of two different contemporaneous views of the character CC from the video frame 201. By way of example and not limitation, two different synchronized cameras may be used to generate two video frames 201 showing different contemporaneous views of the character CC. As used herein, the term "contemporaneous view" generally means that the views are obtained at approximately the same time (e.g., within one or two frame increments of each other for a standard video frame rate). In some implementations, it is possible to obtain two different images at different angles using a single camera that observes the character CC via two or more angled mirrors. In such an implementation, two or more different contemporaneous images and corresponding segmentation masks may be derived from different portions of the same video frame corresponding to different images. The corresponding target segmentation mask 205 may be generated by first generating three-dimensional animation data 203 from the source video frame 201 and using the animation data to generate the target segmentation mask 205. By way of example and not limitation, different views of the current character CC may include views oriented at +45° and -45° relative to a reference plane (e.g., an image plane of a virtual camera used to generate the target segmentation mask 205). Likewise, source mask 207 may be generated from simultaneous video frames of character CC captured with two cameras oriented at +45° and -45° relative to corresponding reference planes.

[0028] exist Figure 2BIn the illustrated implementation, the target segmentation mask 205 may be generated from the animation data 203 as follows. As indicated at 202, the input frame 201 is analyzed by a computer animation program to generate the animation data 203. The animation data 203 corresponds to a three-dimensional model TC of a character CC in a target pose from the video frame 201. The animation program generates the target segmentation mask 205 by a process involving projecting different views of the model TC from virtual cameras VC1, VC2. The orientation of the virtual camera may correspond to the orientation of the real camera that generated the video frame 201. A source segmentation mask 205 is also generated from the input video frame 201, as indicated at 204. In some implementations, a source segmentation mask 207 may be optionally used in the process of generating or refining the animation data 203.

[0029] To determine whether the pose of the three-dimensional model TC shows a pose corresponding to the pose of the character CC in the video frame 201, the target segmentation mask 205 is compared with the corresponding source segmentation mask, as indicated at 206. The results of the comparison are then analyzed, as indicated at 208, by way of example and not limitation, at 206, the IoU of each target / source mask comparison may be calculated. Then, at 208, the result of each IoU calculation may be compared with a certain threshold to determine whether the pose of the model TC corresponds to an indication of the character CC. Based on the analysis results at 208, the animation data 203 may then be adjusted to adjust the pose of the model TC at 202. A new target mask may be generated at 204 and compared to the source mask at 206. Adjusting the animation data may include, but is not limited to, adjusting one or more joint angles of the model TC, rotating the orientation of the virtual cameras VC1, VC2 relative to the reference plane, or some combination of joint angle adjustment and camera orientation adjustment. The process may be iterated until the analysis results indicate a match between the pose of the model TC and the pose of the character CC in the video frame 201. Once a match is obtained, the final pose data 209 may then be used to generate animation frames 211 , as indicated at 210 .

[0030] By way of example and not limitation, the different views of the current character CC may include views oriented at +45° and −45° relative to a reference plane (eg, an image plane of a virtual camera used to generate the current source mask 205 ).

[0031] As indicated at 214, the current segmentation mask 207 may then be compared to each of the target segmentation masks 213, 215 to determine final pose data 217 of the current character CC corresponding to the correct pose of the source character in the video frame 201. By way of example and not limitation, comparing the current mask 205 to the target masks 209, 211 may include calculating an intersection over union (IoU) between each of the target segmentation masks 213, 215 and the current segmentation mask 207. The IoU value may be compared to a threshold, and the correct pose may be determined based on the current mask, e.g., the current mask has an IoU value for each target mask that at least satisfies the threshold. In the event that multiple current masks 207 do not satisfy the threshold, an error condition may be determined and the target mask may be adjusted to correct the problem. For example, if the IoU value is not above the IoU threshold or the difference between the two IoU values ​​is below the difference threshold, the target data 211 may be adjusted to change the pose of the target character TC to a different pose and a new target segmentation mask 213, 215 may be generated, as indicated at 212. If the IoU value is above the threshold for a particular pose but not for other poses, the animation program may generate final pose data 217 corresponding to the particular pose. The animation program may then use the final pose data 217 to generate a final animation frame 219 depicting the current character CC in the correct pose, as indicated at 216. The foregoing process may then be repeated for the next video frame, as indicated at 218.

[0032] Although the foregoing example uses two different views of the model TC and the character CC to generate two target segmentation masks and two corresponding source masks, three or more different views may be used to generate three or more corresponding different target segmentation masks and source segmentation masks. In addition, in an alternative implementation, the target segmentation mask may be generated from two or more contemporaneous video frames of two or more target characters corresponding to different angles obtained using two or more different cameras.

[0033] Posture disambiguation equipment

[0034] Figure 3 Describes the use of, for example, Figure 2A and Figure 2BAn apparatus for computer animation involving gesture disambiguation is described. The apparatus may include a computing device 300 coupled to a user input device 302. The user input device 302 may be a controller, a touch screen, a microphone, a keyboard, a mouse, a joystick, or other device that allows a user to input information including sound data into the system. The user input device may be coupled to or include a tactile feedback device, such as a vibration motor, a force feedback system, an ultrasonic feedback system, or a pneumatic feedback system. Additionally, the system may include a controller 301 for a movable joint, such as, but not limited to, in an implementation involving physics-based animation for controlling a physical robot, the controller may control a motor or actuator for a joint on the robot.

[0035] The computing device 300 may include one or more processor units 303, which may be configured according to well-known architectures (e.g., such as single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, etc.). The computing device may also include one or more memory units 304 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), etc.).

[0036] Processor unit 303 may execute one or more programs 317, portions of which may be stored in memory 304, and processor 303 may be operatively coupled to memory, for example, accessing memory via data bus 305. System 317 may also be stored in mass storage device 315, such as a disk drive, CD-ROM drive, tape drive, flash memory, etc. The program may enable the processor unit to perform an animation method (such as the one described above with respect to Figure 2A and Figure 2B The program may further include a machine learning algorithm configured to adjust weights and transformation values ​​of a neural network (NN) 314 for an implementation involving a neural network in a physics-based animation input control scheme, as discussed elsewhere herein. Additionally, the memory 304 may store video frame data 308 and animation data 309, which may be used to generate a source segmentation mask 310 and a target segmentation mask 312, respectively, as described above. The video frame data 308, the animation data 309, and the segmentation masks 310, 312 may also be stored as data 318 in the mass storage area 315. The processor unit 303 is further configured to execute one or more programs 317 stored in the mass storage area 315 or the memory 304, the one or more programs causing the processor to perform one or more of the above-described methods.

[0037] The computing device 300 may also include well-known support circuits 306, such as input / output (I / O) circuits 307, power supply (P / S) 321, clock (CLK) 322, and cache 323, which may communicate with other components of the system, for example, via bus 305. The computing device 300 may include a network interface 332 to facilitate communication via an electronic communication network 330. The network interface 332 may be configured to implement wired or wireless communication via a local area network and a wide area network such as the Internet. The computing device 300 may send and receive data and / or requests for files via one or more message packets over the network 320. Message packets sent over the network 320 may be temporarily stored in a buffer in the memory 304. Animation frames 308, video frames 309, and segmentation masks 311, 312, 313 may be obtained from a remote computing or storage device via the network 330 and stored in part in the memory 304 and / or the mass storage device 315 for use by the computing device 300.

[0038] The processor unit 303 and the network interface 332 may be configured to implement a local area network (LAN) or a personal area network (PAN) via a suitable network protocol, such as Bluetooth for a PAN. The computing device may also include a user interface 316 for facilitating interaction between the system and a user. The user interface may include a monitor, a television screen, a speaker, headphones, or other means of conveying information to a user.

[0039] Monocular pose estimation

[0040] According to an alternative aspect of the present disclosure, consecutive animation frames may be analyzed as a single problem, rather than analyzing each individual video frame independently. In such an implementation, a candidate pose is constructed for the first animation frame. Each candidate pose has the same segmentation mask. But in 3D space, the candidate poses of the model TC are distributed as far away from each other as possible. Subsequently, a realistic actor-critic trains a neural network (NN) to analyze the candidate poses. There are different methods to evaluate the candidate poses. These methods can be combined in various implementations discussed below.

[0041] Figure 4AAn example of a generalized method for monocular pose prediction in computer animation according to aspects of the present disclosure is depicted. The method may begin with a sequence of input video frames 401. The input video frames may be obtained from a live broadcast or archived footage. Any suitable type of video frames showing a character may be used. Preferably, the sequence of input video frames 401 is in the form of digital video frames. Alternatively, non-digital video frames or motion picture frames may be digitized to provide the sequence of input video frames 401. The animation program may generate a corresponding sequence of segmentation masks 403 for the character in each frame of the sequence of input video frames 401, as indicated at 402. The segmentation mask 403 may be an edge mask. It should be noted that in some implementations, the animation program may receive the segmentation mask 403 from some external source, in which case the segmentation mask does not need to be generated.

[0042] The animation program may generate a three-dimensional animation model 405, as indicated at 404. The animation model 405 includes three-dimensional data representing the joints and limbs of the animated character corresponding to the character in the segmentation mask 403. As indicated at 406, the computer animation program then generates pose sequence data 407 corresponding to possible candidate pose sequences, each sequence containing two or more poses of the character represented by the animation model 405 at different time steps corresponding to consecutive frames of the video sequence. Each pose in each candidate pose sequence is generated in such a way that it has a segmentation mask that matches the segmentation mask of the corresponding frame in the video sequence 401. By way of example and not limitation, the segmentation masks for different possible candidate poses may be edge masks. Due to the above-mentioned pose ambiguity problem, it is desirable to generate candidate poses in such a way that the distance between candidate poses at each time step is maximized. By way of example and not limitation, each candidate pose may be represented as N multidimensional vectors of N joint angles in the three-dimensional model 405, and the distance between poses may be calculated using an N-dimensional distance formula. , the animation program determines the best pose sequence among the plurality of candidate pose sequences, as indicated at 408. At 410, the animation program uses the resulting best pose sequence data 409 to generate an animation frame 411. The animation program may then repeat the foregoing process for another input video frame, as indicated at 412.

[0043] As described above, there are different ways to generate a sequence of poses at 406 and determine the best sequence of poses at 408. According to one implementation, pose candidate pairs from two or more consecutive animation frames in a video sequence 401 can be used as input to a neural network that performs pose optimization 408. As part of pose optimization, a value network (critic) can test a sequence of pose candidates. The sequence that gives the highest value is considered to be the correct sequence. In such an implementation, several three-dimensional (3D) poses of an animation model 405 are generated based on a segmentation mask of the first video frame of the sequence 401. All poses have a segmentation mask that matches the segmentation mask of the first video frame in the sequence. 3D poses are generated, such as to be as far away from each other as possible. The distance between poses can be measured, for example, as the average angular difference between the joints in the 3D model of the animated character. For each 3D pose derived from the first frame in the sequence 401, the optimization process 408 adjusts the 3D pose over time in such a way that for each video frame, the segmentation mask of the pose matches the segmentation mask of the corresponding video frame of the sequence 401. During the optimization process, the movement of the character represented by the model 405 is simulated by the physical simulation environment. The optimization process 408 matches the segmentation masks of the frames of the sequence 401 with the corresponding projections of the candidate poses of the model 405, and at the same time ensures that the movement of the animated character is physically consistent, for example, does not cause the animated character to fall or violate joint constraints. A genetic (evolutionary) algorithm can be used for this purpose.

[0044] In an alternative implementation, several candidate poses may be generated as described above but for each animation frame. All candidate poses for a given animation frame have a segmentation mask that matches the segmentation mask of the corresponding video frame of sequence 401. During the optimization process 408, the pose candidate pairs of consecutive video frames can be fed into a neural network that has been pre-trained to control a character in a physical simulation environment using similar animations. The pose candidate pairs are then evaluated by the neural network. The segmentation mask of the best pose candidate pair should provide the best match with the segmentation mask obtained from the corresponding video frame. At the same time, the movement of the character in the simulated physical environment must not cause the character to fall or violate joint constraints. The solution proceeds continuously from the first frame pair to the end of the video sequence 401.

[0045] In some implementations, the animation program may use the output of the posture optimization process at 408 to generate a robot control input 413, as indicated at 414. The animation program may provide the control input 413 to a robot controller 415, which converts the control input into a control signal that is transmitted to an articulated robot 417. The robot controller 415 may be implemented in hardware or software. For hardware implementations, the optimization process 408 of the animation program provides input in a convenient form and the robot controller may convert the input into a robot command. For software implementations, the robot controller 415 may be implemented by code running on the same computer system as the animation program. Such a robot controller code may be a program separate from the animation program or may be incorporated into the animation program.

[0046] As described above, the pose optimization process 408 may be informed by physics simulation to evaluate the feasibility of various combinations of pose combinations for a pose sequence of the robot 417 or a corresponding animated character. By way of example, the pose optimization process 408 may restrict the movement of the animated character or robot 417 according to one or more physics-based constraints. Alternatively, the pose optimization process 408 may reject poses that would be inconsistent with the operation of the robot 417, for example, poses that would cause the robot to fall or violate joint constraints. In some implementations, the pose optimization process 408 may optionally include using a method such as described above with respect to Figure 2A and Figure 2B This may involve generating two or more different candidate target masks from different views of the 3D animated model 405 and comparing the target mask with a corresponding mask 403 generated from different contemporaneous views of the input video frame 401 .

[0047] Figure 4B An example of pose optimization 408 for using monocular pose prediction in computer animation is depicted in accordance with aspects of the present disclosure. Figure 4B As shown, the posture optimization process 408 can use a neural network 420 to fit the candidate postures in the posture sequence 407 to the corresponding segmentation mask 403, and optionally generate a control input 413. In the illustrated implementation, the input of the neural network 420 is the segmentation mask 403 obtained from the video frame sequence 401. Figure 4B In the depicted example, the target of the neural network 420 is the segmentation masks 421, 423 corresponding to the candidates for the next two poses. Specifically, the target may be two consecutive poses resulting from a target animation imitated by the robot 417. The neural network 420 transforms the target animation in real time in such a way that it can be run on the real robot 417 without causing it to fall.

[0048] The neural network 420 can be trained to determine the next two poses from the current pose. The neural network 420 training can include using a character model in a physical simulation. Motion capture or hand animation poses can be used as targets, and the neural network 420 can be trained to replicate the target poses within the constraints of the physical simulation using a machine learning algorithm. The machine learning algorithm and / or neural network layout can be, for example, but not limited to, a reinforcement learning algorithm, an imitation learning algorithm, or a supervised learning algorithm. The trained neural network can be used to output a score based on each candidate pose. As a result of the training, the score represents the feasibility of the pose in the simulation. The pose is evaluated based on the following factors: stability of the next two frames (e.g., whether the character falls in the simulation), whether any joints violate their constraints (e.g., whether the elbow is bent backward), trying to minimize the distance moved by all joints, whether any limbs collide, whether the limbs are connected to their corresponding joints, etc. Some or all of these evaluation factors can be generated by the neural network and represented by the score, or alternatively, some or all of these factors can be determined by the user and added to the score. Based on the candidate poses from which the best pose set is selected, this can be done by hand or within the neural network using a minimum maximum layer. For more information on the pose determination neural network, see concurrently filed application Ser. No. 17 / 095,586 (U.S. Patent Application Publication No.: 20220143820).

[0049] Based on the selected candidate pose, the robot can be controlled using neural network 420. The output of neural network 420 includes action 425 and value 427. Action 425 corresponds to the control input of robot 415. Value 427 is an internal training algorithm quantity. It is only needed during the training step and is used to estimate the effect of random improvement attempts.

[0050] The robot controller 415 provides commands based on the action 425 to the motors in the robot 417. In general, the robot 417 may include movable joints connected by structural elements and sensors. Each joint may be connected to a sensor configured to generate a sensor value related to information about the joint state. Sensors for physical robots may include, for example, but not limited to, encoders, potentiometers, linear variable differential transformers, pressure sensors, gyroscopes, gravimeters, accelerometers, rotary transformers, speed or velocity sensors. The sensor values ​​of such sensors will correspond to the output of such sensors or information derived therefrom. Examples of sensor values ​​from sensors on the robot include, but are not limited to, joint positions, joint speeds, joint torques, robot orientations, robot linear speeds, robot angular speeds, foot contact points, foot pressures, or two or more thereof. For animated characters, the sensors may be virtual sensors and the sensor values ​​may simply include data related to the state of the movable joints, for example, position, speed, acceleration data. Examples of sensor values ​​from robot simulations include, but are not limited to, joint positions, joint speeds, joint torques, model orientations, model linear speeds, model angular speeds, foot contact points, foot pressures, or two or more thereof. Position data from the controller 415 or animation program may be passed to the motion decision neural network and used as state data during reinforcement learning in conjunction with the pose optimization process 408 .

[0051] The nature of the control input depends on the control parameterization used by the robot controller 415 to control the joints of the robot 417. Common control parameterizations for articulated robots include position control, velocity control, and torque control. One possible implementation uses a hybrid scheme in which the neural network outputs a target joint velocity, which can be labeled as a position derivative v. The integrator block integrates the derivative v to the joint position x according to x=∫vdt, and then directly applies it to the position derivation (PD) controller in the simulation or animation or to the actuator of the robot 417. The output of the integrator block can also be used as a feedback signal by routing it to the neural network as an input. The integration step can advantageously suppress motor jitter in the simulation and control of the robot 417 to a visually unobservable level by smoothing the robot's response to noisy sensors and sensor spikes. Integration can also adjust the movement of the robot when the network input enters the out-of-distribution region of the state space during a fault scenario.

[0052] In the example shown, the neural network that generates actions 425 and values ​​427 divides the policy and value functions into separate networks 422, 424 without shared weights. The policy network 422 and critic network 424 shown can each be composed of three layers, each containing the same number of neurons. Each neuron can have the same activation function. By way of example and not limitation, each of these layers contains 128 neurons and uses softsign as their activation function. The network input (observation) is normalized using a sliding mean and standard deviation. The input may include any or all of the following features: target orientation, joint sensor readings, action at the previous time step, actuator input at the previous time step, gravity vector in the local reference frame, accelerometer readings, gyroscope readings, and foot pressure sensor readings.

[0053] The target orientation may be represented in axis angle form and encoded into a latent representation using two encoding layers 426, 428. By way of example, each encoding layer may include a first layer comprising 128 neurons coupled to a second layer comprising 64 neurons. Each neuron may use a leaky ReLU activation function. Actions 425 specify a set of joint position derivatives output by the neural network. The actuator inputs indicate updated joint positions calculated by integrating the position derivatives. Feeding the actions and actuators from the previous time step into the input network introduces a feedback signal.

[0054] Exploration is performed during training by sampling the policy network output from a learned Gaussian distribution. Sampling in this way introduces jitter during training, which makes learning difficult because it can cause falls. The integration scheme discussed above helps mitigate jitter. Additionally, instead of sampling random actions from a Gaussian distribution at each time step, it is done with a fixed probability ε that a random action can be sampled from the policy network 422 and a probability 1-ε that the robot 417 performs a deterministic action specified by the mean of the Gaussian. Furthermore, updates can be performed using only samples to which the exploration noise is applied.

[0055] Posture prediction device

[0056] Figure 5 Describes the use of, for example, Figure 4A and Figure 4BAn apparatus for computer animation involving monocular pose prediction is described. The apparatus may include a computing device 500 coupled to a user input device 502. The user input device 502 may be a controller, a touch screen, a microphone, a keyboard, a mouse, a joystick, or other device that allows a user to input information including sound data into the system. The user input device may be coupled to or include a tactile feedback device, such as a vibration motor, a force feedback system, an ultrasonic feedback system, or a pneumatic feedback system. Additionally, the system may include a controller 501 for a movable joint, such as, but not limited to, in an implementation involving physics-based animation for controlling a physical robot, the controller may control a motor or actuator for a joint on the robot.

[0057] The computing device 500 may include one or more processor units 503, which may be configured according to well-known architectures (e.g., such as single-core, dual-core, quad-core, multi-core, processor-coprocessor, cell processor, etc.). The computing device may also include one or more memory units 504 (e.g., random access memory (RAM), dynamic random access memory (DRAM), read-only memory (ROM), etc.).

[0058] Processor unit 503 may execute one or more programs 517, portions of which may be stored in memory 504, and processor 503 may be operatively coupled to memory, for example, accessing memory via data bus 505. System 517 may also be stored in mass storage device 515, such as a disk drive, CD-ROM drive, tape drive, flash memory, etc. The program may enable the processor unit to perform animation methods (such as those described above with respect to Figure 4A and Figure 4B The program may further include a machine learning algorithm configured to adjust the weights and transformation values ​​of a neural network (NN) 513 for an implementation involving a neural network in a physics-based animation input control scheme, as discussed elsewhere herein. Additionally, the memory 504 may store video frame data 508 and animation frame data 509. The video frame data 508 may be used to generate a segmentation mask 510 for use in pose prediction as described above. The pose data 511 used in pose prediction may also be stored in the memory 504. When used for control of the robot 530, the memory may also store robot commands 512 and quality values ​​514 generated by the neural network 513, for example, as discussed above. The video frame data 508, the animation data 509, the segmentation mask 510, the pose sequence data 511, the robot commands 512, and the quality values ​​514 may also be stored as data 518 in the mass storage device 515.

[0059] The computing device 500 may also include well-known support circuits 506, such as input / output (I / O) circuits 507, power supply (P / S) 521, clock (CLK) 522, and cache 523, which may communicate with other components of the system, for example, via bus 505. In an implementation involving control of a robot 530, robot commands 512 may be relayed to the robot via the I / O circuits. The computing device 500 may include a network interface 532 to facilitate communication via an electronic communications network 530. The network interface 532 may be configured to implement wired or wireless communications over a local area network and a wide area network such as the Internet. The computing device 500 may send and receive data and / or requests for files via one or more message packets over the network 520. Message packets sent over the network 520 may be temporarily stored in a buffer in the memory 504. Animation frames 508 , video frames 509 , and segmentation masks 511 may be obtained from a remote computing or storage device via network 520 and stored in part in memory 504 and / or mass storage device 315 for use by computing device 500 .

[0060] The processor unit 503 and the network interface 532 may be configured to implement a local area network (LAN) or a personal area network (PAN) via a suitable network protocol, such as Bluetooth for a PAN. The computing device may also include a user interface 516 for facilitating interaction between the system and a user. The user interface may include a monitor, a television screen, a speaker, headphones, or other means of conveying information to a user.

[0061] Although certain implementations are described herein in the context of computer animation for the purpose of controlling a robot, aspects of the disclosure are not limited thereto. Pose disambiguation and monocular pose prediction are useful in many other applications. Furthermore, although certain implementations are described herein in the context of animating animals, aspects of the disclosure are not limited thereto. For example, the techniques described herein can be used to generate computer animations of human and / or robotic characters or other moving objects from archival footage or other situations where motion capture is impractical or impossible.

[0062] While the above is a complete description of the preferred embodiments of the invention, various alternatives, modifications and equivalents may be used. Therefore, the scope of the invention should not be determined with reference to the above description, but should instead be determined with reference to the appended claims and their full range of equivalents. Any feature described herein (whether preferred or not) may be combined with any other feature described herein (whether preferred or not). In the appended claims, Indefinite article "a" or "a kind" Refers to the quantity of one or more of the items following the article unless explicitly stated otherwise. The appended claims should not be read as including means-plus-function limitations unless such a limitation is explicitly recited in a given claim using the phrase "means for..."

Claims

1. A monocular posture prediction method, the method comprising: Generating a plurality of candidate pose sequences of two or more candidate poses of a three-dimensional model of an animated character, such that each of the two or more candidate poses has a segmentation mask that matches a segmentation mask of a corresponding character derived from a corresponding frame of a video sequence at each time step in a plurality of time steps, wherein each time step corresponds to a frame of the video sequence, wherein a distance between the candidate poses at each time step is maximized by using an N-dimensional distance formula, wherein generating the plurality of candidate pose sequences comprises: generating two or more initial poses, the two or more initial poses having corresponding segmentation masks that match a segmentation mask of the corresponding character derived from a first frame of the video sequence; and adjusting each of the two or more initial poses within one or more time steps to generate two or more corresponding candidate pose sequences; Determining an optimal posture sequence among the plurality of candidate posture sequences by inputting the plurality of candidate posture sequences into one or more neural networks, wherein determining the optimal posture sequence includes simulating the movement of the animated character using a physics simulation to determine whether a candidate sequence will violate a condition that causes the animated character to fall; and The optimal pose sequence is used to generate a corresponding animation frame sequence of the animated character. 2 . The method of claim 1 , wherein determining the optimal pose sequence comprises simulating the movement of the animated character using a physics simulation. 3 . The method of claim 1 , wherein determining the optimal pose sequence comprises simulating movement of the animated character using a physics simulation to determine whether a candidate sequence will violate physical constraints.

4. The method of claim 1, wherein determining the optimal posture sequence comprises inputting each posture sequence in the plurality of candidate posture sequences into one or more neural networks.

5. The method of claim 4, wherein each candidate pose of each candidate pose sequence in the plurality of candidate pose sequences is encoded into a latent representation using one or more layers having different numbers of neurons with activation functions.

6. The method of claim 4, wherein each candidate pose of each candidate pose sequence in the plurality of candidate pose sequences is encoded into a latent representation using two layers of different numbers of neurons with a leaky ReLU activation function.

7. The method of claim 4, wherein each candidate pose of each candidate pose sequence in the plurality of candidate pose sequences is encoded into a latent representation using two layers of 128 and 64 neurons with a leaky ReLU activation function.

8. The method of claim 4, further comprising controlling a robot using the output of the one or more neural networks.

9. The method of claim 4, further comprising controlling a robot using outputs of the one or more neural networks, wherein the one or more neural networks include a policy network, wherein outputs of the policy network specify a set of joint position derivatives of one or more joints of the robot.

10. The method of claim 4, further comprising controlling a robot with the outputs of the one or more neural networks, wherein the one or more neural networks include a policy network and a critic network, wherein the output of the policy network specifies a set of joint position derivatives for one or more joints of the robot, and wherein the critic network produces an output during a training phase that represents the quality of random attempts to improve the performance of the policy network.

11. The method of claim 1, wherein generating the plurality of candidate pose sequences comprises generating two or more candidate poses at each time step corresponding to each frame in the video sequence.

12. The method of claim 11, wherein determining the optimal sequence of poses comprises feeding candidate pose pairs of consecutive frames of the video sequence into a neural network that has been pre-trained to control the animated character in a physically simulated environment.

13. The method of claim 12, wherein determining the optimal pose sequence comprises simulating movement of the animated character using a physics simulation.

14. The method of claim 12, wherein determining the optimal pose sequence comprises simulating movement of the animated character with a physics simulation to determine whether a candidate sequence would violate physical constraints.

15. The method of claim 12, wherein determining the optimal pose sequence comprises simulating the movement of the animated character using a physics simulation to determine whether a candidate sequence would violate causing the animated character to fall.

16. A monocular posture prediction device, the device comprising: processor; Memory; Executable instructions are embodied in the memory, which when executed by the processor cause the processor to implement a method for computer animation, the method comprising, Generating a plurality of candidate pose sequences of two or more candidate poses of a three-dimensional model of an animated character, such that each of the two or more candidate poses has a segmentation mask that matches a segmentation mask of a corresponding character derived from a corresponding frame of a video sequence at each time step in a plurality of time steps, wherein each time step corresponds to a frame of the video sequence, wherein a distance between the candidate poses at each time step is maximized by using an N-dimensional distance formula, wherein generating the plurality of candidate pose sequences comprises: generating two or more initial poses, the two or more initial poses having corresponding segmentation masks that match a segmentation mask of the corresponding character derived from a first frame of the video sequence; and adjusting each of the two or more initial poses within one or more time steps to generate two or more corresponding candidate pose sequences; Determining an optimal posture sequence among the plurality of candidate posture sequences by inputting the plurality of candidate posture sequences into one or more neural networks, wherein determining the optimal posture sequence includes simulating the movement of the animated character using a physics simulation to determine whether a candidate sequence will violate a condition that causes the animated character to fall; and The optimal pose sequence is used to generate a corresponding animation frame sequence of the animated character.

17. The apparatus of claim 16, wherein the executable instructions include one or more neural networks, wherein determining the optimal posture sequence includes inputting each of the plurality of posture sequences into the one or more neural networks.

18. The apparatus of claim 16, wherein each candidate pose of each candidate pose sequence in the plurality of candidate pose sequences is encoded into a latent representation using one or more layers having different numbers of neurons with activation functions.

19. The apparatus of claim 16, wherein each candidate pose of each candidate pose sequence in the plurality of candidate pose sequences is encoded into a latent representation using two layers of different numbers of neurons with a leaky ReLU activation function.

20. The apparatus of claim 16, wherein each candidate pose of each candidate pose sequence in the plurality of candidate pose sequences is encoded into a latent representation using two layers of 128 and 64 neurons with a leaky ReLU activation function.

21. The apparatus of claim 16, wherein generating the plurality of candidate pose sequences comprises generating two or more candidate poses at each time step corresponding to each frame in the video sequence.

22. The apparatus of claim 21, wherein determining the optimal sequence of poses comprises feeding candidate pose pairs of consecutive frames of the corresponding animation sequence into a neural network that has been pre-trained to control the animated character in a physically simulated environment.

23. A non-transitory computer-readable medium having executable instructions embodied therein, the executable instructions, when executed by a computer, causing the computer to implement a method for monocular pose prediction, the method comprising: Generating a plurality of candidate pose sequences of two or more candidate poses of a three-dimensional model of an animated character, such that each of the two or more candidate poses has a segmentation mask that matches a segmentation mask of a corresponding character derived from a corresponding frame of a video sequence at each time step in a plurality of time steps, wherein each time step corresponds to a frame of the video sequence, wherein a distance between the candidate poses at each time step is maximized by using an N-dimensional distance formula, wherein generating the plurality of candidate pose sequences comprises: generating two or more initial poses, the two or more initial poses having corresponding segmentation masks that match a segmentation mask of the corresponding character derived from a first frame of the video sequence; and adjusting each of the two or more initial poses within one or more time steps to generate two or more corresponding candidate pose sequences; Determining an optimal posture sequence among the plurality of candidate posture sequences by inputting the plurality of candidate posture sequences into one or more neural networks, wherein determining the optimal posture sequence includes simulating the movement of the animated character using a physics simulation to determine whether a candidate sequence will violate a condition that causes the animated character to fall; as well as The optimal pose sequence is used to generate a corresponding animation frame sequence of the animated character.

Images