Music driven human dancing video synthesis

Abstract

The present disclosure provides a method for generating a video of a body moving in synchronization with music by applying a first artificial neural network (ANN) to a sequence of samples of an audio waveform of the music to generate a first latent vector describing the waveform and a sequence of coordinates of points of body parts of the body, by applying a first stage of a second ANN to the sequence of coordinates to generate a second latent vector describing movement of the body, by applying a second stage of the second ANN to static images of a person in a plurality of different poses to generate a third latent vector describing an appearance of the person, and by applying a third stage of the second ANN to the first latent vector, the second latent vector, and the third latent vector to generate the video.

Description

BACKGROUND1. Technical Field[0001]This disclosure relates to techniques for generating human dancing videos from music, and more particularly to, techniques for generating human dancing videos from music using neural networks.2. Discussion of Related Art[0002]Music and dancing are two related events that are often intertwined. Humans, especially professional dancers, are able to infer some reasonable dance movements when listening to music based on the rhythm, beat, and even genre.[0003]The use of computerized programs to automatically generate reasonable human dancing videos from music may provide rich source materials and inspiration for choreographers. These programs may also facilitate editing of animations and movies, without people having to manually draw each frame for the animation or have a human actor practice a very difficult dance to perform the dance professionally.[0004]For example, a computerized program can generate predicted arms and fingers motion given piano or vi...

AI Technical Summary

Benefits of technology

This disclosed patent system is a way to create videos with realistic and smooth human body movement synchronized with music. It uses a two-stage sequence-to-sequence framework, where a pose decoder, an appearance encoder, and a video decoder work together to generate a video of a target person dancing to the music. The system decouples the generation of the pose and appearance of the target person, making it easier to generate the same dance for different people.

Problems solved by technology

For example, a computerized program can generate predicted arms and fingers motion given piano or violin music, and apply 3D rendering to visualize a predicted motion sequence, but the video generated would not appear to be human body movement that is realistic and smooth, nor synchronized well with music.

Thus, there are several challenges with regard to conventional systems that attempt to synthesize human body movement synchronized with music.

Images