Band command action generation method based on self-supervised cross-modal perception loss

Abstract

The invention relates to the technical field of artificial intelligence, and discloses a band command action generation method based on self-supervision cross-modal perception loss. The method comprises the following steps: firstly, carrying out self-supervised audio-action synchronization learning, and automatically sampling positive and negative sample pairs to train parameters of a two-branch network model; secondly, using the two trained branches for extracting semantic music control signals and calculating perception loss respectively, then using a discriminator for calculating adversarial loss, determining the optimal weight ratio of the perception loss to the adversarial loss according to the output action standard deviation, and training a model; and finally, inputting a test audio into the model, generating a command action sequence synchronized with the music, and visualizing the command action sequence. The method has the important significance that a cross-modal self-supervised learning task is used as a pre-training task of the perception loss network, and the problem of excessive smoothness of traditional regression loss can be avoided, so that natural, attractive and diversified command actions highly synchronous with music are generated.

Description

technical field [0001] The invention relates to a method for generating band conductor actions based on self-supervised cross-modal perceptual loss, and in particular to a method for generating band conductor actions that use music as a conditional control signal to generate rhythm-synchronized and semantically related command actions. , which belongs to the field of human action condition generation. Background technique [0002] The conductor is the soul of the symphony orchestra. From the medieval European church choir to the modern music of the 21st century, conducting technology and art have been continuously developed and have become a discipline with rich content. The conductor's body language is complex and changeable, and it is necessary to convey various information such as tempo, strength, emotion, and playing method in real time when the orchestra is performing, while maintaining a certain style and aesthetic feeling. In recent years, with the development of de...

AI Technical Summary

Problems solved by technology

However, existing perceptual loss networks have their own limitations

Some scholars pointed out that when using the traditional perceptual loss based on ImageNet pre-training VGGNet for image super-resolution, it will lead to unnatural image details

Images