An audio-driven motion synthesis method and apparatus

Abstract

The application discloses an audio driving action synthesis method and device. The method comprises the following steps: determining a plurality of test beat units according to input audio, each test beat unit corresponding to an input audio sequence of the input audio; acquiring audio input corresponding to each test beat unit; acquiring an initial posture of each test beat unit as visual input based on a visual action sequence, the visual action sequence being the visual action sequence synthesized corresponding to a previous test beat unit; and generating an action sequence corresponding to the input audio by using a generator of a generative adversarial network model, the generative adversarial network model being trained by introducing a hybrid loss function, the hybrid loss function comprising a multi-space posture loss, a harmony loss and a generative adversarial network loss, the harmony loss being determined according to beat consistency of an audio-visual beat pair.

Description

Technical Field

[0001] This invention relates to the field of image processing technology, and more specifically, to an audio-driven motion synthesis method and apparatus. Background Technology

[0002] Machine-based generation methods are widely used in tasks such as music video production, voice editing, and animation compositing, where harmony represents the subjective perception of consistency in the rhythm, emotion, or visual presentation of the output.

[0003] As a typical problem in audio-visual cross-domain generation methods, audio-driven motion synthesis has received widespread attention in character animation, video generation, and choreography. Traditional methods address this problem by retrieving video clips with feature-level similarity to given music. However, unlike conventional motion synthesis methods, in the presence of music, people are highly sensitive to discordant synthesized movements, which severely impairs quality evaluation. Harmony is considered one of the most important factors influencing the quality assessment of cross-domain results. However, the perception of harmony relies on subjective judgment. Enhancing audio-visual harmony in audio-driven motion synthesis tasks may be a challenge.

[0004] The methods and systems disclosed in this invention are intended to solve the problems mentioned above and others. Summary of the Invention

[0005] This invention provides a method for synthesizing audio-driven actions based on perceived harmony, applied to a computing device. The method includes: determining multiple test beat units based on input audio, each test beat unit corresponding to an input audio sequence; acquiring audio input corresponding to each test beat unit; acquiring the initial pose of each test beat unit as visual input based on a visual action sequence, wherein the visual action sequence is a synthesized visual action sequence corresponding to the previous test beat unit; and automatically generating a perceived harmony action sequence corresponding to the input audio using a generator of a generative adversarial network (GAN) model. The GAN model is trained by introducing a hybrid loss function. The hybrid loss function includes multi-space pose loss, harmony loss, and GAN loss. The harmony loss is determined based on the beat consistency of the audio-visual beat pair.

[0006] Another aspect of the present invention provides an apparatus for perceptually harmonious audio-driven action synthesis, comprising a memory and a processor coupled to the memory. The processor is configured to perform multiple operations, including determining a plurality of test beat units based on input audio, each test beat unit corresponding to an input audio sequence of the input audio; acquiring audio input corresponding to each of the test beat units; acquiring the initial pose of each test beat unit as visual input based on a visual action sequence, the visual action sequence being the visual action sequence synthesized corresponding to the previous test beat unit; and automatically generating a perceptually harmonious action sequence corresponding to the input audio using a generator of a generative adversarial network (GAN) model. The GAN model is trained by introducing a hybrid loss function. The hybrid loss function includes a multi-space pose loss, a harmony loss, and a GAN loss. The harmony loss is determined based on the beat consistency of the audio-visual beat pair. Attached Figure Description

[0007] Based on the disclosed embodiments, the following figures are merely illustrative examples and are not intended to limit the scope of the invention.

[0008] Figure 1 This is a block diagram of an exemplary computing system described in an embodiment of the present invention.

[0009] Figure 2 An exemplary perceptual harmony audio-driven motion synthesis process 200 according to an embodiment of the present invention is shown.

[0010] Figure 3 An exemplary GAN model training process 300 according to an embodiment of the present invention is shown.

[0011] Figure 4 The beat unit extracted based on the acquired audio beat is shown.

[0012] Figure 5A An exemplary framework diagram of the training process described in an embodiment of the present invention is shown.

[0013] Figure 5B An exemplary framework diagram of the testing phase described in an embodiment of the present invention is shown.

[0014] Figure 5C An exemplary framework diagram of the generator described in an embodiment of the present invention is shown.

[0015] Figure 5D An exemplary framework diagram of a cross-domain frequency discriminator according to an embodiment of the present invention is shown.

[0016] Figure 5E An exemplary framework diagram of a spatiotemporal frequency discriminator according to an embodiment of the present invention is shown.

[0017] Figure 6 An example diagram illustrating the perception of asynchrony between beats is shown.

[0018] Figure 7A The diagram illustrates a comparison, where the left image shows a frame-based visual beat detection method in the prior art, which uses optical flow; the right image shows a skeleton-based beat extraction method in the prior art, which uses action criteria derivation, and this extraction method is used for visual beat detection in human videos.

[0019] Figure 7B The video frames and corresponding skeleton poses extracted from test cases using models in the prior art are shown.

[0020] Figure 8A This is a schematic diagram of harmonic distortion between an initial audio beat and an SD-driven visual beat, where the left image shows the beat extracted from the audio signal and the right image shows the beat extracted from the visual signal.

[0021] Figure 8B This illustrates the asynchronous nature of audio-visual beats under perceptual judgment.

[0022] Figure 9 A diagram showing the relationship between the total joint velocity and the evolution of human movement is presented.

[0023] Figure 10A The diagram illustrates the improved beat synchronization based on a visual beat extraction mechanism, where the left image shows the beat extracted from the audio signal and the right image shows the beat extracted from the visual signal.

[0024] Figure 10B A schematic diagram of the synchronization between audio-visual beats considering joint velocity is shown.

[0025] Figure 11 The percentage of discordant videos accurately extracted according to embodiments of the present invention is shown.

[0026] Figure 12 The illustration shows sample results of the average Frechet Inception Distance (FID) between action sequences generated by models in existing technologies and HarmoGAN models, as described in embodiments of the present invention, and the human-standard ground truth.

[0027] Figure 13 A qualitative example from a dance dataset, as described in an embodiment of the present invention, is illustrated.

[0028] Figure 14AThe audio-visual harmony performance described in an embodiment of the present invention is shown, which is tested on a self-created test dataset using standard values ​​from real dancers and evaluation mechanism results from HarmoGAN models and their variants.

[0029] Figure 14B The audio-visual harmony performance of an embodiment of the present invention is illustrated, which is tested on a self-created test dataset using standard values ​​from real dancers and audio beat hit rates from music sequences in HarmoGAN models and their variants.

[0030] Figure 15A The audio-visual harmony performance of the prior art model described in the embodiments of the present invention and the audio-visual harmony performance of the HarmoGAN model and its variants are shown, the audio-visual harmony performance being tested on a dance hall dataset through an evaluation mechanism.

[0031] Figure 15B The audio-visual harmony performance of the prior art model described in the embodiments of the present invention and the audio-visual harmony performance of the HarmoGAN model and its variants are shown, the audio-visual harmony performance being tested on a dance hall dataset by the audio beat hit rate of a music sequence.

[0032] Figure 16 The results of participants described in an embodiment of the invention are shown, which are more consistent with the perceived harmony based on thirty video pairs created by placing dance videos from three models side by side.

[0033] Figure 17 An example of the qualitative evaluation described in an embodiment of the invention is shown, wherein the generated motion sequence is presented together with the tracked audio beats to demonstrate audio-visual harmony based on different models. Detailed Implementation

[0034] Embodiments of the invention will now be described in detail with reference to the accompanying drawings. Hereinafter, embodiments consistent with the invention will be described with reference to the drawings. Where possible, the same reference numerals will be used throughout the drawings to refer to the same or similar parts. Obviously, the described embodiments are some, but not all, embodiments of the invention. Based on the disclosed embodiments, those skilled in the art can derive other embodiments consistent with the invention, all of which are within the scope of the invention.

[0035] Harmony is a crucial component of artistic creation. Film directors often use songs that enhance emotional expression to create compelling scenes. When musicians arrange different vocal parts in a chorus, they should consider whether the combination is harmonious. Artists strive for harmony in their works to create beauty and comfort. Because such creative work requires specialized skills and technology, automated generative methods are applied to simulate the human creative process using computational models, in order to save financial costs and labor. Similar to human work, machine-based generative methods need to adhere to rules of harmony to produce high-quality results that satisfy human aesthetics.

[0036] Handling harmony in these generative tasks means that the model should strive to ensure consistency among multiple signals, which manifests as the explicit observation of alignment features or their implicit presence in the latent space. The synchronization process of different signal pairs may vary with correlation; therefore, it is easier to capture and align related features in highly correlated signal pairs. In human perception, over 90% of sensations arise from visual or audio stimuli, which are interconnected and interact during brain processing.

[0037] This invention provides a method and apparatus for harmony-aware audio-driven action synthesis. The method and / or apparatus can be applied to any suitable situation where human movement needs to be synthesized with music. The harmony-aware audio-driven action synthesis method is based on a beat-oriented generative adversarial network (GAN) model with a harmony-aware hybrid loss function; specifically, the model is a HarmoGAN model, which uses audio sequences or extracted audio features to generate visual action sequences. The harmony evaluation mechanism is incorporated into the GAN model to quantify the harmony between the audio and visual sequences by analyzing beat consistency.

[0038] Figure 1 This is a block diagram of an exemplary computing system / device capable of implementing the audio-driven action synthesis method for perceptual harmony described in embodiments of the present invention. Figure 1 As shown, the computing system 100 may include a processor 102 and a storage medium 104. In some embodiments, the computing system 100 may also include a display 106, a communication module 108, additional peripheral devices 112, and one or more buses 114 that couple these devices together. Some devices may be omitted in the embodiments, or other devices may be included.

[0039] Processor 102 may include any suitable processor. In some embodiments, processor 102 may include multiple cores for multi-threaded or parallel processing, and / or a graphics processing unit (GPU). Processor 102 may execute sequences of computer program instructions to perform various operations, such as audio-visual harmony evaluation operations and perceptual harmony audio-driven action synthesis programs, GAN model training programs, etc. Storage medium 104 may be a non-transitory computer-readable storage medium and may include storage modules, such as read-only memory (ROM), RAM, flash memory modules, erasable and rewritable memory, and mass storage devices such as optical disc (CD-ROM), USB flash drives, and hard disks. Storage medium 104 may store computer programs for implementing various processing procedures when executed by processor 102. Storage medium 104 may also include one or more databases for storing certain data, including video data, training datasets, test video datasets, and trained GAN model data. The databases may also be used to perform certain operations on the stored data, such as database searches and data retrieval.

[0040] The communication module 108 may include a network device for establishing a connection over a network. The display 106 may include any suitable type of computer display device or electronic device display (e.g., a cathode ray tube (CRT) or liquid crystal (LCD) based device, a touch screen). The peripheral device 112 may include additional input / output (I / O) devices, such as a keyboard, mouse, etc.

[0041] During operation, the processor 102 may be configured to execute instructions stored on the storage medium 104 and perform various operations related to the perceptual harmonic audio-driven motion synthesis method, as described below.

[0042] Figure 2 The illustration shows a motion synthesis process 200 driven by perceptual harmonic audio according to an embodiment of the present invention. The process 200 can be implemented by a motion synthesis device driven by perceptual harmonic audio, which can be any suitable computing device / server having one or more processors and one or more memories, such as computing system 100 (e.g., processor 102).

[0043] like Figure 2 As shown, the motion synthesis method driven by perceptual harmonic audio according to the embodiments of the present invention includes the following process.

[0044] Step S202: Determine multiple test beat units based on the input audio, each test beat unit corresponding to an input audio sequence of the input audio.

[0045] Step S204: Obtain the audio input corresponding to each test beat unit.

[0046] Step S206: Obtain the initial pose of each test beat unit as visual input based on the visual action sequence, wherein the visual action sequence is the visual action sequence synthesized for the previous test beat unit; and

[0047] Step S208: The generator of the GAN model automatically generates a perceptually consonant action sequence corresponding to the input audio. The GAN model is trained by introducing a hybrid loss function. The hybrid loss function includes multi-spatial pose loss, consonance loss, and GAN loss. The consonance loss is determined based on the beat consistency of the audio-visual beat pair.

[0048] like Figure 3 As shown, training the GAN model includes the following process.

[0049] Step S302: Obtain the audio beats and audio beat intensities from the training sample audio. Each audio beat corresponds to an audio beat intensity.

[0050] In audio processing, spectroscopic analysis is widely used to obtain audio beat B. a (t). The spectrogram of a given audio sequence A(t) can be obtained using a Fast Fourier Transform (FFT) with a time window. By estimating the amplitude of the spectrogram and searching for significant amplitude changes in the time domain, the beat can be extracted. The above process can be described as follows:

[0051] g(t)=Amp(FFT(A(t))) (1)

[0052]

[0053] Where Amp represents the function or model used to predict the amplitude of the spectrum. The positive threshold c1 is set to determine the existence of a beat at time t, and is related to the punctured neighborhood. Compared to any other time t′ in the time interval, the beat satisfies condition B. a (t) = 1, the puncture neighborhood Determined by a predefined radius t0. In mainstream methods, the amplitude is estimated by the initial intensity derived from the acquired spectrum. B is determined by the peak values ​​appearing in each initial envelope. a The audio beat (t) = 1.

[0054] In some embodiments, a mainstream method utilizing the initial intensity is employed to obtain the audio beat. All audio beats can actually be processed using methods in the open-source software package LibROSA, which enables the detection of the initial-driven beat of an audio signal. In some embodiments, the tempo is estimated by pre-calculating the audio beat intensity based on the autocorrelation within the initial envelope through analysis of the Mel spectrum. Referring to equations (1) and (2), B can be explained by the obtained tempo and the occurrence of peaks in the initial envelope at a constant t. a The audio beat (t) = 1. (The rest of the text appears to be unrelated and possibly machine-generated.) a The effective beats in (t) are combined to form position-based beats {p}. a (b) Collect data from the set b = 1, 2, ..., N at time t that satisfies B a All positions of N beats where (t) = 1. Simultaneously, beat p... a (b) The intensity is expressed as peak value s a (b) is used to represent this.

[0055] Step S304: Determine multiple training beat units based on the audio beat and audio beat intensity. Each training beat unit corresponds to a sample audio sequence in the training sample audio and a time index based on the time record in the training beat unit.

[0056] Figure 4 The diagram illustrates a beat header extracted based on the acquired audio beats. (Example) Figure 4 As shown, given an audio sequence A(t), the audio beat p can be obtained. a (b) and beat intensity s a (b) Is the rhythm strong or weak? e (b) It can be determined by comparing the intensity with the intensity of the previous beat, as follows:

[0057]

[0058] m e (b) = 1 means a high beat intensity, while 0 represents a low beat intensity. For example, for a quarter note, strong and weak beat combinations are mapped to 3 beat unit types, i.e., all 4 types. and Several beats from the previous beat unit are added to the current beat unit for transitions between beat units, thus creating a uniform beat length to describe the flow of musical rhythm. Figure 4In the illustrated embodiment, each beat unit includes 7 beats (uniform beat length) and one of four musical beat types listed in the mapping table. If the musical beat type is 4 beats, then the previous 3 beats are added to the current beat unit to fill the 7 beats, as shown below. Figure 4 Units 2 and 3 in the [rhythm unit]. Similarly, if the strong and weak beat combination is 5 beats, then the two previous beats are added to the current beat unit to fill 7 beats, for example... Figure 4 Unit 1 in the audio. Because there are no previous beats to fill the uniform length, the first strong-weak beat combination is discarded. The last beat unit ends with the last strong-weak beat combination. In this way, beat units can be extracted based on the identified strong-weak beat combinations of the audio beats. In some embodiments, test beat units can be obtained from the test audio input (e.g., S202) in the same way that trained beat units are extracted from training sample audio (e.g., S304).

[0059] In audio-driven video synthesis, the beat unit is used as the basic unit of the audio-visual pair. For each beat unit MU, the start time and end time are recorded as MU(t), where t∈[t...]. 开始 , t 结束 For a beat unit corresponding to an audio sequence A(t), the method of this invention can use a trained GAN model generator to find an action sequence V(t) that matches the audio sequence, and the match meets the requirements of harmony. Then, the method of this invention repeats a similar process for all beat units. According to the time record, during the training of the GAN model, known audio-visual matching video training samples with ideal harmony are obtained, and the corresponding audio sequence A(t) and action sequence V(t) are extracted as standard value pairs, including the audio beat p. a (b) and audio beat intensity s a (b)

[0060] Simultaneously, a time index TI(t) is formed in binary form to represent the interval between the current beat unit and the previous beat unit. If the time is the priority time of the previous beat unit, then TI(t) is set to 1. For example... Figure 4 As shown, the first three beats of beat unit 2 are marked as priority times belonging to the previous beat unit (TI(t) = 1), and the remaining four beats of beat unit 2 are marked as belonging to the current beat unit (TI(t) = 0).

[0061] In some embodiments, training the GAN model further includes temporally segmenting audio-visual segments based on multiple training beat units and using the segmented audio-visual segments as training input. The initial pose of each training beat unit is obtained from the corresponding audio-visual segment containing the sample audio sequence. In other words, if a sequence of human actions is coordinated with a given audio rhythm, the sequence of human actions can display regular and repetitive action units that are related to the audio beat. It can be considered that there is a correlation between such action units and the acquired audio beat. Therefore, temporally segmenting audio-visual segments based on defined beat units and using the segmented audio-visual segments as training input enhances the learning of beat-driven cross-domain units, which are mapped in a deep model consistent with embodiments of the present invention. This can indirectly benefit the audio-visual harmony in the generation method.

[0062] Step S306: Extract the features of the sample audio sequence in each training beat unit as the sample audio input.

[0063] In some embodiments, features of the input audio sequence A(t) of the current beat unit are extracted as audio input A. f (t). For example, for A(t), the features of Mel-scaled cepstral coefficients (MFCCs) are extracted as the audio input A. f (t). In some embodiments, the audio input corresponding to the test beat unit (e.g., S204) can be obtained in a manner similar to obtaining the audio input of the training samples (e.g., S306).

[0064] Step S308: Based on the time index and the visual action sequence of the training samples, obtain the initial pose of each training beat unit as the visual input of the samples.

[0065] In some embodiments, during training, the initial pose V is obtained from TI(t) and V(t) using the following formula. f (t):

[0066]

[0067] That is, visual feature V f (t) The action of the previous beat unit is taken as the priority action, and the average posture of the previous beat unit is used as the initial posture of the current beat unit. For example... Figure 4As shown, the visual features or initial posture corresponding to the first three beats of beat unit 3 are obtained from the action sequence corresponding to the last three beats of beat unit 2; in the remaining four beats of beat unit 3, the posture of each beat is the average posture corresponding to the last three beats of beat unit 2. This enables the network to enhance the temporal consistency in human motion synthesis methods.

[0068] Audio input (i.e., the audio features extracted in step S304) and visual input (i.e., the initial pose obtained in step S306) can be input into the generator of the GAN model. The structure of the generator G can be summarized as G(A f (t), V f (t))=V'(t), where V'(t) represents the action sequence generated in the beat unit. Figure 5B The testing phase is illustrated in summary. Based on the generator structure, as shown in Figure 5B, during the testing phase, V is obtained by processing the previously synthesized actions. f (t), which helps to repeatedly generate consistent human movements through beat units over random time periods. In some embodiments, the initial pose of the first beat unit can be pre-assigned and input into the generator.

[0069] Step S310: By merging the hybrid loss function, the GAN model is trained using sample audio input and sample visual input from each training beat unit to obtain the trained GAN model. The hybrid loss function includes multi-space pose loss, concordance loss, and GAN loss. The concordance loss is determined based on the beat consistency of the audio-visual beat pairs, which correspond to each training beat unit. Each audio beat in the audio-visual beat pair comes from the sample audio input of the training beat unit. Each visual beat in the audio-visual beat pair comes from the predicted visual action sequence, which corresponds to the training beat unit generated during the GAN model training process.

[0070] Due to their superior capabilities in visual generation tasks, GANs are also widely used in cross-domain generation methods. Figure 5A provides a general overview of the entire training process. In some embodiments, for training a GAN model, the GAN model includes a generator, and a cross-domain frequency discriminator and a spatiotemporal frequency discriminator for jointly monitoring the generator. In some embodiments, to supervise the realism of the generated human actions, in addition to the cross-domain frequency discriminator, a spatiotemporal pose frequency discriminator is used to determine the realism of the actions in space and time. The cross-domain frequency discriminator is widely used to control the effectiveness of feature transfer between different domains. In addition to the GAN loss provided by the frequency discriminator, a multi-spatial pose loss and a beat-driven harmony loss based on an attentional harmony mechanism are combined into the generator's regularization function. This perceptual harmony hybrid loss function can guide the generator to output a sequence of human actions that harmonizes with a given musical score by limiting the consistency between the audio beats and the visual beats, where the audio beats are pre-computed audio beats and the visual beats are visual beats extracted from the generated actions.

[0071] In audio-driven action synthesis tasks, audio sequences or extracted audio features are input into the network to generate visual action sequences. Encouraging efficient feature transformations within the architecture has always been a challenge due to the difficulty of cross-domain synthesis. To address this, an encoder-decoder structure is used to handle sequence-to-sequence transformations. Considering the temporal order of the input and output sequences, recursive structures are introduced into the encoder and decoder architecture to obtain features that take into account temporal correlation. Gated recurrent units (GRUs), as a typical structure of recurrent neural networks (RNNs), outperform general long short-term memory (LSTM) structures in sequence learning due to their fewer parameters and lower computational cost. In some embodiments, such as... Figure 5C As shown, the generator G includes a GRU-based audio encoder and a gesture decoder.

[0072] Furthermore, unlike pose decoding which only analyzes the audio features of the encoder output, initial pose features are concatenated with audio features to enhance the decoder's cross-domain learning. Jump concatenation is also used to intentionally add audio-visual features to future layers.

[0073] Since generator G is designed to generate human motion in 3D (3-Dimensional) poses, it is more difficult to accurately estimate the additional depth dimension compared to synthesizing 2D (2-Dimensional) motion. In some embodiments, the 2D pose is first estimated, and then a depth-enhancing branch is constructed based on the estimated 2D pose to generate the 3D pose. Utilizing the similarity between the 2D and 3D poses, depth can be generated efficiently.

[0074] In music-to-action synthesis methods, it's necessary not only to monitor the consistency of content style between the generated human actions and the target audio sequence, but also to monitor the realism of the synthesized human actions. Therefore, constructing a cross-domain frequency discriminator D... cd and spacetime frequency discriminator D st This is used to guide the network to learn global content consistency between audio-visual pairs and target pose streams in the spatial and temporal domains, respectively.

[0075] In some embodiments, such as Figure 5D As shown, in the cross-domain frequency discriminator, D cd For any audio-visual pair (A(t), V(t)), a two-branch classification network is used to determine the consistency of global style. After extracting audio and video features separately, the audio and video features are concatenated, and similarity is classified based on the acquired audio and video features. A cross-domain frequency discriminator D... cd This can improve the reasonable cross-domain transformation effect of generator G.

[0076] In some embodiments, to penalize unnatural movements, such as distorted human postures and unnatural transitions between movements, a spatiotemporal discriminator D is constructed using a time-progression network. st .like Figure 5E As shown, the input action sequence V(t) is divided into 5 parts on average according to its duration. By progressively extracting and splicing spatial features in chronological order, the spatiotemporal discriminator V(t) can guide the generator G to understand the spatiotemporal relationship of human actions in the standard values.

[0077] Harmony plays a crucial role in evaluating cross-modal outcomes. Since visual and audio are highly correlated and mutually influential in brain processing, harmony is particularly important in generative tasks that generate actions from audio to visual or vice versa. Taking audio-driven human motion synthesis as an example, audio-visual harmony is emphasized in quality assessment; that is, the synthesized actions should be rhythmic and harmonious with the music. In other words, to achieve sensory harmony, the rhythms in the audio and video sequences need to be consistent over time. Since rhythmic perception depends on human subjective perception, given an audio sequence A(t) and a visual sequence V(t) as functions of time t, the perceptual judgment of harmony can be approximated as a quantitative measurement using the following formula:

[0078] h=H(A(t),V(t)) (5)

[0079] Where H represents the algorithm for analyzing harmony between cross-domain signals and h is a scalar representing the quantization judgment of harmony.

[0080] The rules for detecting audio beats are similar to those for analyzing the movement trends between visual sequences V(t) and V(t-t0), extracting the visual beat B. v (t). When the trend of action changes drastically, it is assumed that time t generates rhythm, which is expressed by the following formula:

[0081] d(t)=MT(V(t),V(t-t1)) (6)

[0082]

[0083] Where MT represents a function or model that predicts the trend of movement during the action at time t1 (a predefined constant value), c2 is a positive value used to control the threshold for acquiring the beat at time t, and the puncture neighborhood. Compared to any other time t' in the time interval, the beat satisfies condition B. a (t) = 1, the puncture neighborhood Determined by a predefined radius t2. To process general pixel-based visual signals, optical flow can be used to capture motion trends in moving events. By quantizing the optical flow, visual beats can be obtained by deriving local maxima, which represent significant changes in motion. When focusing only on human motion in such pixel-based signals, skeleton-driven methods can be used to determine motion trends of pure skeletal movements, which are pure skeletal movements extracted from the visual signal. Therefore, by using joint-based standard deviations, the prediction of motion trends can be transformed into an analysis of body motion directions. Therefore, B... v The visual beat (t) = 1 is defined as a significant change in direction in a sequence of actions.

[0084] Based on the observed audio and visual beats, a common assumption is derived to address the rhythmic consistency problem: the occurrence of each audio beat should be synchronized with the occurrence of the visual beat, and vice versa. Based on this assumption, by aligning the extracted beats and evaluating the quantized audio-visual harmony h using existing strategies, equation (5) is extended to:

[0085] h = L(f a (A(t)), f v (V(t)))=L(B a (t), B v (t)) (8)

[0086] Where f a and f v The symbols represent the beat detection functions in the audio and visual signals, respectively. L represents the alignment algorithm. In some embodiments, the algorithm L can describe the alignment problem by analyzing the distance between synchronized beat pairs using warp, cross-entropy, or F-score analysis, which is effective for aligning cross-domain objects.

[0087] In some embodiments, taking human videos as an example, given humans and their actions as the focus, harmony is primarily considered as the alignment between foreground human actions and the associated background music in a visual frame. To better analyze human actions in the foreground, a human skeleton is extracted to represent human actions in the video. Therefore, harmony is evaluated between the audio signal and the skeleton-based human actions.

[0088] In order to visualize subjective evaluations of harmony through objective expressions, a tolerance field adjacent to the audio beat is set based on human reaction time, so as to represent the perceptual judgment of audio-visual harmony through the synchronization effect between beats. Figure 6 An embodiment of asynchronous audio-visual beats is shown, in which the beats of the audio signal and the beats of the video signal are processed separately, without achieving satisfactory alignment.

[0089] In visual scenarios, optical flow is typically used to extract the beat of a general visual signal based on frames. However, this method is less effective when working with human video compared to skeleton-based methods. One reason is that optical flow treats the action of each pixel almost equally, thus giving higher weight to foreground elements. Figure 7A and Figure 7BAs shown, since changes in human motion within video frames are perceptually distinct, visual beats should be obtained through beat extraction methods. However, optical flow methods struggle to detect the beats of foreground human motion due to interference from potential background movement events, while skeleton-based methods significantly outperform optical flow methods in capturing visual beats that better align with human perception. A visual beat detection method based on joint standard deviation (SD) is used to represent the skeleton-based approach, where visual beats are detected by predicting changes in the direction of human motion.

[0090] In some embodiments, beat alignment experiments on dance datasets combined mainstream detection methods based on the initial audio beat with an SD-driven visual beat detection method. Since human reaction time is approximately 0.25 seconds, the tolerance field radius for each audio beat was set to 6 frames, resulting in a total duration of 0.24 seconds at 25 fps (frames per second) to evaluate the audio-visual alignment results. However, the results were not entirely satisfactory. Figure 8A and Figure 8B As shown, due to omissions and redundancies between beat pairs, harmony distortion is quite high. Studies have shown that although SD-driven visual beat detection methods are largely consistent with subjective perception, such cross-domain audio-visual beat pairs are inconsistent when it comes to start-based audio beats. To work with start-based audio beats, this invention provides a novel beat extraction mechanism that takes into account joint velocities in adjacent frames to determine the weights used for detecting visual beats, which can satisfy better beat consistency during audio-visual alignment.

[0091] In some embodiments, training the GAN model further includes detecting the visual beat of the predicted visual action sequence by calculating the difference between the sums of joint velocities of adjacent frames in the predicted visual action sequence.

[0092] For a skeleton-oriented human motion sequence v with j joints and obtained from V(t) at frame t... s (t, j), joint velocity and J v (t) is obtained by calculating the frame difference:

[0093]

[0094] Where i represents the i-th joint. In some embodiments, diversity and frequency can be regularized based on the analysis of joints and .

[0095] In order to define a movement beat that is completely consistent with the audio beat, the evolution of indivisible movement units (such as raising a hand) focuses primarily on analyzing the visual beats in the entire movement sequence.

[0096] Figure 9The correlation between velocity graphs and real human motion is shown. When the changes in joint velocities approach zero, it is usually related to the complementarity of individual motion units. Therefore, the beat of motion can be defined as the peak or trough in the velocity graph where the acceleration is equal to zero. Equations (6) and (7) can then be modified as follows:

[0097]

[0098]

[0099] Similar to the case of audio, for M satisfying conditions The effective beat, a position-based visual beat, can be represented as {p} v (b)|b=1,2,...,M}, and according to the corresponding J v (t) and s v (b) Determine the beat intensity. For example... Figure 10A and Figure 10B As shown, when in such Figure 8A and Figure 8B When tested under the same conditions shown, the visual beats obtained in this way are essentially synchronized with the initial audio beats. This is achieved through comparison. Figure 8B and Figure 10B It can be observed that the novel beat extraction mechanism described in this embodiment of the invention is superior to existing skeleton-based methods that use motion SD by reducing the omission and redundancy of the beat synchronization process.

[0100] Besides the synchronization between beats, another factor that significantly influences the perception of audio-visual harmony is the attention mechanism described in this invention. Because human attention is drawn to more "attractive" things, other elements may be unconsciously ignored during perception. Therefore, to evaluate audio-visual harmony that closely approximates real human perception, an attention mechanism needs to be incorporated into the evaluation framework.

[0101] Attention mechanisms suggest that, without conscious awareness, human perception ignores unwanted objects, which affects both the visual and auditory systems. When it comes to the subjective perception of rhythmic harmony, inattention leading to blindness and deafness can also affect the judgment of the salience distribution of audiovisual rhythms. To approximate the measurement of the perception of harmony, embodiments of the present invention provide an attention-based evaluation framework to emphasize the importance of salience beats, which extends equation (8) to:

[0102] h = L(W a (p a (b)), W v (p v (b))) (12)

[0103] Among them W a and W v These represent attention weight masks derived from audio beat saliency and visual beat saliency, respectively.

[0104] In some embodiments, weights are assigned to each audio beat and each visual beat based on beat salience to determine the beat consistency of the audio-visual beat pairs corresponding to the training beat units. Since salience beats are beneficial for the perception of harmony, weights are assigned to each beat according to its salience to enhance the corresponding attentional influence in the evaluation. Beat salience is determined by beat intensity s. a (b) and s v (b) and the adaptive weight mask representation, which is constructed by taking into account the global SD for beat intensity.

[0105] In audio saliency analysis, the attention weight mask is constructed as follows:

[0106] W a =sign(s) a (b)-SD(s a (b))×λ1) (13)

[0107] Where λ1 represents a constant scaling factor used to adjust the audio saliency threshold.

[0108] Unlike masks used for audio beats, in the visual case, action beats are extracted not only from the peaks of joint velocities but also from their troughs. This means that direct comparisons with SD (Signal Depth) are not suitable for analyzing visual saliency. Therefore, the peak-to-trough difference is used to define visual saliency intensity for detecting the appearance of visually impactful beats, as follows:

[0109] R(b) = |s v (b)-s v (b-1)|,b=2,...,M (14)

[0110] Where R(b) represents the peak-to-valley difference for each beat.

[0111] Then, a visual saliency weight mask W is defined using global SD. v :

[0112] W v =sign(s) v (b)-SD(R(b))×λ2) (15)

[0113] Where λ2 represents a constant scaling factor used to adjust the visual saliency threshold.

[0114] In some embodiments, in the audio-visual beat pairs of the training beat units, attention beats and the beat intensity of each attention beat are obtained according to the weights of the audio beats and the visual beats, to further determine the beat consistency with the audio-visual beat pairs corresponding to the training beat units. The attention beats include one or more attention audio beats and one or more attention visual beats. This is achieved by separately determining the beat intensity at beat p. a (b) and p v (b) Apply weighted mask W a and W v , through W a (p a (b)) and W v (p v (b) Extracting positive results to obtain attention beat p′ a (b) and p′ v (b). Similarly, the beat intensity corresponding to the attention beat is defined as s′. a (b) and s′ v (b)

[0115] The sense of harmony in human audio-visual perception can be described as a fuzzy measurement stemming from how the human brain recognizes sensory signals. In some embodiments, existing warping methods are used to handle the beat alignment process by directly adjusting the intensity curve of the visual beat to match the intensity curve of the audio beat through the application of compensation. In some embodiments, contrast differences are constructed by calculating the cross-entropy distance between the audio amplitude and the action tag. Due to the limitations of the human brain in recognizing signals with precise amplitude, this fine-grained intensity-based mapping between audio-visual beats is inconsistent with true perception.

[0116] In some embodiments, a hit score is constructed by computing labels in the audio and visual domains to represent the aligned attentional beats in the sample audio input and the predicted visual action sequence, thereby further determining the beat consistency of the audio-visual beat pairs corresponding to the training beat units, where one label indicates that, based on human reaction time delay, an attentional audio beat is aligned with the corresponding attentional visual beat. That is, the hit score is determined by binary labels (such as...) used to indicate whether the audio and visual beats are synchronized in the temporal domain. Figure 6 Inspired by this approach, a hit score is constructed by calculating "good" labels in both the audio and visual domains to indicate whether the beats are fuzzily aligned throughout the sequence. To balance audio-visual perception, an F-score is used to fuse cross-domain scores for the final judgment.

[0117] Select N audio beats p′ with high significance. a (b) and M visual beats p′ v(b) Begin by reconstructing equation (12) using F-scoring:

[0118] h=L(p′ a (b), p′ v (b))=F s (E(p′ a (b)), E(p′) v (b))) (16)

[0119] Where E represents the algorithm for obtaining hit scores in the audio and visual domains. F s This represents the measured value of the F-score.

[0120] By observing a delay between visual perception and brain processing for recognition, it can be hypothesized that a beat has been hit as long as the time interval between the beat and the most recent cross-domain beat is less than the human reaction delay. In this way, alignment effectiveness can be measured by making judgments based on fuzzy intervals, rather than relying on precise intensity-based mappings. When synchronized beats occur within an audio-visual pair, the audio beat can be considered the anchor in the hit analysis. To obtain the interval, M visual beats p′ are used... v (b) Repeat N times to construct the position matrix Z(b) a b v ):

[0121]

[0122] Where b a =1, 2, ..., N and b v =1, 2, ..., M.

[0123] Based on Z(b) a b v The column audio-visual interval D(b) a b v By calculating and subtracting p′ a (b a It's absolutely worth it:

[0124] D(b a b v )=|Z(b a b v )-p′ a (b a (18)

[0125] This can be achieved by using the minimum audio-visual interval T(b) based on the row. a The signal is compared with the predetermined response delay to determine whether the audio beat Hp(b) is hit. a ):

[0126] T(b a )=min(D(b a b v (19)

[0127]

[0128] Where T 延迟 It is a constant frame time, and Hp(b) a =1 indicates the existence of synchronized audio-visual beat pairs.

[0129] Finally, the hit score h is obtained by calculating the weights of all hit scores. s :

[0130]

[0131] Considering the standardization of the total number of audio beats and motion beats, the hit score of audio harmony can be constructed as follows: The hit score for visual harmony can be constructed as follows: However h a and h v It varies depending on the source. For example, given a specific input audio-visual sequence, the obtained h a Possibly more than h v However, for another input sequence, the opposite result may be obtained.

[0132] In some embodiments, beat consistency is further determined by using hit scores to determine beat consistency, thereby further determining the beat consistency of the audio-visual beat pairs corresponding to the training beat units. That is, in some embodiments, in order to achieve a balance between audio-visual scores, the final audio-visual harmony h is obtained by calculating the average of the harmony, which modifies equation (16) to:

[0133]

[0134] Where β is a predefined constant. Therefore, equation (22) can be transformed into h s Functions:

[0135]

[0136] From equations (22) and (23), it can be concluded that the quantification of audio-visual harmony in the evaluation method can be described as follows:

[0137] In this invention, the evaluation of harmony is based on the following lemma (1): Given an audio segment with N attentional audio beats and a visual segment with M visual beats, the quantized audio-visual harmony can be obtained from h sUniquely certain

[0138] The hybrid loss function includes multi-spatial pose loss, harmonic loss, and GAN loss. The multi-spatial pose loss is used to adjust the realism of the predicted human motion. In some embodiments, the multi-spatial pose loss includes one or more of Kullback-Leibler (KL) loss, Charbonnier-based mean squared error (MSE) loss, and Charbonnier-based VGG loss.

[0139] For the allocation space, the Kullback-Leibler (KL) loss function Based on intermediate results of 2D pose generation:

[0140]

[0141] in This represents the Kullback-Leibler (KL) loss. This represents a 2D action sequence V with standard values. 2d (t) and intermediate output V 2d The operation of converting ′(t) into a probability distribution.

[0142] In pixel space, Charbonnier-based MSE loss The following is used to constrain the generation of 3D poses:

[0143]

[0144] Where ∈ is a near-zero positive constant used to mitigate gradient vanishing during training. A weight mask is applied based on the time index TI(t). This is to guide the network to focus on the motion generation method of the current beat.

[0145] VGG networks are widely used to generate visual features consistent with human perception, based on Charbonnier's VGG loss. Human motion generated in a deep feature space can be regularized in the following ways:

[0146]

[0147] In some embodiments, feature space pose loss, such as Kullback-Leibler (KL) loss, Charbonnier-based MSE loss, or Charbonnier-based VGG loss, can capture deep features of the action flow and regularize the flow in the synthesized action to be consistent with standard values.

[0148] According to Lemma (1), the harmony between the audio sequence and the human action sequence can be determined by evaluating the audio-visual beat consistency, which depends solely on the hit score h. s Therefore, the harmony loss is generated through the following function:

[0149]

[0150] Here, VB represents the extraction of attentional visual beats with corresponding beat intensities based on the estimated human motion sequence from the generator. These results are then sent to algorithm E, using pre-calculated p′. a (b) and s′ a (b) To calculate the hit score. In addition to minimizing the negative hit score, an L1 distance is added to excessively frequent visual beats by comparing the number of visual beats M with the number of audio beats N.

[0151] By solving the min-max problem, GANs can learn to generate outputs based on the distribution of given data during adversarial training:

[0152]

[0153] Where φ and θ represent the parameters of the frequency discriminator and the generator, respectively. x represents the standard value data, and Y is the input to the generator.

[0154] In some embodiments, training the GAN model further includes minimizing the harmony loss from the generator, the multi-space pose loss, and the GAN loss, as well as maximizing the values ​​of the loss functions of the cross-domain discriminator and the spatiotemporal discriminator to distinguish between real and fake training samples. Therefore, the cross-domain discriminator and the spatiotemporal pose discriminator attempt to distinguish between real and fake samples by maximizing the loss function.

[0155]

[0156]

[0157] Conversely, the generator attempts to deceive the discriminator by minimizing the function:

[0158]

[0159] In summary, considering all the loss functions mentioned above, the final loss function of the generator can be expressed as:

[0160]

[0161] λ kl λ represents the weights of the KL loss function. mse λ represents the weights of the MSE loss function. featλ represents the weights of the VGG loss function. harmo The weights λ represent the harmonic loss function. gan The weights represent the weights of the GAN loss function.

[0162] In some embodiments, such as Figure 2 As shown, step S202 may include determining multiple test beat units based on the audio beat and audio beat intensity, each test beat unit corresponding to an audio sequence of the input audio and a time index corresponding to a time record based on the test beat unit. Step S204 may include extracting features of the audio sequence from each test beat unit as audio input. Step S206 may include obtaining the initial pose of each test beat unit as visual input based on the time index and the visual action sequence synthesized for the previous test beat unit.

[0163] In some embodiments, obtaining the initial pose of each test beat unit includes retaining the action sequence generated by the previous test beat unit in the initial pose of the current test beat unit, and using the average pose of the perceptually harmonious action sequence generated by the previous test beat unit as the initial pose of the current test beat unit.

[0164] In some embodiments, in one implementation, the work "Dance with melody: An LSTM-autoencoder approach to music-oriented dance synthesis, procedures of the 26" published by Tang et al. is utilized. thThe HarmoGAN model was trained using a dance dataset published at the ACM International Conference on Multimedia, 2018 (hereinafter referred to as [Tangental., 2018]). This dataset comprises 61 dance video sequences performed by professional dancers, totaling 94 minutes and 907,200 frames at 25fps. The dance dataset provides 3D human keypoints and corresponding audio trajectories, with the 3D keypoints including 21 joints collected via a wearable device. The dataset includes four typical dance types: Cha-cha, Rumba, Tango, and Waltz. To save memory, all videos were resampled at 15fps to create a sample dataset. 2014 concatenated audio-visual input feature segments and corresponding target poses were extracted from the entire dance dataset; 214 were randomly selected as in-house test data, and the remainder were used for model training. All functions for music feature extraction can be found in McFee et al.'s paper, "Librosa: Audio and music signal analysis in Python, procedures of the 14...". th The Librosa package was found in the paper "Python in Science Conference, 2015" (hereinafter referred to as "[McFee et al., 2015]").

[0165] To evaluate the harmony between the audio and the synthesized audio-driven action sequences, the HarmoGAN model was tested using a dance hall music dataset published by Gouyon et al. in "An experimental comparison of audio tempo induction algorithms, IEEE Transactions on Audio, Speech, and Language Processing, 2006" (hereinafter referred to as [Gouyon et al., 2006]). This dance hall music dataset extracted 698 background music segments from online dance videos, each segment lasting 30 seconds, and included music for seven dances: Cha-cha, Swing, Quickstep, Rumba, Samba, Tango, and Waltz. Within each dance category, six audio sequences were randomly selected to form the test dataset. A beat-based harmony mechanism was used to quantify the audio-visual harmony, and the audio beat information was obtained using the Librosa software package published in [McFee et al., 2015].

[0166] Implementing HarmoGAN in PyTorch. First, the generator is pre-trained to prepare a suitable initialization process for subsequent training. Pre-training is performed through... Training ended after 225 epochs. The optimizer disclosed by Kingma and Ba in "Adam: A method for stochastic optimization, arXir preprint arXir: 1412.6980 (2014)" (hereinafter referred to as [Kingma and Ba, 2014]) was used with a batch size of 10. The initial learning rate was set to 0.001, and subsequently decreased by multiplying the parameter every 50 epochs in the following order: [0.5, 0.2, 0.2, 0.5]. The model was initialized with a pre-trained model, and GAN training began with a generator and discriminator network. The weights of the loss components in the generator's hybrid loss function were set as follows: λ kl =0.0001, λ mse =λ feat =λ gan =0.001, λ harmo =1. The discriminator weight decay was set to 0.001, and the generator weight decay was set to 0.0001. The learning rate of all networks was initialized to 0.0001, and then alternately divided by 2 and 5 every 5 epochs. The optimizer and batch size remained the same as in pre-training. After 45 epochs of adversarial training, the HarmoGAN model was finally obtained. The entire training process was completed in just 53 minutes using an NVIDIA TITAN V GPU, which is quite efficient.

[0167] For the harmony evaluation mechanism, the constant factors λ1 and λ2 in equations (13) and (15) are set to 0.1 and 1, respectively, to obtain attention saliency. Simultaneously, the response delay is set to 0.25 seconds, i.e., T... delay =3.75. The frame rate in Equation (20) is less than 15fps. When evaluating the quantized audio-visual harmony, the β of the F-score in Equation (22) is set to 2 to focus more on the hit rate of the audio beat.

[0168] To confirm that dissonance occurring in audio-visual objects can be observed through human perception, a user study was conducted to test participants' sensitivity to dissonant audio-visual segments. Twenty dance videos were collected, including 10 harmonic segments from the dance dataset published in [Tang et al., 2018], and 10 dissonant segments created by arranging audio or video sequences. Ten invited participants were asked to watch all 20 videos and select all sequences they considered dissonant to provide a perceptual harmony assessment.

[0169] Figure 11 The results of a user study were presented, showing that 78% of the dissonant videos were accurately selected by participants. Due to a lack of background knowledge in professional dance, participants had difficulty distinguishing between all dissonant and harmonious segments. In summary, it can be concluded that audio-visual harmony influences human perception, and in most cases, dissonance can be correctly observed and judged through perception.

[0170] Before analyzing the harmonic performance of the model, the HarmoGAN model should first demonstrate a reasonable ability to synthesize natural motion flows based on the human skeleton. To evaluate the generated motions, the HarmoGAN model was tested on a self-created test dataset derived from a dance dataset published in [Tang et al., 2018], which provides standard values ​​of dance movements performed by real human dancers. The perceptual distance between the estimated motion sequence and human standard values ​​was measured using the FID metric proposed by Heusel et al. in “Gans trained bva two time-scale update rule converge to a local nash equilibrium, arXir preprint arXir: 1706.08500 (2017)” (hereinafter referred to as [Heusel et al., 2015]). Since there is no standard for extracting features from pose sequences, the VGG network proposed by Simonyan and Zisserman in "Very deep convolutional networks for large-scale image recognition, arXir preprint arXir: 1409.1556 (2014)" (hereinafter referred to as [Simonyan and Zisserman, 2014]) is used to obtain pose features for measuring FID. The average results are as follows: Figure 12 As shown. Figure 12As shown, this relates to the dance generation model proposed by Lee et al. in "Self-supervised Dance Video Synthesis Conditioned on Music, proceedings of the 28th ACM International Conference on Multimedia, 2020" (hereinafter referred to as [Lee et al., 2019]), and the dance generation model proposed by Ren et al. in "Self-supervised Dance Video Synthesis Conditioned on Music, proceedings of the 28th ACM International Conference on Multimedia, 2020" (hereinafter referred to as [Lee et al., 2019]). th Compared with the dance generation model proposed in "ACM International Conference on Multimedia, 2020" (hereinafter referred to as "Ren et al., 2020"), it can be seen that the HarmoGAN model is competitive with the state-of-the-art model, which can learn to synthesize human movements and the human movements have a high feature space similarity with the real human movements in the training dataset.

[0171] At the same time, Figure 13 The image shows action sequence pairs used for qualitative assessment. The human action sequences synthesized by the HarmoGAN model exhibit a motion flow similar to standard human values.

[0172] To analyze the enhancement of audio-visual harmony after introducing harmony loss during network training, an ablation study was conducted to evaluate the performance of HarmoGAN models and their variants on a self-created test dataset without affecting the quality of audio-visual harmony. The performance of the dataset includes the relevant initial poses for generating action sequences. Given pre-calculated audio beats from a music sequence, audio-visual beat consistency can be analyzed and harmony evaluated based on estimated human movements.

[0173] In addition to Figure 14A The quantified harmony derived from the harmony evaluation mechanism shown also calculates a hit rate to evaluate performance; this hit rate is commonly used when displaying harmony. For example... Figure 14B As shown, the percentage of musical beats hit by the visual beats within the reaction delay period is calculated to represent the beat hit rate. Human dancers typically hit only about half of the musical beats, which is a reasonable result considering the limited precision of data acquisition when acquiring movement sequences. Figure 14A and Figure 14B As shown, it is clear that the following has been introduced. The audio-visual harmony is far better than without it. The variation exhibits superior audio-visual harmony, surpassing that of a real dancer. This also means that even when deviating from standard values, The need for "privileged" editing of action sequences for coordination can explain why, without using... The reason why the baseline variant performs worse is because it is not designed to strictly simulate the standard values. In summary, tests on a self-built dataset demonstrate the superior performance of the HarmoGAN model, which can improve the audio-visual harmony between a given music sequence and the generated action with the help of harmony loss.

[0174] To further evaluate the coordination capabilities of the HarmoGAN model, in addition to variants, it was compared with two state-of-the-art GAN-based models proposed in [Lee et al., 2019] and [Ren et al., 2020] using an audio-driven action synthesis method. For a fair comparison, all models were tested on the dance hall dataset proposed in [Gouyon et al., 2006], a public music dataset that provides background music for various dance types. Forty-two 6-second audio track clips were randomly collected from the dance hall dataset as the test dataset. Without any given standard human motion values, action sequences from the training dataset were selected as initial poses to generate dance sequences.

[0175] like Figure 15A and Figure 15B As shown, the results of quantifying harmony and the hit rate of musical beats are demonstrated by calculating the average results of seven types of dance music. The results show that the HarmoGAN model significantly outperforms other models in both of these metrics.

[0176] Detailed assessment results for each type of dance are shown in Tables 1 and 2. (Note: The text excludes items used as a baseline.) Compared to the HarmoGAN model, the spatiotemporal GCN model proposed by [Ren et al., 2020] assists and is inherently beneficial to obtaining harmonious results by normalizing the hierarchical representation of the skeleton during the generation of action sequences. However, this improvement lacks robustness and is easily affected by bias in the training dataset. The post-processing beat warper proposed by [Lee et al., 2019] can improve performance relatively uniformly, but the improvement is still limited. Compared to the above models, the HarmoGAN model described in this invention is not limited by dance type and directly produces a significant and robust improvement in audio-visual harmony.

[0177] In some embodiments, a HarmoGAN model can be executed to generate visual sequences based on video frames. In some embodiments, a multi-level system or an end-to-end system can be constructed to achieve audio-visual harmony based on the video frames.

[0178] Table 1 Quantitative Harmony for 7 Types of Dance Music

[0179]

[0180] Table 2 shows the audio beat hit rate for 7 types of dance music.

[0181]

[0182] Furthermore, the cost of the tested models was analyzed based on the number of model parameters and training pairs. The number of parameters in the generator of the HarmoGAN model is close to that in [Ren et al., 2020], and about half the number in [Lee et al., 2019], while [Lee et al., 2019] required a training dataset 10 times larger to obtain the final model. Therefore, considering the harmonious evaluation results of each model, it can be found that the HarmoGAN model can effectively improve performance without adding too much cost in the training and testing phases.

[0183] To qualitatively evaluate audio-visual harmony, dance videos were synthesized by combining audio sequences and generated motions derived from the tested model. A user study was then conducted to compare the perceived harmony of the synthesized videos. Twelve non-expert participants were invited to view pairs of videos from three different models. The non-expert participants were asked to blindly select which model demonstrated better audio-visual harmony.

[0184] like Figure 16 As shown, it can be concluded that the HarmoGAN model performs best in terms of perceptual harmony, which is consistent with the results based on quantitative indicators. The hypothesis that the harmony evaluation mechanism can accurately reflect perceptual audio-visual harmony can also be verified to some extent.

[0185] As an example of qualitative assessment, such as Figure 17As shown, a sequence of movements generated from a test model is presented based on a dance hall dataset. In the movement described in [Ren et al., 2020], given a tracked audio beat, it is difficult to perceive a distinct visual beat from slight body swaying, let alone audio-visual consistency. When it comes to the predicted visual results disclosed in [Lee et al., 2019], a reasonable visual beat can be perceived by observing changes between movements. However, this change is relatively uniform throughout the sequence, which can be affected by inattention and lead to perceptual inconsistencies between audio and visual beats due to misjudgments. In contrast, the HarmoGAN model described in this invention regularizes the harmony loss, thereby generating distinct movement changes near the point where the musical beat is generated, thus providing a visual beat that can attract sufficient attention and is beneficial for harmony assessment based on human perception.

[0186] Other embodiments of the invention will be apparent to those skilled in the art, taking into account the specifications and practices disclosed herein. It is intended that the specification and embodiments be merely exemplary, and that the claims indicate the true scope and subject matter of the invention.

Claims

1. An audio-driven action synthesis method, the method being applied to a computer device, characterized in that, Including the following steps: Multiple test beat units are determined based on the input audio, and each test beat unit corresponds to an input audio sequence of the input audio. Obtain the audio input corresponding to each test beat unit; The initial pose of each test beat unit is obtained as visual input based on the visual action sequence, wherein the visual action sequence is a synthesized visual action sequence corresponding to the previous test beat unit; and The generator of the generative adversarial network model generates an action sequence corresponding to the input audio; The generative adversarial network model is trained by introducing a hybrid loss function, which includes multi-space pose loss, concordance loss and generative adversarial network loss. The concordance loss is determined based on the beat consistency of the audio-visual beat pair. This also includes the following steps: Train a generative adversarial network (GAN) model to obtain the trained GAN model. The training of the generative adversarial network model to obtain the trained generative adversarial network model includes: The audio beats and beat intensities are obtained from the training sample audio, with each audio beat corresponding to an audio beat intensity. Multiple training beat units are determined based on the audio beat and the audio beat intensity. Each training beat unit corresponds to a sample audio sequence in the training sample audio and a time index based on the time record in the training beat unit. Features of the sample audio sequences in each training beat unit are extracted as sample audio input; Based on the time index and the visual action sequence of the training samples, the initial pose of each training beat unit is obtained as the visual input of the samples; and The generative adversarial network (GAN) model is trained by merging the hybrid loss function using the sample audio input and sample visual input of each training beat unit to obtain the trained GAN model. The concordance loss is determined based on the beat consistency of the audio-visual beat pairs, which correspond to each training beat unit. Each audio beat in the audio-visual beat pair comes from the sample audio input of the training beat unit, and each visual beat in the audio-visual beat pair comes from the predicted visual action sequence, which corresponds to the training beat unit generated during the training of the GAN model.

2. The method according to claim 1, characterized in that, Training the generative adversarial network model also includes: The visual beat of the estimated visual motion sequence is detected by calculating the difference between the sums of joint velocities of adjacent frames in the estimated visual motion sequence.

3. The method according to claim 1, characterized in that, The method for determining the tempo consistency of the audio-visual tempo includes the following steps: Weights are assigned to each audio beat and each visual beat based on beat saliency. In the audio-visual beat pair of the training beat unit, attention beats are obtained according to the weights of the audio beats and the visual beats, and the attention beats include one or more attention audio beats and one or more attention visual beats. Obtain the beat intensity of each attention beat; Labels are computed in the audio and visual domains to represent the aligned attentional beats in the sample audio input and the predicted visual action sequence, thereby constructing a hit score, where one label represents the alignment of an attentional audio beat with its corresponding attentional visual beat based on human reaction time delay; and The hit score is used to determine the beat consistency.

4. The method according to claim 1, characterized in that, Training the generative adversarial network also includes: Audio-visual segments are temporally segmented based on multiple trained beat units; The audio-visual segment is input for training; The initial pose of each training beat unit is obtained from the corresponding audio-visual segment containing the sample audio sequence.

5. The method according to claim 1, characterized in that, The generative adversarial network model also includes a cross-domain frequency discriminator and a spatiotemporal frequency discriminator, which jointly monitor the generator. Training the generative adversarial network model also includes: Minimize the multi-space pose loss, the harmony loss, and the generative adversarial network loss from the generator; as well as Maximize the values ​​of the loss functions of the cross-domain frequency discriminator and the spatiotemporal frequency discriminator to distinguish between real training samples and fake training samples.

6. The method according to claim 1, characterized in that, The multi-space pose loss includes at least one of Kullback-Leibler loss, Charbonnier-based MSE loss, and Charbonnier-based VGG loss.

7. The method according to claim 1, characterized in that, The generator includes a GRU-based audio encoder and a gesture decoder; The attitude decoder is used for: Two-dimensional pose is estimated based on the visual motion sequence corresponding to the beat unit; Depth boosting branches are constructed based on the estimated 2D pose to generate 3D pose.

8. The method according to claim 1, characterized in that, The process of determining multiple test beat units based on the input audio includes: Multiple test beat units are determined based on the audio beat and audio beat intensity of the input audio. Each test beat unit corresponds to an audio sequence of the input audio and a time index based on the time record in the test beat unit. The acquisition of audio input corresponding to each test beat unit includes: Extract the features of the audio sequence in each test beat unit as the audio input; The initial pose of each test beat unit is obtained based on the visual action sequence as visual input. The visual action sequence is a synthesized sequence corresponding to the previous test beat unit, including: The initial posture of each test beat unit is obtained as visual input based on the time index and the visual action sequence, and the visual action sequence is the visual action sequence synthesized corresponding to the previous test beat unit.

9. The method according to claim 8, characterized in that, The process of obtaining the initial posture of each test beat unit includes: The motion sequence generated by the previous test beat unit will be preserved in the initial pose of the current test beat unit; and The average pose of the motion sequence generated by the previous test beat unit is used as the initial pose of the current test beat unit.

10. An apparatus for implementing an audio-driven action synthesis method, characterized in that, include: Memory; as well as A processor coupled to the memory performs the following steps: Multiple test beat units are determined based on the input audio, and each test beat unit corresponds to an input audio sequence of the input audio. Obtain the audio input corresponding to each test beat unit; The initial pose of each test beat unit is obtained as visual input based on the visual action sequence, wherein the visual action sequence is a synthesized visual action sequence corresponding to the previous test beat unit; and The generator of the generative adversarial network model generates an action sequence corresponding to the input audio; The processor further implements the following: training the generative adversarial network model by introducing a hybrid loss function, which includes multi-space pose loss, concordance loss and generative adversarial network loss, wherein the concordance loss is determined based on the beat consistency of the audio-visual beat pair; The processor further implements the following steps: Train the generative adversarial network model; Training the generative adversarial network model includes: The audio beats and beat intensities are obtained from the training sample audio, with each audio beat corresponding to an audio beat intensity. Multiple training beat units are determined based on the audio beat and the audio beat intensity. Each training beat unit corresponds to a sample audio sequence in the training sample audio and a time index based on the time record in the training beat unit. Features of the sample audio sequences in each training beat unit are extracted as sample audio input; Based on the time index and the visual action sequence of the training samples, the initial pose of each training beat unit is obtained as the visual input of the samples; and The generative adversarial network (GAN) model is trained by merging the hybrid loss function using the sample audio input and sample visual input of each training beat unit to obtain the trained GAN model. The concordance loss is determined based on the beat consistency of the audio-visual beat pairs, which correspond to each training beat unit. Each audio beat in the audio-visual beat pair comes from the sample audio input of the training beat unit, and each visual beat in the audio-visual beat pair comes from the predicted visual action sequence, which corresponds to the training beat unit generated during the training of the GAN model.

11. The apparatus according to claim 10, characterized in that, Training the generative adversarial network model also includes: The visual beat of the estimated visual motion sequence is detected by calculating the difference between the sums of joint velocities of adjacent frames in the estimated visual motion sequence.

12. The apparatus according to claim 10, characterized in that, The beat consistency of the audio-visual beats is determined by the following steps, wherein the audio-video beat pairs correspond to each training beat unit: Weights are assigned to each audio beat and each visual beat based on beat saliency. In the audio-visual beat pair of the training beat unit, attention beats are obtained according to the weights of the audio beats and the visual beats, and the attention beats include one or more attention audio beats and one or more attention visual beats. Obtain the beat intensity of each attention beat; Calculate labels in the audio and visual domains to represent the aligned attention beats in the sample audio input and the predicted visual action sequence, thereby constructing a hit score, where one label represents the alignment of an attention audio beat with the corresponding attention visual beat based on human reaction time delay; as well as The hit score is used to determine the beat consistency.

13. The apparatus according to claim 10, characterized in that, Training the generative adversarial network also includes: Audio-visual segments are temporally segmented based on multiple trained beat units; The audio-visual segment is input for training; The initial pose of each training beat unit is obtained from the corresponding audio-visual segment containing the sample audio sequence.

14. The apparatus according to claim 10, characterized in that, The generative adversarial network model also includes a cross-domain frequency discriminator and a spatiotemporal frequency discriminator, which jointly monitor the generator. Training the generative adversarial network model also includes: Minimize the multi-space pose loss, the harmony loss, and the generative adversarial network loss from the generator; as well as Maximize the values ​​of the loss functions of the cross-domain frequency discriminator and the spatiotemporal frequency discriminator to distinguish between real training samples and fake training samples.

15. The apparatus according to claim 10, characterized in that, The multi-space pose loss includes at least one of Kullback-Leibler loss, Charbonnier-based MSE loss, and Charbonnier-based VGG loss.

16. The apparatus according to claim 10, characterized in that, The generator includes a GRU-based audio encoder and a gesture decoder; The attitude decoder is used for: Two-dimensional pose is estimated based on the visual motion sequence corresponding to the beat unit; Depth boosting branches are constructed based on the estimated 2D pose to generate 3D pose.

17. The apparatus according to claim 10, characterized in that, The process of determining multiple test beat units based on the input audio includes: Multiple test beat units are determined based on the audio beat and audio beat intensity of the input audio. Each test beat unit corresponds to an audio sequence of the input audio and a time index based on the time record in the test beat unit. The acquisition of audio input corresponding to each test beat unit includes: Extract the features of the audio sequence in each test beat unit as the audio input; The initial pose of each test beat unit is obtained based on the visual action sequence as visual input. The visual action sequence is a synthesized sequence corresponding to the previous test beat unit, including: The initial posture of each test beat unit is obtained as visual input based on the time index and the visual action sequence, and the visual action sequence is the visual action sequence synthesized corresponding to the previous test beat unit.

18. The apparatus according to claim 17, characterized in that, The process of obtaining the initial posture of each test beat unit includes: The motion sequence generated by the previous test beat unit will be preserved in the initial pose of the current test beat unit; and The average pose of the motion sequence generated by the previous test beat unit is used as the initial pose of the current test beat unit.

Images