Audio processing method and device, model training method and device, equipment, storage medium and product

Abstract

Embodiments of the present application disclose an audio processing method, a model training method and device, equipment, a storage medium and a product. The audio processing method comprises: in the embodiments of the present application, obtaining a rhythm condition vector corresponding to a motion video to be matched with music and reference chords of the motion video, performing beat dimension noise reduction processing on noise data based on the rhythm condition vector to obtain a predicted beat of the motion video, performing chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chords to obtain a predicted chord of the motion video, and fusing the predicted beat and the predicted chord to obtain music matched with the motion video. It can be seen that the beats and chords of the music are respectively subjected to noise reduction processing through different constraint conditions, the prediction process of the music can be decoupled, the prediction process of the beats and chords is more targeted, and thus the quality of the music matched with the motion video is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, in particular to an audio processing method and a model training method, an action generation device and a model training device, a computer device, a computer readable storage medium, and an action generation product and a model training product. BACKGROUND

[0002] With the progress of scientific research, the related technology of artificial intelligence model develops rapidly. More and more artificial intelligence models are applied in various fields of life; for example, generative models are applied in the field of audio generation, discriminative models are applied in the field of face recognition, etc. In the field of audio generation, an audio generation model can generate corresponding music according to a video. Research has found that when the video to be accompanied by music is an action video, the quality of the music generated by the audio generation model is low (such as the music rhythm does not match the action, the music style does not match the video style, etc.). SUMMARY

[0003] The embodiments of the present application provide an audio processing method, a model training method and device, equipment, a computer readable storage medium and product, which can improve the quality of the music of the action video.

[0004] In one aspect, the embodiments of the present application provide an audio processing method, comprising:

[0005] obtaining a rhythm condition vector corresponding to the action video to be accompanied by music and a reference chord of the action video, the rhythm condition vector being used to represent the action amplitude and action intensity of a target object in the action video; the reference chord being determined based on the style of the action video;

[0006] based on the rhythm condition vector, performing beat dimension noise reduction processing on the noise data to obtain a predicted beat of the action video;

[0007] based on the rhythm condition vector and the reference chord, performing chord dimension noise reduction processing on the noise data to obtain a predicted chord of the action video;

[0008] fusing the predicted beat and the predicted chord to obtain the music of the action video.

[0009] In the embodiment of the present application, the rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video are obtained, the noise data is processed in the rhythm dimension based on the rhythm condition vector to obtain the predicted rhythm of the action video, the noise data is processed in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video, and the predicted rhythm and the predicted chord are fused to obtain the music matched with the action video. It can be seen that the rhythm and the chord of the music are processed in different constraint conditions, the prediction process of the music is decoupled, the prediction process of the rhythm and the chord is more targeted, and thus the quality of the music matched with the action video is improved.

[0010] In an aspect, the embodiment of the present application provides a model training method, comprising:

[0011] obtaining a sample audio, a rhythm condition vector and a reference chord corresponding to a sample video; the rhythm condition vector is used to represent the action amplitude and the action intensity of a target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0012] calling a first to-be-trained model to process noise data in the rhythm dimension based on the rhythm condition vector to obtain a predicted rhythm of the sample video;

[0013] calling a second to-be-trained model to process noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain a predicted chord of the sample video;

[0014] training the first to-be-trained model and the second to-be-trained model based on the predicted rhythm, the predicted chord and the sample audio to obtain a rhythm prediction model and a chord prediction model; the rhythm prediction model and the chord prediction model are used to generate music matched with a video to be matched with music.

[0015] In the embodiment of the present application, the sample audio, the rhythm condition vector and the reference chord corresponding to the sample video are obtained, the first to-be-trained model is called to process noise data in the rhythm dimension based on the rhythm condition vector to obtain the predicted rhythm of the sample video, the second to-be-trained model is called to process noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the sample video, the first to-be-trained model and the second to-be-trained model are trained based on the predicted rhythm, the predicted chord and the sample audio to obtain the rhythm prediction model and the chord prediction model, and the rhythm prediction model and the chord prediction model are used to generate music matched with a video to be matched with music. It can be seen that the rhythm prediction model and the chord prediction model are trained in different sample data, the model training effect is improved, and the trained model can generate high-quality music.

[0016] In an aspect, the embodiment of the present application provides an audio processing apparatus, comprising:

[0017] an acquisition unit, configured to acquire a rhythm condition vector corresponding to a motion video to be matched with music and reference chords of the motion video, the rhythm condition vector being used to represent motion amplitude and motion intensity of a target object in the motion video, and the reference chords being determined based on a style of the motion video;

[0018] a processing unit, configured to perform beat-dimension noise reduction processing on noise data based on the rhythm condition vector to obtain predicted beats of the motion video;

[0019] and chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chords to obtain predicted chords of the motion video;

[0020] and fuse the predicted beats and the predicted chords to obtain music matched with the motion video.

[0021] In an implementation, the processing unit is configured to acquire the reference chords of the motion video, and specifically configured to:

[0022] extract visual features of the motion video to obtain visual features of the motion video, wherein the visual features contain style information of the motion video;

[0023] select a target candidate video from candidate videos based on the visual features, wherein a feature similarity between the visual features of the target candidate video and the visual features of the motion video is greater than a similarity threshold;

[0024] determine chords of the target candidate video as the reference chords of the motion video.

[0025] In an implementation, the processing unit is configured to acquire the rhythm condition vector corresponding to the motion video to be matched with music, and specifically configured to:

[0026] sample poses of a target object in the motion video to obtain pose information of the target object, wherein the pose information is used to indicate motion changes of the target object within consecutive time;

[0027] encode the pose information of the target object to obtain the rhythm condition vector.

[0028] In an implementation, the processing unit performs beat-dimension noise reduction processing on noise data based on the rhythm condition vector to obtain predicted beats of the motion video, and the process includes:

[0029] invoke a first deep learning network to predict a noise component corresponding to an hth iteration noise reduction result of the noise data based on the rhythm condition vector, wherein h is a positive integer;

[0030] remove the noise component from an (h-1)th iteration noise reduction result to obtain the hth iteration noise reduction result of the noise data.

[0031] In an implementation, the first deep learning network comprises M self-attention layers, each self-attention layer corresponding to a cross-attention layer, M being a positive integer; the process of predicting, by the processing unit, the noise component corresponding to the hth iteration denoising result of the noise data based on the rhythm condition vector comprises:

[0032] performing correlation analysis on the input data by the jth self-attention layer in the first deep learning network to obtain a first intermediate result, j being a positive integer less than M;

[0033] performing feature interaction on the first intermediate result and the rhythm condition vector by the cross-attention layer corresponding to the jth self-attention layer in the first deep learning network to obtain a first feature interaction result;

[0034] taking the first feature interaction result as input data of the (j+1)th self-attention layer in the first deep learning network.

[0035] In an implementation, the process of performing chord dimension denoising processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video by the processing unit comprises:

[0036] calling the second deep learning network to predict the noise component corresponding to the kth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, k being a positive integer;

[0037] removing the noise component from the (k-1)th iteration denoising result to obtain the kth iteration denoising result of the noise data.

[0038] In an implementation, the predicted beat of the action video is obtained by calling the first deep learning network to perform beat dimension denoising processing on the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network each comprise M self-attention layers, each self-attention layer corresponding to a cross-attention layer, M being a positive integer; the process of predicting, by the processing unit, the noise component corresponding to the kth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord by calling the second deep learning network comprises:

[0039] calling the ith self-attention layer in the second deep learning network to perform correlation analysis on the input data based on the output data of the ith self-attention layer in the first deep learning network to obtain a second intermediate result, i being a positive integer less than M;

[0040] performing feature interaction on the second intermediate result, the rhythm condition vector and the reference chord by the cross-attention layer corresponding to the ith self-attention layer in the second deep learning network to obtain a second feature interaction result;

[0041] The second feature interaction result is taken as input data of an (i+1)th self-attention layer in the second deep learning network.

[0042] In an aspect, an embodiment of the present application provides a model training device, which comprises:

[0043] The acquisition unit is configured to acquire a sample audio, a rhythm condition vector, and a reference chord; the rhythm condition vector is used to represent a motion amplitude and a motion intensity of a target object in the sample video, and the reference chord is determined based on a style of the sample video;

[0044] The processing unit is configured to call the first to-be-trained model to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain a predicted beat of the sample video;

[0045] and to call the second to-be-trained model to perform chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord, to obtain a predicted chord of the sample video;

[0046] and to train the first to-be-trained model and the second to-be-trained model based on the predicted beat, the predicted chord, and the sample audio, to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate music for a video to be accompanied.

[0047] In an embodiment, the sample audio comprises a sample beat; the processing unit is configured to train the first to-be-trained model and the second to-be-trained model based on the predicted beat, the predicted chord, and the sample audio, to obtain the beat prediction model and the chord prediction model, specifically for:

[0048] training the first to-be-trained model based on a difference between the predicted beat and the sample beat, to obtain the beat prediction model;

[0049] training the second to-be-trained model based on a difference between the predicted chord and the sample audio, to obtain the chord prediction model.

[0050] In an embodiment, the processing unit is configured to train the second to-be-trained model based on a difference between the predicted chord and the sample audio, to obtain the chord prediction model, specifically for:

[0051] fusing the predicted beat and the predicted chord to obtain a predicted audio of the sample video;

[0052] training the second to-be-trained model based on a difference between the predicted audio and the sample audio, to obtain the chord prediction model.

[0053] Correspondingly, the present application provides a computer device, which comprises:

[0054] A memory, the memory storing a computer program;

[0055] A processor for loading the computer program to implement the above-mentioned audio processing method.

[0056] Correspondingly, the present application provides a computer readable storage medium, the computer readable storage medium storing a computer program, the computer program being adapted to be loaded by a processor and execute the above-mentioned audio processing method, or being adapted to be loaded by a processor and execute the above-mentioned model training method.

[0057] Correspondingly, the present application provides a computer program product or a computer program, the computer program product or the computer program comprising computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions, so that the computer device executes the above-mentioned audio processing method, or executes the above-mentioned model training method.

[0058] In the inference process of the embodiment of the present application, the beat and the chord of the music are respectively processed by different constraint conditions. The prediction process of the music can be decoupled, so that the prediction process of the beat and the chord is more targeted, and the quality of the music of the action video is improved. In the training process, the beat prediction model and the chord prediction model are respectively trained by different sample data. The model training effect can be improved, and the trained model can generate high-quality music. BRIEF DESCRIPTION OF DRAWINGS

[0059] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or the prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0060] Figure 1a A diffusion process schematic diagram provided for the embodiment of the present application;

[0061] Figure 1b An audio processing scene diagram provided for the embodiment of the present application;

[0062] Figure 2 A flowchart of an audio processing method provided for the embodiment of the present application;

[0063] Figure 3a A schematic diagram of a screening module provided for the embodiment of the present application;

[0064] Figure 3b An audio processing model architecture diagram provided for the embodiment of the present application;

[0065] Figure 4 A flowchart of a model training method provided for an embodiment of the present application;

[0066] Figure 5 A model training architecture schematic provided for an embodiment of the present application;

[0067] Figure 6 A structural schematic of an audio processing device provided for an embodiment of the present application;

[0068] Figure 7 A structural schematic of a model training device provided for an embodiment of the present application;

[0069] Figure 8 A structural schematic of a computer device provided for an embodiment of the present application. DETAILED DESCRIPTION

[0070] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative labor fall within the scope of protection of the present application.

[0071] The embodiments of the present application relate to technologies related to diffusion models, and the related terms and concepts of diffusion models will be briefly introduced as follows:

[0072] Diffusion model principle: the inspiration of diffusion models comes from the diffusion process in physics, that is, how to gradually transform an ordered structure into a disordered state, and then reverse this process. Figure 1a A diffusion process schematic provided for an embodiment of the present application. As shown in Figure 1a The working principle of the diffusion model can be divided into two main stages: forward process and reverse process.

[0073] Forward process: this stage simulates the diffusion process from data distribution to random noise distribution. At the beginning, there is a clean sample data (for example, an audio segment), and then noise is gradually added to the sample data. After several times of noise superposition, the sample data completely becomes noise data. This process is usually implemented by applying Gaussian noise, and each step makes the sample data closer to the noise data distribution.

[0074] Reverse process: the reverse process is the inverse process of the forward process, and its goal is to recover the original data from the noise data. In this stage, the diffusion model is used to predict the noise added at each step.

[0075] Based on the above-mentioned technologies related to diffusion models, this application provides an audio processing scheme and a model training scheme, which can improve the quality of background music in action videos. Figure 1b An audio processing scenario diagram provided for an embodiment of this application, such as... Figure 1b As shown, the audio processing scenario provided in this application includes a terminal device 101 and a server 102. The audio processing scheme and model training scheme provided in this application can be executed by either the terminal device 101 or the server 102. When both the audio processing scheme and model training scheme provided in this application are executed by the terminal device 101, the audio processing scenario may not include the server 102. The terminal device may include, but is not limited to: smartphones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile internet devices (MIDs), smart voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., and this application embodiment does not limit this. The server may be an independent physical server, or a server cluster or distributed system composed of two or more physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms, and this application embodiment does not limit this.

[0076] It should be noted that, Figure 1b The number of terminal devices and servers is for illustrative purposes only and does not constitute an actual limitation of this application. Terminal device 101 and server 102 can be connected via wired or wireless means, and this application does not impose any restrictions on this.

[0077] The audio processing scheme and the model training scheme can be independent of each other or combined; for example, the model training scheme can be executed in server 102, and the trained audio processing model can be deployed in terminal device 101, and then the audio processing scheme can be executed in terminal device 101; or, for another example, the model training scheme and the audio processing scheme can be executed in server 102.

[0078] The general flow of the audio processing solution provided in this application is as follows:

[0079] The server 102 acquires the rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video. The action video contains a series of actions performed by a target object in at least one continuous time period, and can be a dance video, a martial arts video, a gymnastics video, a sports video, etc. The target object can be the main character in the action video, and can be a person, a virtual character, a robot, an animal, etc., which is not limited in the present application. The rhythm condition vector is used to represent the action amplitude and action intensity of the target object in the action video, and the reference chord is determined based on the style of the action video. The action video can be sent by the terminal device 101, can be collected by the server 102 through a collection device (such as a camera), or can be generated by the server 102 based on input data (such as an action sequence, a reference video, etc.), which is not limited in the present application.

[0080] After acquiring the rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video, on the one hand, the server 102 performs beat-dimension noise reduction processing on the noise data based on the rhythm condition vector to obtain the predicted beat of the action video. In an embodiment, the processing process includes: the server 102 calls the first deep learning network to predict the noise component corresponding to the hth iteration denoising result of the noise data based on the rhythm condition vector, and removes the noise component corresponding to the (h-1)th iteration denoising result from the (h-1)th iteration denoising result to obtain the hth iteration denoising result of the noise data, h being a positive integer. According to the above embodiment, the server 102 can call the first deep learning network to gradually remove the noise in the noise data until the predicted beat of the action video is obtained.

[0081] On the other hand, the server 102 performs chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video. In an embodiment, the processing process includes: the server 102 calls the second deep learning network to predict the noise component corresponding to the kth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, and removes the noise component from the (k-1)th iteration denoising result to obtain the kth iteration denoising result of the noise data, k being a positive integer. According to the above embodiment, the server 102 can call the second deep learning network to gradually remove the noise in the noise data until the predicted chord of the action video is obtained.

[0082] After obtaining the predicted beat and the predicted chord of the action video, the server 102 fuses the predicted beat and the predicted chord to obtain the music matched with the action video.

[0083] The general process of the model training scheme provided in the present application is as follows:

[0084] The server 102 acquires sample audio corresponding to the sample video, a rhythm condition vector, and reference chords; the rhythm condition vector is used to represent the action amplitude and action intensity of the target object in the sample video, and the reference chords are determined based on the style of the sample video. The sample audio is composed of sample beats and sample chords.

[0085] After acquiring the sample audio corresponding to the sample video, the rhythm condition vector, and the reference chords, on the one hand, the server 102 calls the first to-be-trained model to perform beat-dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain the predicted beats of the sample video. In an embodiment, the first to-be-trained model includes a first deep learning network, and the generation process of the predicted beats includes: calling the first deep learning network to predict the noise component corresponding to the gth iteration denoising result of the noise data based on the rhythm condition vector, and removing the noise component from the (g-1)th iteration denoising result to obtain the gth iteration denoising result of the noise data, g being a positive integer.

[0086] On the other hand, the server 102 calls the second to-be-trained model to perform chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chords, to obtain the predicted chords of the sample video. For details, reference can be made to the embodiment of the audio processing scheme “performing chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chords to obtain the predicted chords of the action video”, which will not be repeated here.

[0087] After obtaining the predicted beats and the predicted chords, the server 102 trains the first to-be-trained model and the second to-be-trained model based on the predicted beats, the predicted chords, and the sample audio, to obtain a beat prediction model and a chord prediction model. In an embodiment, the sample audio includes sample beats, the server 102 trains the first to-be-trained model based on the difference between the predicted beats and the sample beats to obtain the beat prediction model, and trains the second to-be-trained model based on the difference between the predicted chords and the sample audio to obtain the chord prediction model.

[0088] Further, after completing the training of the beat prediction model and the chord prediction model, the server 102 can call the beat prediction model to perform beat-dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain the predicted beats of the action video, and call the chord prediction model to perform chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chords, to obtain the predicted chords of the action video. Through the predicted beats of the action video and the predicted chords of the action video, the music for the action video can be generated. For details, reference can be made to the audio processing scheme, which will not be repeated here.

[0089] In the inference process, the embodiments of the present application respectively perform noise reduction processing on the beats and chords of the music accompaniment through different constraint conditions, can decouple the prediction process of the music accompaniment, make the prediction process of the beats and chords more targeted, and then improve the quality of the music accompaniment of the action video. In the training process, the beat prediction model and the chord prediction model are respectively trained through different sample data, which can improve the model training effect, so that the trained model can generate high-quality music accompaniment.

[0090] Based on the above audio processing scheme and model training scheme, the embodiments of the present application propose more detailed audio processing methods and model training methods, which will be described in detail below in combination with the accompanying drawings.

[0091] Please refer to Figure 2 , Figure 2 A flowchart of an audio processing method provided by the embodiments of the present application, which can be executed by a computer device; for example, by the terminal device 101 or the server 102 shown in Figure 1b . As shown in Figure 2 , the audio processing method can include the following steps S201-S204:

[0092] S201, obtaining the rhythm condition vector corresponding to the action video to be accompanied by music and the reference chord of the action video.

[0093] The computer device first obtains the action video, which can be sent by other devices, or collected by the computer device through the acquisition device (such as a camera), or generated by the computer device based on input data (such as action sequence, reference video, etc.), which is not limited by the present application. The action video contains a series of actions performed by the target object in at least one continuous time period, and the action video can be a dance video, a martial arts video, a gymnastics video, a sports video, etc. The target object can be the main character in the action video, which can be a person, a virtual character, a robot, an animal, etc., which is not limited by the present application.

[0094] In the process of obtaining the reference chord corresponding to the action video to be accompanied by music, the computer device can select the target candidate video from the candidate video based on at least one screening element, and obtain the reference chord of the action video based on the chord of the target candidate video; wherein the screening elements include but are not limited to: whether the same object in the video, visual features, video types, text content in the video, and action similarity.

[0095] In an embodiment, the computer device extracts visual features of the action video to obtain visual features of the action video; wherein the visual features contain style information of the action video. Then the computer device screens target candidate videos from the candidate videos based on the visual features of the action video, and the feature similarity between the visual features of the target candidate videos and the visual features of the action video is greater than a similarity threshold. Further, after screening at least one target candidate video, the computer device can determine the chord of the target candidate video as the reference chord of the action video, and extracts audio features of the reference chord of the action video to obtain chord constraints of the action video. The above process can be represented as:

[0096] Equation (1)

[0097] wherein, represents the chord constraints of the action video, represents the audio features (of the chord) of the candidate video. is the feature similarity between the visual features of the candidate video and the visual features of the action video, in an embodiment, , represents the cosine similarity between the feature vector and the feature vector , is the visual feature of the i-th candidate video in the database, is the visual feature of the action video to be matched with music, is the similarity threshold.

[0098] The above process can be performed by a screening module, Figure 3a is a schematic diagram of a screening module provided by an embodiment of the present application. As shown in Figure 3a , according to the feature similarity between the visual features of the candidate video and the visual features of the action video, the screening module can determine whether to select the chord of the candidate video as the reference chord of the action video (through a retrieval trigger). It can be understood that, since the visual features contain the style information of the action video, the reference chord selected based on the feature similarity has a higher degree of matching with the style of the action video.

[0099] In the process of obtaining the rhythm condition vector corresponding to the action video to be matched with music, the computer device samples the posture of the target object in the action video to obtain posture information of the target object; wherein the posture information is used to indicate the motion change of the target object within a continuous time. In an embodiment, the computer device first extracts the motion sequence of the target object from the action video, and then generates the corresponding posture information for each motion in the motion sequence; for example, the computer device can represent the joint position in each motion, and add all the joint positions corresponding to the same motion to the same matrix to obtain the matrix representation of the motion (i.e. as the posture information of the motion), the position of the jth joint of the tth motion can be represented as P(t,j,x,y), wherein t is the motion (video frame) index, j is the joint index, and x and y are the plane coordinates of the joint. It can be understood that the posture information of the target object can also be represented by three-dimensional coordinates according to actual needs. After obtaining the posture information of the target object, the computer device can encode the posture information of the target object to obtain the rhythm condition vector, which is used to represent the motion amplitude and motion intensity of the target object in the action video.

[0100] The rhythm condition vector can be specifically represented as: ; wherein, represents the rhythm condition vector, T represents the total number of time steps (which can be understood as the number of video frames in the motion sequence), represents that the corresponding time step is a rhythm point; for example, assuming T=10, the values of 2, 5, and 8 indicate that the time points corresponding to the 2nd video frame, the 5th video frame, and the 8th video frame are a rhythm point.

[0101] S202, based on the rhythm condition vector, the noise data is processed in the beat dimension to obtain the predicted beat of the action video.

[0102] The noise data is a kind of random noise, which is generated using Gaussian distribution (normal distribution) and is the starting point of the reverse diffusion process. Gaussian distribution is a kind of probability distribution, which is used to describe the possibility of different values appearing within a given range. The goal of noise reduction processing is to obtain a prediction result that matches the training data of the beat prediction model under the constraint of the rhythm condition vector. The noise reduction processing can be completed at one time, or can be completed through two or more iterations. In the diffusion model, the noise reduction process is completed through two or more iterations, and the object of each iteration noise reduction is the matrix of the noise data in the latent space, and the input data of each iteration noise reduction includes the output data of the last iteration noise reduction.

[0103] In an embodiment, the computer device invokes a first deep learning network to predict, based on the rhythm condition vector, a noise component corresponding to an hth iteration denoising result of the noisy data, h being a positive integer. The first deep learning network is included in the beat prediction model. In an embodiment, the first deep learning network comprises M self-attention layers, each self-attention layer corresponding to a cross-attention layer, M being a positive integer.

[0104] The prediction process of the noise component corresponding to the hth iteration denoising result of the noisy data in the beat dimension comprises: the computer device performs correlation analysis on the input data of the jth self-attention layer in the first deep learning network to obtain a first intermediate result, j being a positive integer less than M. After obtaining the first intermediate result, the computer device performs feature interaction on the first intermediate result and the rhythm condition vector through the cross-attention layer corresponding to the jth self-attention layer in the first deep learning network to obtain a first feature interaction result, and takes the first feature interaction result as the input data of the (j+1)th self-attention layer in the first deep learning network. It can be understood that the input data of the first self-attention layer in the first deep learning network is the (h-1)th iteration denoising result; when h=1, the input data of the first self-attention layer in the first deep learning network is the noisy data. When j=M-1, the result of the feature interaction output by the cross-attention layer corresponding to the (j+1)th self-attention layer in the first deep learning network is the noise component (in vector representation) corresponding to the hth iteration denoising result.

[0105] After obtaining the noise component corresponding to the hth iteration denoising result of the noisy data in the beat dimension, the computer device removes the noise component corresponding to the hth iteration denoising result from the (h-1)th iteration denoising result (in the beat dimension) to obtain the hth iteration denoising result of the noisy data. The above iteration denoising process can be performed in a high-dimensional space. In an embodiment, the computer device calculates the difference between the vector of the (h-1)th iteration denoising result and the vector of the noise component corresponding to the hth iteration denoising result to obtain the vector representation of the hth iteration denoising result of the noisy data. It can be understood that when h=1, after predicting the noise component corresponding to the first iteration denoising result in the beat dimension, the computer device removes the noise component corresponding to the first iteration denoising result from the noisy data to obtain the first iteration denoising result of the noisy data in the beat dimension; assuming that the total number of iteration denoising in the beat dimension is H, the Hth iteration denoising result of the noisy data in the beat dimension is the predicted beat of the action video.

[0106] S203, based on the rhythm condition vector and the reference chord, performing chord dimension denoising on the noisy data to obtain a predicted chord of the action video.

[0107] In an embodiment, the computer device invokes the second deep learning network to predict the noise component corresponding to the kth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, k being a positive integer. The second deep learning network is included in the chord prediction model. In an embodiment, the predicted rhythm of the action video is obtained by invoking the first deep learning network to perform rhythm dimension denoising on the noise data based on the rhythm condition vector, and the first deep learning network and the second deep learning network each include M self-attention layers, each self-attention layer in the first deep learning network and the second deep learning network corresponding to a cross-attention layer, M being a positive integer.

[0108] The prediction process of the noise component corresponding to the kth iteration denoising result of the noise data in the chord dimension includes: the computer device invokes the ith self-attention layer in the second deep learning network to perform correlation analysis on the input data of the ith self-attention layer in the second deep learning network based on the output data of the ith self-attention layer in the first deep learning network, to obtain a second intermediate result, i being a positive integer less than M. After obtaining the second intermediate result, the computer device performs feature interaction on the second intermediate result, the rhythm condition vector and the reference chord through the cross-attention layer corresponding to the ith self-attention layer in the second deep learning network to obtain a second feature interaction result, and takes the second feature interaction result as the input data of the (i+1)th self-attention layer in the second deep learning network. It can be understood that the input data of the first self-attention layer in the second deep learning network is the (k-1)th iteration denoising result; when k=1, the input data of the first self-attention layer in the first deep learning network is the noise data. When i=M-1, the result of the feature interaction output by the cross-attention layer corresponding to the (i+1)th self-attention layer in the second deep learning network is the noise component (in vector representation) corresponding to the kth iteration denoising result.

[0109] After obtaining the noise component corresponding to the kth iteration denoising result of the noise data in the chord dimension, the computer device removes the noise component corresponding to the kth iteration denoising result from the (k-1)th iteration denoising result (in the chord dimension) to obtain the kth iteration denoising result of the noise data. The above iteration denoising process can be performed in a high-dimensional space, and the specific implementation can refer to the implementation of removing the noise component corresponding to the hth iteration denoising result from the (h-1)th iteration denoising result in the rhythm dimension, which will not be described here. It can be understood that when k=1, after predicting the noise component corresponding to the first iteration denoising result in the chord dimension, the computer device removes the noise component corresponding to the first iteration denoising result from the noise data to obtain the first iteration denoising result of the noise data in the chord dimension; assuming that the total number of iteration denoising in the chord dimension is K, the Kth iteration denoising result of the noise data in the chord dimension is the predicted chord of the action video.

[0110] Figure 3b This is a diagram illustrating an audio processing model architecture provided in an embodiment of this application. Figure 3b As shown, structurally, the beat prediction model includes a first deep learning network, which contains M self-attention layers, each corresponding to a cross-attention layer, where M is a positive integer. The chord prediction model includes a second deep learning network, which also contains M self-attention layers, each corresponding to a cross-attention layer. In terms of process, the beat prediction model, based on a rhythm condition vector, performs beat-dimensional denoising on the noisy data to obtain the predicted beat of the action video; whereby the rhythm condition vector (… The motion sequence of the motion video can be encoded using a rhythm encoder. The chord prediction model, based on the rhythm condition vector and reference chords, performs chord-dimensional noise reduction on the noisy data to obtain the predicted chords for the motion video. During this process, the i-th self-attention layer in the second deep learning network, when performing correlation analysis on the input data, is constrained by the first intermediate result output by the i-th self-attention layer in the first deep learning network, where i is a positive integer less than or equal to M. Furthermore, the cross-attention layer corresponding to the i-th self-attention layer in the second deep learning network is used to process the second intermediate result output by the corresponding self-attention layer, the rhythm condition vector, and the chord constraints (…). Feature interaction is performed; where chord constraints are obtained by extracting audio features from reference chords, and the reference videos corresponding to the reference chords are obtained by the filtering module based on the visual features of the action videos. For specific filtering methods, please refer to... Figure 3a This will not be elaborated upon here.

[0111] S204. The predicted beats and predicted chords are blended to obtain the background music for the action video.

[0112] In one implementation, a computer device configures the beat of a predicted chord according to the predicted beat to obtain the background music for the action video. It is understood that by decoupling the prediction process of the background music, the final background music for the action video can be adapted to the action video in both the beat and chord dimensions, effectively improving the quality of the background music for the action video.

[0113] In the embodiments of the present application, the rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video are obtained, the noise data is processed in the rhythm dimension based on the rhythm condition vector to obtain the predicted rhythm of the action video, the noise data is processed in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video, and the predicted rhythm and the predicted chord are fused to obtain the music matched with the action video. It can be seen that the rhythm and the chord of the music are processed in different constraint conditions, which can decouple the prediction process of the music, so that the prediction process of the rhythm and the chord is more targeted, and thus the quality of the music matched with the action video is improved.

[0114] Please refer to Figure 4 , Figure 4 A flowchart of a model training method is provided in the embodiments of the present application, which can be executed by a computer device; for example, executed by the terminal device 101 or the server 102 shown in Figure 1b . As shown in Figure 4 , the audio processing method can include the following steps S401-S404:

[0115] S401, obtaining the sample audio corresponding to the sample video, the rhythm condition vector and the reference chord.

[0116] The sample audio can be the audio accompanying the sample video, or the audio specified by the model training personnel, and the sample audio can be composed of the sample rhythm and the sample chord. The sample audio is composed of the sample rhythm and the sample chord. The rhythm condition vector is used to represent the action amplitude and the action intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video. The specific implementation of obtaining the rhythm condition vector corresponding to the sample video and the reference chord of the sample video can refer to the implementation of step S201 in Figure 2 , which will not be repeated here.

[0117] S402, calling the first to-be-trained model to process the noise data in the rhythm dimension based on the rhythm condition vector to obtain the predicted rhythm of the sample video.

[0118] In an embodiment, the first to-be-trained model comprises a first deep learning network, and the generation process of the predicted beat comprises: calling the first deep learning network to predict a noise component of a beat dimension corresponding to a gth iteration denoising result of the noise data based on the rhythm condition vector, and removing the noise component from a (g-1)th iteration denoising result to obtain the gth iteration denoising result of the noise data in the beat dimension, g being a positive integer. It can be understood that when g = 1, after the noise component corresponding to the 1st iteration denoising result in the beat dimension is predicted, the computer device removes the noise component corresponding to the 1st iteration denoising result from the noise data to obtain the 1st iteration denoising result of the noise data in the beat dimension; assuming that the total iteration denoising number in the beat dimension is G, the Gth iteration denoising result of the noise data in the beat dimension is the predicted beat of the sample video.

[0119] S403, calling the second to-be-trained model to perform chord dimension denoising processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the sample video.

[0120] In an embodiment, the second to-be-trained model comprises a second deep learning network, and the generation process of the predicted chord comprises: calling the second deep learning network to predict a noise component of a chord dimension corresponding to an rth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, and removing the noise component from an (r-1)th iteration denoising result to obtain the rth iteration denoising result of the noise data in the chord dimension, r being a positive integer. It can be understood that when r = 1, after the noise component corresponding to the 1st iteration denoising result in the chord dimension is predicted, the computer device removes the noise component corresponding to the 1st iteration denoising result from the noise data to obtain the 1st iteration denoising result of the noise data in the chord dimension; assuming that the total iteration denoising number in the chord dimension is R, the Rth iteration denoising result of the noise data in the chord dimension is the predicted chord of the action video.

[0121] The specific embodiments of steps S402 and S403 can refer to the embodiments of steps S202 and S203 in Figure 2 The embodiments of steps S202 and S203 in

[0122] S404, training the first to-be-trained model and the second to-be-trained model based on the predicted beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model.

[0123] The beat prediction model and the chord prediction model are used to generate the music for the video to be accompanied. The computer device can call the beat prediction model to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector to obtain the predicted beat of the action video, and call the chord prediction model to perform chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video. Through the predicted beat of the action video and the predicted chord of the action video, the music for the action video can be generated. For specific embodiments, reference can be made to Figure 2 The embodiments of steps S201-S204 are described above and will not be repeated here.

[0124] In an embodiment, the sample audio includes a sample beat. In one aspect, the computer device trains the first to-be-trained model based on the difference between the predicted beat and the sample beat to obtain the beat prediction model. The loss function used in the training process can be represented as:

[0125] Formula (2)

[0126] wherein, is the to-be-solved parameter of the first to-be-trained model, represents the audio feature of the sample audio corresponding to the sample video, represents the actual added noise (such as Gaussian noise), is the noise predicted by the first to-be-trained model (i.e., the noise removed from the noise data), is the noise added to to obtain , represents the time step (which can be understood as the number of times of noise addition), is the rhythm condition vector corresponding to the sample video, represents the expectation.

[0127] On the other hand, the sample audio includes a sample chord, and the computer device trains the second to-be-trained model based on the difference between the predicted chord and the sample audio (or the sample chord) to obtain the chord prediction model.

[0128] In one embodiment, the computer device trains the second to-be-trained model based on the difference between the predicted chord and the sample chord to obtain the chord prediction model.

[0129] ​In another embodiment, the computer device fuses the predicted beat and the predicted chord to obtain the predicted audio of the sample video, and trains a second model based on the difference between the predicted audio and the sample audio to obtain a chord prediction model. The predicted beat can be obtained by the computer device calling the first model to be trained (or the beat prediction model) to perform beat-dimensional noise reduction on the noise data based on the rhythm condition vector. For specific implementation details, please refer to [reference needed]. Figure 2 The implementation method of step S202 will not be described in detail here. The loss function used in the training process can be expressed as:

[0130] Formula (3)

[0131] in, For the second model to be trained ( The parameters to be solved, This represents the audio features of the sample audio corresponding to the sample video. This indicates the actual noise added (such as Gaussian noise). Indicates the number of iterations for noise reduction (i.e., to...) (Number of times noise components are added) Is towards Noise data obtained by superimposing t noise components. This represents the chord constraints of the sample video. The method for obtaining can be referred to formula (1), and will not be repeated here; These are the visual features of the sample video. This represents the predicted beat output by the first model to be trained (or the beat prediction model). This represents the audio features of the predicted audio from the sample video. It is understandable that when... When the predicted beat is output by the first model to be trained, the computer device can perform joint training on the first model to be trained and the second model to be trained based on the above formula (3).

[0132] Figure 5 This is a schematic diagram of a model training architecture provided in an embodiment of this application. Figure 5 As shown, Figure 5 and Figure 3b The main difference is that when training the second training model, it can be trained based on the difference between the predicted chord and the sample chord; it can also be trained based on the difference between the predicted audio (composed of predicted chord and predicted beat) and the sample audio (or the first training model and the second training model can be jointly trained).

[0133] It should be noted that, Figure 4 The model training methods in [the text] can be compared with [other methods].Figure 2 The action generation methods in the code are independent of each other and can also be used in... Figure 2 The audio processing methods in the code are executed before the audio processing methods in the code.

[0134] In this embodiment, sample audio, rhythm condition vector, and reference chords corresponding to the sample video are obtained. A first training model is invoked to perform denoising processing on the noise data based on the rhythm condition vector, yielding the predicted beat of the sample video. A second training model is then invoked to perform denoising processing on the noise data based on the rhythm condition vector and reference chords, yielding the predicted chords of the sample video. Based on the predicted beats, predicted chords, and sample audio, both the first and second training models are trained to obtain a beat prediction model and a chord prediction model. These models are then used to generate background music for the video. It is evident that by specifically training the beat prediction model and chord prediction model using different sample data, the model training effect can be improved, enabling the trained model to generate high-quality background music.

[0135] The methods of the embodiments of this application have been described in detail above. In order to facilitate better implementation of the above solutions of the embodiments of this application, the apparatus of the embodiments of this application is provided below.

[0136] Please see Figure 6 , Figure 6 This is a schematic diagram of the structure of an audio processing device provided in an embodiment of this application. Figure 6 The audio processing device shown can be integrated into a computer device, which may specifically be a terminal device or a server. Figure 6 The audio processing device shown can be used to perform the above. Figure 2 Some or all of the functionality described in the method embodiments. Please refer to [link / reference]. Figure 6 The audio processing device includes:

[0137] The acquisition unit 601 is used to acquire the rhythm condition vector corresponding to the motion video to be set to music and the reference chord of the motion video. The rhythm condition vector is used to characterize the motion amplitude and motion intensity of the target object in the motion video; the reference chord is determined based on the style of the motion video.

[0138] The processing unit 602 is used to perform noise reduction processing on the noise data based on the rhythm condition vector to obtain the predicted beat of the action video.

[0139] And it is used to perform chord-dimensional noise reduction on noisy data based on rhythm condition vectors and reference chords to obtain predicted chords for action videos;

[0140] And used to blend predicted beats and predicted chords to create background music for action videos.

[0141] In an implementation, the processing unit 602 is configured to obtain a reference chord of the action video, and specifically configured to:

[0142] extract visual features of the action video to obtain visual features of the action video, wherein the visual features comprise style information of the action video;

[0143] select a target candidate video from the candidate videos based on the visual features, wherein a feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold;

[0144] determine the chord of the target candidate video as the reference chord of the action video.

[0145] In an implementation, the processing unit 602 is configured to obtain a rhythm condition vector corresponding to the action video to be matched with music, and specifically configured to:

[0146] sample a pose of a target object in the action video to obtain pose information of the target object, wherein the pose information is used to indicate a motion change of the target object in consecutive time;

[0147] encode the pose information of the target object to obtain the rhythm condition vector.

[0148] In an implementation, the processing unit 602 is configured to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector to obtain a predicted beat of the action video, and the process comprises:

[0149] invoke a first deep learning network to predict a noise component corresponding to an hth iteration noise reduction result of the noise data based on the rhythm condition vector, wherein h is a positive integer;

[0150] remove the noise component from an (h-1)th iteration noise reduction result to obtain the hth iteration noise reduction result of the noise data.

[0151] In an implementation, the first deep learning network comprises M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process of invoking the first deep learning network to predict the noise component corresponding to the hth iteration noise reduction result of the noise data based on the rhythm condition vector comprises:

[0152] perform correlation analysis on input data through a jth self-attention layer in the first deep learning network to obtain a first intermediate result, wherein j is a positive integer less than M;

[0153] perform feature interaction on the first intermediate result and the rhythm condition vector through a cross-attention layer corresponding to the jth self-attention layer in the first deep learning network to obtain a first feature interaction result;

[0154] The first feature interaction result is taken as input data of a (j+1)th self-attention layer in the first deep learning network.

[0155] In an embodiment, the process in which the processing unit 602 performs chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video includes:

[0156] The second deep learning network is called to predict a noise component corresponding to the kth iteration noise reduction result of the noise data based on the rhythm condition vector and the reference chord, k being a positive integer.

[0157] The noise component is removed from the (k-1)th iteration noise reduction result to obtain the kth iteration noise reduction result of the noise data.

[0158] In an embodiment, the predicted rhythm of the action video is obtained by calling the first deep learning network to perform rhythm dimension noise reduction processing on the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network each include M self-attention layers, each self-attention layer corresponding to a cross-attention layer, M being a positive integer; the process in which the processing unit 602 calls the second deep learning network to predict a noise component corresponding to the kth iteration noise reduction result of the noise data based on the rhythm condition vector and the reference chord includes:

[0159] The i th self-attention layer in the second deep learning network is called to perform correlation analysis on the input data based on the output data of the i th self-attention layer in the first deep learning network to obtain a second intermediate result, i being a positive integer less than M.

[0160] The second intermediate result, the rhythm condition vector and the reference chord are subjected to feature interaction through the cross-attention layer corresponding to the i th self-attention layer in the second deep learning network to obtain a second feature interaction result.

[0161] The second feature interaction result is taken as input data of a (i+1)th self-attention layer in the second deep learning network.

[0162] According to an embodiment of the present application, Figure 2 Some steps involved in the audio processing method shown can be performed by Figure 6 the various units in the audio processing device shown. For example, Figure 2 Step S201 shown in the method can be performed by Figure 6 the acquisition unit 601 shown, and steps S202-S204 can be performed by Figure 6 the processing unit 602 shown. Figure 6The units in the audio processing apparatus shown can be combined into one or several other units respectively or entirely to constitute, or some of the units can be further split into two or more units that are functionally smaller to constitute, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The units are divided based on logical functions, and in actual applications, the functions of one unit can also be implemented by two or more units, or the functions of two or more units can be implemented by one unit. In other embodiments of the present application, the audio processing apparatus can also include other units, and in actual applications, these functions can also be assisted by other units, and can be implemented by two or more units in cooperation.

[0163] According to another embodiment of the present application, the audio processing apparatus as shown in Figure 2 may be constructed, and the audio processing method of the embodiments of the present application can be implemented, by running a computer program (including program codes) capable of executing each step involved in the corresponding method as shown in Figure 6 . The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium, and run therein.

[0164] Based on the same inventive concept, the principles and beneficial effects of the audio processing apparatus provided in the embodiments of the present application for solving problems are similar to those of the audio processing method provided in the method embodiments of the present application for solving problems, and can be referred to the principles and beneficial effects of the method implementation, which will not be described here for brevity.

[0165] Please refer to Figure 7 , Figure 7 for the structure schematic diagram of a model training apparatus provided in the embodiments of the present application, Figure 7 The model training apparatus shown can be mounted in a computer device, which can be a terminal device or a server. Figure 7 The model training apparatus shown can be used to perform part or all of the functions in the method embodiments described above. Figure 4 Please refer to Figure 7 , the model training apparatus includes:

[0166] The acquisition unit 701 is configured to acquire a sample audio corresponding to a sample video, a rhythm condition vector, and a reference chord; the rhythm condition vector is used to represent the action amplitude and action intensity of a target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0167] The processing unit 702 is configured to invoke the first to-be-trained model to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain predicted beats of the sample video.

[0168] The processing unit 702 is configured to invoke the first to-be-trained model to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain predicted beats of the sample video.

[0169] The processing unit 702 is configured to invoke the first to-be-trained model to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain predicted beats of the sample video.

[0170] In an embodiment, the sample audio includes sample beats. The processing unit 702 is configured to train the first to-be-trained model and the second to-be-trained model based on the predicted beats, the predicted chords, and the sample audio, to obtain the beat prediction model and the chord prediction model, and specifically configured to:

[0171] The processing unit 702 is configured to train the first to-be-trained model based on the difference between the predicted beats and the sample beats, to obtain the beat prediction model.

[0172] The processing unit 702 is configured to train the first to-be-trained model based on the difference between the predicted beats and the sample beats, to obtain the beat prediction model.

[0173] In an embodiment, the processing unit 702 is configured to train the second to-be-trained model based on the difference between the predicted chords and the sample audio, to obtain the chord prediction model, and specifically configured to:

[0174] The processing unit 702 is configured to train the first to-be-trained model based on the difference between the predicted beats and the sample beats, to obtain the beat prediction model.

[0175] The processing unit 702 is configured to train the first to-be-trained model based on the difference between the predicted beats and the sample beats, to obtain the beat prediction model.

[0176] According to an embodiment of the present application, Figure 4 The model training method shown in the figure involves some steps that can be performed by Figure 7 The model training device shown in the figure. For example, Figure 4 The step S401 shown in the figure can be performed by Figure 7 The acquisition unit 701 shown in the figure, and steps S402-S404 can be performed by Figure 7 The processing unit 702 shown in the figure. Figure 7The various units in the model training apparatus shown can be combined into one or several other units respectively or all, or some of the units can be further split into two or more units that are functionally smaller, which can achieve the same operation without affecting the implementation of the technical effects of the embodiments of the present application. The above units are divided based on logical functions, and in actual application, the functions of one unit can also be implemented by two or more units, or the functions of two or more units can be implemented by one unit. In other embodiments of the present application, the model training apparatus can also include other units, and in actual application, these functions can also be assisted by other units to be implemented, and can be implemented by two or more units in cooperation.

[0177] According to another embodiment of the present application, the model training apparatus as shown in Figure 4 may be constructed, and the model training method of the embodiments of the present application can be implemented, by running a computer program (including program codes) capable of executing the steps involved in the corresponding method as shown in Figure 7 . The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above computing device through the computer readable recording medium, and run therein.

[0178] Based on the same inventive concept, the model training apparatus provided in the embodiments of the present application has similar principles and beneficial effects for solving problems as the model training method in the method embodiments of the present application, and the principles and beneficial effects for solving problems can be referred to the principles and beneficial effects for implementing the method, and for brevity, will not be described here.

[0179] Please refer to Figure 8 , Figure 8 The structure schematic diagram of a computer device provided in the embodiments of the present application is shown in Figure 8As shown, the computer device at least includes a processor 801, a communication interface 802 and a memory 803. Among them, the processor 801, the communication interface 802 and the memory 803 can be connected through a bus or other means. Among them, the processor 801 (or called Central Processing Unit (CPU)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device, for example: the CPU can be used to parse the on-off instruction issued by the object to the computer device, and control the computer device to perform the on-off operation; for another example: the CPU can transmit various interactive data between the internal structures of the computer device, etc. The communication interface 802 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be used for receiving and transmitting data under the control of the processor 801; the communication interface 802 can also be used for internal data transmission and interaction of the computer device. The memory 803 (Memory) is a memory device in the computer device, used to store programs and data. It can be understood that the memory 803 here can include the built-in memory of the computer device, and of course can also include the expansion memory supported by the computer device. The memory 803 provides a storage space, which stores the operating system of the computer device, which can include but is not limited to: Android system, iOS system, Windows Phone system, etc., and the present application does not make any limitation thereto.

[0180] The embodiment of the present application also provides a computer readable storage medium (Memory), which is a memory device in the computer device, used to store programs and data. It can be understood that the computer readable storage medium here can include the built-in storage medium in the computer device, and of course can also include the expansion storage medium supported by the computer device. The computer readable storage medium provides a storage space, which stores the processing system of the computer device. And in the storage space, there is also stored a computer program suitable for being loaded and executed by the processor 801. It should be noted that the computer readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, for example, at least one disk memory; optionally, it can also be at least one computer readable storage medium located away from the aforementioned processor.

[0181] In one embodiment, the processor 801 executes the following operations by running the computer program in the memory 803:

[0182] The rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video are obtained, the rhythm condition vector is used to represent the action amplitude and action strength of the target object in the action video, and the reference chord is determined based on the style of the action video.

[0183] Based on the rhythm condition vector, the noise data is processed in the beat dimension to obtain the predicted beat of the action video.

[0184] Based on the rhythm condition vector and the reference chord, the noise data is processed in the chord dimension to obtain the predicted chord of the action video.

[0185] The predicted beat and the predicted chord are fused to obtain the music matched with the action video.

[0186] As an optional embodiment, the specific embodiment of the processor 801 obtaining the reference chord of the action video is:

[0187] The visual features of the action video are extracted to obtain the visual features of the action video, and the visual features contain the style information of the action video.

[0188] Based on the visual features, the target candidate video is selected from the candidate videos, and the feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold.

[0189] The chord of the target candidate video is determined as the reference chord of the action video.

[0190] As an optional embodiment, the specific embodiment of the processor 801 obtaining the rhythm condition vector corresponding to the action video to be matched with music is:

[0191] The posture of the target object in the action video is sampled to obtain the posture information of the target object, and the posture information is used to indicate the action change of the target object in consecutive time.

[0192] The posture information of the target object is feature-encoded to obtain the rhythm condition vector.

[0193] As an optional embodiment, the process of the processor 801 performing noise reduction processing in the beat dimension on the noise data based on the rhythm condition vector to obtain the predicted beat of the action video includes:

[0194] The first deep learning network is called to predict the noise component corresponding to the hth iteration denoising result of the noise data based on the rhythm condition vector, h is a positive integer;

[0195] The noise component is removed from the (h-1)th iteration denoising result to obtain the hth iteration denoising result of the noise data.

[0196] As an optional embodiment, the first deep learning network comprises M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process that the processor 801 calls the first deep learning network to predict the noise component corresponding to the hth iteration denoising result of the noise data based on the rhythm condition vector comprises:

[0197] The input data is analyzed for correlation by the jth self-attention layer in the first deep learning network, and a first intermediate result is obtained, j being a positive integer less than M;

[0198] The first intermediate result and the rhythm condition vector are interacted in features by the cross-attention layer corresponding to the jth self-attention layer in the first deep learning network, and a first feature interaction result is obtained;

[0199] The first feature interaction result is taken as the input data of the j+1th self-attention layer in the first deep learning network.

[0200] As an optional embodiment, the process that the processor 801 performs chord dimension denoising processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video comprises:

[0201] The kth iteration denoising result of the noise data is predicted based on the rhythm condition vector and the reference chord by calling the second deep learning network, k being a positive integer;

[0202] The noise component is removed from the k-1th iteration denoising result to obtain the kth iteration denoising result of the noise data.

[0203] As an optional embodiment, the predicted rhythm of the action video is obtained by calling the first deep learning network to perform rhythm dimension denoising processing on the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network each comprise M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process that the processor 801 calls the second deep learning network to predict the noise component corresponding to the kth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord comprises:

[0204] The input data is analyzed for correlation by the ith self-attention layer in the second deep learning network based on the output data of the ith self-attention layer in the first deep learning network, and a second intermediate result is obtained, i being a positive integer less than M;

[0205] The second intermediate result, the rhythm condition vector and the reference chord are interacted in features by the cross-attention layer corresponding to the ith self-attention layer in the second deep learning network, and a second feature interaction result is obtained;

[0206] The second feature interaction result is taken as input data of an (i+1)th self-attention layer in the second deep learning network.

[0207] In another embodiment, the processor 801 performs the following operations by running a computer program in the memory 803:

[0208] Obtain a sample audio corresponding to the sample video, a rhythm condition vector, and a reference chord; the rhythm condition vector is used to represent the action amplitude and action intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0209] Call the first to-be-trained model to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector, to obtain a predicted beat of the sample video;

[0210] Call the second to-be-trained model to perform chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord, to obtain a predicted chord of the sample video;

[0211] Train the first to-be-trained model and the second to-be-trained model based on the predicted beat, the predicted chord, and the sample audio, to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate music for a video to be accompanied.

[0212] As an optional embodiment, the sample audio includes a sample beat; the processor 801 trains the first to-be-trained model and the second to-be-trained model based on the predicted beat, the predicted chord, and the sample audio, to obtain a specific embodiment of the beat prediction model and the chord prediction model:

[0213] Train the first to-be-trained model based on the difference between the predicted beat and the sample beat, to obtain the beat prediction model;

[0214] Train the second to-be-trained model based on the difference between the predicted chord and the sample audio, to obtain the chord prediction model.

[0215] As an optional embodiment, the processor 801 trains the second to-be-trained model based on the difference between the predicted chord and the sample audio, to obtain a specific embodiment of the chord prediction model:

[0216] Fuse the predicted beat and the predicted chord to obtain a predicted audio of the sample video;

[0217] Train the second to-be-trained model based on the difference between the predicted audio and the sample audio, to obtain the chord prediction model.

[0218] Based on the same inventive concept, the computer device provided in the embodiments of the present application has similar principles and beneficial effects to the principles and beneficial effects of the audio processing method of the method embodiments, and can refer to the principles and beneficial effects of the method embodiments. For brevity, the description is not repeated here.

[0219] The embodiments of the present application also provide a computer readable storage medium, which stores a computer program. The computer program is suitable for being loaded by a processor and performing the audio processing method of the method embodiments.

[0220] The embodiments of the present application also provide a computer program product or a computer program, which includes computer instructions stored in a computer readable storage medium. The processor of the computer device reads the computer instructions from the computer readable storage medium, and the processor executes the computer instructions to make the computer device perform the audio processing method.

[0221] The steps in the method embodiments of the present application can be adjusted in sequence, combined and deleted according to actual needs.

[0222] The modules in the device embodiments of the present application can be combined, divided and deleted according to actual needs.

[0223] In the embodiments of the present application, the "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as processing circuitry or memory) or a combination thereof. Similarly, at least one processor (or memory) can be used to implement at least one module or unit. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0224] Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by a program instructing related hardware, and the program can be stored in a computer readable storage medium, which can include a flash disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0225] The above only discloses a preferred embodiment of the present application, and of course cannot limit the scope of the rights of the present application. Those of ordinary skill in the art can understand that the implementation of all or part of the above embodiments, and the equivalent changes made according to the claims of the present application, still belong to the scope covered by the application.

Claims

1. An audio processing method, characterized by, The method comprises: obtaining a rhythm condition vector corresponding to an action video to be matched with music and reference chords of the action video, the rhythm condition vector being used to represent action amplitude and action intensity of a target object in the action video, and the reference chords being determined based on a style of the action video; based on the rhythm condition vector, performing beat-dimension noise reduction processing on noise data to obtain predicted beats of the action video; based on the rhythm condition vector and the reference chords, performing chord-dimension noise reduction processing on the noise data to obtain predicted chords of the action video; fusing the predicted beats and the predicted chords to obtain music matched with the action video.

2. The method of claim 1, wherein, The reference chords of the action video are obtained by: extracting visual features of the action video to obtain visual features of the action video; the visual features contain style information of the action video; based on the visual features, screening a target candidate video from candidate videos, the visual features of the target candidate video having a feature similarity with the visual features of the action video greater than a similarity threshold; determining chords of the target candidate video as the reference chords of the action video.

3. The method of claim 1, wherein, The rhythm condition vector corresponding to the action video to be matched with music is obtained by: sampling a posture of a target object in the action video to obtain posture information of the target object; the posture information is used to indicate action changes of the target object within consecutive time; feature encoding the posture information of the target object to obtain a rhythm condition vector.

4. The method of claim 1, wherein, The process of performing beat-dimension noise reduction processing on noise data based on the rhythm condition vector to obtain predicted beats of the action video comprises: calling a first deep learning network to predict a noise component corresponding to an hth iteration noise reduction result of noise data based on the rhythm condition vector, h being a positive integer; removing the noise component from an (h-1)th iteration noise reduction result to obtain the hth iteration noise reduction result of the noise data.

5. The method of claim 4, wherein, The first deep learning network comprises M self-attention layers, each self-attention layer corresponding to a cross-attention layer, M being a positive integer; the process of calling the first deep learning network to predict the noise component corresponding to the hth iteration noise reduction result of the noise data based on the rhythm condition vector comprises: performing correlation analysis on input data by a jth self-attention layer in the first deep learning network to obtain a first intermediate result, j being a positive integer less than M; performing feature interaction on the first intermediate result and the rhythm condition vector by a cross-attention layer corresponding to the jth self-attention layer in the first deep learning network to obtain a first feature interaction result; taking the first feature interaction result as input data of a (j+1)th self-attention layer in the first deep learning network.

6. The method of claim 1, wherein, The process of performing chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chords to obtain predicted chords of the action video comprises: calling a second deep learning network to predict, based on the rhythm condition vector and the reference chord, a noise component corresponding to the kth iteration denoising result of the noise data, k being a positive integer; removing the noise component from the k-1th iteration denoising result to obtain the kth iteration denoising result of the noise data.

7. The method of claim 6, wherein, The predicted rhythm of the action video is obtained by calling a first deep learning network to perform rhythm-dimension denoising processing on noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network each include M self-attention layers, each self-attention layer corresponding to a cross-attention layer, M being a positive integer; the process of calling the second deep learning network to predict the noise component corresponding to the kth iteration denoising result of the noise data based on the rhythm condition vector and the reference chord includes: calling an i-th self-attention layer in the second deep learning network to perform correlation analysis on input data based on output data of an i-th self-attention layer in the first deep learning network to obtain a second intermediate result, i being a positive integer less than M; performing feature interaction on the second intermediate result, the rhythm condition vector and the reference chord through a cross-attention layer corresponding to the i-th self-attention layer in the second deep learning network to obtain a second feature interaction result; taking the second feature interaction result as input data of an i+1-th self-attention layer in the second deep learning network.

8. A model training method, comprising: The method includes: obtaining a sample audio, a rhythm condition vector and a reference chord corresponding to a sample video; the rhythm condition vector is used to represent action amplitude and action intensity of a target object in the sample video, and the reference chord is determined based on a style of the sample video; calling a first to-be-trained model to perform rhythm-dimension denoising processing on noise data based on the rhythm condition vector to obtain a predicted rhythm of the sample video; calling a second to-be-trained model to perform chord-dimension denoising processing on noise data based on the rhythm condition vector and the reference chord to obtain a predicted chord of the sample video; training the first to-be-trained model and the second to-be-trained model based on the predicted rhythm, the predicted chord and the sample audio to obtain a rhythm prediction model and a chord prediction model; the rhythm prediction model and the chord prediction model are used to generate music for a video to be accompanied.

9. The method of claim 8, wherein, The sample audio includes a sample rhythm; training the first to-be-trained model and the second to-be-trained model based on the predicted rhythm, the predicted chord and the sample audio to obtain a rhythm prediction model and a chord prediction model includes: training the first to-be-trained model based on a difference between the predicted rhythm and the sample rhythm to obtain a rhythm prediction model; training the second to-be-trained model based on a difference between the predicted chord and the sample audio to obtain a chord prediction model.

10. The method of claim 9, wherein, Training the second to-be-trained model based on a difference between the predicted chord and the sample audio to obtain a chord prediction model includes: fuse the predicted beat and the predicted chord to obtain a predicted audio of the sample video; train the second to-be-trained model based on a difference between the predicted audio and the sample audio to obtain a chord prediction model.

11. An audio processing apparatus, characterized by comprising: The audio processing apparatus comprises: an acquisition unit configured to acquire a rhythm condition vector corresponding to a motion video to be accompanied by music and a reference chord of the motion video, the rhythm condition vector being used to represent a motion amplitude and a motion intensity of a target object in the motion video, and the reference chord being determined based on a style of the motion video; a processing unit configured to perform beat-dimension noise reduction processing on noise data based on the rhythm condition vector to obtain a predicted beat of the motion video, and perform chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain a predicted chord of the motion video; and fuse the predicted beat and the predicted chord to obtain music for the motion video. The model training apparatus comprises:

12. A model training apparatus, comprising: an acquisition unit configured to acquire a sample audio, a rhythm condition vector and a reference chord corresponding to a sample video, the rhythm condition vector being used to represent a motion amplitude and a motion intensity of a target object in the sample video, and the reference chord being determined based on a style of the sample video; a processing unit configured to call a first to-be-trained model to perform beat-dimension noise reduction processing on noise data based on the rhythm condition vector to obtain a predicted beat of the sample video, and call a second to-be-trained model to perform chord-dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain a predicted chord of the sample video; and train the first to-be-trained model and the second to-be-trained model based on the predicted beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model, the beat prediction model and the chord prediction model being used to generate music for a video to be accompanied by music. comprise: a memory having a computer program stored therein; 13. A computer device, comprising: a processor configured to load the computer program to implement the audio processing method according to any one of claims 1-7, or to load the computer program to implement the model training method according to any one of claims 8-10. The computer readable storage medium stores a computer program, and the computer program is adapted to be loaded by a processor and executed to implement the audio processing method according to any one of claims 1-7, or to implement the model training method according to any one of claims 8-10. The computer program product comprises a computer program, and the computer program is adapted to be loaded by a processor and executed to implement the audio processing method according to any one of claims 1-7, or to implement the model training method according to any one of claims 8-10.

14. A computer-readable storage medium, characterized in that, ​ 15. A computer program product, characterised in that, ​

Images