Audio processing method and device, model training method and device, equipment, storage medium and product

Abstract

The embodiment of the invention discloses an audio processing method, a model training method and device, equipment, a storage medium and a product. The audio processing method comprises the steps of obtaining a rhythm condition vector corresponding to an action video to be dubbed with music and a reference chord of the action video, carrying out rhythm dimension noise reduction processing on noise data based on the rhythm condition vector to obtain a predicted rhythm of the action video, and carrying out rhythm dimension noise reduction processing on the predicted rhythm of the action video based on the rhythm condition vector and the reference chord. And carrying out chord dimension noise reduction processing on the noise data to obtain a predicted chord of the action video, and fusing the predicted rhythm and the predicted chord to obtain a score of the action video. Therefore, noise reduction processing is carried out on the takt and the chord of the dubbing music through different constraint conditions, the prediction process of the dubbing music can be decoupled, the prediction process of the takt and the chord is more targeted, and then the quality of the dubbing music of the action video is improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and specifically to an audio processing method, a model training method, an action generation device, a model training device, a computer device, a computer-readable storage medium, and an action generation product and a model training product. Background Art

[0002] With the advancement of scientific and technological research, the relevant technologies of artificial intelligence models are developing rapidly. More and more artificial intelligence models are being applied in various fields of life; for example, generative models are applied in the field of audio generation, and discriminative models are applied in the field of face recognition. In the field of audio generation, the audio generation model can generate corresponding music based on the video. Studies have found that when the video to be accompanied by music is an action video, the quality of the music generated by the audio generation model is low (for example, the rhythm of the music does not match the action, the style of the music is inconsistent with the style of the video, etc.). Summary of the invention

[0003] The embodiments of the present application provide an audio processing method, a model training method and apparatus, a device, a computer-readable storage medium, and a product, which can improve the quality of the soundtrack of an action video.

[0004] On the one hand, an embodiment of the present application provides an audio processing method, including:

[0005] Obtaining a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video, wherein the rhythm condition vector is used to characterize the action amplitude and action intensity of the target object in the action video; and the reference chord is determined based on the style of the action video;

[0006] Based on the rhythm condition vector, the noise data is denoised in the beat dimension to obtain the predicted beat of the action video;

[0007] Based on the rhythm condition vector and the reference chord, the noise data is subjected to denoising in the chord dimension to obtain the predicted chord of the action video;

[0008] The predicted beats and predicted chords are fused to obtain the soundtrack for the action video.

[0009] In an embodiment of the present application, a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video are obtained, and based on the rhythm condition vector, noise data is subjected to noise reduction processing in the beat dimension to obtain a predicted beat of the action video, and based on the rhythm condition vector and the reference chord, noise data is subjected to noise reduction processing in the chord dimension to obtain a predicted chord of the action video, and the predicted beat and predicted chord are fused to obtain the music for the action video. It can be seen that by performing noise reduction processing on the beat and chord of the music respectively through different constraints, the prediction process of the music can be decoupled, making the prediction process of the beat and chord more targeted, thereby improving the quality of the music for the action video.

[0010] On the one hand, an embodiment of the present application provides a model training method, comprising:

[0011] Obtaining sample audio, rhythm condition vector and reference chord corresponding to the sample video; the rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0012] Calling the first model to be trained to perform denoising processing on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the sample video;

[0013] Calling the second to-be-trained model to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the sample video;

[0014] Based on the predicted beat, predicted chord and sample audio, the first model to be trained and the second model to be trained are trained to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate music for the video to be accompanied by music.

[0015] In an embodiment of the present application, sample audio, rhythm condition vectors and reference chords corresponding to a sample video are obtained, a first model to be trained is called to perform noise reduction processing on the noise data in the beat dimension based on the rhythm condition vector to obtain a predicted beat of the sample video, a second model to be trained is called to perform noise reduction processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain a predicted chord of the sample video, and the first model to be trained and the second model to be trained are trained based on the predicted beat, predicted chord and sample audio to obtain a beat prediction model and a chord prediction model, which are used to generate the soundtrack for the video to be soundtracked. It can be seen that by performing targeted training on the beat prediction model and the chord prediction model respectively with different sample data, the model training effect can be improved, so that the trained model can generate high-quality soundtracks.

[0016] On the one hand, an embodiment of the present application provides an audio processing device, the audio processing device comprising:

[0017] An acquisition unit is used to acquire a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video, wherein the rhythm condition vector is used to characterize the action amplitude and action intensity of the target object in the action video; and the reference chord is determined based on the style of the action video;

[0018] A processing unit, configured to perform a denoising process on the noise data in a beat dimension based on the rhythm condition vector, so as to obtain a predicted beat of the action video;

[0019] and for performing denoising processing on the chord dimension of the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video;

[0020] And used to fuse the predicted beats and predicted chords to get the soundtrack for the action video.

[0021] In one implementation, the processing unit is used to obtain a reference chord of the action video, specifically to:

[0022] Extracting visual features of the action video to obtain visual features of the action video; the visual features contain style information of the action video;

[0023] Selecting a target candidate video from the candidate videos based on the visual features, wherein the feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold;

[0024] The chord of the target candidate video is determined as the reference chord of the action video.

[0025] In one implementation, the processing unit is used to obtain a rhythm condition vector corresponding to the action video to be matched with music, specifically to:

[0026] Perform posture sampling on the target object in the action video to obtain the posture information of the target object; the posture information is used to indicate the action changes of the target object in a continuous time;

[0027] The posture information of the target object is feature encoded to obtain a rhythm condition vector.

[0028] In one implementation, the processing unit performs a denoising process on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the action video, including:

[0029] Calling the first deep learning network to predict the noise component corresponding to the h-th iteration denoising result of the noise data based on the rhythm condition vector, where h is a positive integer;

[0030] The noise component is removed from the h-1th iterative denoising result to obtain the hth iterative denoising result of the noise data.

[0031] In one implementation, the first deep learning network includes M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process in which the processing unit calls the first deep learning network to predict the noise component corresponding to the h-th iteration denoising result of the noise data based on the rhythm condition vector includes:

[0032] Perform correlation analysis on the input data through the j-th self-attention layer in the first deep learning network to obtain a first intermediate result, where j is a positive integer less than M;

[0033] Performing feature interaction on the first intermediate result and the rhythm condition vector through a cross attention layer corresponding to the j-th self-attention layer in the first deep learning network to obtain a first feature interaction result;

[0034] The first feature interaction result is used as the input data of the j+1th self-attention layer in the first deep learning network.

[0035] In one implementation, the processing unit performs noise reduction processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video, which includes:

[0036] Calling the second deep learning network to predict the noise component corresponding to the k-th iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, where k is a positive integer;

[0037] The noise component is removed from the k-1th iterative denoising result to obtain the kth iterative denoising result of the noise data.

[0038] In one embodiment, the predicted beat of the action video is obtained by calling the first deep learning network to perform denoising processing on the beat dimension of the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network both include M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the processing unit calls the second deep learning network based on the rhythm condition vector and the reference chord to predict the noise component corresponding to the k-th iteration denoising result of the noise data. The process includes:

[0039] Call the i-th self-attention layer in the second deep learning network to perform correlation analysis on the input data based on the output data of the i-th self-attention layer in the first deep network learning network to obtain a second intermediate result, where i is a positive integer less than M;

[0040] Perform feature interaction on the second intermediate result, the rhythm condition vector, and the reference chord through a cross attention layer corresponding to the i-th self-attention layer in the second deep learning network to obtain a second feature interaction result;

[0041] The second feature interaction result is used as the input data of the i+1th self-attention layer in the second deep learning network.

[0042] On the one hand, an embodiment of the present application provides a model training device, the model training device comprising:

[0043] An acquisition unit, used to acquire sample audio, a rhythm condition vector and a reference chord; the rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0044] A processing unit, configured to call the first model to be trained to perform denoising processing on the noise data in a beat dimension based on the rhythm condition vector, so as to obtain a predicted beat of the sample video;

[0045] and for calling the second to-be-trained model to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord, so as to obtain the predicted chord of the sample video;

[0046] And it is used to train the first model to be trained and the second model to be trained based on the predicted beat, predicted chord and sample audio to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate the soundtrack of the video to be matched with the music.

[0047] In one embodiment, the sample audio includes a sample beat; the processing unit is used to train the first model to be trained and the second model to be trained based on the predicted beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model, specifically for:

[0048] Based on the difference between the predicted beat and the sample beat, the first model to be trained is trained to obtain a beat prediction model;

[0049] Based on the difference between the predicted chord and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

[0050] In one embodiment, the processing unit is used to train the second model to be trained based on the difference between the predicted chord and the sample audio to obtain the chord prediction model, specifically for:

[0051] The predicted beat and the predicted chord are merged to obtain the predicted audio of the sample video;

[0052] Based on the difference between the predicted audio and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

[0053] Accordingly, the present application provides a computer device, the computer device comprising:

[0054] a memory, wherein a computer program is stored in the memory;

[0055] The processor is used to load a computer program to implement the above audio processing method.

[0056] Accordingly, the present application provides a computer-readable storage medium, which stores a computer program, which is suitable for being loaded by a processor and executing the above-mentioned audio processing method, or is suitable for being loaded by a processor and executing the above-mentioned model training method.

[0057] Accordingly, the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned audio processing method, or performs the above-mentioned model training method.

[0058] In the inference process, the embodiment of the present application performs noise reduction processing on the beats and chords of the soundtrack through different constraints, which can decouple the prediction process of the soundtrack, making the prediction process of the beats and chords more targeted, thereby improving the quality of the soundtrack of the action video. In the training process, the beat prediction model and the chord prediction model are trained specifically through different sample data, which can improve the model training effect and enable the trained model to generate high-quality soundtracks. BRIEF DESCRIPTION OF THE DRAWINGS

[0059] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0060] Figure 1a A schematic diagram of a diffusion process provided in an embodiment of the present application;

[0061] Figure 1b An audio processing scene graph provided in an embodiment of the present application;

[0062] Figure 2 A flowchart of an audio processing method provided in an embodiment of the present application;

[0063] Figure 3a A schematic diagram of a screening module provided in an embodiment of the present application;

[0064] Figure 3b An audio processing model architecture diagram provided in an embodiment of the present application;

[0065] Figure 4 A flowchart of a model training method provided in an embodiment of the present application;

[0066] Figure 5 A schematic diagram of a model training architecture provided in an embodiment of the present application;

[0067] Figure 6 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application;

[0068] Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application;

[0069] Figure 8 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0070] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0071] The present application embodiment involves technologies related to diffusion models. The following briefly introduces the relevant terms and concepts of diffusion models:

[0072] Diffusion model principle: The diffusion model is inspired by the diffusion process in physics, that is, how an ordered structure is gradually transformed into a disordered state and then reversed. Figure 1a A diffusion process schematic diagram provided in an embodiment of the present application. Figure 1a As shown in Figure 2, the working principle of the diffusion model can be divided into two main stages: forward process and reverse process.

[0073] Forward process: This stage simulates the diffusion process from data distribution to random noise distribution. At the beginning, there is a clean sample data (such as an audio clip), and then noise is gradually added to the sample data. After several noise superpositions, the sample data completely becomes noise data. This process is usually implemented by applying Gaussian noise, and each step will make the sample data closer to the noise data distribution.

[0074] Reverse process: The reverse process is the inverse of the forward process, and its goal is to recover the original data from the noisy data. In this stage, the diffusion model is used to predict the noise added at each step.

[0075] Based on the above-mentioned technology related to the diffusion model, the embodiment of the present application provides an audio processing solution and a model training solution, which can improve the quality of the soundtrack of action videos. Figure 1b An audio processing scene graph provided in an embodiment of the present application, such as Figure 1b As shown, the audio processing scenario provided by the present application includes a terminal device 101 and a server 102. The audio processing scheme and model training scheme provided by the present application can be executed by the terminal device 101 or the server 102; when the audio processing scheme and model training scheme provided by the present application are both executed by the terminal device 101, the server 102 may not be included in the audio processing scenario. Among them, the terminal device may include but is not limited to: smart phones (such as Android phones, IOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, referred to as MID), intelligent voice interaction devices, smart home appliances, vehicle terminals, aircraft, wearable devices, etc., and the present application embodiment does not limit this; the server can be an independent physical server, or a server cluster or distributed system composed of two or more physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms, and the present application embodiment does not limit this.

[0076] It should be noted that Figure 1b The number of terminal devices and servers is only used as an example and does not constitute an actual limitation of the present application. The terminal device 101 and the server 102 can be connected via wired or wireless means, and the present application does not limit this.

[0077] The audio processing scheme and the model training scheme can be independent of each other or combined with each other; for example, the model training scheme is executed in the server 102, and the trained audio processing model is deployed in the terminal device 101, and then the audio processing scheme is executed in the terminal device 101; for another example, the model training scheme and the audio processing scheme are executed in the server 102.

[0078] The general process of the audio processing solution provided by this application is as follows:

[0079] Server 102 obtains the rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video. The action video contains a series of actions performed by the target object in at least one continuous time period, and the action video can specifically be a dance video, a martial arts video, a gymnastics video, a sports video, etc. The target object can be the protagonist in the action video, and can specifically be a person, a virtual character, a robot, an animal, etc., and this application does not limit this. The rhythm condition vector is used to characterize the action amplitude and action intensity of the target object in the action video, and the reference chord is determined based on the style of the action video. The action video can be sent by the terminal device 101, or it can be collected by the server 102 through an acquisition device (such as a camera), or it can be generated by the server 102 based on input data (such as an action sequence, a reference video, etc.), and this application does not limit this.

[0080] After obtaining the rhythm condition vector corresponding to the action video to be matched with music and the reference chords of the action video, on the one hand, the server 102 performs noise reduction processing on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the action video. In one embodiment, the processing process includes: the server 102 calls the first deep learning network to predict the noise component corresponding to the h-th iteration noise reduction result of the noise data based on the rhythm condition vector, and removes the noise component corresponding to the h-1-th iteration noise reduction result from the h-th iteration noise reduction result to obtain the h-th iteration noise reduction result of the noise data, where h is a positive integer. According to the above embodiment, the server 102 can call the first deep learning network to gradually remove the noise in the noise data until the predicted beat of the action video is obtained.

[0081] On the other hand, the server 102 performs chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video. In one embodiment, the processing process includes: the server 102 calls the second deep learning network to predict the noise component corresponding to the k-th iteration noise reduction result of the noise data based on the rhythm condition vector and the reference chord, and removes the noise component from the k-1-th iteration noise reduction result to obtain the k-th iteration noise reduction result of the noise data, where k is a positive integer. According to the above embodiment, the server 102 can call the second deep learning network to gradually remove the noise in the noise data until the predicted chord of the action video is obtained.

[0082] After obtaining the predicted beat and predicted chord of the action video, the server 102 fuses the predicted beat and predicted chord to obtain the soundtrack of the action video.

[0083] The general process of the model training solution provided in this application is as follows:

[0084] The server 102 obtains the sample audio, rhythm condition vector and reference chord corresponding to the sample video; wherein the rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video. The sample audio is composed of sample beats and sample chords.

[0085] After obtaining the sample audio, rhythm condition vector and reference chord corresponding to the sample video, on the one hand, the server 102 calls the first model to be trained to perform denoising processing on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the sample video. In one embodiment, the first model to be trained includes a first deep learning network, and the process of generating the predicted beat includes: calling the first deep learning network to predict the noise component corresponding to the g-th iteration denoising result of the noise data based on the rhythm condition vector, and removing the noise component from the g-1-th iteration denoising result to obtain the g-th iteration denoising result of the noise data, where g is a positive integer.

[0086] On the other hand, the server 102 calls the second model to be trained to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the sample video. The specific implementation method can refer to the implementation method of "based on the rhythm condition vector and the reference chord, the noise data is subjected to denoising processing in the chord dimension to obtain the predicted chord of the action video" in the audio processing scheme, which will not be repeated here.

[0087] After obtaining the predicted beat and the predicted chord, the server 102 trains the first model to be trained and the second model to be trained based on the predicted beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model. In one embodiment, the sample audio includes a sample beat, and the server 102 trains the first model to be trained based on the difference between the predicted beat and the sample beat to obtain a beat prediction model, and trains the second model to be trained based on the difference between the predicted chord and the sample audio to obtain a chord prediction model.

[0088] Further, after completing the training of the beat prediction model and the chord prediction model, the server 102 can call the beat prediction model based on the rhythm condition vector to perform a denoising process on the noise data in the beat dimension to obtain the predicted beat of the action video, and call the chord prediction model based on the rhythm condition vector and the reference chord to perform a denoising process on the noise data in the chord dimension to obtain the predicted chord of the action video. The soundtrack of the action video can be generated by the predicted beat of the action video and the predicted chord of the action video. The specific implementation method can refer to the audio processing solution, which will not be repeated here.

[0089] In the inference process, the embodiment of the present application performs noise reduction processing on the beats and chords of the soundtrack through different constraints, which can decouple the prediction process of the soundtrack, making the prediction process of the beats and chords more targeted, thereby improving the quality of the soundtrack of the action video. In the training process, the beat prediction model and the chord prediction model are trained specifically through different sample data, which can improve the model training effect and enable the trained model to generate high-quality soundtracks.

[0090] Based on the above-mentioned audio processing scheme and model training scheme, the embodiments of the present application propose a more detailed audio processing method and model training method. The audio processing method and model training method proposed in the embodiments of the present application will be introduced in detail below in conjunction with the accompanying drawings.

[0091] See also Figure 2 , Figure 2 A flowchart of an audio processing method provided in an embodiment of the present application, the audio processing method can be executed by a computer device; for example, Figure 1b The terminal device 101 or the server 102 shown in FIG. Figure 2 As shown, the audio processing method may include the following steps S201-S204:

[0092] S201, obtaining a rhythm condition vector corresponding to an action video to be matched with music and a reference chord of the action video.

[0093] The computer device first obtains an action video, which may be sent by other devices, or may be collected by the computer device through a collection device (such as a camera), or may be generated by the computer device based on input data (such as an action sequence, a reference video, etc.), and this application does not limit this. The action video contains a series of actions performed by the target object in at least one continuous time period, and the action video may specifically be a dance video, a martial arts video, a gymnastics video, a sports video, etc. The target object may be the protagonist in the action video, and may specifically be a person, a virtual character, a robot, an animal, etc., and this application does not limit this.

[0094] In the process of obtaining the reference chords corresponding to the action video to be matched with music, the computer device can filter out the target candidate video from the candidate videos based on at least one screening factor, and obtain the reference chords of the action video based on the chords of the target candidate video; wherein the screening factors include but are not limited to: whether the video is the same object, visual features, video type, text content in the video, and similarity of actions.

[0095] In one embodiment, a computer device extracts visual features of an action video to obtain visual features of the action video, wherein the visual features include style information of the action video. Then, the computer device selects a target candidate video from the candidate videos based on the visual features of the action video, and the feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold. Furthermore, after selecting at least one target candidate video, the computer device may determine the chord of the target candidate video as a reference chord of the action video, and extract audio features of the reference chord of the action video to obtain chord constraints of the action video. The above process can be expressed as:

[0096]

[0097] Among them, S ar represents the chord constraint of the action video, Z rm S represents the (chord) audio features of the candidate video. highest is the feature similarity between the visual features of the candidate video and the visual features of the action video. In one embodiment, <A,B> Indicates the calculation of the cosine similarity between feature vector A and feature vector B. is the visual feature of the i-th candidate video in the database, C v is the visual feature of the action video to be matched with music, and threshold is the similarity threshold.

[0098] The above process can be performed by a screening module. Figure 3a A schematic diagram of a screening module provided in an embodiment of the present application. Figure 3a As shown, the screening module can determine (through a search trigger) whether to select the chord of the candidate video as the reference chord of the action video based on the feature similarity between the visual features of the candidate video and the visual features of the action video. It can be understood that since the visual features contain the style information of the action video, the reference chord obtained by screening based on the feature similarity has a higher degree of style matching with the action video.

[0099] In the process of obtaining the rhythm condition vector corresponding to the action video to be matched with music, the computer device performs posture sampling on the target object in the action video to obtain the posture information of the target object; wherein the posture information is used to indicate the action change of the target object in a continuous time. In one embodiment, the computer device first extracts the action sequence of the target object from the action video, and then generates corresponding posture information for each action in the action sequence; for example, the computer device can represent the joint position in each action, and add all joint positions corresponding to the same action to the same matrix to obtain the matrix representation of the action (i.e., as the posture information of the action), and the position of the jth joint of the tth action can be expressed as P (t, j, x, y), wherein t is the action (video frame) index, j is the joint index, and x, y are the plane coordinates of the joint. It can be understood that according to actual needs, the posture information of the target object can also be represented by three-dimensional coordinates. After obtaining the posture information of the target object, the computer device can feature encode the posture information of the target object to obtain the rhythm condition vector, which is used to characterize the action amplitude and action intensity of the target object in the action video.

[0100] The rhythm condition vector can be specifically expressed as: Among them, C r represents the rhythm condition vector, T represents the total number of time steps (which can be understood as the number of video frames in the action sequence), and l represents that the corresponding time step is a rhythm point; for example, assuming T = 10, the value of l includes 2, 5, and 8, which means that the time points corresponding to the 2nd video frame, the 5th video frame, and the 8th video frame are a rhythm point.

[0101] S202: Based on the rhythm condition vector, perform noise reduction processing on the noise data in the beat dimension to obtain the predicted beat of the action video.

[0102] Noise data is a kind of random noise, which is generated by Gaussian distribution (normal distribution) and is the starting point of the reverse diffusion process. Gaussian distribution is a probability distribution that describes the possibility of different values ​​appearing within a given range. The goal of noise reduction processing is to obtain a prediction result that matches the data distribution with the training data of the beat prediction model under the constraint of the rhythm condition vector. Noise reduction processing can be completed once or through two or more iterations. In the diffusion model, the noise reduction process is completed through two or more iterations. Each iterative noise reduction process is carried out in the latent space. The object of iterative noise reduction is the matrix of noise data in the latent space, and the input data of each iterative noise reduction includes the output data of the previous iterative noise reduction.

[0103] In one embodiment, the computer device calls the first deep learning network to predict the noise component corresponding to the h-th iteration noise reduction result of the noise data based on the rhythm condition vector, where h is a positive integer. The first deep learning network is included in the beat prediction model. In one embodiment, the first deep learning network includes M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer.

[0104] The prediction process of the noise component corresponding to the h-th iteration denoising result of the noise data in the beat dimension includes: the computer device performs correlation analysis on the input data of the attention layer through the j-th self-attention layer in the first deep learning network to obtain a first intermediate result, where j is a positive integer less than M. After obtaining the first intermediate result, the computer device performs feature interaction on the first intermediate result and the rhythm condition vector through the cross attention layer corresponding to the j-th self-attention layer in the first deep learning network to obtain a first feature interaction result, and uses the first feature interaction result as the input data of the j+1-th self-attention layer in the first deep learning network. It can be understood that the input data of the first self-attention layer in the first deep learning network is the h-1-th iteration denoising result; when h=1, the input data of the first self-attention layer in the first deep learning network is the noise data. When j=M-1, the result of the feature interaction output by the cross attention layer corresponding to the j+1-th self-attention layer in the first deep learning network is the noise component corresponding to the h-th iteration denoising result (vector representation).

[0105] After obtaining the noise component corresponding to the h-th iteration denoising result of the noise data in the beat dimension, the computer device removes the noise component corresponding to the h-th iteration denoising result from the h-1-th iteration denoising result (beat dimension) to obtain the h-th iteration denoising result of the noise data. The above-mentioned iterative denoising process can be performed in a high-dimensional space. In one embodiment, the computer device calculates the difference between the vector of the h-1-th iteration denoising result and the vector of the noise component corresponding to the h-th iteration denoising result to obtain the vector representation of the h-th iteration denoising result of the noise data. It can be understood that when h=1, after predicting the noise component corresponding to the 1st iteration denoising result of the beat dimension, the computer device removes the noise component corresponding to the 1st iteration denoising result from the noise data to obtain the 1st iteration denoising result of the noise data in the beat dimension; assuming that the total number of iterative denoising times in the beat dimension is H, the H-th iteration denoising result of the noise data in the beat dimension is the predicted beat of the action video.

[0106] S203: Based on the rhythm condition vector and the reference chord, perform noise reduction processing on the noise data in the chord dimension to obtain the predicted chord of the action video.

[0107] In one embodiment, the computer device calls the second deep learning network to predict the noise component corresponding to the kth iteration noise reduction result of the noise data based on the rhythm condition vector and the reference chord, where k is a positive integer. The second deep learning network is included in the chord prediction model. In one embodiment, the predicted beat of the action video is obtained by calling the first deep learning network to perform beat dimension noise reduction processing on the noise data based on the rhythm condition vector, and the first deep learning network and the second deep learning network both include M self-attention layers, each self-attention layer in the first deep learning network and the second deep learning network corresponds to a cross-attention layer, and M is a positive integer.

[0108] The prediction process of the noise component corresponding to the k-th iteration denoising result of the noise data in the chord dimension includes: the computer device calls the i-th self-attention layer in the second deep learning network, and based on the output data of the i-th self-attention layer in the first deep network learning network, performs correlation analysis on the input data of the i-th self-attention layer in the second deep learning network to obtain a second intermediate result, where i is a positive integer less than M. After obtaining the second intermediate result, the computer device performs feature interaction on the second intermediate result, the rhythm condition vector and the reference chord through the cross attention layer corresponding to the i-th self-attention layer in the second deep learning network to obtain a second feature interaction result, and uses the second feature interaction result as the input data of the i+1-th self-attention layer in the second deep learning network. It can be understood that the input data of the first self-attention layer in the second deep learning network is the k-1-th iteration denoising result; when k=1, the input data of the first self-attention layer in the first deep learning network is noise data. When i=M-1, the result of the feature interaction output by the cross-attention layer corresponding to the i+1th self-attention layer in the second deep learning network is the noise component (vector representation) corresponding to the kth iterative denoising result.

[0109] After obtaining the noise component corresponding to the kth iteration denoising result of the noise data in the chord dimension, the computer device removes the noise component corresponding to the kth iteration denoising result from the k-1th iteration denoising result (in the chord dimension) to obtain the kth iteration denoising result of the noise data. The above-mentioned iterative denoising process can be performed in a high-dimensional space. The specific implementation method can refer to the implementation method of removing the noise component corresponding to the hth iteration denoising result from the h-1th iteration denoising result in the beat dimension, which will not be repeated here. It can be understood that when k=1, after predicting the noise component corresponding to the 1st iteration denoising result in the chord dimension, the computer device removes the noise component corresponding to the 1st iteration denoising result from the noise data to obtain the 1st iteration denoising result of the noise data in the chord dimension; assuming that the total number of iterative denoising in the chord dimension is K, the Kth iteration denoising result of the noise data in the chord dimension is the predicted chord of the action video.

[0110] Figure 3b This is a diagram of an audio processing model architecture provided in an embodiment of the present application. Figure 3b As shown in FIG. 1 , structurally, the beat prediction model includes a first deep learning network, which includes M self-attention layers, each of which corresponds to a cross-attention layer, and M is a positive integer; the chord prediction model includes a second deep learning network, which also includes M self-attention layers, and each of which corresponds to a cross-attention layer. In terms of process, the beat prediction model performs beat dimension denoising on the noise data based on the rhythm condition vector, and can obtain the predicted beat of the action video; wherein, the rhythm condition vector (C r ) can be obtained by encoding the action sequence of the action video through the rhythm encoder. The chord prediction model performs denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video; in this process, when the i-th self-attention layer in the second deep learning network performs correlation analysis on the input data, it is constrained by the first intermediate result output by the i-th self-attention layer in the first deep learning network, where i is a positive integer less than or equal to M. Furthermore, the cross-attention layer corresponding to the i-th self-attention layer in the second deep learning network is used to perform correlation analysis on the second intermediate result output by the corresponding self-attention layer, the rhythm condition vector and the chord constraint condition (C ar ) for feature interaction; wherein the chord constraint condition is obtained by extracting audio features of the reference chord, and the reference video corresponding to the reference chord is obtained by screening the visual features of the action video by the screening module. For specific screening methods, please refer to Figure 3a , I will not go into details here.

[0111] S204: Fusing the predicted beats and the predicted chords to obtain the soundtrack of the action video.

[0112] In one embodiment, the computer device configures the beat of the predicted chord according to the predicted beat to obtain the soundtrack of the action video. It can be understood that by decoupling the prediction process of the soundtrack, the final soundtrack of the action video can be adapted to the action video in both the beat dimension and the chord dimension, effectively improving the quality of the soundtrack of the action video.

[0113] In an embodiment of the present application, a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video are obtained, and based on the rhythm condition vector, noise data is subjected to noise reduction processing in the beat dimension to obtain a predicted beat of the action video, and based on the rhythm condition vector and the reference chord, noise data is subjected to noise reduction processing in the chord dimension to obtain a predicted chord of the action video, and the predicted beat and predicted chord are fused to obtain the music for the action video. It can be seen that by performing noise reduction processing on the beat and chord of the music respectively through different constraints, the prediction process of the music can be decoupled, making the prediction process of the beat and chord more targeted, thereby improving the quality of the music for the action video.

[0114] See also Figure 4 , Figure 4 A flowchart of a model training method provided in an embodiment of the present application, the model training method can be executed by a computer device; for example, Figure 1b The terminal device 101 or the server 102 shown in FIG. Figure 4 As shown, the audio processing method may include the following steps S401-S404:

[0115] S401: Obtain sample audio, rhythm condition vector and reference chord corresponding to the sample video.

[0116] The sample audio can be the audio of the sample video itself or the audio specified by the model trainer. The sample audio can be composed of sample beats and sample chords. The sample audio is composed of sample beats and sample chords. The rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video. The reference chord is determined based on the style of the sample video. The specific implementation method of obtaining the rhythm condition vector corresponding to the sample video and the reference chord of the sample video can be referred to. Figure 2 The implementation of step S201 will not be repeated here.

[0117] S402: Call the first model to be trained to perform denoising processing on the noise data in the beat dimension based on the rhythm condition vector to obtain a predicted beat of the sample video.

[0118] In one embodiment, the first model to be trained includes a first deep learning network, and the generation process of the predicted beat includes: calling the first deep learning network based on the rhythm condition vector, predicting the noise component of the beat dimension corresponding to the g-th iteration denoising result of the noise data, and removing the noise component from the g-1-th iteration denoising result to obtain the g-th iteration denoising result of the noise data in the beat dimension, where g is a positive integer. It can be understood that when g=1, after predicting the noise component corresponding to the first iteration denoising result in the beat dimension, the computer device removes the noise component corresponding to the first iteration denoising result from the noise data to obtain the first iteration denoising result of the noise data in the beat dimension; assuming that the total number of iterative denoising in the beat dimension is G, the G-th iteration denoising result of the noise data in the beat dimension is the predicted beat of the sample video.

[0119] S403: Call the second model to be trained to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord, and obtain the predicted chord of the sample video.

[0120] In one embodiment, the second model to be trained includes a second deep learning network, and the process of generating the predicted chord includes: calling the second deep learning network to predict the noise component of the chord dimension corresponding to the r-th iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, and removing the noise component from the r-1-th iteration denoising result to obtain the r-th iteration denoising result of the noise data in the chord dimension, where r is a positive integer. It can be understood that when r=1, after predicting the noise component corresponding to the first iteration denoising result in the chord dimension, the computer device removes the noise component corresponding to the first iteration denoising result from the noise data to obtain the first iteration denoising result of the noise data in the chord dimension; assuming that the total number of iterative denoising in the chord dimension is R, the R-th iteration denoising result of the noise data in the chord dimension is the predicted chord of the action video.

[0121] The specific implementation of step S402 and step S403 can be referred to Figure 2 The implementation of step S202 and step S203 will not be repeated here.

[0122] S404: Based on the predicted beat, the predicted chord and the sample audio, the first model to be trained and the second model to be trained are trained to obtain a beat prediction model and a chord prediction model.

[0123] The beat prediction model and the chord prediction model are used to generate the soundtrack of the video to be matched with the music. The computer device can call the beat prediction model to perform a denoising process on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the action video, and call the chord prediction model to perform a denoising process on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video. The soundtrack of the action video can be generated by the predicted beat of the action video and the predicted chord of the action video. For the specific implementation method, please refer to Figure 2 The implementation of steps S201 to S204 will not be repeated here.

[0124] In one embodiment, the sample audio includes a sample beat. On the one hand, the computer device trains the first model to be trained based on the difference between the predicted beat and the sample beat to obtain a beat prediction model. The loss function used in the training process can be expressed as:

[0125]

[0126] Among them, θ is the first model to be trained (L A ) is the parameter to be solved, Z m represents the audio features of the sample audio corresponding to the sample video, ∈ represents the actual added noise (such as Gaussian noise), the noise predicted by the first training model (i.e., the noise removed from the noise data), and Z t is to superimpose ∈ onto Z m obtained above, t represents the time step (which can be understood as the number of noise superpositions), C r is the rhythm condition vector corresponding to the sample video, Express expectations.

[0127] On the other hand, the sample audio includes sample chords, and the computer device trains the second model to be trained based on the difference between the predicted chords and the sample audio (or sample chords) to obtain a chord prediction model.

[0128] In one embodiment, the computer device trains the second model to be trained based on the difference between the predicted chord and the sample chord to obtain a chord prediction model.

[0129] In another embodiment, the computer device fuses the predicted beat and the predicted chord to obtain the predicted audio of the sample video, and trains the second model to be trained based on the difference between the predicted audio and the sample audio to obtain the chord prediction model; wherein the predicted beat can be obtained by the computer device calling the first model to be trained (or the beat prediction model) based on the rhythm condition vector to perform beat dimension denoising on the noise data. For a specific implementation method, please refer to Figure 2The implementation of step S202 in the training process is not described in detail here. The loss function used in the training process can be expressed as:

[0130]

[0131] Among them, θ is the second model to be trained (L B ) is the parameter to be solved, Z m represents the audio features of the sample audio corresponding to the sample video, ∈ represents the actual added noise (such as Gaussian noise), and t represents the number of iterative noise reduction (i.e., the number of iterations to Z m The number of times the noise component is added), Z t It is to Z m Noise data obtained by superimposing t noise components. ar represents the chord constraint of the sample video, C ar The method of obtaining C can refer to formula (1), which will not be repeated here; v is the visual feature of the sample video, represents the predicted beat output by the first model to be trained (or beat prediction model), represents the audio features of the predicted audio of the sample video. It can be understood that when When the predicted rhythm is output by the first model to be trained, the computer device can jointly train the first model to be trained and the second model to be trained based on the above formula (3).

[0132] Figure 5 A schematic diagram of a model training architecture provided in an embodiment of the present application. Figure 5 As shown, Figure 5 and Figure 3b The main difference is that when training the second model to be trained, the second model to be trained can be trained based on the difference between the predicted chords and the sample chords; the second model to be trained can also be trained based on the difference between the predicted audio (consisting of predicted chords and predicted beats) and the sample audio (or the first model to be trained and the second model to be trained can be jointly trained).

[0133] It should be noted that Figure 4 The model training method in can be used with Figure 2 The action generation methods in are independent of each other and can also be Figure 2 Executed before the audio processing methods in .

[0134] In an embodiment of the present application, sample audio, rhythm condition vectors and reference chords corresponding to a sample video are obtained, a first model to be trained is called to perform noise reduction processing on the noise data in the beat dimension based on the rhythm condition vector to obtain a predicted beat of the sample video, a second model to be trained is called to perform noise reduction processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain a predicted chord of the sample video, and the first model to be trained and the second model to be trained are trained based on the predicted beat, predicted chord and sample audio to obtain a beat prediction model and a chord prediction model, which are used to generate the soundtrack for the video to be soundtracked. It can be seen that by performing targeted training on the beat prediction model and the chord prediction model respectively with different sample data, the model training effect can be improved, so that the trained model can generate high-quality soundtracks.

[0135] The method of the embodiment of the present application is described in detail above. In order to facilitate better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly.

[0136] See also Figure 6 , Figure 6 A schematic diagram of the structure of an audio processing device provided in an embodiment of the present application is shown in FIG. Figure 6 The audio processing apparatus shown may be mounted in a computer device, which may specifically be a terminal device or a server. Figure 6 The audio processing device shown can be used to perform the above Figure 2 Some or all of the functions of the described method embodiments. Figure 6 , the audio processing device comprises:

[0137] The acquisition unit 601 is used to acquire the rhythm condition vector corresponding to the action video to be matched with music and the reference chord of the action video, wherein the rhythm condition vector is used to represent the action amplitude and action intensity of the target object in the action video; the reference chord is determined based on the style of the action video;

[0138] A processing unit 602 is used to perform a denoising process on the noise data in a beat dimension based on the rhythm condition vector to obtain a predicted beat of the action video;

[0139] and for performing denoising processing on the chord dimension of the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video;

[0140] And used to fuse the predicted beats and predicted chords to get the soundtrack for the action video.

[0141] In one implementation, the processing unit 602 is used to obtain a reference chord of the action video, specifically to:

[0142] Extracting visual features of the action video to obtain visual features of the action video; the visual features contain style information of the action video;

[0143] Selecting a target candidate video from the candidate videos based on the visual features, wherein the feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold;

[0144] The chord of the target candidate video is determined as the reference chord of the action video.

[0145] In one implementation, the processing unit 602 is used to obtain a rhythm condition vector corresponding to the action video to be matched with music, specifically to:

[0146] Perform posture sampling on the target object in the action video to obtain the posture information of the target object; the posture information is used to indicate the action changes of the target object in a continuous time;

[0147] The posture information of the target object is feature encoded to obtain a rhythm condition vector.

[0148] In one implementation, the processing unit 602 performs a denoising process on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the action video, including:

[0149] Calling the first deep learning network to predict the noise component corresponding to the h-th iteration denoising result of the noise data based on the rhythm condition vector, where h is a positive integer;

[0150] The noise component is removed from the h-1th iterative denoising result to obtain the hth iterative denoising result of the noise data.

[0151] In one implementation, the first deep learning network includes M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process in which the processing unit 602 calls the first deep learning network to predict the noise component corresponding to the h-th iteration denoising result of the noise data based on the rhythm condition vector includes:

[0152] Perform correlation analysis on the input data through the j-th self-attention layer in the first deep learning network to obtain a first intermediate result, where j is a positive integer less than M;

[0153] Performing feature interaction on the first intermediate result and the rhythm condition vector through a cross attention layer corresponding to the j-th self-attention layer in the first deep learning network to obtain a first feature interaction result;

[0154] The first feature interaction result is used as the input data of the j+1th self-attention layer in the first deep learning network.

[0155] In one implementation, the processing unit 602 performs denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video, including:

[0156] Calling the second deep learning network to predict the noise component corresponding to the k-th iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, where k is a positive integer;

[0157] The noise component is removed from the k-1th iterative denoising result to obtain the kth iterative denoising result of the noise data.

[0158] In one implementation, the predicted beat of the action video is obtained by calling the first deep learning network to perform denoising processing on the beat dimension of the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network both include M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the processing unit 602 calls the second deep learning network based on the rhythm condition vector and the reference chord to predict the noise component corresponding to the k-th iteration denoising result of the noise data, and the process includes:

[0159] Call the i-th self-attention layer in the second deep learning network to perform correlation analysis on the input data based on the output data of the i-th self-attention layer in the first deep network learning network to obtain a second intermediate result, where i is a positive integer less than M;

[0160] Perform feature interaction on the second intermediate result, the rhythm condition vector, and the reference chord through a cross attention layer corresponding to the i-th self-attention layer in the second deep learning network to obtain a second feature interaction result;

[0161] The second feature interaction result is used as the input data of the i+1th self-attention layer in the second deep learning network.

[0162] According to one embodiment of the present application, Figure 2 Some of the steps involved in the audio processing method shown can be represented by Figure 6 The audio processing device shown in the figure is executed by each unit. For example, Figure 2 The step S201 shown in FIG. 1 can be performed by Figure 6 The acquisition unit 601 shown in FIG. 6 is executed, and steps S202 to S204 can be performed by Figure 6 Processing unit 602 is shown executing. Figure 6The various units in the audio processing device shown can be separately or completely combined into one or several other units to form, or one (some) of the units can also be split into two or more functionally smaller units to form, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the functions of one unit can also be implemented by two or more units, or the functions of two or more units can be implemented by one unit. In other embodiments of the present application, the audio processing device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of two or more units.

[0163] According to another embodiment of the present application, the program can be executed by running a program on a general computing device such as a computer device including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 2 A computer program (including program code) for each step involved in the corresponding method shown in Figure 6 The audio processing device shown in and the audio processing method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0164] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the audio processing device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the audio processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0165] See also Figure 7 , Figure 7 A schematic diagram of the structure of a model training device provided in an embodiment of the present application, Figure 7 The model training apparatus shown may be mounted in a computer device, which may specifically be a terminal device or a server. Figure 7 The model training apparatus shown can be used to perform the above Figure 4 Some or all of the functions of the described method embodiments. Figure 7 , the model training device comprises:

[0166] The acquisition unit 701 is used to acquire the sample audio, the rhythm condition vector and the reference chord corresponding to the sample video; the rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0167] The processing unit 702 is used to call the first model to be trained to perform a denoising process on the noise data in the beat dimension based on the rhythm condition vector to obtain a predicted beat of the sample video;

[0168] and for calling the second to-be-trained model to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord, so as to obtain the predicted chord of the sample video;

[0169] And it is used to train the first model to be trained and the second model to be trained based on the predicted beat, predicted chord and sample audio to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate the soundtrack of the video to be matched with the music.

[0170] In one implementation, the sample audio includes a sample beat; the processing unit 702 is used to train the first model to be trained and the second model to be trained based on the predicted beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model, specifically for:

[0171] Based on the difference between the predicted beat and the sample beat, the first model to be trained is trained to obtain a beat prediction model;

[0172] Based on the difference between the predicted chord and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

[0173] In one implementation, the processing unit 702 is used to train the second model to be trained based on the difference between the predicted chord and the sample audio to obtain the chord prediction model, specifically for:

[0174] The predicted beat and the predicted chord are merged to obtain the predicted audio of the sample video;

[0175] Based on the difference between the predicted audio and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

[0176] According to one embodiment of the present application, Figure 4 Some of the steps involved in the model training method shown can be represented by Figure 7 The model training device shown is executed by each unit. For example, Figure 4 The step S401 shown in FIG. 4 can be performed by Figure 7 The acquisition unit 701 shown in FIG. 1 is executed, and steps S402 to S404 can be performed by Figure 7 Processing unit 702 is shown executing. Figure 7The various units in the model training device shown can be respectively or completely merged into one or several other units to constitute, or some of the units (some) can also be split into two or more functionally smaller units to constitute, which can achieve the same operation without affecting the realization of the technical effects of the embodiments of the present application. The above-mentioned units are divided based on logical functions. In practical applications, the function of a unit can also be realized by two or more units, or the function of two or more units is realized by one unit. In other embodiments of the present application, the model training device may also include other units. In practical applications, these functions can also be implemented with the assistance of other units, and can be implemented by the collaboration of two or more units.

[0177] According to another embodiment of the present application, the program can be executed by running a program on a general computing device such as a computer device including a central processing unit (CPU), a random access memory medium (RAM), a read-only memory medium (ROM), and other processing elements and storage elements. Figure 4 A computer program (including program code) for each step involved in the corresponding method shown in Figure 7 The model training device shown in and the model training method of the embodiment of the present application are implemented. The computer program can be recorded on, for example, a computer readable recording medium, and loaded into the above-mentioned computing device through the computer readable recording medium and run therein.

[0178] Based on the same inventive concept, the principles and beneficial effects of solving the problems by the model training device provided in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems by the model training method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0179] See also Figure 8 , Figure 8 The following is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device may be a terminal device or a server. Figure 8As shown, the computer device at least includes a processor 801, a communication interface 802 and a memory 803. The processor 801, the communication interface 802 and the memory 803 can be connected via a bus or other means. The processor 801 (or central processing unit (CPU)) is the computing core and control core of the computer device, which can parse various instructions in the computer device and process various data of the computer device. For example, the CPU can be used to parse the power on and off instructions issued by the object to the computer device, and control the computer device to perform power on and off operations; for example, the CPU can transmit various interactive data between the internal structures of the computer device, and so on. The communication interface 802 can optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which can be used to send and receive data under the control of the processor 801; the communication interface 802 can also be used for the transmission and interaction of data within the computer device. The memory 803 (Memory) is a memory device in the computer device, which is used to store programs and data. It can be understood that the memory 803 here can include both the built-in memory of the computer device and the extended memory supported by the computer device. The memory 803 provides a storage space, which stores the operating system of the computer device, including but not limited to: Android system, iOS system, Windows Phone system, etc., which is not limited in this application.

[0180] The embodiment of the present application also provides a computer-readable storage medium (Memory), which is a memory device in a computer device for storing programs and data. It is understandable that the computer-readable storage medium here can include both built-in storage media in the computer device and, of course, extended storage media supported by the computer device. The computer-readable storage medium provides a storage space that stores the processing system of the computer device. In addition, a computer program suitable for being loaded and executed by the processor 801 is also stored in the storage space. It should be noted that the computer-readable storage medium here can be a high-speed RAM memory, or a non-volatile memory, such as at least one disk storage; optionally, it can also be at least one computer-readable storage medium located away from the aforementioned processor.

[0181] In one embodiment, the processor 801 performs the following operations by running the computer program in the memory 803:

[0182] Obtaining a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video, wherein the rhythm condition vector is used to characterize the action amplitude and action intensity of the target object in the action video; and the reference chord is determined based on the style of the action video;

[0183] Based on the rhythm condition vector, the noise data is denoised in the beat dimension to obtain the predicted beat of the action video;

[0184] Based on the rhythm condition vector and the reference chord, the noise data is subjected to denoising in the chord dimension to obtain the predicted chord of the action video;

[0185] The predicted beats and predicted chords are fused to obtain the soundtrack for the action video.

[0186] As an optional embodiment, the specific embodiment of the processor 801 obtaining the reference chord of the action video is:

[0187] Extracting visual features of the action video to obtain visual features of the action video; the visual features contain style information of the action video;

[0188] Selecting a target candidate video from the candidate videos based on the visual features, wherein the feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold;

[0189] The chord of the target candidate video is determined as the reference chord of the action video.

[0190] As an optional embodiment, a specific embodiment of the processor 801 obtaining the rhythm condition vector corresponding to the action video to be matched with music is:

[0191] Perform posture sampling on the target object in the action video to obtain the posture information of the target object; the posture information is used to indicate the action changes of the target object in a continuous time;

[0192] The posture information of the target object is feature encoded to obtain a rhythm condition vector.

[0193] As an optional embodiment, the processor 801 performs a denoising process on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the action video, including:

[0194] Calling the first deep learning network to predict the noise component corresponding to the h-th iteration denoising result of the noise data based on the rhythm condition vector, where h is a positive integer;

[0195] The noise component is removed from the h-1th iterative denoising result to obtain the hth iterative denoising result of the noise data.

[0196] As an optional embodiment, the first deep learning network includes M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process in which the processor 801 calls the first deep learning network to predict the noise component corresponding to the h-th iteration denoising result of the noise data based on the rhythm condition vector includes:

[0197] Perform correlation analysis on the input data through the j-th self-attention layer in the first deep learning network to obtain a first intermediate result, where j is a positive integer less than M;

[0198] Performing feature interaction on the first intermediate result and the rhythm condition vector through a cross attention layer corresponding to the j-th self-attention layer in the first deep learning network to obtain a first feature interaction result;

[0199] The first feature interaction result is used as the input data of the j+1th self-attention layer in the first deep learning network.

[0200] As an optional embodiment, the processor 801 performs denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video, including:

[0201] Calling the second deep learning network to predict the noise component corresponding to the k-th iteration denoising result of the noise data based on the rhythm condition vector and the reference chord, where k is a positive integer;

[0202] The noise component is removed from the k-1th iterative denoising result to obtain the kth iterative denoising result of the noise data.

[0203] As an optional embodiment, the predicted beat of the action video is obtained by calling the first deep learning network to perform denoising on the beat dimension of the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network both include M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the processor 801 calls the second deep learning network based on the rhythm condition vector and the reference chord to predict the noise component corresponding to the k-th iteration denoising result of the noise data. The process includes:

[0204] Call the i-th self-attention layer in the second deep learning network to perform correlation analysis on the input data based on the output data of the i-th self-attention layer in the first deep network learning network to obtain a second intermediate result, where i is a positive integer less than M;

[0205] Perform feature interaction on the second intermediate result, the rhythm condition vector, and the reference chord through a cross attention layer corresponding to the i-th self-attention layer in the second deep learning network to obtain a second feature interaction result;

[0206] The second feature interaction result is used as the input data of the i+1th self-attention layer in the second deep learning network.

[0207] In another embodiment, the processor 801 performs the following operations by running the computer program in the memory 803:

[0208] Obtaining sample audio, rhythm condition vector and reference chord corresponding to the sample video; the rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video;

[0209] Calling the first model to be trained to perform denoising processing on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the sample video;

[0210] Calling the second to-be-trained model to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord to obtain the predicted chord of the sample video;

[0211] Based on the predicted beat, predicted chord and sample audio, the first model to be trained and the second model to be trained are trained to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate music for the video to be matched with music.

[0212] As an optional embodiment, the sample audio includes a sample beat; the processor 801 trains the first model to be trained and the second model to be trained based on the predicted beat, the predicted chord and the sample audio, and obtains a specific embodiment of the beat prediction model and the chord prediction model as follows:

[0213] Based on the difference between the predicted beat and the sample beat, the first model to be trained is trained to obtain a beat prediction model;

[0214] Based on the difference between the predicted chord and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

[0215] As an optional embodiment, the processor 801 trains the second model to be trained based on the difference between the predicted chord and the sample audio, and a specific embodiment of obtaining the chord prediction model is:

[0216] The predicted beat and the predicted chord are merged to obtain the predicted audio of the sample video;

[0217] Based on the difference between the predicted audio and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

[0218] Based on the same inventive concept, the principles and beneficial effects of solving the problems provided by the computer device in the embodiment of the present application are similar to the principles and beneficial effects of solving the problems provided by the audio processing method in the method embodiment of the present application. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.

[0219] An embodiment of the present application further provides a computer-readable storage medium, in which a computer program is stored. The computer program is suitable for being loaded by a processor and executing the audio processing method of the above method embodiment.

[0220] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the computer device reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the computer device performs the above-mentioned audio processing method.

[0221] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0222] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0223] In the embodiments of the present application, the "module" or "unit" involved refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, at least one processor (or memory) can be used to implement at least one module or unit. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0224] A person of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing related hardware through a program, and the program can be stored in a computer-readable storage medium, and the readable storage medium can include: a flash drive, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0225] What is disclosed above is only a preferred embodiment of the present application, and it certainly cannot be used to limit the scope of rights of the present application. Ordinary technicians in this field can understand that all or part of the processes of implementing the above embodiment and equivalent changes made according to the claims of the present application are still within the scope covered by the application.

Claims

1. An audio processing method, characterized in that: The method comprises: Acquire a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video, wherein the rhythm condition vector is used to characterize the action amplitude and action intensity of the target object in the action video; and the reference chord is determined based on the style of the action video; Based on the rhythm condition vector, performing a denoising process on the noise data in terms of the beat dimension to obtain a predicted beat of the action video; Based on the rhythm condition vector and the reference chord, performing denoising processing on the noise data in a chord dimension to obtain a predicted chord of the action video; The predicted beat and the predicted chord are merged to obtain the soundtrack of the action video.

2. The method according to claim 1, characterized in that Obtaining a reference chord of the action video, including: Extracting visual features from the action video to obtain visual features of the action video; the visual features include style information of the action video; Filtering a target candidate video from the candidate videos based on the visual features, wherein a feature similarity between the visual features of the target candidate video and the visual features of the action video is greater than a similarity threshold; The chord of the target candidate video is determined as the reference chord of the action video.

3. The method according to claim 1, characterized in that The step of obtaining a rhythm condition vector corresponding to the action video to be matched with music includes: Performing posture sampling on the target object in the action video to obtain posture information of the target object; the posture information is used to indicate the action change of the target object within a continuous time; Feature encoding is performed on the posture information of the target object to obtain a rhythm condition vector.

4. The method according to claim 1, characterized in that The process of performing a denoising process on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the action video includes: Calling the first deep learning network to predict the noise component corresponding to the h-th iteration noise reduction result of the noise data based on the rhythm condition vector, where h is a positive integer; The noise component is removed from the h-1th iterative denoising result to obtain the hth iterative denoising result of the noise data.

5. The method according to claim 4, characterized in that The first deep learning network includes M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process of calling the first deep learning network to predict the noise component corresponding to the h-th iterative denoising result of the noise data based on the rhythm condition vector includes: Performing correlation analysis on the input data through the j-th self-attention layer in the first deep learning network to obtain a first intermediate result, where j is a positive integer less than M; Performing feature interaction on the first intermediate result and the rhythm condition vector through a cross attention layer corresponding to the j-th self-attention layer in the first deep learning network to obtain a first feature interaction result; The first feature interaction result is used as input data of the j+1th self-attention layer in the first deep learning network.

6. The method according to claim 1, characterized in that The process of performing chord dimension noise reduction processing on the noise data based on the rhythm condition vector and the reference chord to obtain the predicted chord of the action video includes: Calling a second deep learning network to predict the noise component corresponding to the k-th iteration noise reduction result of the noise data based on the rhythm condition vector and the reference chord, where k is a positive integer; The noise component is removed from the k-1th iterative denoising result to obtain the kth iterative denoising result of the noise data.

7. The method according to claim 6, characterized in that The predicted beat of the action video is obtained by calling the first deep learning network to perform denoising processing on the beat dimension of the noise data based on the rhythm condition vector; the first deep learning network and the second deep learning network both include M self-attention layers, each self-attention layer corresponds to a cross-attention layer, and M is a positive integer; the process of calling the second deep learning network to predict the noise component corresponding to the kth iterative denoising result of the noise data based on the rhythm condition vector and the reference chord includes: Calling the i-th self-attention layer in the second deep learning network to perform correlation analysis on the input data based on the output data of the i-th self-attention layer in the first deep network learning network to obtain a second intermediate result, where i is a positive integer less than M; Performing feature interaction on the second intermediate result, the rhythm condition vector, and the reference chord through a cross attention layer corresponding to the i-th self-attention layer in the second deep learning network to obtain a second feature interaction result; The second feature interaction result is used as input data of the i+1th self-attention layer in the second deep learning network.

8. A model training method, characterized in that: The method comprises: Acquire sample audio, a rhythm condition vector and a reference chord corresponding to the sample video; the rhythm condition vector is used to characterize the movement amplitude and movement intensity of the target object in the sample video, and the reference chord is determined based on the style of the sample video; Calling the first model to be trained to perform denoising processing on the noise data in the beat dimension based on the rhythm condition vector to obtain the predicted beat of the sample video; Calling the second to-be-trained model to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord, so as to obtain the predicted chord of the sample video; Based on the predicted beat, the predicted chord and the sample audio, the first model to be trained and the second model to be trained are trained to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate music for the video to be accompanied by music.

9. The method according to claim 8, characterized in that The sample audio includes a sample beat; the first model to be trained and the second model to be trained are trained based on the beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model, including: Based on the difference between the predicted beat and the sample beat, the first to-be-trained model is trained to obtain a beat prediction model; Based on the difference between the predicted chord and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

10. The method according to claim 9, characterized in that The step of training the second model to be trained based on the difference between the predicted chord and the sample audio to obtain a chord prediction model includes: fusing the predicted beat and the predicted chord to obtain predicted audio of the sample video; Based on the difference between the predicted audio and the sample audio, the second model to be trained is trained to obtain a chord prediction model.

11. An audio processing device, characterized in that: The audio processing device comprises: An acquisition unit, used to acquire a rhythm condition vector corresponding to the action video to be matched with music and a reference chord of the action video, wherein the rhythm condition vector is used to characterize the action amplitude and action intensity of the target object in the action video; and the reference chord is determined based on the style of the action video; A processing unit, configured to perform a denoising process on the noise data in a beat dimension based on the rhythm condition vector, so as to obtain a predicted beat of the action video; and for performing noise reduction processing on the noise data in a chord dimension based on the rhythm condition vector and the reference chord to obtain a predicted chord of the action video; And used to merge the predicted beat and the predicted chord to obtain the soundtrack of the action video.

12. A model training device, characterized in that: The model training device comprises: An acquisition unit, used to acquire sample audio, a rhythm condition vector and a reference chord corresponding to a sample video; the rhythm condition vector is used to characterize the movement amplitude and movement intensity of a target object in the sample video, and the reference chord is determined based on the style of the sample video; A processing unit, configured to call the first model to be trained to perform denoising processing on the noise data in a beat dimension based on the rhythm condition vector, so as to obtain a predicted beat of the sample video; and for calling a second model to be trained to perform denoising processing on the noise data in the chord dimension based on the rhythm condition vector and the reference chord, so as to obtain the predicted chord of the sample video; And for training the first model to be trained and the second model to be trained based on the predicted beat, the predicted chord and the sample audio to obtain a beat prediction model and a chord prediction model; the beat prediction model and the chord prediction model are used to generate music for the video to be accompanied by music.

13. A computer device, characterized in that: include: a memory, wherein a computer program is stored in the memory; A processor, used to load the computer program to implement the audio processing method as described in any one of claims 1-7; or used to load the computer program to implement the model training method as described in any one of claims 8-10.

14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which is suitable for being loaded by a processor and executing the audio processing method as described in any one of claims 1-7; or is suitable for being loaded by a processor and executing the model training method as described in any one of claims 8-10.

15. A computer program product, characterized in that The computer program product comprises a computer program, which is suitable for being loaded by a processor and executing the audio processing method as described in any one of claims 1 to 7; or is suitable for being loaded by a processor and executing the model training method as described in any one of claims 8 to 10.

Images