Audio processing method and device, nonvolatile storage medium and electronic equipment

Abstract

The invention discloses an audio processing method and device, a nonvolatile storage medium and electronic equipment. The method comprises the following steps: determining a first sound feature of a target sounding object and first semantic information of an audio to be processed, the audio to be processed being an audio emitted by the target sounding object; a to-be-processed audio feature map of the to-be-processed audio is determined, and the to-be-processed audio feature map is used for reflecting audio features of the to-be-processed audio; splicing the first sound feature, the first semantic information and the audio feature map to obtain a first splicing result; and processing the first splicing result by using an audio processing model to obtain a target audio corresponding to the to-be-processed audio, the audio processing model comprising a diffusion model. The technical problems of high cost and low efficiency due to the fact that the audio is beautified in a manual adjustment mode in the related technology are solved.

Description

Technical Field

[0001] This application relates to the field of audio processing, and more specifically, to an audio processing method, apparatus, non-volatile storage medium, and electronic device. Background Technology

[0002] In related technologies, audio enhancement typically requires manual adjustments by professional engineers. The problem with this approach is the high labor costs and low enhancement efficiency.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides an audio processing method, apparatus, non-volatile storage medium, and electronic device to at least solve the technical problems of high cost and low efficiency caused by the use of manual adjustment methods for audio enhancement in related technologies.

[0005] According to one aspect of the embodiments of this application, an audio processing method is provided, comprising: determining a first sound feature of a target sound-emitting object and first semantic information of an audio to be processed, wherein the audio to be processed is audio emitted by the target sound-emitting object; determining an audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed; concatenating the first sound feature, the first semantic information and the audio feature map to obtain a first concatenation result; and processing the first concatenation result using an audio processing model to obtain a target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

[0006] Optionally, concatenating sound features, semantic information, and the audio feature map to be processed to obtain a first concatenation result includes: determining the sound features, semantic information, and the audio feature map to be processed corresponding to each time step; determining the time step features used to represent the time step; and concatenating the time step features, the first sound features, the first semantic information, and the audio feature map to be processed to obtain the first concatenation result. The first layer of the first concatenation result includes the time step features and the first sound features; the second layer of the first concatenation result includes the audio feature map to be processed and the first mask; the third layer of the first concatenation result includes a noise feature map and the first mask; and the fourth layer of the first concatenation result includes the first semantic information. The first mask is an all-zero matrix, and the noise feature map is a feature map of random noise determined based on the time step.

[0007] Optionally, before concatenating the time step features, the first sound features, the first semantic information, and the audio feature map to be processed to obtain the first concatenated audio feature map, the method further includes: performing length normalization processing on the first sound features, the first semantic information, and the audio feature map to be processed so that the time periods corresponding to the processed first sound features, the first semantic information, and the audio feature map to be processed are the same.

[0008] Optionally, the audio processing model is trained in the following manner: A training dataset is acquired, comprising multiple training data groups, each group including first and second audio data corresponding to the same vocal object, with the second audio data receiving a better evaluation result than the first audio data; for each training data group, a second sound feature corresponding to the first and second audio data, a second semantic information corresponding to the first audio data, and a third semantic information corresponding to the second audio data are determined; a second splicing result is determined based on the second sound feature, second semantic information, third semantic information, first audio, and second audio, and the audio processing model is trained based on the second splicing result, wherein the audio processing model adjusts the audio feature map of the first audio based on the second audio during training.

[0009] Optionally, determining the second concatenation result of the audio processing model based on the second sound feature, second semantic information, third semantic information, first audio, and second audio includes: determining the first audio feature map corresponding to the first audio and the second audio feature map corresponding to the second audio; adding random noise corresponding to the time step to the first audio feature map to obtain the third audio feature map, and adding random noise corresponding to the time step to the second audio feature map to obtain the fourth audio feature map; concatenating the second sound feature, the first audio feature map, the second audio feature map, the third audio feature map, the fourth audio feature map, the second semantic information, and the third semantic information to obtain the second concatenation result. The first layer of the second concatenation result includes the second sound feature and a time step feature representing the time step; the second layer of the second concatenation result includes the first audio feature map and a second mask of the second audio feature map; the third layer of the second concatenation result includes the third audio feature map, the second mask, and the fourth audio feature map; and the fourth layer of the second concatenation result includes the first semantic information and the second semantic information.

[0010] Optionally, the method further includes: determining a first mask based on a first audio feature map and a second audio feature map, wherein the first mask is used to indicate a target audio feature in the second audio feature map, the target audio feature being an audio feature not included in the first audio feature map, and the second mask is an all-zero matrix.

[0011] Optionally, obtaining the training dataset includes: obtaining first audio data; determining second audio data corresponding to the first audio data, wherein the second audio is obtained by optimizing the first audio using a timbre optimization tool.

[0012] According to another aspect of the embodiments of this application, an audio processing apparatus is also provided, comprising: a first processing module, configured to determine a first sound feature of a target sound-emitting object and first semantic information of an audio to be processed, wherein the audio to be processed is audio emitted by the target sound-emitting object; a second processing module, configured to determine an audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed; a third processing module, configured to splice the first sound feature, the first semantic information and the audio feature map to be processed to obtain a first splicing result; and a fourth processing module, configured to process the first splicing result using an audio processing model to obtain a target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

[0013] According to another aspect of the embodiments of this application, a non-volatile storage medium is also provided, wherein a program is stored in the non-volatile storage medium, and the program controls the device where the non-volatile storage medium is located to execute an audio processing method when it runs.

[0014] According to another aspect of the embodiments of this application, an electronic device is also provided, including a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes an audio processing method when it runs.

[0015] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that implements an audio processing method when executed by a processor.

[0016] In this embodiment, a first sound feature of the target speaker and first semantic information of the audio to be processed are determined, wherein the audio to be processed is the audio emitted by the target speaker. A feature map of the audio to be processed is determined, wherein the feature map of the audio to be processed is used to represent the audio features of the audio to be processed. The first sound feature, the first semantic information and the feature map of the audio are spliced ​​together to obtain a first splicing result. The first splicing result is processed by an audio processing model to obtain the target audio corresponding to the audio to be processed. The audio processing model includes a diffusion model. By using the diffusion model to process the feature map of the audio to be processed based on the first semantic information and the first sound feature of the target speaker, the purpose of automatically processing the audio to be processed is achieved, thereby improving the audio processing efficiency and reducing the processing cost. This solves the technical problem of high cost and low efficiency caused by the use of manual adjustment for audio enhancement in related technologies. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a schematic diagram of the structure of a computer terminal (mobile device) according to an embodiment of this application;

[0019] Figure 2 This is a flowchart illustrating an audio processing method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of a splicing result provided according to an embodiment of this application;

[0021] Figure 4 This is a flowchart illustrating an audio processing procedure according to an embodiment of this application;

[0022] Figure 5 This is a flowchart illustrating an audio processing model training process according to an embodiment of this application;

[0023] Figure 6 This is a schematic diagram of the structure of an audio processing device according to an embodiment of this application. Detailed Implementation

[0024] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0025] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0026] To better understand the embodiments of this application, the technical terms involved in the embodiments of this application are explained below:

[0027] Audio refers to sound recorded by a recording device, including speech and non-speech sounds.

[0028] Speech refers to audio recordings of people speaking.

[0029] Voice conversion: Speech conversion is typically used to switch between speaker A and speaker B. It involves converting the original voice to a target voice while masking the original voice. Evaluation metrics include using voice similarity to determine if it's the target voice. This can be represented as follows: User voice + Semantic 1 & Original voice + Semantic 2 → User voice pro + Semantic 2.

[0030] Diffusion: A diffusion model. In a diffusion model, random noise is progressively added to an initial clean signal; this process is called the "Forward Process." This process is deterministic, typically using a Gaussian-distributed noise superposition. Subsequently, in the "Reverse Process," the model is trained to progressively remove the noise and recover the original signal. This Reverse Process is nondeterministic; the model must learn how to recover the signal from the noise, which is usually achieved through complex network structures and large amounts of training data.

[0031] A key advantage of diffusion models is their ability to generate high-quality samples while remaining relatively stable during training. They are less prone to getting stuck in local optima and are less susceptible to common generative model problems such as mode collapse or vanishing gradients. Furthermore, they offer flexibility in controlling uncertainties during the generation process, allowing users to adjust model parameters to control the diversity and fidelity of the output.

[0032] Diffusion Transformer: The Diffusion Transformer combines the advantages of the diffusion model and the Transformer architecture, representing a more complex and efficient model developed based on the diffusion model. The Transformer architecture initially shone brightly in the field of natural language processing, significantly improving the processing capabilities of sequential data with its self-attention mechanism. Introducing the Transformer architecture into the diffusion model can enhance the model's ability to handle long sequence dependencies and improve the quality and consistency of generated samples.

[0033] U-DiT Model: The U-DiT model combines U-Net and a diffusion model. In U-DiT, the U-Net architecture is used to efficiently extract and process features at different scales, while the diffusion model mechanism is used to guide the noise addition and removal process to achieve high-quality generation results.

[0034] In audio enhancement schemes, U-DiT can be used to progressively recover and optimize signals from noise spectra. Specifically, the U-Net structure can be used for feature extraction and conditional generation to understand the local and global characteristics of audio signals at different scales; while the diffusion model gradually removes noise, restoring the audio signal from a state of random noise to a clear, optimized state, and finally converting the processed mel spectrum into audio through a vocoder.

[0035] U-DiT, which combines U-Net and a diffusion model, can not only handle complex audio features but also maintain high-quality sound details during the generation process.

[0036] Currently, audio processing includes two main methods: voice enhancement and speech conversion. Audio enhancement refers to processing the target speaker's own audio, adjusting features such as timbre to obtain an enhanced version. Speech conversion, on the other hand, directly changes the timbre and other features of an audio file to resemble the timbre of another speaker. Evaluation metrics include using timbre similarity to determine if the timbre is the target timbre.

[0037] For voice enhancement, related techniques are primarily handled manually by professional audio engineers with sufficient domain knowledge. They use commercial voice correction tools such as Melodyne3 and Autotune4. Alternatively, they compare the fundamental frequency f0 and the first formant frequency f1 to utilize latent spatial representations to improve the pitch and timbre of amateur singing. Both methods suffer from low optimization efficiency, and manual processing also incurs high costs.

[0038] To address the aforementioned issues, this application provides relevant solutions, which are detailed below.

[0039] According to an embodiment of this application, an embodiment of an audio processing method is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0040] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1A hardware block diagram of a computer terminal (or mobile device) for implementing an audio processing method is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0041] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuits are generally referred to herein as "data processing circuits". These data processing circuits may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuits may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). As involved in the embodiments of this application, the data processing circuits serve as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0042] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the audio processing method in this embodiment. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the aforementioned audio processing method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0043] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0044] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0045] Under the above operating environment, this application provides an audio processing method, such as... Figure 2 As shown, the method includes the following steps:

[0046] Step S202: Determine the first sound feature of the target sound-emitting object and the first semantic information of the audio to be processed, wherein the audio to be processed is the audio emitted by the target sound-emitting object.

[0047] In some embodiments of this application, the first vocal feature of the target vocal object refers to the speaker embedding of the target vocal object. When extracting semantic information from audio, a semantic extractor can be used. Speaker embedding refers to a set of numerical vectors representing the vocal characteristics of a specific speaker, extracted from the speech signal during Chinese speech processing. This vector contains personalized acoustic features such as the speaker's timbre, intonation, and pronunciation habits, and can distinguish the voices of different speakers, even when they are saying the same content. In other words, speaker embedding reflects the vocal characteristics that distinguish the target vocal object from other objects.

[0048] Step S204: Determine the audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed.

[0049] In some embodiments of this application, the audio feature map of the audio to be processed can have various representations, such as an audio mel spectrogram. The audio feature maps of other audios mentioned in the embodiments of this application can also be audio mel spectrograms.

[0050] Step S206: Concatenate the first sound feature, the first semantic information, and the audio feature map to obtain the first concatenation result;

[0051] In the technical solution provided in step S206, the step of splicing sound features, semantic information, and audio feature maps to be processed to obtain a first splicing result includes: determining the sound features, semantic information, and audio feature maps to be processed corresponding to each time step; determining the time step features used to represent the time step; splicing the time step features, the first sound features, the first semantic information, and the audio feature maps to be processed to obtain a first splicing result, wherein the first layer of the first splicing result includes the time step features and the first sound features, the second layer of the first splicing result includes the audio feature maps to be processed and the first mask, the third layer of the first splicing result includes the noise feature maps and the first mask, and the fourth layer of the first splicing result includes the first semantic information, the first mask is an all-zero matrix, and the noise feature maps are feature maps of random noise determined according to the time steps.

[0052] The first splicing result is as follows: Figure 3 As shown. The "S" at the top represents the first sound characteristic. Additionally, from... Figure 3 As can be seen, the left and right sides of the fourth layer contain the same first semantic information. The reason why the fourth layer in the inference stage needs to be concatenated with the same semantic information is to maintain consistency with the format of the concatenation result in the training stage. The fourth layer of the concatenation result in the training stage contains the semantic information of the audio to be optimized, as well as the semantic information of the reference audio.

[0053] As an optional implementation, before splicing time step features, first sound features, first semantic information and audio feature map to be processed to obtain the first spliced ​​audio feature map, the method further includes: performing length normalization processing on the first sound features, first semantic information and audio feature map to be processed so that the time periods corresponding to the processed first sound features, first semantic information and audio feature map to be processed are the same.

[0054] An arbitrary length procedure module can be used to process features that need to be spliced, excluding time step features, so that the length of these features on the time axis can be kept consistent.

[0055] Step S208: The first splicing result is processed using an audio processing model to obtain the target audio corresponding to the audio to be processed. The audio processing model includes a diffusion model.

[0056] As an optional implementation, the above diffusion model can be the U-DiT model or other types of diffusion models.

[0057] In some embodiments of this application, the processing flow for determining the target audio is as follows: Figure 4 As shown. Among them. Figure 4The upper right section shows the speaker embedding (i.e., the first sound feature of the target speaker) determined using CAM++. The lower right section shows the process of obtaining the semantic features of the speaker using Semantic Feature Exractor. The left side, from bottom to top, shows the process of processing the extracted features to obtain the final target audio. The lower left section represents the concatenation process, where 't' represents the time-step feature and 's' represents the speaker embedding, which is the first layer of the concatenation result. Figure 4 As can be seen, in order to ensure the consistency of the input data structure between the inference stage and the training stage, semantic features are spliced ​​with their own copy results.

[0058] The splicing results can then be further processed using diffusion models such as U-DiT to obtain the optimized audio mel spectrum. Finally, a vocoder can be used to convert the optimized audio mel spectrum into the target audio converter speech.

[0059] In the technical solution provided in step S208, the audio processing model is trained in the following manner: A training dataset is obtained, comprising multiple training data groups, each including first and second audio data corresponding to the same vocal object, with the evaluation result of the second audio data being superior to that of the first audio data; for each training data group, the second sound features of the vocal object corresponding to the first and second audio data, the second semantic information corresponding to the first audio data, and the third semantic information corresponding to the second audio data are determined; based on the second sound features, the second semantic information, the third semantic information, the first audio, and the second audio, a second splicing result is determined, and the audio processing model is trained based on the second splicing result, wherein the audio processing model adjusts the audio feature map of the first audio based on the second audio during the training process.

[0060] In some embodiments of this application, the second audio data exhibits superior evaluation results in terms of aesthetic quality, timbre fidelity, and pitch accuracy. The evaluation results for different audio files can be determined by combining an expert scoring system with automated audio quality assessment. The evaluation results can be determined using the following methods:

[0061] Audio recordings of the same speaker can be objectively and subjectively scored using tools such as Perceptual Evaluation of Speech Quality (PESQ) and Mean Opinion Score (MOS). PESQ measures sound quality, while MOS assesses the naturalness and aesthetics of the audio by collecting listener feedback. The goal is to ensure that the final second audio recording scores higher than the first audio recording in terms of timbre similarity, pitch accuracy, and overall sound quality.

[0062] PESQ is a widely used objective measurement method for evaluating the quality of compressed or processed speech signals relative to the original signal, focusing on simulating the human auditory system's perception of speech quality. MOS is a subjective evaluation method typically used to measure signal quality or user experience satisfaction, including in areas such as voice communication, audio transmission, and video conferencing. MOS scores are calculated by presenting the signal to be evaluated to a group of testers, asking them to rate it according to certain criteria (such as clarity, naturalness, and satisfaction), and then averaging these ratings. MOS scores generally range from 1 to 5, with higher numbers indicating better signal quality or a higher user experience.

[0063] Additionally, deep learning models can be designed and applied, pre-trained to distinguish timbre and pitch. The audio data to be evaluated can then be input into the model, which outputs scores for timbre difference and pitch accuracy, quantifying the evaluation results for different audio samples. Simultaneously, the model also assesses pronunciation clarity and semantic information preservation, further validating the accuracy of the evaluation results.

[0064] As an alternative implementation, a self-supervised learning method can also be used to analyze the pronunciation accuracy and semantic coherence of each audio file based on a pre-trained speech recognition model. By comparing the recognition rate and consistency score, it can be ensured that the final recognized second audio data is superior in both pronunciation clarity and semantic information retention.

[0065] It should be noted that different groups of audio can correspond to different speakers.

[0066] In some embodiments of this application, the step of determining the second splicing result of the audio processing model based on the second sound feature, the second semantic information, the third semantic information, the first audio, and the second audio includes: determining the first audio feature map corresponding to the first audio and the second audio feature map corresponding to the second audio; adding random noise corresponding to the time step to the first audio feature map to obtain the third audio feature map, and adding random noise corresponding to the time step to the second audio feature map to obtain the fourth audio feature map; splicing the second sound feature, the first audio feature map, the second audio feature map, the third audio feature map, the fourth audio feature map, the second semantic information, and the third semantic information to obtain the second splicing result, wherein the first layer of the second splicing result includes the second sound feature and the time step feature used to represent the time step, the second layer of the second splicing result includes the first audio feature map and the second audio feature map and the second mask, the third layer of the second splicing result includes the third audio feature map and the second mask and the fourth audio feature map, and the fourth layer of the second splicing result includes the first semantic information and the second semantic information.

[0067] As an optional implementation, the method further includes: determining a first mask based on a first audio feature map and a second audio feature map, wherein the first mask is used to indicate a target audio feature in the second audio feature map, the target audio feature being an audio feature not included in the first audio feature map, and the second mask is an all-zero matrix.

[0068] In some embodiments of this application, both the first mask in the inference phase and the second mask in the training phase are all-zero matrices, and the length of the all-zero matrix can be determined based on the second audio feature map. The first audio is the audio to be optimized, which can also be amateur audio. The second audio is the audio used as a reference, which can also be professional audio. For example, the first audio can be a flawed singing voice (pronunciation problems, etc.), while the second audio can be a higher-quality, more pleasant singing voice from the same vocalist. Here, "higher-quality, more pleasant" can be quantified as timbre indicators or pitch indicators, etc.

[0069] In some embodiments of this application, the optimization process for the audio processing model is as follows: Figure 5 As shown. From Figure 5 As can be seen, since the first and second audio samples are from the same speaker, only the speakerembedding needs to be extracted once. Random noise corresponding to the time step can be generated and added using the flow model. Furthermore, the addition method and location can be determined based on the second mask (noise_mel mask). It's also important to note that all features except the time step features are processed using a length normalization module before concatenation to ensure consistent lengths across the time axis.

[0070] Optionally, when adding random noise, a random time step t can be determined, and noise can be added according to the transformation at step t. The training process (also called the forward process) of the audio processing model provided in this application is a process of gradually increasing noise from step 0 to step T, while the inference process (also called the backward process) is a process of gradually decreasing noise from step T to step 0. Therefore, the noise feature map added in the third layer of the inference stage is the Mel spectrum of the random noise determined according to the transformation at step T. In other words, the model in the inference stage has a fixed inference time step T, gradually inferring the optimized feature map from the noise.

[0071] In some embodiments of this application, random noise can be added during the training phase in the following ways:

[0072] The first step is to extract a sample x0 from the training data;

[0073] The second step is to randomly select a time step t from time steps 1-T.

[0074] The third step involves feeding x0 and t into GausssionDiffusion. GausssionDiffusion samples random noise, adds it to x0 to form xt, and then feeds xt and t into UNet. UNet generates a sinusoidal position code based on t and combines it with xt. The audio processing model predicts the added noise and returns the noise. GausssionDiffusion calculates the loss between this noise and the random noise.

[0075] The fourth step is to calculate the L2 loss between the noise predicted by the audio processing model and the random noise sampled by GausssionDiffusion, calculate the gradient, and update the weights.

[0076] Repeat the above steps until the audio processing model is trained.

[0077] In some embodiments of this application, the step of obtaining the training dataset includes: obtaining first audio data; determining second audio data corresponding to the first audio data, wherein the second audio is obtained by optimizing the first audio using a timbre optimization tool.

[0078] In some embodiments of this application, the audio processing model can be trained using L1 or L2 loss backpropagation gradient.

[0079] By employing a first sound feature of the target speaker and first semantic information of the audio to be processed, wherein the audio to be processed is the audio emitted by the target speaker; determining the audio feature map to be processed of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed; splicing the first sound feature, the first semantic information, and the audio feature map to obtain a first splicing result; and processing the first splicing result using an audio processing model to obtain the target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model, by using a diffusion model to process the audio feature map to be processed of the audio to be processed based on the first semantic information and the first sound feature of the target speaker, the purpose of automatically processing the audio to be processed is achieved, thereby realizing the technical effect of improving audio processing efficiency and reducing processing costs, and thus solving the technical problem of high cost and low efficiency caused by the use of manual adjustment for audio beautification processing in related technologies.

[0080] Furthermore, the method provided in this application embodiment is superior to audio optimization methods in related technologies in terms of similarity index, alignment, and aesthetic quality.

[0081] This application provides an audio processing device. Figure 6 This is a schematic diagram of the device. From Figure 6 As can be seen from the diagram, the device includes: a first processing module 60, used to determine a first sound feature of the target sound-emitting object and a first semantic information of the audio to be processed, wherein the audio to be processed is the audio emitted by the target sound-emitting object; a second processing module 62, used to determine a feature map of the audio to be processed, wherein the feature map of the audio to be processed is used to represent the audio features of the audio to be processed; a third processing module 64, used to splice the first sound feature, the first semantic information and the feature map of the audio to be processed to obtain a first splicing result; and a fourth processing module 66, used to process the first splicing result using an audio processing model to obtain a target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

[0082] In some embodiments of this application, the third processing module 64 concatenates sound features, semantic information, and audio feature maps to be processed to obtain a first concatenation result, including: determining the sound features, semantic information, and audio feature maps to be processed corresponding to each time step; determining the time step features used to represent the time step; and concatenating the time step features, the first sound features, the first semantic information, and the audio feature maps to be processed to obtain the first concatenation result. The first layer of the first concatenation result includes the time step features and the first sound features; the second layer of the first concatenation result includes the audio feature maps to be processed and a first mask; the third layer of the first concatenation result includes a noise feature map and a first mask; and the fourth layer of the first concatenation result includes the first semantic information. The first mask is an all-zero matrix, and the noise feature map is a feature map of random noise determined based on the time step.

[0083] In some embodiments of this application, before splicing time step features, first sound features, first semantic information and audio feature map to be processed to obtain first spliced ​​audio feature map, the third processing module 64 is further configured to: perform length normalization processing on the first sound features, first semantic information and audio feature map to be processed so that the time periods corresponding to the processed first sound features, first semantic information and audio feature map to be processed are the same.

[0084] In some embodiments of this application, the audio processing model is trained in the following manner: A training dataset is acquired, comprising multiple training data groups, each group including first audio data and second audio data corresponding to the same vocal object, wherein the evaluation result of the second audio data is better than that of the first audio data; for each training data group, a second sound feature of the vocal object corresponding to the first and second audio data is determined, along with second semantic information corresponding to the first audio data and third semantic information corresponding to the second audio data; a second splicing result is determined based on the second sound feature, second semantic information, third semantic information, first audio, and second audio, and the audio processing model is trained based on the second splicing result, wherein the audio processing model adjusts the audio feature map of the first audio based on the second audio during training.

[0085] In some embodiments of this application, the step of the audio processing device determining the second splicing result of the audio processing model based on the second sound feature, the second semantic information, the third semantic information, the first audio, and the second audio includes: determining the first audio feature map corresponding to the first audio and the second audio feature map corresponding to the second audio; adding random noise corresponding to the time step to the first audio feature map to obtain the third audio feature map, and adding random noise corresponding to the time step to the second audio feature map to obtain the fourth audio feature map; splicing the second sound feature, the first audio feature map, the second audio feature map, the third audio feature map, the fourth audio feature map, the second semantic information, and the third semantic information to obtain the second splicing result, wherein the first layer of the second splicing result includes the second sound feature and the time step feature used to represent the time step, the second layer of the second splicing result includes the first audio feature map and the second audio feature map and the second mask, the third layer of the second splicing result includes the third audio feature map and the second mask and the fourth audio feature map, and the fourth layer of the second splicing result includes the first semantic information and the second semantic information.

[0086] In some embodiments of this application, the audio processing apparatus is further configured to: determine a first mask based on a first audio feature map and a second audio feature map, wherein the first mask is used to indicate a target audio feature in the second audio feature map, the target audio feature being an audio feature not included in the first audio feature map, and the second mask is an all-zero matrix.

[0087] In some embodiments of this application, the step of the audio processing device acquiring the training dataset includes: acquiring first audio data; determining second audio data corresponding to the first audio data, wherein the second audio is obtained by optimizing the first audio using a timbre optimization tool.

[0088] It should be noted that each module in the above-mentioned audio processing device can be a program module (for example, a set of program instructions to implement a certain function) or a hardware module. For the latter, it can be manifested in the following forms, but is not limited to them: each of the above modules is manifested as a processor, or the functions of each of the above modules are implemented by a processor.

[0089] In addition, the audio processing device can run in an electronic device with audio acquisition capabilities, or in an electronic device that is connected to the audio acquisition device, thereby enabling the enhancement of the audio acquired by the audio acquisition device.

[0090] According to an embodiment of this application, a non-volatile storage medium is also provided, which stores a program. During program execution, the program controls the device containing the non-volatile storage medium to perform the following audio processing method: determining a first sound feature of a target sound-emitting object and first semantic information of the audio to be processed, wherein the audio to be processed is the audio emitted by the target sound-emitting object; determining a feature map of the audio to be processed, wherein the feature map is used to represent the audio features of the audio to be processed; concatenating the first sound feature, the first semantic information, and the feature map to obtain a first concatenation result; and processing the first concatenation result using an audio processing model to obtain a target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

[0091] According to an embodiment of this application, an electronic device is also provided, including a memory and a processor. The processor is used to run a program stored in the memory, wherein the program executes the following audio processing method: determining a first sound feature of a target sound-emitting object and first semantic information of an audio to be processed, wherein the audio to be processed is the audio emitted by the target sound-emitting object; determining an audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed; concatenating the first sound feature, the first semantic information, and the audio feature map to obtain a first concatenation result; and processing the first concatenation result using an audio processing model to obtain a target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

[0092] According to another aspect of the embodiments of this application, a computer program product is also provided, including a computer program that, when executed by a processor, implements the following audio processing method: determining a first sound feature of a target sound-emitting object and first semantic information of an audio to be processed, wherein the audio to be processed is the audio emitted by the target sound-emitting object; determining an audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed; concatenating the first sound feature, the first semantic information, and the audio feature map to obtain a first concatenation result; and processing the first concatenation result using an audio processing model to obtain a target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

[0093] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0094] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For instance, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to related technologies, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0098] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An audio processing method, characterized in that, include: The first sound feature of the target sound-emitting object and the first semantic information of the audio to be processed are determined, wherein the audio to be processed is the audio emitted by the target sound-emitting object; Determine the audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed; By concatenating the first sound feature, the first semantic information, and the audio feature map, a first concatenation result is obtained; The first splicing result is processed using an audio processing model to obtain the target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

2. The audio processing method according to claim 1, characterized in that, By concatenating the sound features, the semantic information, and the audio feature map to be processed, the first concatenation result is obtained, including: Determine the sound features, semantic information, and audio feature map to be processed corresponding to each time step; Determine the time step features used to represent the time step; The time step features, the first sound features, the first semantic information, and the audio feature map to be processed are concatenated to obtain the first concatenation result. The first layer of the first concatenation result includes the time step features and the first sound features. The second layer of the first concatenation result includes the audio feature map to be processed and the first mask. The third layer of the first concatenation result includes the noise feature map and the first mask. The fourth layer of the first concatenation result includes the first semantic information. The first mask is an all-zero matrix. The noise feature map is a feature map of random noise determined based on the time step.

3. The audio processing method according to claim 2, characterized in that, Before concatenating the time step features, the first sound features, the first semantic information, and the audio feature map to be processed to obtain the first concatenated audio feature map, the method further includes: The first sound feature, the first semantic information, and the audio feature map to be processed are subjected to length normalization processing so that the time periods corresponding to the first sound feature, the first semantic information, and the audio feature map to be processed are the same after processing.

4. The audio processing method according to claim 1, characterized in that, The audio processing model is trained in the following way: Obtain a training dataset, wherein the training dataset includes multiple training data groups, and each training data group includes first audio data and second audio data corresponding to the same vocal object, wherein the evaluation result of the second audio data is better than that of the first audio data; For each training data set, determine the second sound features of the vocal object corresponding to the first audio data and the second audio data in the training data set, as well as the second semantic information corresponding to the first audio data and the third semantic information corresponding to the second audio data; Based on the second sound feature, the second semantic information, the third semantic information, the first audio, and the second audio, a second splicing result is determined, and the audio processing model is trained based on the second splicing result. During the training process, the audio processing model adjusts the audio feature map of the first audio based on the second audio.

5. The audio processing method according to claim 4, characterized in that, Based on the second sound feature, the second semantic information, the third semantic information, and the first and second audio, the second splicing result of the audio processing model is determined to include: Determine the first audio feature map corresponding to the first audio and the second audio feature map corresponding to the second audio; Add random noise corresponding to the time step to the first audio feature map to obtain the third audio feature map, and add random noise corresponding to the time step to the second audio feature map to obtain the fourth audio feature map; By concatenating the second sound feature, the first audio feature map, the second audio feature map, the third audio feature map, the fourth audio feature map, the second semantic information, and the third semantic information, a second concatenation result is obtained. The first layer of the second concatenation result includes the second sound feature and a time step feature representing a time step. The second layer of the second concatenation result includes the first audio feature map and a second mask. The third layer of the second concatenation result includes the third audio feature map, the second mask, and the fourth audio feature map. The fourth layer of the second concatenation result includes the first semantic information and the second semantic information.

6. The audio processing method according to claim 5, characterized in that, The method further includes: The second mask is determined based on the second audio feature map, wherein the second mask is an all-zero matrix.

7. The audio processing method according to claim 4, characterized in that, Obtaining the training dataset includes: Obtain the first audio data; Determine the second audio data corresponding to the first audio data, wherein the second audio data is obtained by optimizing the first audio data using a timbre optimization tool.

8. An audio processing apparatus, characterized in that, include: The first processing module is used to determine the first sound feature of the target sound-emitting object and the first semantic information of the audio to be processed, wherein the audio to be processed is the audio emitted by the target sound-emitting object; The second processing module is used to determine the audio feature map of the audio to be processed, wherein the audio feature map is used to represent the audio features of the audio to be processed. The third processing module is used to splice the first sound feature, the first semantic information and the audio feature map to be processed to obtain the first splicing result; The fourth processing module is used to process the first splicing result using an audio processing model to obtain the target audio corresponding to the audio to be processed, wherein the audio processing model includes a diffusion model.

9. A non-volatile storage medium, characterized in that, The non-volatile storage medium stores a program, wherein when the program is executed, it controls the device containing the non-volatile storage medium to perform the audio processing method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, include: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when executed, performs the audio processing method according to any one of claims 1 to 7.

11. A computer program product, characterized in that, Includes a computer program that, when executed by a processor, implements the audio processing method according to any one of claims 1 to 7.

Images