Audio generation method and device, electronic equipment and storage medium
Through audio feature reconstruction technology, the problem of low efficiency in line modification in film and television works is solved, automatic audio modification is realized, efficiency and sound consistency is improved, and the production quality of film and television works is improved.
Patent Information
- Application Number
- CN202410989641.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-12
AI Technical Summary
In the post-production of film and television works such as movies and TV series, when modifying lines, actors need to be reconvened for audio recording, resulting in low efficiency, high time and economic costs, and possible conflicts in actor schedules.
By obtaining the target text and original audio, converting it into phoneme sequences and encoding, removing the sub-audio encoding data corresponding to the characters to be replaced, acquiring audio features and performing audio reconstruction, realizing automatic modification of the original audio.
Improve the efficiency of modifying the original audio, avoid the time and economic costs in traditional methods, ensure the consistency between the target audio and the original audio in sound characteristics, achieve seamless connection, and improve the production quality of film and television works and audience experience. .
Smart Images

Figure CN118942466A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of audio processing technology, and in particular to an audio generation method, device, electronic device and storage medium. Background Art
[0002] During the post-production of movies, TV series and other film and television works, it is often necessary to modify some lines due to plot changes or review requirements. The existing solution is to re-record the lines of the actors according to their schedules, thereby modifying the original audio of the film and television works.
[0003] However, when re-recruiting actors to re-record their lines, not only do they need to rearrange the recording equipment and venue, but there may also be conflicts in the actors' schedules, resulting in very low efficiency in modifying the original audio, significantly lengthening the overall production cycle of film and television works, and increasing a lot of time and economic costs. Summary of the invention
[0004] The purpose of the embodiments of the present application is to provide an audio generation method, device, electronic device and storage medium to improve the efficiency of modifying the original audio. The specific technical solution is as follows:
[0005] In a first aspect, an embodiment of the present application provides an audio generation method, the method comprising:
[0006] Obtaining a target text and an original audio, wherein the target text is obtained by replacing characters to be replaced in an original text corresponding to the original audio;
[0007] Converting the target text into a phoneme sequence, and encoding the original audio to obtain first audio encoding data;
[0008] removing the sub-audio coded data corresponding to the character to be replaced from the first audio coded data to obtain second audio coded data;
[0009] Acquire a first audio feature, wherein the first audio feature is an audio feature used when predicting the audio encoding data corresponding to the phoneme sequence according to the second audio feature, and the second audio feature is an audio feature of the second audio encoding data;
[0010] Audio reconstruction is performed based on the first audio feature to obtain target audio corresponding to the target text.
[0011] Optionally, the performing audio reconstruction based on the first audio feature to obtain a target audio corresponding to the target text includes:
[0012] The first audio feature is used as prior information, and according to the prior information and the noise level corresponding to each time step, the noise in the initial noise matrix is gradually removed within multiple time steps, and the initial noise matrix after denoising is determined as the spectrum data corresponding to the first audio feature, wherein the initial noise matrix is a noise matrix that matches the spectrum data in dimension and structure and is completely composed of random noise;
[0013] The spectrum data is converted into target audio.
[0014] Optionally, taking the first audio feature as prior information and gradually removing noise in the initial noise matrix within multiple time steps according to the prior information and a noise level corresponding to each time step includes:
[0015] inputting the first audio feature into a diffusion model;
[0016] An initial noise matrix after denoising output by the diffusion model is obtained, wherein the initial noise matrix after denoising is obtained by the diffusion model taking the first audio feature as prior information and gradually removing noise in the initial noise matrix within multiple time steps according to the prior information and the noise level corresponding to each time step.
[0017] Optionally, obtaining the first audio feature includes:
[0018] Inputting the phoneme sequence and the second audio encoding data into a pre-trained speech model;
[0019] Obtain a first audio feature output by a previous layer of an output layer of the speech model, wherein the first audio feature is an audio feature used by the speech model when predicting audio coding data corresponding to the phoneme sequence based on the second audio feature.
[0020] Optionally, the training method of the large speech model includes:
[0021] Obtaining a sample text and a sample audio corresponding to the sample text;
[0022] Converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio encoding data;
[0023] Randomly remove part of the sub-audio coded data in the third audio coded data to obtain fourth audio coded data;
[0024] Inputting the sample phoneme sequence and the fourth audio coding data into an initial speech big model;
[0025] Acquire fifth audio coding data output by the initial speech large model, wherein the fifth audio coding data is audio coding data corresponding to the sample phoneme sequence predicted by the initial speech large model according to the audio features of the fourth audio coding data;
[0026] Based on the predicted difference between the fifth audio coding data and the third audio coding data, the parameters of the initial speech model are adjusted to obtain a trained speech model.
[0027] Optionally, the first audio encoding data includes each sub-audio encoding data, and each sub-audio encoding data corresponds to one time information;
[0028] The step of removing the sub-audio coded data corresponding to the character to be replaced in the first audio coded data to obtain the second audio coded data includes:
[0029] Determine the time range of the audio segment corresponding to the character to be replaced in the original audio;
[0030] Among the sub-audio coded data included in the first audio coded data, the sub-audio coded data whose corresponding time information is within the time range is removed to obtain the second audio coded data.
[0031] Optionally, determining the time range of the audio segment corresponding to the character to be replaced in the original audio includes:
[0032] Aligning the original text with the original audio to obtain an alignment result, wherein the alignment result includes: a time range of the audio segment corresponding to each character in the original text in the original audio; in the alignment result, determining a time range of the audio segment corresponding to the character to be replaced in the original text in the original audio;
[0033] or,
[0034] Perform speech recognition on the original audio to obtain a speech recognition result, wherein the speech recognition result includes: the time range of each recognized character and the audio segment corresponding to each character in the original audio; determine the time range of the audio segment corresponding to the target character in the original audio as the time range of the audio segment corresponding to the character to be replaced in the original audio, wherein the target character is the character in the speech recognition result that matches the character to be replaced.
[0035] In a second aspect, an embodiment of the present application provides an audio generation device, the device comprising:
[0036] A first acquisition module is used to acquire a target text and an original audio, wherein the target text is obtained by replacing characters to be replaced in an original text corresponding to the original audio;
[0037] A data conversion module, used for converting the target text into a phoneme sequence, and encoding the original audio to obtain first audio encoding data;
[0038] A data deletion module, used for removing the sub-audio coded data corresponding to the character to be replaced in the first audio coded data to obtain second audio coded data;
[0039] A second acquisition module is configured to acquire a first audio feature, wherein the first audio feature is an audio feature used when predicting the audio encoding data corresponding to the phoneme sequence according to the second audio feature, and the second audio feature is an audio feature of the second audio encoding data;
[0040] An audio generation module is used to reconstruct the audio based on the first audio feature to obtain the target audio corresponding to the target text.
[0041] Optionally, the audio generation module includes:
[0042] a spectrum data generating unit, configured to use the first audio feature as prior information, and gradually remove noise in an initial noise matrix within a plurality of time steps according to the prior information and a noise level corresponding to each time step, and determine the initial noise matrix after denoising as spectrum data corresponding to the first audio feature, wherein the initial noise matrix is a noise matrix that matches the spectrum data in dimension and structure and is completely composed of random noise;
[0043] The audio generation unit is used to convert the spectrum data into target audio.
[0044] Optionally, the spectrum data generating unit is specifically used to: input the first audio feature into a diffusion model; obtain an initial noise matrix after denoising output by the diffusion model, wherein the initial noise matrix after denoising is obtained by the diffusion model taking the first audio feature as prior information and gradually removing the noise in the initial noise matrix within multiple time steps based on the prior information and the noise level corresponding to each time step.
[0045] Optionally, the second acquisition module is specifically used to: input the phoneme sequence and the second audio coding data into a pre-trained speech model; obtain the first audio feature output by a previous layer of the output layer of the speech model, wherein the first audio feature is the audio feature used by the speech model when predicting the audio coding data corresponding to the phoneme sequence based on the second audio feature.
[0046] Optionally, the training method of the speech large model includes: obtaining a sample text and a sample audio corresponding to the sample text; converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio coding data; randomly removing part of the sub-audio coding data in the third audio coding data to obtain fourth audio coding data; inputting the sample phoneme sequence and the fourth audio coding data into the initial speech large model; obtaining fifth audio coding data output by the initial speech large model, wherein the fifth audio coding data is the audio coding data corresponding to the sample phoneme sequence predicted by the initial speech large model according to the audio features of the fourth audio coding data; adjusting the parameters of the initial speech large model based on the difference between the predicted fifth audio coding data and the third audio coding data to obtain a trained speech large model.
[0047] Optionally, the first audio encoding data includes various sub-audio encoding data, and one sub-audio encoding data corresponds to one time information; the data deletion module includes: a time range determination unit, used to determine the time range of the audio segment corresponding to the character to be replaced in the original audio; a data deletion unit, used to remove the sub-audio encoding data whose corresponding time information is within the time range from the various sub-audio encoding data included in the first audio encoding data, to obtain the second audio encoding data.
[0048] Optionally, the time range determination unit includes: a first determination subunit, used to align the original text with the original audio to obtain an alignment result, wherein the alignment result includes: the time range of the audio segment corresponding to each character in the original text in the original audio; in the alignment result, determining the time range of the audio segment corresponding to the character to be replaced in the original text in the original audio; a second determination subunit, used to perform speech recognition on the original audio to obtain a speech recognition result, wherein the speech recognition result includes: the time range of each recognized character and the audio segment corresponding to each character in the original audio; determining the time range of the audio segment corresponding to the target character in the original audio as the time range of the audio segment corresponding to the character to be replaced in the original audio, wherein the target character is the character that matches the character to be replaced in the speech recognition result.
[0049] In a third aspect, an embodiment of the present application provides an electronic device, including a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other through the communication bus;
[0050] Memory, used to store computer programs;
[0051] The processor is used to implement any method step described in the first aspect when executing the program stored in the memory.
[0052] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of the first aspects are implemented.
[0053] In the solution provided by the embodiment of the present application, the electronic device can obtain the target text and the original audio, wherein the target text is obtained by replacing the characters to be replaced in the original text corresponding to the original audio; converting the target text into a phoneme sequence, and encoding the original audio to obtain the first audio encoding data; removing the sub-audio encoding data corresponding to the characters to be replaced in the first audio encoding data to obtain the second audio encoding data; obtaining the first audio feature, wherein the first audio feature is the audio feature used when predicting the audio encoding data corresponding to the phoneme sequence based on the second audio feature, and the second audio feature is the audio feature of the second audio encoding data; reconstructing the audio based on the first audio feature to obtain the target audio corresponding to the target text. The above scheme realizes the automatic modification of the original audio, avoids the time and economic cost of re-convening the actors to re-record the audio in the traditional method, and greatly improves the efficiency of modifying the original audio. In addition, the first audio feature is essentially a latent variable feature, which can more accurately capture and retain the sound characteristics of the original audio, thereby ensuring that when the target audio is reconstructed based on the first audio feature, the target audio and the original audio can maintain consistency in sound characteristics. Further, when the target audio is used to replace the original audio in a film or television work, a seamless connection can be achieved in sound, avoiding abruptness and improving the overall production quality of the film or television work and the audience experience. BRIEF DESCRIPTION OF THE DRAWINGS
[0054] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art are briefly introduced below.
[0055] Figure 1 A flowchart of an audio generation method provided in an embodiment of the present application;
[0056] Figure 2 Based on Figure 1 A schematic flow chart of an audio reconstruction method of the illustrated embodiment;
[0057] Figure 3 Based on Figure 2 A schematic diagram of a specific process of an audio reconstruction method in the illustrated embodiment;
[0058] Figure 4 Based on Figure 1 A schematic diagram of a flow chart of an audio feature acquisition method in the illustrated embodiment;
[0059] Figure 5 Based on Figure 4 A schematic diagram of a specific process of obtaining audio features in the illustrated embodiment;
[0060] Figure 6 Based on Figure 1 A schematic diagram of a flow chart of a model training method in the illustrated embodiment;
[0061] Figure 7 Based on Figure 1 A schematic diagram of a process flow of a data removal method of the illustrated embodiment;
[0062] Figure 8 A schematic diagram of the structure of an audio generating device provided in an embodiment of the present application;
[0063] Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0064] The technical solutions in the embodiments of the present application will be described below in conjunction with the drawings in the embodiments of the present application.
[0065] In order to improve the efficiency of modifying original audio, an embodiment of the present application provides an audio generation method, device, electronic device, computer-readable storage medium and computer program product. The following first introduces an audio generation method provided in an embodiment of the present application.
[0066] An audio generation method provided in an embodiment of the present application can be applied to any electronic device that needs to modify the original audio, for example, it can be a terminal, a server, etc., which is not specifically limited here. For the sake of clear description, it is referred to as an electronic device below.
[0067] like Figure 1 As shown, an audio generation method includes:
[0068] S101, obtaining target text and original audio.
[0069] The target text is obtained by replacing the characters to be replaced in the original text corresponding to the original audio.
[0070] S102, converting the target text into a phoneme sequence, and encoding the original audio to obtain first audio encoding data.
[0071] S103, removing the sub-audio coded data corresponding to the character to be replaced in the first audio coded data to obtain second audio coded data.
[0072] S104: Acquire a first audio feature.
[0073] The first audio feature is an audio feature used when predicting the audio coding data corresponding to the phoneme sequence according to the second audio feature, and the second audio feature is an audio feature of the second audio coding data.
[0074] S105: Reconstruct the audio based on the first audio feature to obtain the target audio corresponding to the target text.
[0075] In the solution provided by the embodiment of the present application, the electronic device can obtain the target text and the original audio, wherein the target text is obtained by replacing the characters to be replaced in the original text corresponding to the original audio; converting the target text into a phoneme sequence, and encoding the original audio to obtain the first audio encoding data; removing the sub-audio encoding data corresponding to the characters to be replaced in the first audio encoding data to obtain the second audio encoding data; obtaining the first audio feature, wherein the first audio feature is the audio feature used when predicting the audio encoding data corresponding to the phoneme sequence based on the second audio feature, and the second audio feature is the audio feature of the second audio encoding data; reconstructing the audio based on the first audio feature to obtain the target audio corresponding to the target text. The above scheme realizes the automatic modification of the original audio, avoids the time and economic cost of re-convening the actors to re-record the audio in the traditional method, and greatly improves the efficiency of modifying the original audio. In addition, the first audio feature is essentially a latent variable feature, which can more accurately capture and retain the sound characteristics of the original audio, thereby ensuring that when the target audio is reconstructed based on the first audio feature, the target audio and the original audio can maintain consistency in sound characteristics. Further, when the target audio is used to replace the original audio in a film or television work, a seamless connection can be achieved in sound, avoiding abruptness and improving the overall production quality of the film or television work and the audience experience.
[0076] When the original audio needs to be modified, the user can replace the characters to be replaced in the original text corresponding to the original audio with the target characters according to the actual modification requirements, thereby obtaining the target text. Then the user can input the target text and the original audio into the electronic device together, so that the electronic device obtains the target text and the original audio.
[0077] For example, the original text is "I am going to the amusement park today". According to the actual modification requirements, the user replaces the character "today" in the original text with the target character "this weekend", thereby obtaining the target text "I am going to the amusement park this weekend". Then, the user can input the audio corresponding to the target text "I am going to the amusement park this weekend" and "I am going to the amusement park today" (i.e., the original audio) into the electronic device.
[0078] After obtaining the target text, the electronic device converts the character sequence in the text into a phoneme sequence, because phonemes are the smallest pronunciation units in a language. The phoneme sequence can more accurately control the generation of each pronunciation unit, thereby synthesizing a more natural and accurate speech. Therefore, the electronic device can convert the character sequence composed of each character in the target text into a phoneme sequence through a preset mapping table between characters and phonemes, or a pre-trained model for converting characters into phonemes.
[0079] For example, the electronic device can convert the target text "I am going to the amusement park this weekend" into the phoneme sequence "wo3 zhe4 zhou1 mo4 yao4 qu4 you2 le4chang3" through a preset mapping table between characters and phonemes.
[0080] After the electronic device obtains the original audio, because the original audio usually contains a huge amount of information, taking the original audio with a sampling rate of 44.1kHz as an example, 44,100 data are recorded per second in the original audio. Such a high amount of data is both complex and time-consuming to process directly, so the electronic device can encode the original audio through an encoder to obtain the first audio encoding data corresponding to the original audio to reduce data redundancy and greatly reduce the amount of data required to be processed without significantly affecting the sound quality.
[0081] For example, the electronic device can encode the original audio to obtain the audio token (i.e., the first audio coded data) through models such as Encodec or Descript Audio Codec, thereby compressing the original 44,100 data per second into only tens to hundreds of coded data per second. Among them, the audio token is a discrete and standardized representation obtained after the original audio is encoded.
[0082] After obtaining the first audio coded data, the electronic device may further remove the sub-audio coded data corresponding to the character to be replaced from each sub-audio coded data included in the first audio coded data to obtain the second audio coded data. The specific method of removing the sub-audio coded data corresponding to the character to be replaced will be described in detail in subsequent embodiments and will not be repeated here.
[0083] After obtaining the phoneme sequence and the second audio coding data, since the audio feature can reflect the sound characteristics of the audio, which may include information such as the timbre, intonation, rhythm and volume of the human voice in the audio, as well as information such as potential noise and reverberation in the audio, the electronic device can extract the second audio feature of the second audio coding data, and then predict the audio coding data corresponding to the phoneme sequence according to the sound characteristics such as timbre, intonation, rhythm, volume, potential noise and reverberation reflected by the second audio feature.
[0084] In one embodiment, a model for predicting audio coding data corresponding to a phoneme sequence can be pre-trained, and then after the electronic device obtains the phoneme sequence and the second audio coding data, the phoneme sequence and the second audio coding data can be input into the model, so that the model learns the sound characteristics of the original audio such as timbre, intonation, rhythm, potential noise and reverberation based on the second audio coding data, and predicts the audio coding data corresponding to the phoneme sequence based on the sound characteristics of the original audio.
[0085] When an electronic device predicts the audio coding data corresponding to a phoneme sequence, it does not directly obtain the predicted audio coding data to generate the target audio, but obtains the audio features used when predicting the audio coding data corresponding to the phoneme sequence, and then reconstructs the audio based on the audio features to obtain the target audio corresponding to the target text. This is because for audio features, when encoding the audio features to generate audio coding data, some information will be lost due to the encoding operation, so compared to audio coding data, audio features include richer and more fine-grained sound characteristics. On this basis, reconstructing the audio based on the audio features to obtain the target audio corresponding to the target text can better ensure that the target audio and the original audio can be consistent in sound characteristics.
[0086] Regarding the method of reconstructing audio based on audio features, in one implementation, the electronic device can input the audio features into pre-trained models such as GAN (Generative Adversarial Networks) and VAE (Variational Auto-Encoder), and then obtain the audio output by the model as the target audio corresponding to the target text.
[0087] On the one hand, the solution provided by the embodiment of the present application realizes the automatic modification of the original audio, avoiding the time and economic cost of re-recruiting actors to re-record audio in the traditional method, and greatly improving the efficiency of modifying the original audio. On the other hand, because the first audio feature is essentially a latent variable feature, it can more accurately capture and retain the sound characteristics of the original audio. Therefore, when the audio is reconstructed based on the first audio feature to obtain the target audio, it is ensured that the target audio and the original audio can be consistent in sound characteristics. When the target audio is used to replace the original audio in the film and television work, a seamless connection can be achieved in sound, avoiding abruptness, and improving the overall production quality of the film and television work and the audience experience.
[0088] As an implementation method of the present application, Figure 2 As shown, the above step 105 may include:
[0089] S201, taking the first audio feature as prior information, and gradually removing the noise in the initial noise matrix within multiple time steps according to the prior information and the noise level corresponding to each time step, and determining the initial noise matrix after denoising as the frequency spectrum data corresponding to the first audio feature.
[0090] When performing audio reconstruction, the electronic device can first construct a noise matrix that matches the dimensionality and structure of the spectral data based on the dimensionality and structure of the spectral data, and then gradually add random noise to the noise matrix at each time step, so that the noise matrix is completely composed of random noise, thereby obtaining an initial noise matrix. In addition, when the electronic device adds random noise to the noise matrix at each time step, it can also record the noise level corresponding to the noise matrix at the time step. The dimensionality and structure of the spectral data are pre-constructed according to the desired spectral data.
[0091] Among them, the matching of the initial noise matrix and the spectrum data in terms of dimension and structure mainly includes the following aspects:
[0092] 1. Dimension matching:
[0093] Data dimension: Spectral data is usually stored in some form (such as array, matrix), which contains a series of frequency-related information (such as amplitude, phase, etc.). The noise matrix also needs to be a data structure corresponding to the spectrum data, so that it can be operated point by point or element by element with the spectrum data in subsequent processing.
[0094] Time / frequency dimension: In signal processing, spectrum data is often expanded in the time or frequency domain. The noise matrix also needs to be consistent with the spectrum data in the corresponding time or frequency dimension to ensure that it can correctly correspond to each time point or frequency point when removing noise.
[0095] 2. Structural matching:
[0096] Data format: The spectrum data and the noise matrix need to be in the same or compatible data format. For example, if the spectrum data is represented in complex form (including real and imaginary parts), then the noise matrix should also be represented in the same way to perform point-by-point noise removal operations in the complex plane.
[0097] Data structure: Spectral data may contain information of multiple channels (such as stereo, multi-channel audio) or multiple frequency bands. The noise matrix needs to be able to reflect this data structure, that is, each channel or frequency band should have a corresponding noise component so that the integrity and consistency of the data can be maintained when removing noise.
[0098] After obtaining the initial noise matrix, the electronic device can use the first audio feature as prior information, and determine the denoising strategy corresponding to the current time step according to the prior information and the noise level corresponding to each time step recorded in advance, and then gradually remove the noise in the initial noise matrix within multiple time steps according to the denoising strategy corresponding to each time step. Finally, the initial noise matrix after denoising is determined as the spectrum data corresponding to the first audio feature.
[0099] In one embodiment, Figure 3 As shown, when the electronic device executes the above step S201, it can specifically input the first audio feature into the Diffusion model, and implement the processing in the above step S201 through the diffusion model, that is, the diffusion model uses the first audio feature as prior information, and performs denoising in multiple time steps based on the prior information and the noise level corresponding to each time step t, gradually removes the noise in the initial noise matrix, and obtains the initial noise matrix after denoising. Then the electronic device can obtain the initial noise matrix after denoising output by the diffusion model, and determine the initial noise matrix after denoising as the spectrum data corresponding to the first audio feature.
[0100] S202, converting the spectrum data into target audio.
[0101] After obtaining the spectrum data, the electronic device can determine the amplitude and phase information of the target audio according to the spectrum data represented in complex form, and determine the sampling rate of the target audio according to the sampling rate of the original audio. Then, the electronic device can restore the time domain signal through inverse Fourier transform according to the amplitude and phase information, and determine the time interval between each data in the time domain signal according to the sampling rate, so as to obtain the target audio.
[0102] In one implementation, when the electronic device executes the above step S202, it may specifically input the spectrum data into a voice encoder-decoder (voice encoder-decoder), and implement the processing in the above step S202 through the voice encoder-decoder. The electronic device may then obtain the audio output by the voice encoder-decoder, and determine the audio as the target audio.
[0103] In the solution provided in the embodiment of the present application, the electronic device uses the first audio feature as prior information and gradually removes the noise in the initial noise matrix within multiple time steps based on the prior information and the noise level corresponding to each time step. It can accurately simulate and restore the spectral characteristics of the sound signal under different speech noise or reverberation conditions, so that the reconstructed audio is closer to the auditory effect in the actual scene.
[0104] As an implementation method of the present application, Figure 4 As shown, the above step S104 may include:
[0105] S401, inputting the phoneme sequence and the second audio coding data into a pre-trained speech model.
[0106] A speech model for predicting the audio coding data corresponding to the phoneme sequence according to the audio features of the audio coding data can be pre-trained, and then the pre-trained speech model can be deployed in the electronic device. In this way, when the electronic device needs to obtain the first audio feature, the phoneme sequence and the second audio coding data can be input into the pre-trained speech model, and then the audio feature used by the speech model when predicting the audio coding data corresponding to the phoneme sequence according to the second audio feature is obtained, that is, the first audio feature is obtained.
[0107] Among them, the training method of the large speech model will be described in detail in subsequent embodiments and will not be repeated here.
[0108] S402, obtaining a first audio feature output by a previous layer of the output layer of the speech model.
[0109] Because when the large speech model predicts the audio coding data corresponding to the phoneme sequence based on the second audio feature, it will encode the audio feature at its output layer to obtain the audio coding data, and the encoding operation causes some information to be lost. Therefore, the electronic device can directly obtain the audio feature output by the previous layer of the output layer of the large speech model, which includes richer and finer-grained audio information, and use it as the first audio feature.
[0110] like Figure 5As shown, the electronic device can input the phoneme sequence and the audio coding data into the input layer of the speech model. The speech model can output the audio coding data corresponding to the predicted phoneme sequence through the output layer through the processing of each level inside. In this process, the electronic device can not obtain the audio coding data output by the output layer of the speech model, but obtain the audio coding data of the previous layer (i.e., the output layer of the speech model) of the output layer of the speech model. Figure 5 The audio features output by the "XX layer" in the audio processing module are used to reconstruct the audio based on the audio features to obtain the target audio.
[0111] In the solution provided in the embodiment of the present application, it is considered that if the target audio is generated directly based on the audio coding data predicted by the large language model, it may be limited by the ability of the large language model to directly process audio data and its ability to capture audio details (such as noise and reverberation), resulting in the generated target audio not being able to be well close to the auditory effect in the actual scene. Therefore, the electronic device extracts the audio features output by the previous layer of the output layer of the large language model. The audio features output by this layer include richer and more fine-grained audio information than the audio coding data output by the output layer, and the target audio generated according to the audio features can be well close to the auditory effect in the actual scene.
[0112] As an implementation method of the present application, Figure 6 As shown, the training method of the above-mentioned large speech model may include:
[0113] S601, obtaining sample text and sample audio corresponding to the sample text.
[0114] When training a large speech model, you can first obtain a text and the audio corresponding to the text, and use them as sample text and sample audio corresponding to the sample text. For example, obtain the text "I'm going to the amusement park today" and the audio corresponding to "I'm going to the amusement park today" as sample text and sample audio corresponding to the sample text.
[0115] S602, converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio encoding data.
[0116] After obtaining the sample text and sample audio, the sample text can be converted into a sample phoneme sequence, and the sample audio can be encoded to obtain third audio encoding data, wherein the method of "converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio encoding data" is similar to the above-mentioned method of "converting the target text into a phoneme sequence, and encoding the original audio to obtain first audio encoding data", and will not be repeated here.
[0117] S603: Randomly remove part of the sub-audio coded data in the third audio coded data to obtain fourth audio coded data.
[0118] After obtaining the third audio coded data, the sub-audio coded data of a preset time length can be randomly removed, or a preset number of sub-audio coded data can be randomly removed, or the sub-audio coded data corresponding to some characters in the sample text can be randomly removed, thereby achieving random removal of some sub-audio coded data in the third audio coded data to obtain the fourth audio coded data.
[0119] S604, input the sample phoneme sequence and the fourth audio coding data into the initial speech large model.
[0120] S605, obtaining the fifth audio coding data output by the initial speech model.
[0121] After obtaining the sample phoneme sequence and the fourth audio coding data, they can be input into the initial speech large model so that the initial speech large model can predict part of the sub-audio coding data that is randomly removed from the third audio coding data based on the audio features of the fourth audio coding data, and adaptively adjust the remaining sub-audio coding data in the third audio coding data, so as to obtain the fifth audio coding data corresponding to the sample phoneme sequence.
[0122] S606: Based on the difference between the predicted fifth audio coding data and the third audio coding data, adjust the parameters of the initial speech model to obtain a trained speech model.
[0123] The ideal large speech model predicts, based on the audio features of the fourth audio coding data, that the fifth audio coding data corresponding to the sample phoneme sequence should be the same as the third audio coding data. Therefore, the performance of the large speech model in the current training process can be determined based on the difference between the predicted fifth audio coding data and the third audio coding data.
[0124] Specifically, it can be determined whether the difference between the predicted fifth audio coding data and the third audio coding data meets the preset convergence condition. If so, the speech large model obtained in the current training process can be used as the trained speech large model.
[0125] If it is not satisfied, the parameters of the initial speech model can be adjusted and the above step of obtaining the sample text and the sample audio corresponding to the sample text can be returned until the initial speech model converges or the training times of the initial speech model reaches the preset times.
[0126] Because the audio coding data is generated based on the audio features, the audio coding data can also reflect the audio's timbre, intonation, rhythm, volume, potential noise and reverberation and other sound characteristics.
[0127] When adjusting the parameters of the initial speech model based on the difference between the predicted fifth audio coding data and the third audio coding data, the fifth audio coding data predicted by the speech model can be gradually approached to the third audio coding data. That is, the sound characteristics of the audio extracted by the speech model, such as timbre, intonation, rhythm, volume, potential noise and reverberation, can be gradually approached to the sound characteristics of the sample audio.
[0128] Through the above training process, the speech model can learn how to accurately extract the audio's timbre, intonation, rhythm, volume, potential noise, reverberation and other sound characteristics from the audio encoding data corresponding to the audio.
[0129] It should be noted here that the training method of the above-mentioned large speech model can be performed by other electronic devices, and then the trained large speech model can be deployed to the electronic device that executes the audio generation method provided by the embodiment of the present application. It can also be performed by the electronic device that executes the audio generation method provided by the embodiment of the present application itself, and no specific limitation is made here.
[0130] In the solution provided in the embodiment of the present application, through the above-mentioned training method, not only can a large speech model that can accurately predict audio coding data be obtained, but the internal structure of the large speech model, especially the first audio feature extracted by the layer before the output layer, can also accurately capture and reflect richer and finer-grained audio information, and then the target audio generated according to the first audio feature can be very close to the auditory effect in the actual scene.
[0131] As an implementation method of the present application, Figure 7 As shown, the above step S103 may include:
[0132] S701, determining a time range of the audio segment corresponding to the character to be replaced in the original audio.
[0133] In one implementation, the electronic device may first align the original text with the original audio through an MFA (Montreal Forced Aligner) or a pre-trained alignment neural network to obtain an alignment result, wherein the alignment result includes: the time range of the audio segment corresponding to each character in the original text in the original audio.
[0134] For example, the original text includes character A, character B, character C and character D, and the alignment result may include: character A in the original text corresponds to audio segment a in the original audio, and the time range corresponding to audio segment a is [0s, 1s); character B in the original text corresponds to audio segment b in the original audio, and the time range corresponding to audio segment b is [1s, 2s); character C in the original text corresponds to audio segment c in the original audio, and the time range corresponding to audio segment c is [2s, 3s); character D in the original text corresponds to audio segment d in the original audio, and the time range corresponding to audio segment d is [3s, 4s].
[0135] After obtaining the alignment result, the electronic device can determine the time range of the audio segment corresponding to the character to be replaced in the original text in the original audio.
[0136] For example, if the characters to be replaced in the original text are characters B and C, then according to the above alignment results, it can be determined that the audio segments corresponding to the characters to be replaced are audio segment b and audio segment c, the time range corresponding to audio segment b is [1s, 2s), and the time range corresponding to audio segment c is [2s, 3s), and then it can be determined that the time range of the characters to be replaced (character B and character C) in the original text in the original audio is [1s, 3s).
[0137] In another implementation, the electronic device may perform speech recognition on the original audio to obtain a speech recognition result, wherein the speech recognition result includes: each recognized character and the time range of the audio segment corresponding to each character in the original audio.
[0138] For example, speech recognition is performed on the original audio to obtain character A, character B, character C and character D. The speech recognition results may include: character A recognized from the original audio corresponds to audio segment a in the original audio, and the time range corresponding to audio segment a is [0s, 1s); character B recognized from the original audio corresponds to audio segment b in the original audio, and the time range corresponding to audio segment b is [1s, 2s); character C recognized from the original audio corresponds to audio segment c in the original audio, and the time range corresponding to audio segment c is [2s, 3s); character D recognized from the original audio corresponds to audio segment d in the original audio, and the time range corresponding to audio segment d is [3s, 4s].
[0139] After obtaining the speech recognition result, the electronic device can first determine the target character that matches the character to be replaced in the original text from the various characters obtained by the speech recognition, and then determine the time range of the audio segment corresponding to the target character in the original audio as the time range of the audio segment corresponding to the character to be replaced in the original audio.
[0140] Regarding the method of determining the target character that matches the character to be replaced in the original text, in one embodiment, if the characters included in the original text are the same as the characters obtained by speech recognition of the original audio, then the characters in the speech recognition result that are the same as the characters to be replaced can be directly used as the target character that matches the characters to be replaced.
[0141] In another embodiment, if the characters included in the original text are different from the characters obtained by voice recognition of the original audio, a prompt message can be output to prompt the user to manually determine the target characters that match the characters to be replaced in the original text from the characters obtained by voice recognition. The similarity between the characters included in the original text and the characters obtained by voice recognition of the original audio can also be calculated, and the character with the highest similarity to the characters to be replaced in the original text among the characters obtained by voice recognition can be used as the target character.
[0142] S702: Remove the sub-audio coded data whose corresponding time information is within the time range from among the sub-audio coded data included in the first audio coded data, to obtain second audio coded data.
[0143] The first audio encoding data obtained by the electronic device for the original audio may include various sub-audio encoding data, and one sub-audio encoding data corresponds to one time information.
[0144] For example, the first audio encoding data may include sub-audio encoding data 1, sub-audio encoding data 2...sub-audio encoding data N, wherein the time information corresponding to the sub-audio encoding data 1 is [0s, 0.01s), the time information corresponding to the sub-audio encoding data 2 is [0.01s, 0.02s)...the time information corresponding to the sub-audio encoding data N is [{0.01*(N-1)}s,{0.01*N}s].
[0145] After determining the time range of the audio segment corresponding to the character to be replaced in the original audio, the electronic device can determine which sub-audio encoded data's time information is within the time range based on the time information corresponding to each sub-audio encoded data, and then remove the sub-audio encoded data whose corresponding time information is within the time range from the sub-audio encoded data included in the first audio encoded data to obtain the second audio encoded data.
[0146] For example, when the electronic device determines that the time range of the audio segment corresponding to the character to be replaced in the original audio is [0s, 0.1s), then because the sub-audio coding data 1 to the sub-audio coding data 10 are all within this time range, the electronic device removes the sub-audio coding data 1 to the sub-audio coding data 10 from the sub-audio coding data included in the first audio coding data, thereby obtaining the second audio coding data.
[0147] In the solution provided in the embodiment of the present application, the electronic device can accurately determine the sub-audio coding data corresponding to the character to be replaced in the first audio coding data through alignment or voice recognition, thereby removing the part of the sub-audio coding data to obtain accurate second audio coding data.
[0148] Corresponding to the above-mentioned audio generation method, an embodiment of the present application also provides an audio generation device. The audio generation device provided by the embodiment of the present application is introduced below.
[0149] like Figure 8 As shown, an audio generating device comprises:
[0150] A first acquisition module 810 is used to acquire a target text and an original audio, wherein the target text is obtained by replacing characters to be replaced in the original text corresponding to the original audio;
[0151] A data conversion module 820, configured to convert the target text into a phoneme sequence, and encode the original audio to obtain first audio encoding data;
[0152] A data deletion module 830 is used to remove the sub-audio coded data corresponding to the character to be replaced in the first audio coded data to obtain the second audio coded data;
[0153] A second acquisition module 840 acquires a first audio feature, wherein the first audio feature is an audio feature used when predicting audio encoding data corresponding to a phoneme sequence according to a second audio feature, and the second audio feature is an audio feature of the second audio encoding data;
[0154] The audio generation module 850 is used to reconstruct the audio based on the first audio feature to obtain the target audio corresponding to the target text.
[0155] In the solution provided by the embodiment of the present application, the electronic device can obtain the target text and the original audio, wherein the target text is obtained by replacing the characters to be replaced in the original text corresponding to the original audio; converting the target text into a phoneme sequence, and encoding the original audio to obtain the first audio encoding data; removing the sub-audio encoding data corresponding to the characters to be replaced in the first audio encoding data to obtain the second audio encoding data; obtaining the first audio feature, wherein the first audio feature is the audio feature used when predicting the audio encoding data corresponding to the phoneme sequence based on the second audio feature, and the second audio feature is the audio feature of the second audio encoding data; reconstructing the audio based on the first audio feature to obtain the target audio corresponding to the target text. The above scheme realizes the automatic modification of the original audio, avoids the time and economic cost of re-convening the actors to re-record the audio in the traditional method, and greatly improves the efficiency of modifying the original audio. In addition, the first audio feature is essentially a latent variable feature, which can more accurately capture and retain the sound characteristics of the original audio, thereby ensuring that when the target audio is reconstructed based on the first audio feature, the target audio and the original audio can maintain consistency in sound characteristics. Further, when the target audio is used to replace the original audio in a film or television work, a seamless connection can be achieved in sound, avoiding abruptness and improving the overall production quality of the film or television work and the audience experience.
[0156] As an implementation of the embodiment of the present application, the above-mentioned audio generation module 850 may include:
[0157] a spectrum data generating unit, configured to use the first audio feature as prior information, and gradually remove noise in an initial noise matrix within a plurality of time steps according to the prior information and a noise level corresponding to each time step, and determine the initial noise matrix after denoising as spectrum data corresponding to the first audio feature, wherein the initial noise matrix is a noise matrix that matches the spectrum data in dimension and structure and is completely composed of random noise;
[0158] The audio generation unit is used to convert the spectrum data into target audio.
[0159] As an implementation manner of the embodiment of the present application, the above-mentioned spectrum data generating unit can be specifically used for:
[0160] The first audio feature is input into the diffusion model; and an initial noise matrix after denoising outputted by the diffusion model is obtained, wherein the initial noise matrix after denoising is obtained by the diffusion model taking the first audio feature as prior information and gradually removing noise in the initial noise matrix within multiple time steps according to the prior information and the noise level corresponding to each time step.
[0161] As an implementation method of an embodiment of the present application, the above-mentioned second acquisition module 840 can be specifically used to: input the phoneme sequence and the second audio coding data into a pre-trained speech model; obtain the first audio feature output by the previous layer of the output layer of the speech model, wherein the first audio feature is the audio feature used by the speech model when predicting the audio coding data corresponding to the phoneme sequence based on the second audio feature.
[0162] As an implementation method of the embodiment of the present application, the training method of the above-mentioned large speech model may include:
[0163] Obtain sample text and sample audio corresponding to the sample text;
[0164] Converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio encoding data;
[0165] Randomly remove part of the sub-audio coded data in the third audio coded data to obtain fourth audio coded data;
[0166] Inputting the sample phoneme sequence and the fourth audio coding data into the initial speech big model;
[0167] Acquire fifth audio coding data output by the initial speech large model, wherein the fifth audio coding data is audio coding data corresponding to a sample phoneme sequence predicted by the initial speech large model according to audio features of the fourth audio coding data;
[0168] Based on the difference between the predicted fifth audio coding data and the third audio coding data, the parameters of the initial speech model are adjusted to obtain the trained speech model.
[0169] As an implementation manner of the embodiment of the present application, the first audio encoding data may include various sub-audio encoding data, and one sub-audio encoding data corresponds to one time information;
[0170] The data deletion module 830 may include:
[0171] A time range determination unit, used to determine the time range of the audio segment corresponding to the character to be replaced in the original audio;
[0172] The data deleting unit is used to remove the sub-audio coded data whose corresponding time information is within the time range from each sub-audio coded data included in the first audio coded data, so as to obtain the second audio coded data.
[0173] As an implementation of the embodiment of the present application, the time range determination unit may include:
[0174] The first determination subunit is used to align the original text with the original audio to obtain an alignment result, wherein the alignment result includes: the time range of the audio segment corresponding to each character in the original text in the original audio; in the alignment result, the time range of the audio segment corresponding to the character to be replaced in the original text in the original audio is determined;
[0175] The second determination subunit is used to perform speech recognition on the original audio to obtain a speech recognition result, wherein the speech recognition result includes: each recognized character and the time range of the audio segment corresponding to each character in the original audio; the time range of the audio segment corresponding to the target character in the original audio is determined as the time range of the audio segment corresponding to the character to be replaced in the original audio, wherein the target character is the character in the speech recognition result that matches the character to be replaced.
[0176] The present application also provides an electronic device, such as Fig. 9 As shown, it includes a processor 901, a communication interface 902, a memory 903 and a communication bus 904, wherein the processor 901, the communication interface 902, and the memory 903 communicate with each other through the communication bus 904.
[0177] Memory 903, used for storing computer programs;
[0178] The processor 901 is used to implement the audio generation method described in any of the above embodiments when executing the program stored in the memory 903.
[0179] In the solution provided by the embodiment of the present application, the electronic device can obtain the target text and the original audio, wherein the target text is obtained by replacing the characters to be replaced in the original text corresponding to the original audio; converting the target text into a phoneme sequence, and encoding the original audio to obtain the first audio encoding data; removing the sub-audio encoding data corresponding to the characters to be replaced in the first audio encoding data to obtain the second audio encoding data; obtaining the first audio feature, wherein the first audio feature is the audio feature used when predicting the audio encoding data corresponding to the phoneme sequence based on the second audio feature, and the second audio feature is the audio feature of the second audio encoding data; reconstructing the audio based on the first audio feature to obtain the target audio corresponding to the target text. The above scheme realizes the automatic modification of the original audio, avoids the time and economic cost of re-convening the actors to re-record the audio in the traditional method, and greatly improves the efficiency of modifying the original audio. In addition, the first audio feature is essentially a latent variable feature, which can more accurately capture and retain the sound characteristics of the original audio, thereby ensuring that when the target audio is reconstructed based on the first audio feature, the target audio and the original audio can maintain consistency in sound characteristics. Further, when the target audio is used to replace the original audio in a film or television work, a seamless connection can be achieved in sound, avoiding abruptness and improving the overall production quality of the film or television work and the audience experience.
[0180] The communication bus mentioned in the above terminal can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.
[0181] The communication interface is used for communication between the above terminal and other devices.
[0182] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.
[0183] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.
[0184] In another embodiment provided in the present application, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio generation method described in any one of the above embodiments is implemented.
[0185] In another embodiment provided in the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute the audio generation method described in any one of the above embodiments.
[0186] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.
[0187] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.
[0188] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device, electronic device, computer-readable storage medium, and computer program product embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.
[0189] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.
Claims
1. An audio generation method, characterized in that: The method comprises: Obtaining a target text and an original audio, wherein the target text is obtained by replacing characters to be replaced in an original text corresponding to the original audio; Converting the target text into a phoneme sequence, and encoding the original audio to obtain first audio encoding data; removing the sub-audio coded data corresponding to the character to be replaced from the first audio coded data to obtain second audio coded data; Acquire a first audio feature, wherein the first audio feature is an audio feature used when predicting the audio encoding data corresponding to the phoneme sequence according to the second audio feature, and the second audio feature is an audio feature of the second audio encoding data; Audio reconstruction is performed based on the first audio feature to obtain target audio corresponding to the target text.
2. The method according to claim 1, characterized in that The reconstructing the audio based on the first audio feature to obtain the target audio corresponding to the target text includes: The first audio feature is used as prior information, and according to the prior information and the noise level corresponding to each time step, the noise in the initial noise matrix is gradually removed within multiple time steps, and the initial noise matrix after denoising is determined as the spectrum data corresponding to the first audio feature, wherein the initial noise matrix is a noise matrix that matches the spectrum data in dimension and structure and is completely composed of random noise; The spectrum data is converted into target audio.
3. The method according to claim 2, characterized in that The method of using the first audio feature as prior information and gradually removing noise in the initial noise matrix within multiple time steps according to the prior information and the noise level corresponding to each time step includes: inputting the first audio feature into a diffusion model; An initial noise matrix after denoising output by the diffusion model is obtained, wherein the initial noise matrix after denoising is obtained by the diffusion model taking the first audio feature as prior information and gradually removing noise in the initial noise matrix within multiple time steps according to the prior information and the noise level corresponding to each time step.
4. The method according to claim 1, characterized in that: The obtaining of the first audio feature includes: Inputting the phoneme sequence and the second audio encoding data into a pre-trained speech model; Obtain a first audio feature output by a previous layer of an output layer of the speech model, wherein the first audio feature is an audio feature used by the speech model when predicting audio coding data corresponding to the phoneme sequence based on the second audio feature.
5. The method according to claim 4, characterized in that The training method of the large speech model includes: Obtaining a sample text and a sample audio corresponding to the sample text; Converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio encoding data; Randomly remove part of the sub-audio coded data in the third audio coded data to obtain fourth audio coded data; Inputting the sample phoneme sequence and the fourth audio coding data into an initial speech big model; Acquire fifth audio coding data output by the initial speech large model, wherein the fifth audio coding data is audio coding data corresponding to the sample phoneme sequence predicted by the initial speech large model according to the audio features of the fourth audio coding data; Based on the predicted difference between the fifth audio coding data and the third audio coding data, the parameters of the initial speech model are adjusted to obtain a trained speech model.
6. The method according to any one of claims 1 to 5, characterized in that: The first audio encoding data includes each sub-audio encoding data, and each sub-audio encoding data corresponds to one time information; The step of removing the sub-audio coded data corresponding to the character to be replaced in the first audio coded data to obtain the second audio coded data includes: Determine the time range of the audio segment corresponding to the character to be replaced in the original audio; Among the sub-audio coded data included in the first audio coded data, the sub-audio coded data whose corresponding time information is within the time range is removed to obtain the second audio coded data.
7. The method according to claim 6, characterized in that The step of determining the time range of the audio segment corresponding to the character to be replaced in the original audio includes: Aligning the original text with the original audio to obtain an alignment result, wherein the alignment result includes: a time range of the audio segment corresponding to each character in the original text in the original audio; in the alignment result, determining a time range of the audio segment corresponding to the character to be replaced in the original text in the original audio; or, Perform speech recognition on the original audio to obtain a speech recognition result, wherein the speech recognition result includes: the time range of each recognized character and the audio segment corresponding to each character in the original audio; determine the time range of the audio segment corresponding to the target character in the original audio as the time range of the audio segment corresponding to the character to be replaced in the original audio, wherein the target character is the character in the speech recognition result that matches the character to be replaced.
8. An audio generating device, characterized in that: The device comprises: A first acquisition module is used to acquire a target text and an original audio, wherein the target text is obtained by replacing characters to be replaced in an original text corresponding to the original audio; A data conversion module, used for converting the target text into a phoneme sequence, and encoding the original audio to obtain first audio encoding data; A data deletion module, used for removing the sub-audio coded data corresponding to the character to be replaced in the first audio coded data to obtain second audio coded data; A second acquisition module acquires a first audio feature, wherein the first audio feature is an audio feature used when predicting the audio encoding data corresponding to the phoneme sequence according to the second audio feature, and the second audio feature is an audio feature of the second audio encoding data; An audio generation module is used to reconstruct the audio based on the first audio feature to obtain the target audio corresponding to the target text.
9. The device according to claim 8, characterized in that The audio generation module includes: a spectrum data generation unit, which is used to use the first audio feature as prior information, and gradually remove the noise in the initial noise matrix within multiple time steps according to the prior information and the noise level corresponding to each time step, and determine the initial noise matrix after denoising as the spectrum data corresponding to the first audio feature, wherein the initial noise matrix is a noise matrix that matches the spectrum data in dimension and structure and is completely composed of random noise; an audio generation unit, which is used to convert the spectrum data into target audio; or, The frequency spectrum data generating unit is specifically used to: input the first audio feature into a diffusion model; obtain an initial noise matrix after denoising output by the diffusion model, wherein the initial noise matrix after denoising is obtained by the diffusion model taking the first audio feature as prior information and gradually removing noise in the initial noise matrix in multiple time steps according to the prior information and the noise level corresponding to each time step; or, The second acquisition module is specifically used to: input the phoneme sequence and the second audio coding data into a pre-trained speech model; obtain the first audio feature output by a previous layer of the output layer of the speech model, wherein the first audio feature is an audio feature used by the speech model when predicting the audio coding data corresponding to the phoneme sequence according to the second audio feature; or, The training method of the speech large model includes: obtaining a sample text and a sample audio corresponding to the sample text; converting the sample text into a sample phoneme sequence, and encoding the sample audio to obtain third audio encoding data; randomly removing part of the sub-audio encoding data in the third audio encoding data to obtain fourth audio encoding data; inputting the sample phoneme sequence and the fourth audio encoding data into the initial speech large model; obtaining fifth audio encoding data output by the initial speech large model, wherein the fifth audio encoding data is the audio encoding data corresponding to the sample phoneme sequence predicted by the initial speech large model according to the audio features of the fourth audio encoding data; adjusting the parameters of the initial speech large model based on the difference between the predicted fifth audio encoding data and the third audio encoding data to obtain a trained speech large model; or, The first audio coded data includes various sub-audio coded data, each sub-audio coded data corresponds to one time information; the data deletion module includes: a time range determination unit, used to determine the time range of the audio segment corresponding to the character to be replaced in the original audio; a data deletion unit, used to remove the sub-audio coded data whose corresponding time information is within the time range from the various sub-audio coded data included in the first audio coded data, to obtain the second audio coded data; or, The time range determination unit includes: a first determination subunit, used to align the original text with the original audio to obtain an alignment result, wherein the alignment result includes: the time range of the audio segment corresponding to each character in the original text in the original audio; in the alignment result, determining the time range of the audio segment corresponding to the to-be-replaced character in the original text in the original audio; a second determination subunit, used to perform speech recognition on the original audio to obtain a speech recognition result, wherein the speech recognition result includes: the time range of each recognized character and the audio segment corresponding to each character in the original audio; determining the time range of the audio segment corresponding to the target character in the original audio as the time range of the audio segment corresponding to the to-be-replaced character in the original audio, wherein the target character is the character that matches the to-be-replaced character in the speech recognition result.
10. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 7 when executing a program stored in a memory.
11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 7 are implemented.
