Audio processing method and device, equipment and storage medium
Patent Information
- Application Number
- CN202411114313.X
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-13
- Publication Date
- 2024-12-03
Smart Images

Figure CN119068890A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of digital medicine, and in particular to an audio processing method and apparatus, device and storage medium. Background Art
[0002] At present, music is often used in many medical scenarios to relieve the emotions of each subject. For example, in medical consultations, music is provided to each medical expert to improve the thinking activity of each subject during the consultation; in rehabilitation training, music is provided to each training subject to reduce the emotional stress of each training subject.
[0003] However, different subjects often have different preferences for music, which is not only reflected in the type of music, but also in the preference for the singing object of the same audio. In this regard, the relevant technology considers a method of replacing the original timbre information in the audio template with the timbre information of the object to be switched to by presetting the audio template of the audio, so that the audio is changed from being sung by subject A to being sung by subject B. However, the voiceprint conversion in this method is based on a fixed template, and there is a serious problem of audio homogeneity, which will lead to low accuracy of voiceprint conversion. Summary of the invention
[0004] The main purpose of the embodiments of the present application is to propose an audio processing method and apparatus, a device and a storage medium, aiming to improve the accuracy of voiceprint conversion of the target audio, thereby improving the audio quality of the synthesized audio.
[0005] To achieve the above object, a first aspect of an embodiment of the present application provides an audio processing method, the method comprising:
[0006] Receiving a voiceprint conversion request for a target audio from an interactive object, wherein the voiceprint conversion request is used to instruct to convert a current voiceprint feature of the target audio into a voiceprint feature of a target object;
[0007] Based on the voiceprint conversion request, obtaining a Mel-cepstrum feature of the target audio;
[0008] Encoding the Mel-cepstrum feature based on a preset voiceprint conversion model to obtain an audio basic feature latent vector and an audio harmonic feature latent vector of the target audio;
[0009] The audio harmonic feature latent vector is updated based on the pre-acquired reference voiceprint feature data to obtain a target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on the reference audio sung by the target object;
[0010] Performing enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and a combination result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain a target audio feature latent vector;
[0011] The target audio feature latent vector is decoded based on the voiceprint conversion model to obtain synthesized audio data.
[0012] In some embodiments, the voiceprint conversion model includes a first encoder and a second encoder;
[0013] The encoding process of the Mel-cepstrum feature based on the preset voiceprint conversion model to obtain the audio basic feature latent vector and the audio harmonic feature latent vector of the target audio includes:
[0014] Performing a first encoding on the Mel-frequency cepstrum feature based on the first encoder to obtain the audio basic feature latent vector;
[0015] The Mel-frequency cepstrum feature is second-encoded based on the second encoder to obtain the audio harmonic feature latent vector.
[0016] In some embodiments, the reference voiceprint feature data includes emotion feature data and phoneme feature data;
[0017] The updating of the audio harmonic feature latent vector based on the pre-acquired reference voiceprint feature data to obtain the target harmonic feature latent vector includes:
[0018] Embedding the emotion feature data to obtain an emotion embedding feature vector;
[0019] Performing harmonic feature extraction on the phoneme feature data to obtain a reference harmonic feature latent vector;
[0020] Performing feature concatenation on the emotion embedding feature vector and the reference harmonic feature latent vector to obtain a reference concatenated feature vector;
[0021] The audio harmonic feature latent vector is updated using the reference concatenated feature vector to obtain the target harmonic feature latent vector.
[0022] In some embodiments, the step of performing enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and a combination result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain the target audio feature latent vector includes:
[0023] Performing a first enhancement process on the audio basic feature latent vector to obtain an audio basic feature enhanced latent vector;
[0024] Performing a second enhancement process on the target harmonic feature latent vector to obtain a target harmonic feature enhanced latent vector;
[0025] Performing a third enhancement process on a result of combining the audio basic feature latent vector and the target harmonic feature latent vector to obtain an enhanced combined feature vector;
[0026] The audio basic feature enhanced latent vector, the target harmonic feature enhanced latent vector, and the enhanced merged feature vector are feature merged to obtain the target audio feature latent vector.
[0027] In some embodiments, the voiceprint conversion model includes a decoder and a vocoder;
[0028] The decoding process of the target audio feature latent vector based on the voiceprint conversion model to obtain synthesized audio data includes:
[0029] Performing feature decoding on the target audio feature latent vector based on the decoder to obtain an audio decoding sequence;
[0030] The audio decoding sequence is audio-converted based on the vocoder to obtain the synthesized audio data.
[0031] In some embodiments, the reference voiceprint feature data is determined by:
[0032] Parsing the voiceprint conversion request to obtain interaction intention data of the interaction object;
[0033] Based on the interaction intention data, determining the reference audio from a plurality of candidate songs sung by the target object, wherein the reference audio and the target audio have the same audio type;
[0034] Extracting emotional features from the reference audio to obtain emotional feature data;
[0035] Extracting phoneme features from the reference audio to obtain phoneme feature data;
[0036] The emotion feature data and the phoneme feature data are integrated into the reference voiceprint feature data.
[0037] In some embodiments, obtaining the Mel-cepstrum feature of the target audio based on the voiceprint conversion request includes:
[0038] Performing Fourier transform on the target audio to obtain a Fourier transform result, and determining a target power spectrum based on the square of the modulus of the Fourier transform result;
[0039] Performing Mel filtering on the target power spectrum to obtain a target audio filtering result, and performing logarithmic compression on the target audio filtering result to obtain a target audio compression result;
[0040] Discretization mapping is performed on the target audio compression result to obtain the Mel-frequency cepstrum feature.
[0041] To achieve the above object, a second aspect of an embodiment of the present application provides an audio processing device, the device comprising:
[0042] A receiving unit, configured to receive a voiceprint conversion request from an interactive object for a target audio, wherein the voiceprint conversion request is used to instruct to convert a current voiceprint feature of the target audio into a voiceprint feature of a target object;
[0043] An acquiring unit, configured to acquire a Mel-cepstrum feature of the target audio based on the voiceprint conversion request;
[0044] An encoding unit, configured to encode the Mel-cepstrum feature based on a preset voiceprint conversion model to obtain an audio basic feature latent vector and an audio harmonic feature latent vector of the target audio;
[0045] an updating unit, configured to update the audio harmonic feature latent vector based on pre-acquired reference voiceprint feature data to obtain a target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on a reference song sung by the target object;
[0046] an enhancement unit, configured to perform enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and a combination result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain a target audio feature latent vector;
[0047] A decoding unit is used to decode the target audio feature latent vector based on the voiceprint conversion model to obtain synthesized audio data.
[0048] To achieve the above-mentioned purpose, the third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the audio processing method described in the first aspect is implemented.
[0049] To achieve the above objectives, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the audio processing method described in the first aspect is implemented.
[0050] The audio processing method and device, electronic device and storage medium proposed in the present application receive a voiceprint conversion request for the target audio from an interactive object, wherein the voiceprint conversion request is used to indicate that the current voiceprint feature of the target audio is converted into the voiceprint feature of the target object. Further, based on the voiceprint conversion request, the Mel-cepstral feature of the target audio is obtained, and the Mel-cepstral feature is encoded based on a preset voiceprint conversion model to obtain the audio basic feature latent vector and the audio harmonic feature latent vector of the target audio, and the different feature information of the target audio is encoded respectively. Then, the audio harmonic feature latent vector is updated based on the pre-acquired reference voiceprint feature data to obtain the target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on the reference audio sung by the target object. This method can flexibly adjust and optimize the audio harmonic feature latent vector through the voiceprint characteristics of the target object contained in the reference voiceprint feature data, so as to realize the rapid conversion of the voiceprint feature and improve the accuracy of the voiceprint conversion of the target audio. Furthermore, the audio basic feature latent vector, the target harmonic feature latent vector, and the combined result of the audio basic feature latent vector and the target harmonic feature latent vector are enhanced to obtain the target audio feature latent vector; and the target audio feature latent vector is decoded based on the voiceprint conversion model to obtain the synthesized audio data. The enhancement link of multiple latent vectors is introduced before the decoding link, which can improve the utilization efficiency of the effective information of each latent vector, improve the feature accuracy and feature comprehensiveness of the target audio feature latent vector used for decoding, and thus improve the audio quality of the decoded synthesized audio data. In addition, the present application is a voiceprint conversion based on human-computer interaction, which can meet the real-time interaction needs of various objects at different times and in different scenarios. BRIEF DESCRIPTION OF THE DRAWINGS
[0051] Figure 1 is a flowchart of an audio processing method provided in an embodiment of the present application;
[0052] Figure 2 yes Figure 1 Flow chart of step S102 in FIG.
[0053] Figure 3 yes Figure 1 Flow chart of step S103 in FIG.
[0054] Figure 4 yes Figure 1 Flow chart of step S104 in FIG.
[0055] Figure 5 is a flow chart of determining reference voiceprint feature data according to an embodiment of the present application;
[0056] Figure 6 yes Figure 1 Flow chart of step S105 in FIG.
[0057] Figure 7 yes Figure 1 Flow chart of step S106 in FIG.
[0058] Figure 8 is a structural diagram of an audio processing device provided in an embodiment of the present application;
[0059] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION
[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.
[0063] The audio processing method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the data processing method in the embodiments of the present application is described.
[0064] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.
[0065] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0066] The audio processing method provided in the embodiment of the present application relates to the field of digital medical technology. The audio processing method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or can be configured as a server cluster or a distributed system composed of multiple physical servers, and can also be configured as a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the audio processing method, etc., but is not limited to the above forms.
[0067] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.
[0068] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to the identity or characteristics of the object, such as object information, object behavior data, object historical data, and object location information, the permission or consent of the object will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the personal information of the object, the separate permission or separate consent of the object will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the separate permission or separate consent of the object, the necessary object-related data for the normal operation of the embodiment of the present application will be obtained.
[0069] Figure 1 is an optional flowchart of the audio processing method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S106.
[0070] Step S101, receiving a voiceprint conversion request for a target audio from an interactive object;
[0071] Step S102, obtaining Mel-cepstrum features of the target audio based on the voiceprint conversion request;
[0072] Step S103, encoding the Mel-frequency cepstrum feature based on a preset voiceprint conversion model to obtain an audio basic feature latent vector and an audio harmonic feature latent vector of the target audio;
[0073] Step S104, updating the audio harmonic feature latent vector based on the pre-acquired reference voiceprint feature data to obtain a target harmonic feature latent vector;
[0074] Step S105, performing enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and the combined result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain the target audio feature latent vector;
[0075] Step S106: decoding the target audio feature latent vector based on the voiceprint conversion model to obtain synthesized audio data.
[0076] Steps S101 to S106 shown in the embodiment of the present application are performed by receiving a voiceprint conversion request for the target audio from the interactive object, wherein the voiceprint conversion request is used to indicate that the current voiceprint feature of the target audio is converted into the voiceprint feature of the target object. Further, based on the voiceprint conversion request, the Mel-cepstrum feature of the target audio is obtained, and the Mel-cepstrum feature is encoded based on a preset voiceprint conversion model to obtain the audio basic feature latent vector and the audio harmonic feature latent vector of the target audio, and the different feature information of the target audio is encoded respectively. Then, the audio harmonic feature latent vector is updated based on the pre-acquired reference voiceprint feature data to obtain the target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on the reference audio sung by the target object. This method can flexibly adjust and optimize the audio harmonic feature latent vector through the voiceprint characteristics of the target object contained in the reference voiceprint feature data, thereby realizing the rapid conversion of the voiceprint feature and improving the accuracy of the voiceprint conversion of the target audio. Furthermore, the audio basic feature latent vector, the target harmonic feature latent vector, and the combined result of the audio basic feature latent vector and the target harmonic feature latent vector are enhanced to obtain the target audio feature latent vector; and the target audio feature latent vector is decoded based on the voiceprint conversion model to obtain the synthesized audio data. The enhancement link of multiple latent vectors is introduced before the decoding link, which can improve the utilization efficiency of the effective information of each latent vector, improve the feature accuracy and feature comprehensiveness of the target audio feature latent vector used for decoding, and thus improve the audio quality of the decoded synthesized audio data. In addition, the present application is a voiceprint conversion based on human-computer interaction, which can meet the real-time interaction needs of various objects at different times and in different scenarios.
[0077] It should be noted that the audio processing method of the embodiment of the present application can be used in various medical scenarios in the field of digital medicine. For example, when conducting rehabilitation training for a certain object, the object can initiate a voiceprint conversion request based on the object's interest preference, and change the target audio played in the rehabilitation training from object A to object B. The object enters the intention "change the singer of music 1 to object B" in the intelligent robot. At this time, the intelligent robot will search for the music type of music 1, and then search for the music of the same type sung by object B before the current time, and select one from them as a reference audio, use music 1 as the target music, use the selected music as the reference audio, and call the voiceprint conversion model of the present application, perform the voiceprint conversion operation according to the audio processing method of the present application, and push the final generated synthetic audio that meets the object's needs to the object, where the synthetic audio is the music 1 sung by object B that the object wants to play during rehabilitation training.
[0078] In step S101 of some embodiments, a voiceprint conversion request for target audio is received from an interactive object.
[0079] The interactive object refers to a user who wants to use human-computer interaction to transform the target audio from being sung by object A to being sung by object B. The target audio can be a song consisting of human voice and background music, etc. The voiceprint conversion request is used to instruct to convert the current voiceprint features of the target audio into the voiceprint features of the target object.
[0080] The current voiceprint feature is used to indicate the sound characteristics of the original object in the target audio. Among them, the sound characteristics include pitch, sound intensity, timbre, etc. The voiceprint features of different objects are often different. The voiceprint feature is the unique attribute of an individual in the audio.
[0081] The voiceprint feature of the target object is used to indicate the sound characteristics of the target object. The target object is not the same object as the original object in the target audio.
[0082] In the specific implementation of this embodiment, when the interactive object wants to use human-computer interaction to transform the target audio from the target A to the target B, the interactive object will initiate a voiceprint conversion request to the server through the object terminal. Based on this, the server will receive the voiceprint conversion request for the target audio, and determine the target audio to be converted and the identity information of the target object (for example, the object name, object identifier, etc. of the target object) according to the voiceprint conversion request.
[0083] See also Figure 2 In some embodiments, step S102 may include but is not limited to steps S201 to S203:
[0084] Step S201, performing Fourier transform on the target audio to obtain a Fourier transform result, and determining a target power spectrum based on the square of the modulus of the Fourier transform result;
[0085] Step S202, performing Mel filtering on the target power spectrum to obtain a target audio filtering result, and performing logarithmic compression on the target audio filtering result to obtain a target audio compression result;
[0086] Step S203, discretizing and mapping the target audio compression result to obtain Mel-frequency cepstrum features.
[0087] Steps S201 to S203 are described in detail below.
[0088] In step S201 of some embodiments, first, the target audio is denoised to eliminate the ambient noise in the target audio and obtain the denoised audio. Then, the denoised audio (continuous-time audio signal) is converted into a discrete-time signal, and the discrete-time signal is truncated and multiplied by a window function to obtain a truncated audio signal. Among them, the window function includes but is not limited to a Hanning window, a Hamming window, and the like. Further, the truncated audio signal is Fourier transformed using a fast Fourier transform algorithm to provide the frequency components in the truncated audio signal, and the target audio is converted from the time domain to the frequency domain to obtain a Fourier transform result. Among them, the Fourier transform result is used to represent the amplitude and phase of different frequency components of the target audio, and is used to indicate the frequency distribution information of the target audio. Finally, the Fourier transform result is modulo, and the modulus of the Fourier transform result is bisected to obtain the target power spectrum.
[0089] In step S202 of some embodiments, first, the Mel filter bank can be called with authorization, and the target power spectrum can be filtered using the Mel filter bank to map the conventional frequency to the Mel frequency scale to obtain the target audio filtering result. Then, the target audio filtering result is logarithmically compressed to simulate the nonlinear perception of the human ear to the sound intensity to obtain the target audio compression result.
[0090] In step S203 of some embodiments, when discretizing and mapping the target audio compression result, the target audio compression result is firstly subjected to discrete cosine transform to obtain a discrete cosine transform result, wherein the discrete cosine transform result has a plurality of coefficients for characterizing the audio information of the target audio. Then, the first predetermined number of coefficients are selected from the discrete cosine transform result as Mel-cepstral features. The predetermined number is not greater than 13, and the selected coefficients represent the important audio information of the target audio. By screening the coefficients in the discrete cosine transform result, redundant information can be reduced well.
[0091] The advantage of this embodiment is that it uses operations such as Fourier transform, Mel filtering and discretization mapping to convert the target audio from the time domain to the frequency domain, and extracts important audio information of the target audio in the frequency domain, which can improve the feature quality of the Mel cepstrum feature used for voiceprint conversion.
[0092] See also Figure 3 In some embodiments, the voiceprint conversion model includes a first encoder and a second encoder; step S103 may include but is not limited to steps S301 to S302:
[0093] Step S301, performing a first encoding on the Mel-frequency cepstrum feature based on a first encoder to obtain an audio basic feature latent vector;
[0094] Step S302: Perform a second encoding on the Mel-frequency cepstrum feature based on a second encoder to obtain an audio harmonic feature latent vector.
[0095] Steps S301 to S302 are described in detail below.
[0096] In step S301 of some embodiments, the Mel-cepstral feature is input into a voiceprint conversion model, and the Mel-cepstral feature is first encoded by a first encoder to capture feature information related to the inherent information of the target audio (such as lyrics, rhythm, melody, etc.) in the Mel-cepstral feature, and obtain the audio basic feature latent vector, wherein the audio basic feature latent vector is used to indicate the representation of the inherent feature information of the target audio in the latent space (latent space).
[0097] In step S302 of some embodiments, the Mel-cepstral feature is input into a voiceprint conversion model, and a second encoder performs a second encoding on the Mel-cepstral feature to capture feature information related to the voiceprint characteristics of the original object (such as tone, pitch, timbre, etc.) in the Mel-cepstral feature, and obtain an audio harmonic feature latent vector, wherein the audio harmonic feature latent vector is used to indicate the representation of the voiceprint feature information of the original object in the target audio in the latent space (latent space).
[0098] The advantage of this embodiment is that by separately encoding the Mel-cepstral features using the first encoder and the second encoder, different feature information of the Mel-cepstral features can be extracted separately, and the voiceprint feature information (for example, the tone, timbre, pitch, etc. of the human voice) and the audio inherent feature information (for example, the genre of music, text content, etc.) in the target audio can be separated more conveniently and accurately. This is beneficial for separately processing the audio harmonic feature latent vectors while keeping the basic audio feature latent vectors unchanged, so that when the target audio is converted into a voiceprint, the impact on the audio inherent characteristics of the target audio is reduced as much as possible, thereby improving the audio quality of the generated synthetic audio data.
[0099] See also Figure 4 In some embodiments, the reference voiceprint feature data includes emotion feature data and phoneme feature data; step S104 may include but is not limited to steps S401 to S404:
[0100] Step S401, embedding the emotion feature data to obtain an emotion embedding feature vector;
[0101] Step S402, extracting harmonic features from the phoneme feature data to obtain a reference harmonic feature latent vector;
[0102] Step S403, performing feature concatenation on the emotion embedding feature vector and the reference harmonic feature latent vector to obtain a reference concatenated feature vector;
[0103] Step S404: update the audio harmonic feature latent vector using the reference concatenated feature vector to obtain the target harmonic feature latent vector.
[0104] Steps S401 to S404 are described in detail below.
[0105] Among them, the reference voiceprint feature data is determined based on the reference audio sung by the target object.
[0106] In step S401 of some embodiments, the emotion feature data is embedded and mapped from the data space to the latent vector space to obtain an emotion embedding feature vector, which is used to indicate the feature representation of the emotion feature data in the latent vector space.
[0107] In step S402 of some embodiments, harmonic feature extraction is performed on the phoneme feature data to obtain harmonic information related to the voiceprint characteristics of the target object in the phoneme feature data, and obtain a reference harmonic feature latent vector.
[0108] In step S403 of some embodiments, feature splicing is performed on the emotion embedding feature vector and the reference harmonic feature latent vector, and the emotion embedding feature vector and the reference harmonic feature latent vector are converted into latent vectors of the same dimension, so that under the same feature dimension, the transformed emotion embedding feature vector and the transformed reference harmonic feature latent vector are vector spliced to obtain a reference spliced feature vector, wherein the reference spliced feature vector contains the voiceprint characteristics of the target object and the emotion representation characteristics when singing audio of the audio type to which the target audio belongs.
[0109] In step S404 of some embodiments, when the audio harmonic feature latent vector is updated with the reference splicing feature vector, first, the element types of the reference splicing feature vector and the audio harmonic feature latent vector are compared, wherein the comparison result may have three situations, for example, a first element type that is present in both the reference splicing feature vector and the audio harmonic feature latent vector; a second element type that appears in the reference splicing feature vector but does not appear in the audio harmonic feature latent vector; a third element type that does not appear in the reference splicing feature vector but appears in the audio harmonic feature latent vector. Based on this, for the first element type that is present in both the reference splicing feature vector and the audio harmonic feature latent vector, the element value of the first element type in the audio harmonic feature latent vector is replaced with the element value in the reference splicing feature vector. For the second element type that appears in the reference splicing feature vector but does not appear in the audio harmonic feature latent vector, the element value of the second element type in the reference splicing feature vector is added as a new element and element value to the audio harmonic feature latent vector. For the third element type that does not appear in the reference splicing feature vector but appears in the audio harmonic feature latent vector, the element value of the third element type in the audio harmonic feature latent vector is kept unchanged. At this time, the updated target harmonic feature latent vector will contain the above three element types (first element type, second element type, and third element type) at the same time, and the element values of the first element type and the second element type are both derived from the reference splicing feature vector, and the element value of the third element type is derived from the audio harmonic feature latent vector.
[0110] The benefit of this embodiment is that the reference harmonic feature latent vector containing the emotional characteristics and acoustic characteristics of the target object and the audio harmonic feature latent vector are feature fused, and the voiceprint conversion model can be used to learn the potential representation of the target audio, so that the voiceprint conversion model can achieve multi-dimensional adjustment and optimization of pitch, timbre, emotion, etc. in the voiceprint conversion of the target audio, and the voiceprint features in the audio harmonic feature latent vector are fully replaced with the voiceprint features of the target object contained in the reference harmonic feature latent vector, and the emotional feature representation of the target object is also integrated into it, which can improve the feature accuracy and feature comprehensiveness of the target harmonic feature latent vector.
[0111] See also Figure 5 In some embodiments, the specific process of determining the reference voiceprint feature data may include but is not limited to steps S501 to S505:
[0112] Step S501, parsing the voiceprint conversion request to obtain the interaction intention data of the interaction object;
[0113] Step S502, determining reference audio from a plurality of candidate songs sung by the target object based on the interaction intention data;
[0114] Step S503, extracting emotional features from the reference audio to obtain emotional feature data;
[0115] Step S504, extracting phoneme features from the reference audio to obtain phoneme feature data;
[0116] Step S505: Integrate the emotion feature data and the phoneme feature data into reference voiceprint feature data.
[0117] Steps S501 to S505 are described in detail below.
[0118] In step S501 of some embodiments, with authorization, the voiceprint conversion request is parsed, the key code segments in the code data corresponding to the voiceprint conversion request are analyzed, and the interaction intention data of the interactive object is obtained according to the logical meaning of the key code segments, wherein the interaction intention data is used to indicate the object identity of the target object and the source of the voiceprint feature data of the target object to be used. For example, the interaction intention data may be "to realize the voiceprint conversion of the target audio according to the voiceprint feature of song A sung by the target object, the name of song A is xxx, and the download path is xxxxx".
[0119] In step S502 of some embodiments, first, the interaction intention data is extracted to obtain the audio name of the reference audio specified by the interaction object and to be applied in the voiceprint conversion. Then, with authorization, all candidate songs sung by the target object are obtained, and according to the audio name of the reference audio, the candidate songs specified by the interaction object and whose song names are consistent with the audio names are searched in the candidate songs, and the audio files of the searched candidate songs are extracted. The searched candidate songs are the reference audio. The reference audio and the target audio have the same audio type.
[0120] In step S503 of some embodiments, a convolutional neural network or the like may be used to extract emotional feature information from the audio file of the candidate song (reference audio), obtain emotional feature elements contained in each audio frame of the reference audio, and integrate all acquired emotional feature elements to obtain emotional feature data, wherein the emotional feature data is used to indicate the emotional representation characteristics of the target object when singing a song of the audio type to which the target audio belongs.
[0121] In step S504 of some embodiments, a convolutional neural network or the like may be used to extract phoneme feature information from an audio file of a candidate song (reference audio), obtain phoneme feature elements contained in each audio frame of the reference audio, and integrate all obtained phoneme feature elements to obtain phoneme feature data, wherein the phoneme feature data is used to indicate the phoneme representation characteristics (e.g., speed, timbre, etc.) of a target object when singing a song of the audio type to which the target audio belongs.
[0122] In step S505 of some embodiments, the emotion feature data and the phoneme feature data are aggregated into a whole to form a feature data set, thereby integrating the emotion feature data and the phoneme feature data, and all feature data related to the target object in the feature data set are used as reference voiceprint feature data.
[0123] The benefit of this embodiment is that the reference audio specified by the interactive object and belonging to the same audio type as the target audio is screened out from the multiple candidate songs sung by the target object, which can improve the extraction accuracy of the reference voiceprint feature data (the voiceprint feature information of the target object). At the same time, when obtaining the reference voiceprint feature data, the emotional feature information and phoneme feature information of the target object are introduced, so that when the reference voiceprint feature data is used for voiceprint conversion, the emotional features of the target object can be integrated into it, and the character emotions of the target audio can be finely controlled during voiceprint conversion, which can improve the emotional expression of the generated synthetic audio to a certain extent.
[0124] See also Figure 6 In some embodiments, step S105 includes but is not limited to steps S601 to S604:
[0125] Step S601, performing a first enhancement process on the audio basic feature latent vector to obtain an audio basic feature enhanced latent vector;
[0126] Step S602, performing a second enhancement process on the target harmonic feature latent vector to obtain a target harmonic feature enhanced latent vector;
[0127] Step S603, performing a third enhancement process on the merging result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain an enhanced merged feature vector;
[0128] Step S604: Merge the audio basic feature enhanced latent vector, the target harmonic feature enhanced latent vector, and the enhanced merged feature vector to obtain the target audio feature latent vector.
[0129] Steps S601 to S604 are described in detail below.
[0130] In step S601 of some embodiments, when performing the first enhancement processing on the audio basic feature latent vector, the audio basic feature latent vector is mapped from the high-dimensional vector space to the low-dimensional vector space to eliminate redundant information in the audio basic feature latent vector, obtain a more compact and specific audio basic feature representation, and obtain the audio basic feature enhanced latent vector.
[0131] In step S602 of some embodiments, when performing the second enhancement processing on the target harmonic feature latent vector, the target harmonic feature latent vector is mapped from the high-dimensional vector space to the low-dimensional vector space to eliminate redundant information in the target harmonic feature latent vector, obtain a more compact and specific target voiceprint feature representation, and obtain the target harmonic feature enhanced latent vector.
[0132] In step S603 of some embodiments, first, the audio basic feature latent vector and the target harmonic feature latent vector are vector-concatenated to obtain a merged result of the audio basic feature latent vector and the target harmonic feature latent vector, wherein the merged result is used to indicate the fusion of the audio basic feature information of the target audio and the voiceprint feature information of the target object in the latent space. Then, the merged result is mapped from the high-dimensional vector space to the low-dimensional vector space to eliminate redundant information in the merged result, obtain a more compact and specific feature representation, and obtain an enhanced merged feature vector.
[0133] In step S604 of some embodiments, feature merging is performed on the audio basic feature enhanced latent vector, the target harmonic feature enhanced latent vector, and the enhanced merged feature vector, and the audio basic feature enhanced latent vector, the target harmonic feature enhanced latent vector, and the enhanced merged feature vector are spliced into a longer feature vector to obtain the target audio feature latent vector, wherein the target audio feature latent vector includes not only the audio inherent feature information of the target audio and the voiceprint feature information of the target object, but also feature fusion information of the audio inherent feature information of the target audio and the voiceprint feature information of the target object.
[0134] The benefit of this embodiment is that the basic audio feature latent vector, the target harmonic feature latent vector, and the merged result of the basic audio feature latent vector and the target harmonic feature latent vector are respectively enhanced, which can effectively eliminate the redundant information in each latent vector, making the feature information in each latent vector simplified and effective. Furthermore, the enhanced audio basic feature latent vector, the enhanced target harmonic feature latent vector, and the enhanced merged feature vector are also feature merged, so that the target audio feature latent vector not only contains the local feature information of each part (the audio inherent feature information of the individual target audio, the voiceprint feature information of the target object), but also has the global feature information of the synthetic audio to be generated (the feature fusion information of the audio inherent feature information of the target audio and the voiceprint feature information of the target object), which can improve the comprehensiveness of the content of the target audio feature latent vector, facilitate the decoder to fully learn and fully extract the feature information distribution of the synthetic audio to be generated, thereby improving the accuracy of feature decoding.
[0135] See also Figure 7 In some embodiments, the voiceprint conversion model includes a decoder and a vocoder; step S106 may include but is not limited to steps S701 to S702:
[0136] Step S701, feature decoding is performed on the target audio feature latent vector based on a decoder to obtain an audio decoding sequence;
[0137] Step S702: Perform audio conversion on the audio decoding sequence based on the vocoder to obtain synthesized audio data.
[0138] Steps S701 to S702 are described in detail below.
[0139] The synthesized audio data is the target audio sung by the target object.
[0140] In step S701 of some embodiments, when feature decoding is performed on the target audio feature latent vector based on the decoder, key features are first extracted from the target audio feature latent vector based on the decoder to obtain feature information representing the inherent audio features of the target audio and feature information representing the voiceprint features of the target object, thereby obtaining key audio features. Next, based on the decoder, feature decoding is performed on the key audio features using a hidden Markov algorithm or a Gaussian mixture algorithm, and the key audio features are converted into a series of speech parameters (e.g., fundamental frequency, vocal channel features, and other parameters representing the time domain and frequency domain characteristics of the speech signal) to obtain an audio decoding sequence.
[0141] In step S702 of some embodiments, when performing audio conversion on an audio decoding sequence based on a vocoder, first, extract key information from the audio decoding sequence based on the vocoder to obtain audio decoding features, wherein the audio decoding features are used to indicate the fundamental frequency, spectrum envelope, non-periodic signal parameters, etc. of the synthetic audio to be generated. Next, reconstruct the audio decoding features based on the vocoder, adjust the fundamental frequency, spectrum envelope, non-periodic signal parameters, etc. in the audio decoding features, and generate a reconstructed audio waveform. Further, noise reduction and equalization are performed on the reconstructed audio waveform based on the vocoder to improve the naturalness and intelligibility of the audio, and obtain synthetic audio data.
[0142] The advantage of this embodiment is that a decoder is used to convert the target audio feature latent vector into an audio decoding sequence that can define the time domain and frequency domain characteristics of the synthetic audio to be generated, and then a vocoder is used to convert the audio decoding sequence into a waveform form. In this embodiment, two key feature extractions are introduced to eliminate the influence of irrelevant information on audio conversion. At the same time, after the audio waveform is generated, post-processing such as noise reduction and equalization is introduced, which can better improve the quality of the generated synthetic audio data and improve the audio quality of the generated synthetic audio data.
[0143] See also Figure 8 The present application also provides an audio processing device that can implement the above audio processing method. The device includes:
[0144] A receiving unit 801 is used to receive a voiceprint conversion request for a target audio from an interactive object, wherein the voiceprint conversion request is used to instruct to convert a current voiceprint feature of the target audio into a voiceprint feature of a target object;
[0145] An acquiring unit 802 is used to acquire a Mel-cepstrum feature of a target audio based on a voiceprint conversion request;
[0146] The encoding unit 803 is used to encode the Mel-frequency cepstrum feature based on a preset voiceprint conversion model to obtain an audio basic feature latent vector and an audio harmonic feature latent vector of the target audio;
[0147] An updating unit 804 is used to update the audio harmonic feature latent vector based on the pre-acquired reference voiceprint feature data to obtain a target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on a reference song sung by the target object;
[0148] An enhancement unit 805 is used to enhance the audio basic feature latent vector, the target harmonic feature latent vector, and the combined result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain a target audio feature latent vector;
[0149] The decoding unit 806 is used to decode the target audio feature latent vector based on the voiceprint conversion model to obtain synthesized audio data.
[0150] In some embodiments, the acquisition unit 802 is used to:
[0151] Performing Fourier transform on the target audio to obtain a Fourier transform result, and determining a target power spectrum based on the square of the modulus of the Fourier transform result;
[0152] Performing Mel filtering on the target power spectrum to obtain a target audio filtering result, and performing logarithmic compression on the target audio filtering result to obtain a target audio compression result;
[0153] The target audio compression result is discretized and mapped to obtain the Mel-frequency cepstrum feature.
[0154] In some embodiments, the voiceprint conversion model includes a first encoder and a second encoder;
[0155] The encoding unit 803 is used for:
[0156] Performing a first encoding on the Mel-frequency cepstrum feature based on a first encoder to obtain a basic audio feature latent vector;
[0157] The Mel-frequency cepstrum feature is second-encoded based on the second encoder to obtain an audio harmonic feature latent vector.
[0158] In some embodiments, the reference voiceprint feature data includes emotion feature data and phoneme feature data;
[0159] The updating unit 804 is used for:
[0160] Embed the emotional feature data to obtain the emotional embedding feature vector;
[0161] Performing harmonic feature extraction on the phoneme feature data to obtain a reference harmonic feature latent vector;
[0162] Perform feature concatenation on the emotion embedding feature vector and the reference harmonic feature latent vector to obtain a reference concatenated feature vector;
[0163] The audio harmonic feature latent vector is updated with the reference concatenated feature vector to obtain the target harmonic feature latent vector.
[0164] In some embodiments, the enhancement unit 805 is used to:
[0165] Performing a first enhancement process on the audio basic feature latent vector to obtain an audio basic feature enhanced latent vector;
[0166] Performing a second enhancement process on the target harmonic feature latent vector to obtain a target harmonic feature enhanced latent vector;
[0167] Performing a third enhancement process on the merged result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain an enhanced merged feature vector;
[0168] The audio basic feature enhanced latent vector, the target harmonic feature enhanced latent vector, and the enhanced merged feature vector are merged to obtain the target audio feature latent vector.
[0169] In some embodiments, the voiceprint conversion model includes a decoder and a vocoder;
[0170] The decoding unit 806 is used for:
[0171] Based on the decoder, feature decoding is performed on the target audio feature latent vector to obtain an audio decoding sequence;
[0172] The audio decoded sequence is converted into audio data based on the vocoder to obtain synthesized audio data.
[0173] In some embodiments, the reference voiceprint feature data is determined by:
[0174] Parse the voiceprint conversion request to obtain the interaction intention data of the interaction object;
[0175] Based on the interaction intention data, determining reference audio from a plurality of candidate songs sung by the target object;
[0176] Extract emotional features from the reference audio to obtain emotional feature data;
[0177] Extracting phoneme features from the reference audio to obtain phoneme feature data;
[0178] Integrate the emotional feature data and phoneme feature data into reference voiceprint feature data.
[0179] The specific implementation of the audio processing device is substantially the same as the specific implementation of the above-mentioned audio processing method, and will not be described in detail here.
[0180] The embodiment of the present application also provides an electronic device, the electronic device comprising: a memory, a processor, a program stored in the memory and executable on the processor, and a data bus for realizing connection and communication between the processor and the memory, and the program is executed by the processor to realize the above-mentioned audio processing method. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.
[0181] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:
[0182] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;
[0183] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the audio processing method of the embodiment of this application;
[0184] Input / output interface 903, used to implement information input and output;
[0185] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);
[0186] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);
[0187] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .
[0188] An embodiment of the present application also provides a computer-readable storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the above-mentioned audio processing method.
[0189] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.
[0190] The audio processing method, audio processing device, electronic device and computer-readable storage medium provided by the embodiment of the present application receive a voiceprint conversion request for the target audio from the interactive object, wherein the voiceprint conversion request is used to indicate that the current voiceprint feature of the target audio is converted into the voiceprint feature of the target object. Further, based on the voiceprint conversion request, the Mel-cepstral feature of the target audio is obtained, and the Mel-cepstral feature is encoded based on the preset voiceprint conversion model to obtain the audio basic feature latent vector and the audio harmonic feature latent vector of the target audio, and the different feature information of the target audio is encoded respectively. Then, the audio harmonic feature latent vector is updated based on the pre-acquired reference voiceprint feature data to obtain the target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on the reference audio sung by the target object. This method can flexibly adjust and optimize the audio harmonic feature latent vector through the voiceprint characteristics of the target object contained in the reference voiceprint feature data, so as to realize the rapid conversion of the voiceprint feature and improve the accuracy of the voiceprint conversion of the target audio. Furthermore, the audio basic feature latent vector, the target harmonic feature latent vector, and the combined result of the audio basic feature latent vector and the target harmonic feature latent vector are enhanced to obtain the target audio feature latent vector; and the target audio feature latent vector is decoded based on the voiceprint conversion model to obtain the synthesized audio data. The enhancement link of multiple latent vectors is introduced before the decoding link, which can improve the utilization efficiency of the effective information of each latent vector, improve the feature accuracy and feature comprehensiveness of the target audio feature latent vector used for decoding, and thus improve the audio quality of the decoded synthesized audio data. In addition, the present application is a voiceprint conversion based on human-computer interaction, which can meet the real-time interaction needs of various objects at different times and in different scenarios.
[0191] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.
[0192] It can be understood by those skilled in the art that Figure 1-7 The technical solutions shown in the figure do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than those shown in the figure, or a combination of certain steps, or different steps.
[0193] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0194] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.
[0195] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.
[0196] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.
[0197] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0198] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
[0199] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0200] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.
[0201] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.
Claims
1. An audio processing method, characterized in that: The method comprises: Receiving a voiceprint conversion request for a target audio from an interactive object, wherein the voiceprint conversion request is used to instruct to convert a current voiceprint feature of the target audio into a voiceprint feature of a target object; Based on the voiceprint conversion request, obtaining a Mel-cepstrum feature of the target audio; Encoding the Mel-cepstrum feature based on a preset voiceprint conversion model to obtain an audio basic feature latent vector and an audio harmonic feature latent vector of the target audio; The audio harmonic feature latent vector is updated based on the pre-acquired reference voiceprint feature data to obtain a target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on the reference audio sung by the target object; Performing enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and a combination result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain a target audio feature latent vector; The target audio feature latent vector is decoded based on the voiceprint conversion model to obtain synthesized audio data.
2. The method according to claim 1, characterized in that The voiceprint conversion model includes a first encoder and a second encoder; The encoding process of the Mel-cepstrum feature based on the preset voiceprint conversion model to obtain the audio basic feature latent vector and the audio harmonic feature latent vector of the target audio includes: Performing a first encoding on the Mel-frequency cepstrum feature based on the first encoder to obtain the audio basic feature latent vector; The Mel-frequency cepstrum feature is second-encoded based on the second encoder to obtain the audio harmonic feature latent vector.
3. The method according to claim 1, characterized in that The reference voiceprint feature data includes emotion feature data and phoneme feature data; The updating of the audio harmonic feature latent vector based on the pre-acquired reference voiceprint feature data to obtain the target harmonic feature latent vector includes: Embedding the emotion feature data to obtain an emotion embedding feature vector; Performing harmonic feature extraction on the phoneme feature data to obtain a reference harmonic feature latent vector; Performing feature concatenation on the emotion embedding feature vector and the reference harmonic feature latent vector to obtain a reference concatenated feature vector; The audio harmonic feature latent vector is updated using the reference concatenated feature vector to obtain the target harmonic feature latent vector.
4. The method according to claim 1, characterized in that The step of performing enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and a combination result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain a target audio feature latent vector includes: Performing a first enhancement process on the audio basic feature latent vector to obtain an audio basic feature enhanced latent vector; Performing a second enhancement process on the target harmonic feature latent vector to obtain a target harmonic feature enhanced latent vector; Performing a third enhancement process on a result of combining the audio basic feature latent vector and the target harmonic feature latent vector to obtain an enhanced combined feature vector; The audio basic feature enhanced latent vector, the target harmonic feature enhanced latent vector, and the enhanced merged feature vector are feature merged to obtain the target audio feature latent vector.
5. The method according to claim 1, characterized in that The voiceprint conversion model includes a decoder and a vocoder; The decoding process of the target audio feature latent vector based on the voiceprint conversion model to obtain synthesized audio data includes: Performing feature decoding on the target audio feature latent vector based on the decoder to obtain an audio decoding sequence; The audio decoding sequence is audio-converted based on the vocoder to obtain the synthesized audio data.
6. The method according to claim 1, characterized in that The reference voiceprint feature data is determined by: Parsing the voiceprint conversion request to obtain interaction intention data of the interaction object; Based on the interaction intention data, determining the reference audio from a plurality of candidate songs sung by the target object, wherein the reference audio and the target audio have the same audio type; Extracting emotional features from the reference audio to obtain emotional feature data; Extracting phoneme features from the reference audio to obtain phoneme feature data; The emotion feature data and the phoneme feature data are integrated into the reference voiceprint feature data.
7. The method according to any one of claims 1 to 6, characterized in that: The obtaining, based on the voiceprint conversion request, the Mel-cepstrum feature of the target audio includes: Performing Fourier transform on the target audio to obtain a Fourier transform result, and determining a target power spectrum based on the square of the modulus of the Fourier transform result; Performing Mel filtering on the target power spectrum to obtain a target audio filtering result, and performing logarithmic compression on the target audio filtering result to obtain a target audio compression result; Discretization mapping is performed on the target audio compression result to obtain the Mel-frequency cepstrum feature.
8. An audio processing device, characterized in that: The device comprises: A receiving unit, configured to receive a voiceprint conversion request from an interactive object for a target audio, wherein the voiceprint conversion request is used to instruct to convert a current voiceprint feature of the target audio into a voiceprint feature of a target object; An acquiring unit, configured to acquire a Mel-cepstrum feature of the target audio based on the voiceprint conversion request; An encoding unit, configured to encode the Mel-cepstrum feature based on a preset voiceprint conversion model to obtain an audio basic feature latent vector and an audio harmonic feature latent vector of the target audio; an updating unit, configured to update the audio harmonic feature latent vector based on pre-acquired reference voiceprint feature data to obtain a target harmonic feature latent vector, wherein the reference voiceprint feature data is determined based on a reference song sung by the target object; an enhancement unit, configured to perform enhancement processing on the audio basic feature latent vector, the target harmonic feature latent vector, and a combination result of the audio basic feature latent vector and the target harmonic feature latent vector to obtain a target audio feature latent vector; A decoding unit is used to decode the target audio feature latent vector based on the voiceprint conversion model to obtain synthesized audio data.
9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the audio processing method according to any one of claims 1 to 7 when executing the computer program.
10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the audio processing method according to any one of claims 1 to 7 is implemented.
Citation Information
Cited By
Human voice quality optimization method and device, equipment and medium
CN120375838A
