Voice optimization method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a voice optimization method and device, electronic equipment and a storage medium, and belongs to the technical field of voice optimization and artificial intelligence. The method comprises the following steps: acquiring current audio data of a target object, and performing feature extraction on the current audio data by using a pre-constructed target voice optimization model to obtain current timbre features of the target object; obtaining target music score data according to the current audio data, and performing feature extraction on the target music score data to obtain target tone features; performing feature splicing on the current tone feature and the target tone feature to obtain a spliced audio feature; performing random noise adding processing on the spliced audio features to obtain noise-added audio features; and performing voice optimization on the current audio data by using the target voice optimization model based on the noise-added audio characteristics. According to the embodiment of the invention, the voice optimization efficiency can be improved.

Description

Technical Field

[0001] The present application relates to the field of speech optimization and artificial intelligence technology, and in particular to a speech optimization method and device, an electronic device and a storage medium. Background Art

[0002] In the postoperative rehabilitation scenario in the medical field, patients can relax through entertainment, thereby speeding up their recovery. For example, patients can relax by singing.

[0003] When patients sing, they are often out of tune, causing the patient's singing pitch to be inconsistent with the standard repertoire, which can easily reduce the patient's mood and comfort, thereby slowing down the patient's recovery. At present, the common voice optimization method is usually for the tuner to manually adjust the pitch in the voice data, but this method involves a large amount of manual operation, resulting in low voice optimization efficiency. Therefore, how to improve the efficiency of voice optimization has become a technical problem that needs to be solved urgently. Summary of the invention

[0004] The main purpose of the embodiments of the present application is to propose a voice optimization method and device, an electronic device and a storage medium, aiming to improve the efficiency of voice optimization.

[0005] To achieve the above object, a first aspect of an embodiment of the present application proposes a voice optimization method, the method comprising:

[0006] Acquire current audio data of a target object, and use a pre-built target speech optimization model to perform feature extraction on the current audio data to obtain current timbre features of the target object;

[0007] Acquire target music score data according to the current audio data, and perform feature extraction on the target music score data to obtain target tone features;

[0008] Performing feature splicing on the current timbre feature and the target tone feature to obtain a spliced ​​audio feature;

[0009] Performing random noise processing on the spliced ​​audio features to obtain noisy audio features;

[0010] Based on the noise-added frequency features, the target speech optimization model is used to perform speech optimization on the current audio data.

[0011] In some embodiments, the method further includes training the target speech optimization model, specifically including:

[0012] Obtain audio sample data and music score sample data;

[0013] Performing spectrum conversion on the audio sample data to obtain an initial Mel spectrum;

[0014] Based on the pre-built initial speech optimization model, the audio sample data, the music score sample data and the preset random Gaussian noise, the initial Mel spectrum is forwardly noised to obtain a noisy Mel spectrum;

[0015] Based on the noisy Mel spectrum and the random Gaussian noise, a preset initial noise prediction model is trained to obtain a target noise prediction model;

[0016] Based on the target noise prediction model, performing reverse denoising processing on the noisy Mel spectrum to obtain a denoised Mel spectrum;

[0017] Calculating mel spectrum loss data based on the initial mel spectrum, the denoised mel spectrum and a preset mel spectrum loss function;

[0018] Based on the Mel spectrum loss data, the parameters of the initial speech optimization model are adjusted to obtain the target speech optimization model.

[0019] In some embodiments, the forward noise addition processing is performed on the initial Mel spectrum based on the pre-built initial speech optimization model, the audio sample data, the music score sample data and the preset random Gaussian noise to obtain the noisy Mel spectrum, including:

[0020] Based on the initial speech optimization model and the random Gaussian noise, the initial Mel spectrum is subjected to noise injection processing to obtain a noise-injected Mel spectrum;

[0021] Extracting features from the audio sample data to obtain timbre features, and extracting features from the music score sample data to obtain pitch features;

[0022] The timbre feature, the pitch feature and the noise-injected Mel spectrum are feature-concatenated to obtain the noise-added Mel spectrum.

[0023] In some embodiments, the step of performing noise injection processing on the initial Mel spectrum based on the initial speech optimization model and the random Gaussian noise to obtain a noise-injected Mel spectrum includes:

[0024] Obtaining the number of noise additions of the initial Mel spectrum and the noise addition time step of the initial speech optimization model, wherein the noise addition time step represents the index of the number of noise additions;

[0025] Based on a preset noise adjustment strategy and the number of noise additions, generating a noise adjustment coefficient of the initial speech optimization model;

[0026] Based on the noise adjustment coefficient and the noise addition time step, the random Gaussian noise and the initial Mel spectrum are combined to obtain the noise-injected Mel spectrum.

[0027] In some embodiments, performing reverse denoising on the noisy Mel spectrum based on the target noise prediction model to obtain a denoised Mel spectrum includes:

[0028] Using the target noise prediction model to perform noise prediction on the noisy Mel spectrum to obtain target noise;

[0029] Determining a non-noised feature from the noisy Mel spectrum based on the timbre feature and the pitch feature;

[0030] Based on the non-noised feature, masking the noisy Mel spectrum to obtain a noisy feature;

[0031] Based on the target noise, the noise-added feature is de-noised to obtain the denoised Mel spectrum.

[0032] In some embodiments, performing spectrum conversion on the audio sample data to obtain an initial Mel spectrum includes:

[0033] Performing Fourier transform on the audio sample data to obtain audio spectrum data;

[0034] Mel-scale mapping is performed on the audio spectrum data to obtain the initial Mel-scale spectrum.

[0035] In some embodiments, the randomly adding noise to the concatenated audio features to obtain the noisy audio features includes:

[0036] Constructing a noise space, wherein the noise space contains noise data;

[0037] Randomly selecting the noise data to obtain the target injection noise;

[0038] Noise is injected into the concatenated audio feature based on the target injected noise to obtain the noisy audio feature.

[0039] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speech optimization device, the device comprising:

[0040] A video acquisition module, used to acquire current video data containing a target insured object, wherein the target insured object includes an animal;

[0041] A target detection module, used to perform target detection based on the current video data to obtain a target image of the target insured object;

[0042] A key point extraction module is used to perform skeleton positioning on the target image to obtain the object skeleton key points;

[0043] A skeleton assessment module, used to perform skeleton posture assessment on the target insured object based on the skeleton key points of the object to obtain a skeleton posture score;

[0044] An obesity assessment module, used to assess the body obesity of the target insured person based on the current video data and the object's skeletal key points to obtain an obesity score;

[0045] An appearance assessment module, configured to perform an appearance hair assessment on the target insured person based on the current video data to obtain an appearance hair score;

[0046] A health status assessment module, used to perform a health assessment on the target insured person based on the skeletal posture score, the obesity score and the appearance hair score. A timbre feature extraction module, used to obtain the current audio data of the target object, perform feature extraction on the current audio data using a pre-built target voice optimization model, and obtain the current timbre feature of the target object;

[0047] A tone feature extraction module, used for acquiring target music score data according to the current audio data, and performing feature extraction on the target music score data to obtain target tone features;

[0048] An audio feature splicing module, used for performing feature splicing on the current timbre feature and the target tone feature to obtain a spliced ​​audio feature;

[0049] An audio feature noise adding module, used for performing random noise adding processing on the spliced ​​audio features to obtain a noise added audio feature;

[0050] The audio data optimization module is used to perform speech optimization on the current audio data based on the noise-added frequency characteristics and using the target speech optimization model.

[0051] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0052] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0053] The speech optimization method and device, electronic device and storage medium proposed in the present application perform feature extraction on the current audio data of the target object by using a pre-built target speech optimization model to obtain the current timbre feature of the target object, then obtain the target music score data according to the current audio data, and perform feature extraction on the target music score data to obtain the target pitch feature, clarify the correct pitch of the audio sung by the target object, further, perform feature splicing on the current timbre feature and the target pitch feature to obtain the spliced ​​audio feature, perform random noise processing on the spliced ​​audio feature to obtain the noisy audio feature, and perform speech optimization on the current audio data according to the noisy audio feature using the target speech optimization model, so as to replace the pitch in the current audio data with the target pitch data in the target music score data, keep the current timbre feature of the target object unchanged, so that the current audio data of the target object is optimized, and the steps of manually adjusting the pitch of the audio data are reduced, thereby improving the efficiency of speech optimization. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 is a flow chart of the voice optimization method provided by an embodiment of the present application;

[0055] Figure 2 is a flow chart of a target speech optimization model training method provided by another embodiment of the present application;

[0056] Figure 3 yes Figure 2 Flow chart of step S202 in FIG.

[0057] Figure 4 yes Figure 2 Flow chart of step S203 in FIG.

[0058] Figure 5 yes Figure 4 Flow chart of step S401 in FIG.

[0059] Figure 6 yes Figure 2 Flow chart of step S205 in FIG.

[0060] Figure 7 yes Figure 1 Flow chart of step S104 in FIG.

[0061] Figure 8 is a structural diagram of a speech optimization device provided in an embodiment of the present application;

[0062] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0063] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0064] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0065] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0066] First, some nouns involved in this application are analyzed:

[0067] Diffusion model: Diffusion model is a generative model that generates data from random noise by gradually adding and removing noise. Its main benefits include the ability to generate high-quality and detailed samples, strong flexibility and diversity, relatively stable training process, and no prone to mode collapse. In addition, the diffusion model is easy to control the generation process, shows good robustness, and can effectively handle noisy data. These characteristics make it perform well in fields such as image generation and text generation, and it has become an important tool for generation tasks in recent years.

[0068] In the postoperative rehabilitation scenario in the medical field, patients can relax through entertainment, thereby speeding up their recovery. For example, patients can relax by singing.

[0069] When patients sing, they are often out of tune, causing the patient's singing pitch to be inconsistent with the standard repertoire, which can easily reduce the patient's mood and comfort, thereby slowing down the patient's recovery. At present, the common voice optimization method is usually for the tuner to manually adjust the pitch in the voice data, but this method involves a large amount of manual operation, resulting in low voice optimization efficiency. Therefore, how to improve the efficiency of voice optimization has become a technical problem that needs to be solved urgently.

[0070] Based on this, the embodiments of the present application provide a voice optimization method and device, an electronic device and a storage medium, aiming to improve the efficiency of voice optimization.

[0071] The speech optimization method and device, electronic device and storage medium provided in the embodiments of the present application are specifically illustrated through the following embodiments. First, the speech optimization method in the embodiments of the present application is described.

[0072] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0073] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0074] The speech optimization method provided in the embodiment of the present application relates to the field of speech optimization and artificial intelligence technology. The speech optimization method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speech optimization method, etc., but is not limited to the above forms.

[0075] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0076] Figure 1 is an optional flow chart of the voice optimization method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S105.

[0077] Step S101, obtaining current audio data of a target object, and extracting features of the current audio data using a pre-built target speech optimization model to obtain current timbre features of the target object;

[0078] Step S102, acquiring target music score data according to the current audio data, and performing feature extraction on the target music score data to obtain target tone features;

[0079] Step S103, performing feature splicing on the current timbre feature and the target tone feature to obtain a spliced ​​audio feature;

[0080] Step S104, performing random noise processing on the spliced ​​audio features to obtain noisy audio features;

[0081] Step S105: Based on the noise-added frequency features, the target speech optimization model is used to perform speech optimization on the current audio data.

[0082] In steps S101 to S105 shown in the embodiment of the present application, a pre-constructed target speech optimization model is used to perform feature extraction on the acquired current audio data of the target object to obtain the current timbre features of the target object, and then the corresponding target music score data is obtained based on the above current audio data, and the target speech optimization model is used to perform feature extraction on the target music score data to obtain the target pitch features. Furthermore, the current timbre features and the target pitch features are feature spliced ​​to obtain spliced ​​audio features. Secondly, the spliced ​​audio features are randomly noised to obtain noisy audio features. Finally, based on the noisy audio features, the target speech optimization model is used to perform speech optimization on the current audio data. Therefore, the present application utilizes a pre-built target speech optimization model to perform feature extraction on the current audio data of the target object to obtain the current timbre features of the target object, then obtains the target music score data based on the current audio data, and performs feature extraction on the target music score data to obtain the target pitch features, and clarifies the correct pitch of the audio sung by the target object. Furthermore, the current timbre features and the target pitch features are feature spliced ​​to obtain spliced ​​audio features, and the spliced ​​audio features are randomly noised to obtain noisy audio features. According to the noisy audio features, the target speech optimization model is used to perform speech optimization on the current audio data, so as to replace the pitch in the current audio data with the target pitch data in the target music score data, keep the current timbre features of the target object unchanged, optimize the current audio data of the target object, reduce the steps of manually adjusting the pitch of the audio data, and improve the efficiency of speech optimization.

[0083] In step S101 of some embodiments, the current audio data may be a fragment of a song audio or a complete song audio containing the timbre of the target object. The target speech optimization model may be a diffusion model, which is used to perform speech optimization on the current audio data. The target speech optimization model includes a noise scheduling module, a feature extraction module, a denoising network module and a spectrum generation module, wherein the noise scheduling module can determine the noise level of the audio data input to the target speech optimization model at each time step, and is used to add noise or remove noise from the audio data input to the target speech optimization model. The feature extraction module is used to extract features from the audio data input to the target speech optimization model. The feature extraction module may generally be a deep learning model such as a convolutional neural network. The denoising network module is responsible for restoring the audio data input to the target speech optimization model according to the noise and feature input at the current time step. The denoising network module may be constructed using a transformer or a convolutional network. The spectrum generation module performs a generation process from noise to spectrum data. The spectrum generation module may use the output result of the denoising network module to gradually generate optimized audio data.

[0084] The present application can obtain the current audio data of the target object through a device with audio recording function such as a microphone, a recorder, or a camera, and then input the current audio data into a pre-trained target speech optimization model, and use the feature extraction module of the target optimization model to extract the timbre features of the current audio data to obtain the current timbre features of the target object.

[0085] In one application scenario, taking the singing of a patient in the postoperative recovery period as an example, the song sung by the patient is recorded by a recorder or camera and other equipment, so that the current singing data containing the patient's timbre characteristics can be obtained, and then the obtained current singing data is input into the trained target speech optimization model to extract the timbre characteristics, so that the current timbre characteristics of the patient when recording the current singing data can be obtained.

[0086] It is important to note that before using the target speech optimization model to extract the current timbre features of the target object in the current audio data, it is necessary to complete model training of the target speech optimization model to improve the speech optimization accuracy of the current audio data.

[0087] See also Figure 2 In some embodiments, the training method of the target speech optimization model in step S101 may include but is not limited to steps S201 to S207:

[0088] Step S201, obtaining audio sample data and music score sample data;

[0089] Step S202, performing spectrum conversion on the audio sample data to obtain an initial Mel spectrum;

[0090] Step S203, based on the pre-built initial speech optimization model, audio sample data, music score sample data and preset random Gaussian noise, the initial Mel spectrum is forwardly noised to obtain a noisy Mel spectrum;

[0091] Step S204, training a preset initial noise prediction model based on the noisy Mel spectrum and random Gaussian noise to obtain a target noise prediction model;

[0092] Step S205, based on the target noise prediction model, reverse denoising is performed on the noisy Mel spectrum to obtain a denoised Mel spectrum;

[0093] Step S206, calculating Mel spectrum loss data based on the initial Mel spectrum, the denoised Mel spectrum and a preset Mel spectrum loss function;

[0094] Step S207: adjusting parameters of the initial speech optimization model based on the Mel spectrum loss data to obtain a target speech optimization model.

[0095] In step S201 of some embodiments, the audio sample data may be the audio data of a song sung by any object. The music score sample data may be the music score data of the song in the audio sample data.

[0096] The embodiment of the present application can obtain audio sample data with non-standard pitch through a recording device, and can also obtain audio sample data with non-standard pitch through a network query. Further, the music score sample data can be obtained by querying the music score of the song in the audio sample data.

[0097] In step S202 of some embodiments, before training the target speech optimization model, in order to facilitate data processing of the target speech optimization model and improve the model calculation efficiency, it is necessary to convert the format of the audio sample data into a spectrum format. Therefore, it is also necessary to perform spectrum conversion on the audio sample data to obtain audio sample data in a spectrum format, that is, an initial Mel spectrum.

[0098] For details, see Figure 3 In some embodiments, step S202 may include but is not limited to steps S301 to S302:

[0099] Step S301, performing Fourier transform on the audio sample data to obtain audio spectrum data;

[0100] Step S302: Perform Mel-scale mapping on the audio spectrum data to obtain an initial Mel-scale spectrum.

[0101] In step S301 of some embodiments, audio sample data suitable for spectrum conversion is obtained by preprocessing the audio sample data, wherein the preprocessing operation may be resampling, normalization and the like. Further, the preprocessed audio sample data is divided into small overlapping frames to facilitate Fourier transform of the audio sample data. Finally, Fourier transform is used to convert each frame in the overlapping frames into a frequency domain signal to obtain audio spectrum data.

[0102] In step S302 of some embodiments, the audio spectrum data is mapped to a pre-constructed Mel scale space, so as to obtain an initial Mel spectrum of the audio sample data, wherein the Mel scale is a non-linear frequency scale.

[0103] In step S301 to step S302 shown in this embodiment, audio spectrum data is obtained by performing Fourier transform on the Mel spectrum, and then Mel scale mapping is performed on the audio spectrum data to obtain an initial Mel spectrum, so that the initial Mel spectrum is more consistent with human auditory characteristics.

[0104] In step S203 of some embodiments, the random Gaussian noise may be noise data randomly selected from Gaussian noise with a normal distribution, and the random Gaussian noise may be randomly generated by a noise scheduling module in the initial speech optimization model.

[0105] After obtaining the initial Mel spectrum of the audio sample data, the initial Mel spectrum, the audio sample data and the music score sample data are input into a pre-built initial speech optimization model, and then the randomly selected random Gaussian noise is added to the initial Mel spectrum to obtain the noisy Mel spectrum.

[0106] See also Figure 4 In some embodiments, step S203 may include but is not limited to steps S401 to S403:

[0107] Step S401, based on the initial speech optimization model and random Gaussian noise, the initial Mel spectrum is subjected to noise injection processing to obtain a noise-injected Mel spectrum;

[0108] Step S402, extracting features from the audio sample data to obtain timbre features, and extracting features from the music score sample data to obtain pitch features;

[0109] Step S403, feature concatenating the timbre feature, the pitch feature and the noise-injected Mel spectrum to obtain the noise-added Mel spectrum.

[0110] In step S401 of some embodiments, the initial Mel spectrum is synthesized with randomly selected random Gaussian noise using the initial speech optimization model to obtain a noisy Mel spectrum containing random Gaussian noise.

[0111] See also Figure 5 In some embodiments, step S401 may include but is not limited to steps S501 to S503:

[0112] Step S501, obtaining the number of noise additions of the initial Mel spectrum and the noise addition time step of the initial speech optimization model, wherein the noise addition time step represents the index of the number of noise additions;

[0113] Step S502, generating a noise adjustment coefficient of an initial speech optimization model based on a preset noise adjustment strategy and a number of noise additions;

[0114] Step S503: Based on the noise adjustment coefficient and the noise addition time step, the random Gaussian noise and the initial Mel spectrum are combined to obtain a noise-injected Mel spectrum.

[0115] In step S501 of some embodiments, before the initial Mel spectrum is subjected to noise injection processing, in order to determine the random Gaussian noise that needs to be added in each step, it is necessary to determine the number of times the initial Mel spectrum is noised in the initial speech optimization model. In order to facilitate determination of the number of noise adding steps of the initial speech optimization model at the current moment, it is also necessary to obtain the index of the number of noise adding times of the initial speech optimization model, that is, the noise adding time step.

[0116] In step S502 of some embodiments, the preset noise adjustment strategy may be a preset random Gaussian noise selection strategy, and the noise adjustment strategy includes linear attenuation, exponential attenuation, cosine attenuation and other strategies.

[0117] The embodiment of the present application can generate an array containing the number of noise addition times according to a preset noise adjustment strategy. The value in the array is the noise adjustment coefficient. It should be noted that the noise adjustment coefficient is generally greater than 0 and less than 1.

[0118] In step S503 of some embodiments, after determining the noise adjustment coefficient and the noise addition time step of the initial speech optimization model, the noise-injected Mel spectrum after the random Gaussian noise is combined with the initial Mel spectrum can be calculated using the following formula:

[0119]

[0120] Where t represents the noise adding time step, x t represents the Mel spectrum with a noise injection time step of t, α t represents the noise adjustment coefficient with a noise adding time step of t, x0 represents the initial Mel spectrum, and ∈ represents random Gaussian noise.

[0121] In steps S501 to S503 shown in this embodiment, the noise adjustment coefficient of each noise adding time step of the initial speech optimization model can be determined according to the obtained number of noise adding times of the initial Mel spectrum, the noise adding time step of the initial speech optimization model and the pre-set noise adjustment strategy, and then the random Gaussian noise and the initial Mel spectrum are merged according to the noise adjustment coefficient and the noise adding time step, so that the random Gaussian noise can be clearly generated, the intensity of the random Gaussian noise can be adjusted, and the random Gaussian noise can be effectively merged with the Mel spectrum to obtain the noise injected Mel spectrum.

[0122] In step S402 of some embodiments, in order to achieve speech optimization of the audio sample data, it is necessary to ensure that the timbre in the audio sample data remains unchanged and the pitch becomes more standard. Therefore, it is necessary to use an initial speech optimization model to extract the timbre features of the audio sample data, and to use an initial speech optimization model to extract the pitch features of the music score sample data.

[0123] In step S403 of some embodiments, the noise-injected Mel spectrum is feature-concatenated with the timbre features of the audio sample data to obtain concatenated feature data, and then the pitch features of the music score sample data are feature-concatenated with the concatenated feature data to obtain the noise-added Mel spectrum.

[0124] It should be noted that the forward noise addition process includes multiple noise addition process steps, and the method of each noise addition process step is similar, so the above steps S401 to S403 only describe the method of one noise addition process step.

[0125] In steps S401 to S403 shown in this embodiment, the initial Mel spectrum is synthesized with the random Gaussian noise randomly generated in the initial speech optimization model to obtain a noise-injected noisy Mel spectrum, and then the noisy Mel spectrum is feature-spliced ​​with the timbre features of the audio sample data and the pitch features of the music score sample data to obtain a noisy Mel spectrum, which can improve the randomness of the noisy Mel spectrum and ensure that the timbre and pitch in the noisy Mel spectrum are the spliced ​​timbre features and pitch features.

[0126] In step S204 of some embodiments, the initial noise prediction model may be a neural network model for predicting noise in spectrum data, and the initial noise prediction model usually only includes a coding layer and a decoding layer. In the process of forward noise addition processing of the initial Mel spectrum, the input data of the initial noise prediction model can be obtained by acquiring the random Gaussian noise of each time step and acquiring the noisy Mel spectrum of the time step, and the random Gaussian noise of each time step and the noisy Mel spectrum data form a set of training data of the initial noise prediction model, so as to realize model training of the initial noise prediction model, thereby obtaining a trained target noise prediction model.

[0127] In step S205 of some embodiments, the noise in the noisy Mel spectrum is predicted using a trained target noise prediction model, and the noisy Mel spectrum is inversely denoised according to the predicted noise to obtain a denoised Mel spectrum.

[0128] See also Figure 6 In some embodiments, step S205 may include but is not limited to steps S601 to S604:

[0129] Step S601, using a target noise prediction model to perform noise prediction on the noisy Mel spectrum to obtain target noise;

[0130] Step S602, determining non-noised features from the noisy Mel spectrum based on the timbre features and the pitch features;

[0131] Step S603, masking the noisy Mel spectrum based on the non-noise feature to obtain the noisy feature;

[0132] Step S604: based on the target noise, the noise-added feature is de-noised to obtain a denoised Mel-spectrogram.

[0133] In step S601 of some embodiments, the noisy Mel spectrum is input into a trained target noise prediction model for noise prediction, and the noise part in the noisy Mel spectrum, ie, the target noise, can be obtained.

[0134] In step S602 of some embodiments, the noisy Mel spectrum is formed by concatenating the timbre feature, the pitch feature and the noisy Mel spectrum. Therefore, in the noisy Mel spectrum, the timbre feature and the pitch feature do not have noise added. Furthermore, according to the timbre feature and the pitch feature, the non-denoised part of the noisy Mel spectrum that does not need to be denoised, i.e., the non-noised feature, can be determined.

[0135] In step S603 of some embodiments, after the non-noised features in the noisy Mel spectrum are masked, the unmasked portion of the noisy Mel spectrum is the noisy features.

[0136] In step S604 of some embodiments, the target noise is eliminated from the noisy feature to obtain a denoised Mel spectrum.

[0137] It should be noted that, similar to the forward denoising method of the initial Mel spectrum, the reverse denoising process of the noisy Mel spectrum also includes multiple denoising steps, and the method of each denoising step is similar, so the above steps S601 to S604 only describe the method of one denoising step.

[0138] In steps S601 to S604 shown in this embodiment, the target noise prediction model is used to predict the target noise in the noisy Mel spectrum, and then the noisy Mel spectrum is de-noised according to the target noise, timbre characteristics and pitch characteristics to obtain a denoised Mel spectrum, which can significantly improve the clarity and realism of the audio generated by the model and make the generated result closer to the audio sample data.

[0139] In step S206 of some embodiments, the loss between the denoised mel spectrum and the initial mel spectrum is calculated using a pre-designed mel spectrum loss function to obtain mel spectrum loss data.

[0140] In step S207 of some embodiments, after obtaining the Mel spectrum loss data, the Mel spectrum loss data is compared with a pre-set loss value threshold. If the Mel spectrum loss data is greater than or equal to the loss value threshold, the parameters in the initial speech optimization model are adjusted according to the Mel spectrum loss data, and the above steps S203 to S206 are repeated until the Mel spectrum loss data between the denoised Mel spectrum and the initial Mel spectrum is less than the loss value threshold, so that the initial speech optimization model is more suitable for speech optimization of the audio sample data. If the Mel spectrum loss data is less than the loss value threshold, the initial speech optimization model in the current state is used as the target speech optimization model suitable for speech optimization of the audio sample data.

[0141] In steps S201 to S207 shown in this embodiment, by adding random Gaussian noise to the initial Mel spectrum of the audio sample data, and then determining the noisy Mel spectrum based on the audio sample data and the music score sample data, the model can generate more diverse samples, and then the noisy Mel spectrum is reversely denoised based on the target noise prediction model, which makes full use of randomness and improves the performance of the target speech optimization model and the quality of the generated spectrum data.

[0142] In step S102 of some embodiments, the target music score data may be music score data corresponding to a song in the current audio data.

[0143] The embodiment of the present application can utilize the feature extraction module in the target speech optimization model to extract the pitch features in the target music score data to obtain the target pitch features.

[0144] In step S103 of some embodiments, the current timbre feature of the target object extracted by the target speech optimization model is concatenated with the target pitch feature to obtain a concatenated audio feature, so as to achieve the purpose of replacing the pitch in the current audio data.

[0145] In step S104 of some embodiments, after the concatenated audio features are obtained, in order to improve the quality of the audio data generated by the target speech optimization model, it is necessary to perform one or more random noise addition processes on the concatenated audio features.

[0146] See also Figure 7 In some embodiments, step S104 may include but is not limited to steps S701 to S703:

[0147] Step S701, constructing a noise space, wherein the noise space contains noise data;

[0148] Step S702, randomly selecting noise data to obtain target injection noise;

[0149] Step S703: inject noise into the concatenated audio features based on the target injected noise to obtain a noisy audio feature.

[0150] In step S701 of some embodiments, the noise type in the noise space to be constructed is first determined, and then a mathematical model or function is used to generate noise data of the noise type, and finally, the generated noise data is integrated to obtain the noise space. It should be noted that noise types generally include Gaussian noise, impulse noise, speckle noise, salt and pepper noise, white noise, etc. The noise type of the noise data required in the noise space in the embodiment of the present application is Gaussian noise, and the mathematical model or function for generating noise data of different noise types can be a mathematical model or function in a library such as numpy and scipy in the Python programming language.

[0151] In step S702 of some embodiments, one or more noise data are randomly selected from the noise space as target injection data for splicing audio features.

[0152] In step S703 of some embodiments, the target injected noise is synthesized with the spliced ​​audio feature to obtain a noise-added audio feature after noise addition processing.

[0153] In steps S701 to S702 shown in this embodiment, noise is injected into the spliced ​​audio features using the target injection noise in the preset noise space to obtain the noisy audio features, which can ensure that the spliced ​​audio features can be denoised in the target speech optimization model, thereby improving the quality of the audio generated by the target speech optimization model.

[0154] In step S105 of some embodiments, after obtaining the noisy audio features, the noisy audio features can be reversely denoised through a denoising network module. It should be noted that the reverse denoising method of the noisy audio features is similar to the method in steps S601 to S602 above, so it is not repeated here. In addition, the spectrum generation module can also be used to generate audio data containing current timbre features and target pitch features, thereby achieving voice optimization of the current audio data.

[0155] The present application utilizes a pre-built target speech optimization model to perform feature extraction on the current audio data of the target object to obtain the current timbre features of the target object, then obtains target music score data based on the current audio data, and performs feature extraction on the target music score data to obtain target pitch features, thereby clarifying the correct pitch of the audio sung by the target object, and further, performs feature splicing on the current timbre features and the target pitch features to obtain spliced ​​audio features, performs random noise processing on the spliced ​​audio features to obtain noisy audio features, and performs speech optimization on the current audio data based on the noisy audio features using the target speech optimization model, thereby replacing the pitch in the current audio data with the target pitch data in the target music score data, keeping the current timbre features of the target object unchanged, thereby optimizing the current audio data of the target object, reducing the steps of manually adjusting the pitch of the audio data, and improving the efficiency of speech optimization.

[0156] See also Figure 8 The embodiment of the present application also provides a voice optimization device, which can implement the above-mentioned voice optimization method, and the device includes:

[0157] The timbre feature extraction module 801 is used to obtain the current audio data of the target object, and use the pre-built target speech optimization model to extract features from the current audio data to obtain the current timbre features of the target object;

[0158] The tone feature extraction module 802 is used to obtain target music score data according to the current audio data, and perform feature extraction on the target music score data to obtain target tone features;

[0159] The audio feature splicing module 803 is used to perform feature splicing on the current timbre feature and the target tone feature to obtain a spliced ​​audio feature;

[0160] The audio feature noise adding module 804 is used to perform random noise adding processing on the spliced ​​audio features to obtain the noise added audio features;

[0161] The audio data optimization module 805 is used to perform speech optimization on the current audio data based on the noise-added frequency features using a target speech optimization model.

[0162] The specific implementation of the speech optimization device is basically the same as the specific implementation of the above-mentioned speech optimization method, and will not be repeated here.

[0163] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned voice optimization method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0164] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0165] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0166] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the voice optimization method of the embodiment of this application;

[0167] Input / output interface 903, used to implement information input and output;

[0168] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0169] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0170] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0171] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program, and when the computer program is executed by a processor, the above-mentioned voice optimization method is implemented.

[0172] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0173] The speech optimization method, speech optimization device, electronic device and storage medium provided in the embodiments of the present application use a pre-constructed target speech optimization model to perform feature extraction on the acquired current audio data of the target object to obtain the current timbre features of the target object, then obtain the corresponding target music score data based on the above current audio data, and use the above target speech optimization model to perform feature extraction on the target music score data to obtain the target pitch features, further, perform feature splicing on the current timbre features and the target pitch features to obtain spliced ​​audio features, then perform random noise processing on the spliced ​​audio features to obtain noisy audio features, and finally, use the target speech optimization model to perform speech optimization on the current audio data based on the noisy audio features.

[0174] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0175] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0176] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0177] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0178] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0179] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0180] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. The mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0181] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0182] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0183] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0184] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A voice optimization method, characterized in that: The method comprises: Acquire current audio data of a target object, and use a pre-built target speech optimization model to perform feature extraction on the current audio data to obtain current timbre features of the target object; Acquire target music score data according to the current audio data, and perform feature extraction on the target music score data to obtain target tone features; Performing feature splicing on the current timbre feature and the target tone feature to obtain a spliced ​​audio feature; Performing random noise processing on the spliced ​​audio features to obtain noisy audio features; Based on the noise-added frequency features, the target speech optimization model is used to perform speech optimization on the current audio data.

2. The method according to claim 1, characterized in that The method further includes training the target speech optimization model, specifically including: Obtain audio sample data and music score sample data; Performing spectrum conversion on the audio sample data to obtain an initial Mel spectrum; Based on the pre-built initial speech optimization model, the audio sample data, the music score sample data and the preset random Gaussian noise, the initial Mel spectrum is forwardly noised to obtain a noisy Mel spectrum; Based on the noisy Mel spectrum and the random Gaussian noise, a preset initial noise prediction model is trained to obtain a target noise prediction model; Based on the target noise prediction model, performing reverse denoising processing on the noisy Mel spectrum to obtain a denoised Mel spectrum; Calculating mel spectrum loss data based on the initial mel spectrum, the denoised mel spectrum and a preset mel spectrum loss function; Based on the Mel spectrum loss data, the parameters of the initial speech optimization model are adjusted to obtain the target speech optimization model.

3. The method according to claim 2, characterized in that The method of performing forward noise addition processing on the initial Mel spectrum based on the pre-constructed initial speech optimization model, the audio sample data, the music score sample data and the preset random Gaussian noise to obtain the noisy Mel spectrum includes: Based on the initial speech optimization model and the random Gaussian noise, the initial Mel spectrum is subjected to noise injection processing to obtain a noise-injected Mel spectrum; Extracting features from the audio sample data to obtain timbre features, and extracting features from the music score sample data to obtain pitch features; The timbre feature, the pitch feature and the noise-injected Mel spectrum are feature-concatenated to obtain the noise-added Mel spectrum.

4. The method according to claim 3, characterized in that The step of injecting noise into the initial Mel spectrum based on the initial speech optimization model and the random Gaussian noise to obtain a noise-injected Mel spectrum includes: Obtaining the number of noise additions of the initial Mel spectrum and the noise addition time step of the initial speech optimization model, wherein the noise addition time step represents the index of the number of noise additions; Based on a preset noise adjustment strategy and the number of noise additions, generating a noise adjustment coefficient of the initial speech optimization model; Based on the noise adjustment coefficient and the noise addition time step, the random Gaussian noise and the initial Mel spectrum are combined to obtain the noise-injected Mel spectrum.

5. The method according to claim 2, characterized in that: The step of performing reverse denoising on the noisy Mel spectrum based on the target noise prediction model to obtain a denoised Mel spectrum includes: Using the target noise prediction model to perform noise prediction on the noisy Mel spectrum to obtain target noise; Determining a non-noised feature from the noisy Mel spectrum based on the timbre feature and the pitch feature; Based on the non-noised feature, masking the noisy Mel spectrum to obtain a noisy feature; Based on the target noise, the noise-added feature is de-noised to obtain the denoised Mel spectrum.

6. The method according to claim 2, characterized in that The performing spectrum conversion on the audio sample data to obtain an initial Mel spectrum includes: Performing Fourier transform on the audio sample data to obtain audio spectrum data; Mel-scale mapping is performed on the audio spectrum data to obtain the initial Mel-scale spectrum.

7. The method according to any one of claims 1 to 6, characterized in that: The step of performing random noise processing on the spliced ​​audio features to obtain the noise-added audio features includes: Constructing a noise space, wherein the noise space contains noise data; Randomly selecting the noise data to obtain the target injection noise; Noise is injected into the concatenated audio feature based on the target injected noise to obtain the noisy audio feature.

8. A speech optimization device, characterized in that: The device comprises: A timbre feature extraction module is used to obtain current audio data of a target object, and extract features from the current audio data using a pre-built target speech optimization model to obtain current timbre features of the target object; A tone feature extraction module, used for acquiring target music score data according to the current audio data, and performing feature extraction on the target music score data to obtain target tone features; An audio feature splicing module, used for performing feature splicing on the current timbre feature and the target tone feature to obtain a spliced ​​audio feature; An audio feature noise adding module, used for performing random noise adding processing on the spliced ​​audio features to obtain a noise added audio feature; The audio data optimization module is used to perform speech optimization on the current audio data based on the noise-added frequency characteristics and using the target speech optimization model.

9. An electronic device, characterized in that: The electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the speech optimization method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the speech optimization method according to any one of claims 1 to 7 is implemented.

Images