Audio correction method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an audio correction method and device, electronic equipment and a storage medium, and belongs to the technical field of voice recognition. The method comprises the following steps: acquiring a sample audio, and acquiring an original audio correction model; performing pitch feature extraction on the sample audio according to a pitch encoder; performing pitch correction processing on the original pitch vector according to the generator; performing content feature extraction on the sample audio according to a content encoder; performing content correction processing on the original content vector according to the generator; performing audio fusion processing on the corrected pitch vector and the corrected content vector according to a generator; performing audio quality discrimination on the corrected audio according to a target discriminator; performing parameter adjustment on the original audio correction model according to the target discrimination data to obtain a target audio correction model; and performing audio correction on a pre-acquired target audio according to the target audio correction model. The embodiment of the invention can be applied to financial science and technology, the audio quality is improved, and the accuracy of speech recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of speech recognition and financial technology, and in particular to an audio correction method and device, an electronic device and a storage medium. Background Art

[0002] At present, in scenarios such as finance, consulting services can be provided through customer service. In the above scenarios, since the speaking audio of the person being asked or the person being answered may have problems such as inaccurate pronunciation and intonation, the speaking audio quality is poor, which affects the accuracy of speech recognition and further affects the consulting service. Therefore, how to provide an audio correction method to improve the audio quality and thus improve the accuracy of speech recognition has become a technical problem that needs to be solved urgently. Summary of the invention

[0003] The main purpose of the embodiments of the present application is to provide an audio correction method and device, an electronic device and a storage medium, aiming to improve the audio quality and thereby improve the accuracy of speech recognition.

[0004] To achieve the above object, a first aspect of an embodiment of the present application provides an audio correction method, the method comprising:

[0005] Acquire sample audio and acquire a preset original audio correction model; wherein the original audio correction model includes a pitch encoder, a content encoder and a generator;

[0006] Extracting pitch features from the sample audio according to the pitch encoder to obtain an original pitch vector;

[0007] Performing pitch correction processing on the original pitch vector according to the generator to correct the pitch vector;

[0008] Extracting content features from the sample audio according to the content encoder to obtain an original content vector;

[0009] Performing content correction processing on the original content vector according to the generator to correct the content vector;

[0010] Performing audio fusion processing on the modified pitch vector and the modified content vector according to the generator to obtain modified audio;

[0011] Performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data;

[0012] Adjusting parameters of the original audio correction model according to the target discrimination data to obtain a target audio correction model;

[0013] Audio correction is performed on the pre-acquired target audio according to the target audio correction model.

[0014] In some embodiments, the original audio correction model further includes a timbre encoder, and before performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data, the method further includes:

[0015] Extracting timbre features of the sample audio according to the timbre encoder to obtain a timbre vector;

[0016] The timbre of the modified audio is corrected according to the generator and the timbre vector.

[0017] In some embodiments, extracting content features from the sample audio according to the content encoder to obtain an original content vector includes:

[0018] Performing speech recognition on the sample audio according to the content encoder to obtain a text vector;

[0019] Extracting rhythm features from the sample audio according to the content encoder to obtain a rhythm vector;

[0020] Extracting vocalization features from the sample audio according to the content encoder to obtain a vocalization vector;

[0021] The text vector, the rhythm vector, and the utterance vector are concatenated to obtain the original content vector.

[0022] In some embodiments, before performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data, the method further includes: training the target discriminator, specifically including:

[0023] Get the training audio;

[0024] Performing audio quality discrimination on the training audio according to a preset original discriminator to obtain first training discrimination data;

[0025] Perform at least one of the following adjustments on the training audio: pitch adjustment, content adjustment, to obtain comparison audio;

[0026] Performing audio quality discrimination on the comparison audio according to the original discriminator to obtain second training discrimination data;

[0027] The parameters of the original discriminator are adjusted according to the first training discriminant data and the second training discriminant data to obtain the target discriminator.

[0028] In some embodiments, obtaining the sample audio includes:

[0029] Get the original audio;

[0030] The original audio is processed by dimensionality upgrading to obtain the sample audio.

[0031] In some embodiments, performing dimension-upgrading processing on the original audio to obtain the sample audio includes:

[0032] Performing denoising on the original audio to obtain preliminary audio;

[0033] Acoustic parameters are extracted from the preliminary audio to obtain Mel spectrum data, and the Mel spectrum data is used as the sample audio.

[0034] In some embodiments, obtaining the original audio includes:

[0035] Acquire initial audio; wherein the speaking object of the initial audio includes a target object and a non-target object;

[0036] Performing speaker object separation processing on the initial audio to obtain a first audio and a second audio; wherein the speaker object of the first audio includes the target object, and the speaker object of the second audio includes the non-target object;

[0037] The second audio is filtered out, and the first audio is used as the original audio.

[0038] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides an audio correction device, the device comprising:

[0039] An audio acquisition module, used to acquire sample audio and obtain a preset original audio correction model; wherein the original audio correction model includes a pitch encoder, a content encoder and a generator;

[0040] A pitch extraction module, used for extracting pitch features from the sample audio according to the pitch encoder to obtain an original pitch vector;

[0041] A pitch correction module, used for performing pitch correction processing on the original pitch vector according to the generator to correct the pitch vector;

[0042] A content extraction module, configured to extract content features from the sample audio according to the content encoder to obtain an original content vector;

[0043] A content correction module, performing content correction processing on the original content vector according to the generator to correct the content vector;

[0044] An audio fusion module, used for performing audio fusion processing on the modified pitch vector and the modified content vector according to the generator to obtain modified audio;

[0045] A quality discrimination module, used for performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data;

[0046] A parameter adjustment module, used for adjusting the parameters of the original audio correction model according to the target discrimination data to obtain a target audio correction model;

[0047] The audio correction module is used to perform audio correction on the pre-acquired target audio according to the target audio correction model.

[0048] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0049] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0050] The audio correction method and device, electronic device and storage medium proposed in the present application discriminate the audio quality of the corrected audio through a target discriminator to obtain target discrimination data, so that after adjusting the parameters of the original audio correction model according to the target discrimination data, a target audio correction model with better audio quality correction capability can be obtained. Therefore, when the audio correction is performed on the target audio according to the target audio correction model, the audio quality of the target audio can be improved. When the present application is applied to customer service scenarios in the financial field, the impact on consulting services caused by poor target audio quality can be reduced. In addition, the target discriminator and the generator in the original audio correction model constitute a generative adversarial network, so compared with the method of uniformly correcting all audio according to a correction template in the related art, the serious problem of homogenization of the corrected audio can be reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] Figure 1 is a flow chart of the audio correction method provided in an embodiment of the present application;

[0052] Figure 2 yes Figure 1 Flow chart of step S101 in FIG.

[0053] Figure 3 yes Figure 2 Flow chart of step S201 in FIG.

[0054] Figure 4 yes Figure 2Flow chart of step S202 in FIG.

[0055] Figure 5 yes Figure 1 Flow chart of step S104 in FIG.

[0056] Figure 6 yes Figure 1 A flowchart of another embodiment of an audio correction method;

[0057] Figure 7 yes Figure 1 A flowchart of another embodiment of an audio correction method;

[0058] Figure 8 is a schematic diagram of the structure of an audio correction device provided in an embodiment of the present application;

[0059] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0060] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0061] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0062] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0063] First, some nouns involved in this application are analyzed:

[0064] Artificial intelligence (AI) is a new technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. AI is a branch of computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence. Research in this field includes robotics, speech recognition, image recognition, natural language processing and expert systems. AI can simulate the information process of human consciousness and thinking. AI is also a theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0065] Pre-emphasis: It is a preprocessing method in feature extraction. Since the energy of high-frequency signals in speech signals is usually low and is greatly affected by suppression, it is necessary to increase the energy of the high-frequency part, that is, to compensate for the amplitude of the high-frequency part of the speech signal, so that the energy distribution of the speech signal is more balanced, thereby preventing the numerical calculation of Fourier transform from being unstable.

[0066] Framing: It is a preprocessing method in feature extraction. The speech signal is a non-stationary signal, but considering that the vocal cords have regular rhythms when making voiced sounds, that is, the fundamental frequency is relatively fixed in a short time range, the speech signal has a short-term stable characteristic.

[0067] Windowing: It is a preprocessing method in feature extraction. Considering the short-term stability of speech signals, each frame of the signal is windowed. Common window functions include Hamming window, Hanning window, Blackman window, etc. Specifically, after the speech signal is divided into frames, each frame is substituted into the window function, and the value outside the window is set to 0 to eliminate the signal discontinuity that may appear at both ends of each frame. For example, a Hamming window of size n is added to m frames of signal. By adding a Hamming window to each frame of data, the Hamming window matrix C(m, n) is obtained.

[0068] Fourier transform: refers to the ability to express a function that meets certain conditions as a trigonometric function (sine function or cosine function) or its integral linear combination.

[0069] Mel filter bank: Define a filter bank with M filters, where the number of filters is close to the number of critical bands, and the value of M is usually 22 to 26. The energy spectrum of the speech signal is constructed by the above method and has a set of Mel-scale triangular filter banks to extract features from the speech signal.

[0070] Speech recognition (Automatic Speech Recognition, ASR): used to convert the vocabulary content in the target speech into computer-readable input, such as keystrokes, binary codes or character sequences. The speech recognition system consists of four parts: signal processing and feature extraction, acoustic model, language model and decoding search. Among them, signal processing and feature extraction take audio signals as input, enhance speech by eliminating noise and channel distortion, convert signals from time domain to frequency domain, and extract suitable representative feature vectors for the acoustic model. The acoustic model integrates the knowledge of acoustics and phonetics, takes the features generated by the feature extraction part as input, and generates an acoustic model score for a variable-length feature sequence. The language model is used to learn the relationship between words through training corpus to estimate the possibility of a hypothesized word sequence, which is also called the language model score.

[0071] At present, in scenarios such as finance, consulting services can be provided through customer service. In the above scenarios, since the speaking audio of the person being asked or the person being answered may have inaccurate pronunciation and intonation, the speaking audio quality is poor, which affects the accuracy of speech recognition and further affects the consulting service. Therefore, how to provide an audio correction method to improve the accuracy of speech recognition has become a technical problem that needs to be solved urgently.

[0072] It should be noted that in the following embodiments, the application of this application to the intelligent customer service scenario is taken as an example. That is to say, in the following embodiments, the audio to be corrected is the audio generated by the consulting object, and the system corresponding to the intelligent customer service can perform speech recognition on the corrected audio, and obtain the reply audio or reply text from the corresponding corpus according to the speech recognition result. Among them, the consulting audio includes but is not limited to insurance purchase consultation, financial product consultation, etc. For example, when the consulting audio is a consultation on matter A, but due to problems such as inaccurate pronunciation and tone of the inquiry object, the system corresponding to the intelligent customer service performs speech recognition on the consulting audio and identifies the consulting content as a consultation on matter B. At this time, the intelligent customer service will match the reply audio or reply text from the corresponding corpus according to matter B, resulting in a phenomenon that the reply content does not match the inquiry content of the inquiry object.

[0073] Based on this, the embodiments of the present application provide an audio correction method and device, an electronic device and a storage medium, which aim to perform audio correction on the consultation audio, improve the audio quality, and enable the intelligent customer service to recognize that the content of the consultation audio is consultation matter A.

[0074] The audio correction method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the audio correction method in the embodiments of the present application is described.

[0075] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Among them, artificial intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0076] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0077] The audio correction method provided in the embodiment of the present application relates to the field of speech recognition and financial technology. The audio correction method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, or can be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the audio correction method, etc., but is not limited to the above forms.

[0078] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0079] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0080] Figure 1 is an optional flowchart of the audio correction method provided in an embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S109.

[0081] Step S101, obtaining sample audio and obtaining a preset original audio correction model; wherein the original audio correction model includes a pitch encoder, a content encoder and a generator;

[0082] Step S102, extracting pitch features from the sample audio according to the pitch encoder to obtain an original pitch vector;

[0083] Step S103, performing pitch correction processing on the original pitch vector according to the generator to correct the pitch vector;

[0084] Step S104, extracting content features from the sample audio using the content encoder to obtain an original content vector;

[0085] Step S105, performing content correction processing on the original content vector according to the generator to correct the content vector;

[0086] Step S106, performing audio fusion processing on the modified pitch vector and the modified content vector according to the generator to obtain modified audio;

[0087] Step S107, performing audio quality discrimination on the corrected audio according to a preset target discriminator to obtain target discrimination data;

[0088] Step S108, adjusting parameters of the original audio correction model according to the target discrimination data to obtain a target audio correction model;

[0089] Step S109: performing audio correction on the pre-acquired target audio according to the target audio correction model.

[0090] In steps S101 to S109 shown in the embodiment of the present application, the audio quality of the corrected audio is discriminated by a target discriminator to obtain target discrimination data, so that after the parameters of the original audio correction model are adjusted according to the target discrimination data, a target audio correction model with better audio quality correction capability can be obtained. Therefore, when the target audio is corrected according to the target audio correction model, the audio quality of the target audio can be improved. When the present application is applied to customer service scenarios in the financial field, the impact on consulting services caused by poor target audio quality can be reduced. In addition, the target discriminator and the generator in the original audio correction model constitute a generative adversarial network (GAN), so compared with the method of uniformly correcting all audio according to a correction template in the related art, the serious problem of homogeneity of the corrected audio can be reduced.

[0091] In step S101 of some embodiments, sample audio is obtained, and the sample audio refers to audio with poor audio quality. For example, when the present application is applied to the customer service scenario in the financial field, the sample audio can be the consulting audio of the consulting object. Since the consulting object may have problems such as accent and inaccurate pronunciation and tone, the audio quality of the sample audio is poor. When voice recognition is performed based on the poor quality sample audio to realize the intelligent customer service reply, there may be a phenomenon that the voice recognition is inaccurate and the reply is wrong. For example, the ideal consulting audio of the consulting object is "understanding the vehicle premium", but due to the problems such as the accent and inaccurate pronunciation and tone of the consulting object, the actual consulting audio of the consulting object is "understanding the scrapping of the vehicle". According to the above two types of consulting audio, the intelligent customer service will generate different reply contents. For example, for the ideal consulting audio, the reply content includes what vehicle insurances are there, how much is the premium of each type of vehicle insurance, what is the compensation ratio, etc. However, in actual situations, the audio recognized by the intelligent customer service voice is "understanding the scrapping of the vehicle", and the reply content includes the judgment criteria for the scrapping of the vehicle, etc. Therefore, it is necessary to correct the poor quality audio. In the embodiment of the present application, the correction processing is performed by the original audio correction model, and the correction processing method will be described in the embodiments of other steps. It should be noted that the sample audio can be obtained through the corresponding application programming interface (Application Programming Interface, API), such as obtaining the sample audio through the customer service application API interface. Alternatively, in order to enrich the training samples of the original audio correction model, other audios of poor quality can also be obtained from big data as sample audio, which is not specifically limited in the embodiment of the present application.

[0092] In addition, the embodiment of the present application also needs to obtain a preset original audio correction model, which includes a pitch encoder, a content encoder and a generator.

[0093] Reference Figure 2 In some embodiments, “obtaining sample audio” in step S101 includes but is not limited to steps S201 to S202.

[0094] Step S201, obtaining original audio;

[0095] Step S202: perform dimensionality upscaling on the original audio to obtain the sample audio.

[0096] In step S201 of some embodiments, the original audio refers to audio with poor audio quality. For example, when the present application is applied to a customer service scenario in the financial field, the original audio may be a consulting audio of the consulting object. Since the consulting object may have problems such as accent and inaccurate pronunciation, the audio quality of the original audio is poor. The original audio can be obtained through the corresponding API interface, such as obtaining the original audio through the customer service application API interface. Alternatively, in order to enrich the training samples of the original audio correction model, other audios of poor quality can also be obtained from big data as the original audio, which is not specifically limited in the embodiments of the present application.

[0097] Reference Figure 3 In some embodiments, step S201 includes but is not limited to steps S301 to S303.

[0098] Step S301, obtaining initial audio; wherein the speaking object of the initial audio includes a target object and a non-target object;

[0099] Step S302, performing speaker object separation processing on the initial audio to obtain a first audio and a second audio; wherein the speaker object of the first audio includes a target object, and the speaker object of the second audio includes a non-target object;

[0100] Step S303: filter out the second audio and use the first audio as the original audio.

[0101] In step S301 of some embodiments, the initial audio refers to audio with poor audio quality. For example, when the present application is applied to a customer service scenario in the financial field, the initial audio may be a consulting audio of the consulting object. Since the consulting object may have problems such as accent and inaccurate pronunciation, the audio quality of the initial audio is poor. In addition, the initial audio is a combined audio of multiple speaking objects, that is, the initial audio includes both the audio of the target object and the audio of the non-target object. For example, in a customer service scenario in the financial field, when the target object performs a consulting operation in an indoor scene, the target object is an adult and the non-target object is a child. Alternatively, when the target object performs a consulting operation in an outdoor scene, the non-target object is a passerby, etc. It can be understood that the format of the initial audio can be any of the following: MP3, WAV, AAC, FLAC, etc., which is not specifically limited in the embodiments of the present application.

[0102] In steps S302 to S303 of some embodiments, in order to improve the accuracy of audio quality correction and reduce the amount of calculation during audio quality correction, the initial audio is subjected to speech object separation processing, that is, the audio of the target object is separated from the audio of the non-target object, and a first audio including only the speech audio of the target object and a second audio including only the speech audio of the non-target object are obtained. The second audio is filtered out, and the first audio is used as the original audio. Among them, the speech object separation processing method can adopt any of the following: blind source separation (BSS) method, speech segmentation (Speech Segmentation) method, method based on deep learning, method based on time-frequency analysis, method based on statistical modeling, etc. Among them, the method based on blind source separation is based on blind source analysis theory, and independent component analysis (ICA) is performed on the original audio. In the method based on deep learning, a deep neural network (DNN) or a convolutional neural network (CNN) can be used. In the method based on time-frequency analysis, the initial audio is analyzed from two perspectives: time domain and frequency domain. In the method based on statistical modeling, a Gaussian mixture model (GMM) or a hidden Markov model (HMM) can be used to perform statistical modeling on the initial audio. It is understandable that the method for determining the target object and the non-target object is related to the method for separating the speaking objects, which is not described in detail in the embodiments of the present application.

[0103] In step S202 of some embodiments, since the original audio data is one-dimensional data, in order to obtain more audio features from the original audio data, the original audio data is processed to obtain a sample audio with a higher dimension, for example, a two-dimensional sample audio.

[0104] Reference Figure 4 In some embodiments, step S202 includes but is not limited to steps S401 to S402.

[0105] Step S401, denoising the original audio to obtain preliminary audio;

[0106] Step S402: extract acoustic parameters from the preliminary audio to obtain Mel-spectrogram data, and use the Mel-spectrogram data as sample audio.

[0107] In step S401 of some embodiments, in order to improve the accuracy of audio correction, the original audio is subjected to denoising, that is, the noise in the original audio is filtered out to obtain preliminary audio. It can be understood that the difference between denoising and speaking object separation is that speaking object separation takes speaking object as the separation basis and separates the audio corresponding to different speaking objects. The denoising takes the speaking audio and background sound of the target object as the separation basis and separates the speaking audio and background sound of the target object. Among them, the background sound includes wind, whistle, signal noise, etc. The denoising method can adopt any of the following: a method based on frequency domain filtering, a method based on time domain filtering, a method based on deep learning, and a method based on statistical modeling. In the method based on frequency domain filtering, the original audio is converted from time domain to frequency domain by performing short-time Fourier transform or wavelet transform on the original audio, and then the frequency domain signal is filtered and denoised. Frequency domain filtering methods include spectral subtraction, minimum mean square error (MMSE) method, etc. Methods based on time domain filtering include Kalman filtering, adaptive noise estimation (ANE), etc.

[0108] Filtering-based methods, spectral subtraction-based methods, wavelet transform-based methods, deep learning-based methods, etc.

[0109] In step S402 of some embodiments, since the preliminary audio is one-dimensional time series data, in order to obtain more audio features from the initial audio, acoustic parameters are extracted from the preliminary audio to obtain Mel spectrum data (MFCC) representing the two-dimensional frequency domain. The Mel spectrum data is used as the sample audio. Among them, the acoustic parameter extraction includes the following steps: pre-emphasis, framing, windowing, Fourier transform, Mel filter bank, etc. It can be understood that in addition to Mel spectrum data (MFCC), any of the following forms of two-dimensional frequency domain data can also be extracted from the initial audio: linear prediction system (LPC), linear prediction cepstral coefficients (LPCC), line spectrum frequency (LSF), discrete wavelet transform (SWT).

[0110] In other embodiments, the preliminary audio may be converted into a Fourier spectrum using a function in an audio processing library (librosa library), and other functions may be used to convert the Fourier spectrum into Mel spectrum data, thereby obtaining sample audio.

[0111] The advantage of using Mel spectrum data as sample audio in the embodiment of the present application is that Mel spectrum data has better noise resistance and robustness than other two-dimensional frequency domain data, and can improve the subsequent training effect of the original audio correction model based on Mel spectrum data.

[0112] In step S102 of some embodiments, pitch refers to the range of sound used by the subject when speaking. Pitch determines the vibration frequency of the sound-emitting body. The higher the frequency, the higher the pitch. The height of the pitch will affect the accuracy and clarity of pronunciation, thereby affecting the audio quality. The sample audio is used as input data of the pitch encoder, and the pitch features of the sample audio are extracted according to the pitch encoder to obtain the original pitch vector.

[0113] In step S103 of some embodiments, since the original pitch vector may be a factor causing poor audio quality of the sample audio, the original pitch vector is used as input data of the generator, and the original pitch vector is corrected by the generator to obtain a corrected pitch vector.

[0114] In step S104 of some embodiments, the sample audio is used as input data of the content encoder, and the content encoder extracts content features from the sample audio to obtain an original content vector. Content features refer to features that constitute the audio characteristics of the sample audio, such as the speech content represented by the sample audio, the audio rhythm of the sample audio, etc. The accuracy of the content features will affect the accuracy and clarity of pronunciation, thereby affecting the audio quality.

[0115] Reference Figure 5 In some embodiments, step S104 includes but is not limited to steps S501 to S504.

[0116] Step S501, performing speech recognition on the sample audio according to the content encoder to obtain a text vector;

[0117] Step S502, extracting rhythm features from the sample audio according to the content encoder to obtain a rhythm vector;

[0118] Step S503, extracting vocal features from the sample audio according to the content encoder to obtain a vocalization vector;

[0119] Step S504, concatenating the text vector, the rhythm vector, and the utterance vector to obtain the original content vector.

[0120] In step S501 of some embodiments, speech recognition is performed on the sample audio according to the content encoder to determine the spoken content represented by the sample audio and obtain a text vector.

[0121] In step S502 of some embodiments, speaking rhythm is related to factors such as accent, intonation, and voice tone, so it can be seen that speaking rhythm affects the accuracy and clarity of pronunciation, that is, affects the audio quality. The content encoder extracts rhythm features from the sample audio to obtain a rhythm vector. The rhythm vector is used to represent information such as speaking speed, pauses, and connected reading.

[0122] In step S503 of some embodiments, speaking and voicing refers to the pronunciation and articulation of the spoken content. For example, for the same word, different objects may have different pronunciations. For example, for some objects, "premium" may be pronounced as "scrapped". It can be seen that speaking and voicing will affect the accuracy and clarity of pronunciation, that is, the audio quality. The sample audio is subjected to vocalization feature extraction according to the content encoder to determine the vocalization characteristics of the corresponding object and obtain a vocalization vector.

[0123] In step S504 of some embodiments, the text vector, the rhythm vector, and the utterance vector are concatenated to obtain an original content vector that can represent the text vector, the rhythm vector, and the utterance vector as a whole.

[0124] In step S105 of some embodiments, since the original content vector may be a factor causing poor audio quality of the sample audio, the original content vector is used as input data of the generator, and the original content vector is modified by the generator to obtain a modified content vector.

[0125] In step S106 of some embodiments, the generator is further configured to perform audio fusion processing on the modified pitch vector and the modified content vector to fuse them to obtain a new audio, namely, the modified audio.

[0126] Reference Figure 6 In some embodiments, the original audio correction model also includes a timbre encoder. Before step S107, the method provided in the embodiment of the present application also includes but is not limited to steps S601 to S602.

[0127] Step S601, extracting timbre features of the sample audio according to the timbre encoder to obtain a timbre vector;

[0128] Step S602, performing timbre correction on the corrected audio according to the generator and the timbre vector.

[0129] In step S601 of some embodiments, in order to make the original audio modification model only perform audio quality modification on the sample audio without changing the speech characteristics of the corresponding object, the original audio modification model further includes a timbre encoder. The sample audio is used as input data of the timbre encoder, and the timbre features of the sample audio are extracted through the timbre encoder to obtain a timbre vector representing the speech characteristics of the object.

[0130] In step S602 of some embodiments, the timbre vector is used as input data of the generator to perform timbre correction on the generated modified audio according to the timbre vector, so that the timbre of the modified audio is consistent with the timbre corresponding to the timbre vector, that is, the modified audio retains the speaking characteristics of the corresponding object, thereby reducing the problem of timbre homogeneity of the modified audio.

[0131] Reference Figure 7 In some embodiments, before step S107, the method provided in the embodiment of the present application further includes training a target discriminator, including but not limited to steps S701 to S705.

[0132] Step S701, obtaining training audio;

[0133] Step S702, performing audio quality discrimination on the training audio according to a preset original discriminator to obtain first training discrimination data;

[0134] Step S703, performing at least one of the following adjustments on the training audio: pitch adjustment, content adjustment, to obtain a comparison audio;

[0135] Step S704, performing audio quality discrimination on the comparison audio according to the original discriminator to obtain second training discrimination data;

[0136] Step S705, adjusting the parameters of the original discriminator according to the first training discriminant data and the second training discriminant data to obtain a target discriminator.

[0137] In step S701 of some embodiments, training audio is obtained, and the training audio is audio used to train the target discriminator. Therefore, in order for the target discriminator to have the ability to discriminate audio with good quality, the training audio should be audio with good audio quality. It is understandable that when the expression content of a certain audio is clear, the pronunciation and tone are accurate, etc., the audio is regarded as audio with good audio quality; otherwise, the audio is regarded as audio with poor audio quality. The training audio can be obtained through the corresponding API interface, or the training audio can be generated by a professional reading object, which is not specifically limited in the embodiments of the present application.

[0138] In step S702 of some embodiments, the training audio is used as input data of the original discriminator, and the audio quality of the training audio is discriminated by the original discriminator to obtain first training discriminant data.

[0139] In step S703 of some embodiments, in order to enable the target discriminator to also have the ability to discriminate poor audio quality, the audio quality of the training audio is adjusted to obtain audio indicating poor audio quality, i.e., comparison audio. The audio quality adjustment can be performed in at least one of the following ways: pitch adjustment, content adjustment, and content adjustment includes rhythm adjustment, voicing adjustment, etc. Since pitch and content will affect the accuracy and clarity of pronunciation, the audio quality of the training audio can be changed according to the above adjustment method.

[0140] In step S704 of some embodiments, the comparison audio is used as input data of the original discriminator, and the audio quality of the comparison audio is discriminated by the original discriminator to obtain second training discriminant data.

[0141] In step S705 of some embodiments, parameters of the original discriminator are adjusted according to the first training discriminant data and the second discriminant data, respectively, so as to reduce the discrimination error of the original discriminator for audio with good audio quality, and reduce the discrimination error of the original discriminator for audio with poor audio quality, so as to obtain a target discriminator capable of discriminating audio with good audio quality and audio with poor audio quality.

[0142] In step S107 of some embodiments, a target discriminator capable of discriminating audio quality is pre-set, and the modified audio is used as input data of the target discriminator to obtain target discrimination data indicating good audio quality or poor audio quality. It can be understood that in some embodiments, the modified audio input to the target discriminator is audio obtained after timbre correction.

[0143] In step S108 of some embodiments, it can be determined whether the audio quality of the modified audio is improved compared to the audio quality of the sample audio based on the target discrimination data. For example, when the target discrimination data indicates that the audio quality is poor, it indicates that the audio quality of the modified audio has not been improved, or the degree of improvement is small; when the target discrimination data indicates that the audio quality is good, it indicates that the audio quality of the modified audio has been improved. Therefore, the parameters of the original audio correction model can be adjusted according to the target discrimination data to improve the audio quality correction ability of the original audio correction model to obtain the target audio correction model. It can be understood that when the target discrimination data is less than a preset threshold, it is determined that the target discrimination data indicates poor audio quality; when the target discrimination data is greater than or equal to the preset threshold, it is determined that the target discrimination data indicates good audio quality.

[0144] As can be seen from the above description, the purpose of the correction in the embodiment of the present application is to improve the audio quality of the corrected audio compared to the audio before correction, without requiring the corrected audio to meet a unified correction standard. The advantage of doing so is that it can reduce the serious problem of homogeneity of the corrected audio, that is, it can retain the speaking characteristics of the corresponding object while ensuring that the corrected audio is accurate and clear.

[0145] In step S109 of some embodiments, the target audio is the audio that needs to be corrected in actual applications. For example, in the customer service scenario in the financial field, the customer service application is loaded on the terminal. The consulting object sends audio through the voice control of the consulting interface on the touch terminal, and the audio that can be obtained is used as the target audio. The target audio is used as the input data of the target audio correction model to perform audio quality correction processing on the target audio through the target audio correction model. According to the corrected audio, downstream tasks such as speech recognition and intelligent customer service replies can be performed, thereby improving the accuracy of speech recognition and intelligent customer service replies. Among them, the method of speech recognition includes any of the following: a recognition method based on a speech model, a recognition method based on a neural network, an end-to-end recognition, etc. In the recognition method based on a speech model, the speech model includes a hidden Markov model (HMM), a conditional random field (CRF), etc. In the recognition method based on a neural network, the neural network includes a recurrent neural network (RNN), a convolutional neural network (CNN), etc.

[0146] In some embodiments, downstream tasks may also include real-time translation, online consultation in medical scenarios, etc., which is not specifically limited in the embodiments of the present application.

[0147] The audio correction method provided in the embodiment of the present application discriminates the audio quality of the corrected audio through a target discriminator to obtain target discrimination data, so that after adjusting the parameters of the original audio correction module according to the target discrimination data, a target audio correction model with better audio quality correction capability can be obtained. Therefore, when the target audio is corrected according to the target audio correction model, the audio quality of the target audio can be improved. When the present application is applied to customer service scenarios in the financial field, the impact on consulting services caused by poor target audio quality can be reduced. In addition, the target discriminator and the generator in the original audio correction model constitute a generative adversarial network, so compared with the method of uniformly correcting all audio according to a correction template in the related art, the serious problem of homogenization of the corrected audio can be reduced.

[0148] In some specific embodiments, the consulting object sends a target audio through the customer service window of the insurance APP, and the target audio is "understand the vehicle scrapping". The target audio is obtained, and the target audio is processed by dimensionality upgrading and object separation. The processed audio is used as the input data of the target audio correction model to perform audio correction on the target audio through the target audio correction model to obtain a corrected audio, and the corrected audio is "understand the vehicle premium". The insurance APP calls the corpus stored locally or in the cloud according to the corrected audio, and semantically matches "understand the vehicle premium" with the corpus in the corpus to obtain the corresponding reply content. The reply content includes but is not limited to: what vehicle insurances are available, how much is the premium for each type of vehicle insurance, what is the compensation ratio, etc. The reply content is displayed in the customer service window in the form of text or voice, thereby realizing an intelligent reply to the consulting object and improving the accuracy of the reply.

[0149] Reference Figure 8 The embodiment of the present application also provides an audio correction device, which can implement the above audio correction method, and the device includes:

[0150] The audio acquisition module 801 is used to acquire sample audio and obtain a preset original audio correction model; wherein the original audio correction model includes a pitch encoder, a content encoder and a generator;

[0151] A pitch extraction module 802 is used to extract pitch features from the sample audio according to the pitch encoder to obtain an original pitch vector;

[0152] A pitch correction module 803 is used to perform pitch correction processing on the original pitch vector according to the generator to correct the pitch vector;

[0153] The content extraction module 804 is used to extract content features from the sample audio according to the content encoder to obtain an original content vector;

[0154] The content correction module 805 performs content correction processing on the original content vector according to the generator to correct the content vector;

[0155] An audio fusion module 806, configured to perform audio fusion processing on the modified pitch vector and the modified content vector according to the generator to obtain modified audio;

[0156] The quality determination module 807 is used to determine the audio quality of the modified audio according to a preset target discriminator to obtain target determination data;

[0157] A parameter adjustment module 808 is used to adjust the parameters of the original audio correction model according to the target discrimination data to obtain a target audio correction model;

[0158] The audio correction module 809 is used to perform audio correction on the pre-acquired target audio according to the target audio correction model.

[0159] The specific implementation of the audio correction device is basically the same as the specific implementation of the above-mentioned audio correction method, and will not be repeated here.

[0160] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above audio correction method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0161] Reference Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0162] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0163] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the audio correction method of the embodiment of this application;

[0164] Input / output interface 903, used to implement information input and output;

[0165] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized by wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.);

[0166] A bus 905 that transmits information between various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0167] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0168] An embodiment of the present application also provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned audio correction method is implemented.

[0169] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0170] The audio correction method, audio correction device, electronic device and storage medium provided in the embodiments of the present application discriminate the audio quality of the corrected audio through a target discriminator to obtain target discrimination data, so that after adjusting the parameters of the original audio correction module according to the target discrimination data, a target audio correction model with better audio quality correction capability can be obtained. Therefore, when the target audio is corrected according to the target audio correction model, the audio quality of the target audio can be improved. When the present application is applied to customer service scenarios in the financial field, the impact on consulting services caused by poor target audio quality can be reduced. In addition, the target discriminator and the generator in the original audio correction model constitute a generative adversarial network, so compared with the method of uniformly correcting all audio according to a correction template in the related art, the serious problem of homogeneity of the corrected audio can be reduced.

[0171] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0172] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0173] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0174] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0175] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0176] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0177] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0178] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0179] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0180] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0181] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. An audio correction method, characterized in that: include: Acquire sample audio and acquire a preset original audio correction model; wherein the original audio correction model includes a pitch encoder, a content encoder and a generator; Extracting pitch features from the sample audio according to the pitch encoder to obtain an original pitch vector; Performing pitch correction processing on the original pitch vector according to the generator to correct the pitch vector; Extracting content features from the sample audio according to the content encoder to obtain an original content vector; Performing content correction processing on the original content vector according to the generator to correct the content vector; Performing audio fusion processing on the modified pitch vector and the modified content vector according to the generator to obtain modified audio; Performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data; Adjusting parameters of the original audio correction model according to the target discrimination data to obtain a target audio correction model; Audio correction is performed on the pre-acquired target audio according to the target audio correction model.

2. The method according to claim 1, characterized in that The original audio correction model further includes a timbre encoder. Before performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data, the method further includes: Extracting timbre features of the sample audio according to the timbre encoder to obtain a timbre vector; The timbre of the modified audio is corrected according to the generator and the timbre vector.

3. The method according to claim 1, characterized in that The extracting content features of the sample audio according to the content encoder to obtain an original content vector includes: Performing speech recognition on the sample audio according to the content encoder to obtain a text vector; Extracting rhythm features from the sample audio according to the content encoder to obtain a rhythm vector; Extracting vocal features from the sample audio according to the content encoder to obtain a vocalization vector; The text vector, the rhythm vector, and the utterance vector are concatenated to obtain the original content vector.

4. The method according to claim 1, characterized in that: Before performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data, the method further includes: training the target discriminator, specifically including: Get the training audio; Performing audio quality discrimination on the training audio according to a preset original discriminator to obtain first training discrimination data; Perform at least one of the following adjustments on the training audio: pitch adjustment, content adjustment, to obtain comparison audio; Performing audio quality discrimination on the comparison audio according to the original discriminator to obtain second training discrimination data; The parameters of the original discriminator are adjusted according to the first training discriminant data and the second training discriminant data to obtain the target discriminator.

5. The method according to any one of claims 1 to 4, characterized in that: The obtaining of sample audio comprises: Get the original audio; The original audio is processed by dimensionality upgrading to obtain the sample audio.

6. The method according to claim 5, characterized in that The step of performing dimension-upgrading processing on the original audio to obtain the sample audio includes: Performing denoising on the original audio to obtain preliminary audio; Acoustic parameters are extracted from the preliminary audio to obtain Mel spectrum data, and the Mel spectrum data is used as the sample audio.

7. The method according to claim 5, characterized in that The obtaining of the original audio comprises: Acquire initial audio; wherein the speaking object of the initial audio includes a target object and a non-target object; Performing speaker object separation processing on the initial audio to obtain a first audio and a second audio; wherein the speaker object of the first audio includes the target object, and the speaker object of the second audio includes the non-target object; The second audio is filtered out, and the first audio is used as the original audio.

8. An audio correction device, characterized in that: The device comprises: An audio acquisition module, used to acquire sample audio and obtain a preset original audio correction model; wherein the original audio correction model includes a pitch encoder, a content encoder and a generator; A pitch extraction module, used for extracting pitch features from the sample audio according to the pitch encoder to obtain an original pitch vector; A pitch correction module, used for performing pitch correction processing on the original pitch vector according to the generator to correct the pitch vector; A content extraction module, configured to extract content features from the sample audio according to the content encoder to obtain an original content vector; A content correction module, performing content correction processing on the original content vector according to the generator to correct the content vector; An audio fusion module, used for performing audio fusion processing on the modified pitch vector and the modified content vector according to the generator to obtain modified audio; A quality discrimination module, used for performing audio quality discrimination on the modified audio according to a preset target discriminator to obtain target discrimination data; A parameter adjustment module, used for adjusting the parameters of the original audio correction model according to the target discrimination data to obtain a target audio correction model; The audio correction module is used to perform audio correction on the pre-acquired target audio according to the target audio correction model.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images