Audio correction method and related device

Abstract

The present application is about an audio correction method and related devices, and relates to the field of artificial intelligence technology. The method includes: separating audio into a vocal signal and an accompaniment signal; extracting the spectral information of the audio; inputting the spectral information into a vocal and accompaniment harmony evaluation model to obtain the harmony between the vocals and the accompaniment in the audio, and the harmony reflects the degree of deviation of the vocal accuracy when the vocals and the accompaniment in the audio work together; inputting the vocal signal and the harmony into a correction model to obtain a corrected vocal signal; mixing the corrected vocal signal with the accompaniment signal to obtain the corrected audio. The present application effectively improves the accuracy of audio correction.

Description

Background Art

[0002] Generally speaking, audio correction is very common in digital recordings nowadays. In particular, in the learning of singing, how to automatically beautify the human voice is also very important. Currently, there is a method of detecting the fundamental frequency of the human voice in the audio file and then synthesizing the corrected audio, that is, correcting the fundamental frequency to a discrete pitch, which will change the singing details that the singer intends to make. Therefore, it is necessary to manually correct the pitch in a gradual manner on the timeline to fine-tune the singing details, which will introduce the problem of difficult trade-offs between pitch accuracy and variability.

[0003] At the same time, the current tuning requires the vocals to correspond to the musical score. In many cases, the musical score information is actually missing. If the musical score is used as a reference and basis for tuning, and the standard pitch of the vocal melody is not known in advance, then the tuning will be ineffective; further, the musical score is composed of a series of notes, and the pitch and duration of these notes are discrete. This concise symbolic representation leaves a considerable performance space for the singer. Singing is actually a process of artistic re-creation. Therefore, although the singer will follow the basic information of the musical score, they will generally add personalized elements, such as adding vibrato according to needs, which will further lead to the accuracy and effectiveness of the audio correction. Summary of the invention

[0004] The purpose of this application is to provide an audio correction solution, thereby effectively improving the accuracy of audio correction at least to a certain extent.

[0005] According to one aspect of the present application, there is provided an audio correction method, comprising:

[0006] Separate the audio into vocal signal and accompaniment signal;

[0007] Extracting spectrum information of the audio;

[0008] Inputting the spectrum information into a model for evaluating the harmony between human voice and accompaniment to obtain the harmony between human voice and accompaniment in the audio, wherein the harmony reflects the degree of deviation of human voice pitch when the human voice and accompaniment in the audio are coordinated;

[0009] Inputting the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal;

[0010] The modified vocal signal is mixed with the accompaniment signal to obtain modified audio.

[0011] In an exemplary embodiment of the present application, separating the audio into a vocal signal and an accompaniment signal includes:

[0012] The audio is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model.

[0013] In an exemplary embodiment of the present application, inputting the audio into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model includes:

[0014] Obtaining a waveform graph of the audio, and determining a waveform compression coefficient according to the number of troughs or peaks in the waveform graph;

[0015] Compressing the height of the waveform according to the compression coefficient to obtain a wave to be separated, and outputting the audio to be separated according to the wave to be separated;

[0016] The audio to be separated is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model.

[0017] In an exemplary embodiment of the present application, extracting the spectrum information of the audio includes:

[0018] Inputting the human voice signal into a trajectory prediction model to obtain a human voice fundamental frequency trajectory;

[0019] Determining the segmentation points of the audio based on the fundamental frequency trajectory of the human voice according to a predetermined fundamental frequency magnitude change rule;

[0020] Splitting the audio into a plurality of audio segments according to the segmentation points;

[0021] Extracting the spectral information of the plurality of audio segments.

[0022] In an exemplary embodiment of the present application, determining the segmentation point of the audio based on the fundamental frequency trajectory of the human voice according to a predetermined fundamental frequency magnitude change rule includes:

[0023] The segmentation points of the audio for breathing and word change are determined based on the fundamental frequency trajectory of the human voice and according to the fundamental frequency size change rules during breathing and word change.

[0024] In an exemplary embodiment of the present application, the human voice and accompaniment harmony evaluation model includes a CNN model and an RNN model connected to the CNN model; the step of inputting the spectrum information into the human voice and accompaniment harmony evaluation model to obtain the harmony between the human voice and the accompaniment in the audio includes:

[0025] Inputting the spectrum information of the plurality of audio segments into the CNN model respectively, and obtaining the acoustic features corresponding to each of the audio segments output by the CNN model;

[0026] The acoustic features are input into the RNN model to obtain the harmony degree between the human voice and the audio corresponding to each of the audio segments output by the RNN model.

[0027] In an exemplary embodiment of the present application, inputting the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal includes:

[0028] Obtain the type of music from which the audio is sourced;

[0029] Determining the exact score of the harmony level according to the music type;

[0030] When the accuracy score is greater than a predetermined threshold, the human voice signal and the harmony degree are input into a correction model to obtain a corrected human voice signal.

[0031] According to one aspect of the present application, an audio correction device is characterized by comprising:

[0032] A separation module, used for separating the audio into a vocal signal and an accompaniment signal;

[0033] An extraction module, used for extracting spectrum information of the audio;

[0034] An evaluation module, used for inputting the spectrum information into a harmony evaluation model of human voice and accompaniment to obtain the harmony between human voice and accompaniment in the audio, wherein the harmony reflects the degree of deviation of human voice pitch when the human voice and accompaniment in the audio are coordinated;

[0035] A correction module, used for inputting the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal;

[0036] The mixing module is used to mix the modified vocal signal with the accompaniment signal to obtain modified audio.

[0037] According to one aspect of the present application, a computer-readable storage medium is provided, on which program instructions are stored, wherein the program instructions implement any of the above-mentioned methods when executed by a processor.

[0038] According to one aspect of the present application, there is provided an electronic device, characterized in that it includes:

[0039] Processor; and

[0040] A memory is used to store program instructions of the processor; wherein the processor is configured to perform any of the above methods by executing the program instructions.

[0041] The present application discloses an audio correction scheme. First, the audio is separated into a vocal signal and an accompaniment signal. In this way, in the subsequent steps, the audio correction can be performed based on the characteristics of the vocals and the accompaniment, without the need for accurate score information. Accurate scores are often difficult to obtain, which can effectively save the cost of obtaining the scores. At the same time, the correction failure is avoided when the correction is performed according to the score information. Then, the spectrum information of the audio is extracted. The spectrum information is input into a vocal and accompaniment harmony evaluation model to obtain the harmony between the vocals and the accompaniment in the audio. The harmony reflects the degree of deviation of the vocal accuracy when the vocals and the accompaniment are coordinated in the audio. Then, the vocal signal and the harmony are input into the correction model to obtain the corrected vocal signal. The vocal signal can be accurately corrected according to the degree of deviation of the vocal accuracy when the vocals and the accompaniment are coordinated in the audio. Finally, the corrected vocal signal and the accompaniment signal are mixed to obtain the corrected audio. The sound correction based on the harmony between the vocals and the accompaniment is realized, and the singing details and singing style of the singer are retained to the maximum extent, and the accuracy of the sound correction is effectively improved.

[0042] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] The drawings herein are incorporated into the specification and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0044] Figure 1 A flow chart of an audio correction method is schematically shown.

[0045] Figure 2 An example diagram schematically shows an application scenario of an audio correction method.

[0046] Figure 3 A flow chart of a method for obtaining sound spectrum information is schematically shown.

[0047] Figure 4 A block diagram of an audio correction device is schematically shown.

[0048] Figure 5 An example block diagram of an electronic device for implementing the above audio correction method is schematically shown.

[0049] Figure 6 A computer-readable storage medium for implementing the above audio correction method is schematically shown. DETAILED DESCRIPTION

[0050] Example embodiments will now be described more fully with reference to the accompanying drawings. However, example embodiments can be implemented in a variety of forms and should not be construed as being limited to the examples set forth herein; on the contrary, these embodiments are provided so that the present application will be more comprehensive and complete, and the concept of the example embodiments will be fully conveyed to those skilled in the art. The described features, structures, or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present application. However, those skilled in the art will appreciate that the technical solutions of the present application may be practiced while omitting one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known technical solutions are not shown or described in detail to avoid obscuring various aspects of the present application.

[0051] In addition, the accompanying drawings are only schematic illustrations of the present application and are not necessarily drawn to scale. The same reference numerals in the figures represent the same or similar parts, and thus their repeated description will be omitted. Some of the block diagrams shown in the accompanying drawings are functional entities and do not necessarily correspond to physically or logically independent entities. These functional entities can be implemented in software form, or implemented in one or more hardware modules or integrated circuits, or implemented in different networks and / or processor devices and / or microcontroller devices.

[0052] In this exemplary embodiment, an audio correction method is first provided, which can be run on a server, a server cluster, a cloud server, etc. Of course, those skilled in the art can also run the method of the present invention on other platforms as required, which is not particularly limited in this exemplary embodiment. Figure 1 As shown, the audio correction method may include the following steps:

[0053] Step S110, separating the audio into a vocal signal and an accompaniment signal;

[0054] Step S120, extracting spectrum information of the audio;

[0055] Step S130, inputting the spectrum information into a model for evaluating the harmony between human voice and accompaniment, and obtaining the harmony between human voice and accompaniment in the audio, wherein the harmony reflects the degree of deviation of human voice pitch when the human voice and accompaniment in the audio are coordinated;

[0056] Step S140, inputting the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal;

[0057] Step S150, mixing the modified vocal signal with the accompaniment signal to obtain modified audio.

[0058] In the above audio correction method, first, the audio is separated into a vocal signal and an accompaniment signal; in this way, in the subsequent steps, the audio correction can be performed based on the characteristics of the vocals and the accompaniment, without the need for accurate score information, which is often difficult to obtain, and can effectively save the cost of obtaining the score; at the same time, the correction failure is avoided when the correction is performed according to the score information; then, the spectral information of the audio is extracted; the spectral information is input into the vocal and accompaniment harmony evaluation model to obtain the harmony between the vocals and the accompaniment in the audio, and the harmony reflects the degree of deviation of the vocal accuracy when the vocals and the accompaniment work together in the audio; then, the vocal signal and the harmony are input into the correction model to obtain the corrected vocal signal; the vocal signal can be accurately corrected according to the degree of deviation of the vocal accuracy when the vocals and the accompaniment work together in the audio; finally, the corrected vocal signal and the accompaniment signal are mixed to obtain the corrected audio. The sound correction based on the harmony between the vocals and the accompaniment is realized, and the singing details and singing style of the singer are retained to the maximum extent, and the accuracy of the sound correction is effectively improved.

[0059] Below, each step in the above audio correction method in this example implementation will be explained and illustrated in detail with reference to the accompanying drawings.

[0060] In step S110 , the audio is separated into a vocal signal and an accompaniment signal.

[0061] In the implementation of this example, reference Figure 2 As shown, the server 210 can obtain audio from the terminal 220; then, the server 210 can separate the audio into a vocal signal and an accompaniment signal, and perform audio correction based on the vocal signal and the accompaniment signal. The server 210 can be a cloud server or a central node server of a blockchain; the terminal 220 can be a computer, a mobile phone, or any other terminal device with a command processing function and a data storage function, which is not specifically limited here.

[0062] Audio, for example, is audio of monks singing Brahma Chan, in which the singing of Brahma Chan is often accompanied by accompaniment consisting of melody instruments, bass, percussion instruments, etc., as well as the singer's own voice.

[0063] The human voice signal included in the audio may be a human voice time domain spectrum, and the accompaniment signal may be an accompaniment time domain spectrum. The trained machine learning model can be used to accurately separate the audio into a human voice signal and an accompaniment signal, wherein the human voice signal and the accompaniment signal have the same duration and the signal amplitude is half of the original audio.

[0064] In one embodiment, separating the audio into a vocal signal and an accompaniment signal includes:

[0065] The audio is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model.

[0066] The vocal accompaniment separation model takes audio as input and the vocal signal and accompaniment information in the audio as the expected output. The trained machine learning model with the required accuracy can reliably perform automatic audio separation. The vocal accompaniment separation model can perform Fourier transform on the audio to obtain the spectrum, and then extract the vocals and music sounds from the spectrum space.

[0067] In one embodiment, a waveform graph of the audio is obtained, and a waveform compression coefficient is determined according to the number of troughs or peaks in the waveform graph;

[0068] Compressing the height of the waveform according to the compression coefficient to obtain a wave to be separated, and outputting the audio to be separated according to the wave to be separated;

[0069] The audio to be separated is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model.

[0070] The applicant has discovered that pre-compression of the audio, separation of the life signal and the accompaniment signal, post-processing in subsequent steps, and merging and correcting the corrected audio can effectively ensure the correction effect. The compression coefficient during pre-compression is related to the number of troughs or peaks in the waveform graph. The waveform compression coefficient corresponding to the range can be determined from a preset coefficient table based on the range of the number of troughs or peaks. In one embodiment, different types of music correspond to different types of preset coefficient tables.

[0071] In one embodiment, the audio is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model, including:

[0072] The audio is input into a U-net model to obtain a vocal signal and an accompaniment signal output by the U-net model.

[0073] The cropping and splicing process of the U-Net model is very intuitive and reasonable. The splicing of high-level feature maps and low-level feature maps and the repeated and continuous convolution operations enable the model to obtain more accurate output feature maps from the combination of context information and detail information. Experiments have shown that the U-Net model can obtain more accurate classification results even with fewer training samples. Therefore, the U-net model can accurately extract the human voice and music sound according to the audio spectrum, separate the original audio into the human voice time domain signal and the accompaniment time domain signal, and effectively ensure the accuracy of the separation of the human voice signal and the accompaniment signal. The existing spleeter model can also be used to achieve accurate separation of the human voice signal and the accompaniment signal.

[0074] Step S120: extracting spectrum information of the audio.

[0075] In the implementation of this example, the spectral information of the audio can be extracted by obtaining the audio spectrum through Fourier transform; it can also be a CQT spectrum calculated by an existing time-frequency transform algorithm - CQT with the same exponential distribution law. The distribution of the CQT spectrum is the same as the scale frequency, so by calculating the CQT spectrum of the music signal, the amplitude value of the music signal at each note frequency can be directly obtained for music signal processing.

[0076] In one embodiment, reference Figure 3 As shown, extracting the spectrum information of the audio includes:

[0077] Step S310, inputting the human voice signal into a trajectory prediction model to obtain a human voice fundamental frequency trajectory;

[0078] Step S320, determining the segmentation points of the audio based on the fundamental frequency trajectory of the human voice according to a predetermined fundamental frequency magnitude change rule;

[0079] Step S330, dividing the audio into a plurality of audio segments according to the segmentation points;

[0080] Step S340: extracting spectrum information of the plurality of audio segments.

[0081] The trajectory prediction model can specifically use a CNN model. The input of the CNN model can be the human voice time domain signal obtained after the aforementioned U-NET separation, and then the CNN model is used to predict the fundamental frequency trajectory of the human voice.

[0082] The segmentation point of the audio is determined based on the fundamental frequency trajectory of the human voice according to the predetermined fundamental frequency size change rule. The segmentation point can be the fundamental frequency trajectory change rule of the human voice, for example, the time point corresponding to the mutation point in the fundamental frequency trajectory of the human voice is used as the segmentation point; it can also be that the fundamental frequency in the fundamental frequency trajectory of the human voice is converted into pitch; the critical point of pitch switching is determined as the segmentation point, and the critical point of pitch switching is determined as the segmentation point, including: determining a pitch higher than a predetermined threshold; determining the critical point of pitch switching in a pitch higher than the predetermined threshold as the segmentation point.

[0083] For example, after obtaining the fundamental frequency trajectory of human voice, convert the fundamental frequency into the pitch of the twelve-tone equal temperament according to the principle of proximity. For example, if the predicted fundamental frequency is 443.121Hz or 438.654Hz, the corresponding pitch is converted to A4. Then, based on the short-term stability characteristics, it can be assumed that the stable pitch greater than 30 milliseconds is the effective pitch. Find out where the pitch changes (for example, the area from 5 to 6 seconds is C4, and the area from 6 seconds to 6.3 seconds is D4, so the time point of the pitch change is 5.95 seconds), and set it as the split point.

[0084] After the split point is determined, the split point can reflect the moment when a singer's audio changes. The audio can be split into multiple audio segments according to the split point, and the spectral information of the multiple audio segments can be extracted. The degree of deviation of the vocal pitch when the human voice and accompaniment work together in each audio segment can be analyzed based on the spectral information of each audio segment, thereby further ensuring the accuracy of the sound correction.

[0085] In one embodiment, determining the segmentation point of the audio based on the human voice fundamental frequency trajectory according to a predetermined fundamental frequency magnitude change rule includes:

[0086] The segmentation points of the audio for breathing and word change are determined based on the fundamental frequency trajectory of the human voice and according to the fundamental frequency size change rules during breathing and word change.

[0087] After obtaining the fundamental frequency trajectory of the human voice, convert the fundamental frequency into the pitch of the twelve-tone equal temperament according to the principle of proximity. For example, if the predicted fundamental frequency is 443.121Hz or 438.654Hz, the corresponding pitch is converted to A4. Then, based on the short-term stability characteristics, it can be assumed that the stable pitch greater than 30 milliseconds is the effective pitch. Find out where the pitch changes (for example, the area of ​​5-6 seconds is C4, the area of ​​6 seconds to 6.3 seconds is D4, then the time point of the pitch change is 5.95 seconds), and set it as the split point, which is the split point for breathing and changing words.

[0088] Step S130, inputting the spectrum information into a model for evaluating the harmony between the human voice and the accompaniment, and obtaining the harmony between the human voice and the accompaniment in the audio, wherein the harmony reflects the degree of deviation of the human voice pitch when the human voice and the accompaniment in the audio work together.

[0089] In the implementation of this example, the model for evaluating the harmony between human voice and accompaniment is a machine learning model that meets the requirements and is trained by taking the spectral information sample as input in advance and the harmony between human voice and accompaniment in the audio corresponding to the spectral information sample as the expected output. The harmony between human voice and accompaniment in the audio output by the machine learning model can reflect the degree of deviation of human voice accuracy (i.e., pitch deviation) when the human voice and accompaniment in the audio work together. Specifically, the spectral information of each audio segment in the aforementioned embodiment can be input into the model for evaluating the harmony between human voice and accompaniment to obtain the harmony between human voice and accompaniment in each audio segment.

[0090] In one embodiment, the human voice and accompaniment harmony evaluation model includes a CNN model and an RNN model connected to the CNN model; the step of inputting the spectrum information into the human voice and accompaniment harmony evaluation model to obtain the harmony between the human voice and accompaniment in the audio includes:

[0091] Inputting the spectrum information of the plurality of audio segments into the CNN model respectively, and obtaining the acoustic features corresponding to each of the audio segments output by the CNN model;

[0092] The acoustic features are input into the RNN model to obtain the harmony degree between the human voice and the audio corresponding to each of the audio segments output by the RNN model.

[0093] The CNN model is used to learn the spectral relationship between human voice and accompaniment. The input to the CNN model is the spectral information (such as CQT spectrum). The CNN model plays the role of feature preprocessing, which is used to reduce the dimension of the spectral information and extract abstract acoustic features. Then, the role of the RNN model is to learn the spectral relationship in the front and back time sequence of an audio segment. It can convert the variable-length sound features into a vector of fixed length, and finally output the pitch deviation value through the full connection layer (that is, it is implemented as a classification model to predict the pitch deviation). Among them, the loss function in the model uses the mean square error MSE, and its evaluation process is the mean square error between the actual pitch of the sound signal corresponding to the duration of a note and the predicted pitch.

[0094] During the model training process, the human voice signal can be randomly moved up / down within a predetermined number of cents (e.g., a 50-cent range), and the human voice and accompaniment harmony evaluation model is allowed to make predictions and compare them with the original human voice fundamental frequency to minimize the mean square error between the two. In the inference stage, the value predicted by the human voice and accompaniment harmony evaluation model is the deviation value of the human voice accuracy.

[0095] Step S140: input the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal.

[0096] In the implementation of this example, the degree of harmony reflects the degree of deviation in the pitch of the human voice when the human voice and the accompaniment in the audio are coordinated. The degree of harmony information can indicate the amplitude of the fundamental frequency correction, and then accurately guide the correction model to change the speed and tone of the human voice signal, thereby achieving pitch adjustment of the human voice signal and obtaining a corrected human voice signal.

[0097] Among them, the correction model is a machine learning model that takes the human voice signal sample and the harmony degree sample as input and the corrected human voice signal as the expected output and is trained to achieve the predetermined accuracy requirement.

[0098] The human voice correction model may specifically be a TD-PSOLA model. The TD-PSOLA model may change the speed and pitch of the human voice signal according to the harmony degree information, and the harmony degree information indicates the amplitude of the fundamental frequency correction to achieve the pitch adjustment of the human voice signal.

[0099] In one embodiment, the human voice signal and the harmony degree are input into a correction model to obtain a corrected human voice signal, including:

[0100] Obtain the type of music from which the audio is sourced;

[0101] Determining the exact score of the harmony level according to the music type;

[0102] When the accuracy score is greater than a predetermined threshold, the human voice signal and the harmony degree are input into a correction model to obtain a corrected human voice signal.

[0103] The music type can be rock music or Buddhist chants, etc. Different types of music have different singing styles. According to the music type, the harmony degree range under the corresponding singing style can be determined, and then the accurate score of the harmony degree can be calculated based on the acquired harmony degree and the harmony degree range. When the accurate score is greater than a predetermined threshold, the human voice signal and the harmony degree are input into the correction model to obtain the corrected human voice signal, which can ensure the effectiveness of the correction.

[0104] Step S150, mixing the modified vocal signal with the accompaniment signal to obtain modified audio.

[0105] In the implementation of this example, the signal amplitudes of the vocal signal and the accompaniment signal are both half of the original audio, and then the addition is performed in the time domain to achieve the correction of the vocals in the audio. The tuning method based on the harmony between the vocals and the accompaniment can retain the singer's singing details and singing style to the greatest extent, effectively improving the accuracy of tuning.

[0106] The present application also provides an audio correction device. Figure 4 As shown, the audio modification device may include a separation module 410, an extraction module 420, an evaluation module 430, a modification module 440 and a mixing module 450. Among them:

[0107] The separation module 410 may be used to separate the audio into a vocal signal and an accompaniment signal;

[0108] The extraction module 420 may be used to extract spectrum information of the audio;

[0109] The evaluation module 430 may be used to input the spectrum information into a human voice and accompaniment harmony evaluation model to obtain the harmony between the human voice and the accompaniment in the audio, wherein the harmony reflects the degree of deviation of the human voice pitch when the human voice and the accompaniment in the audio are coordinated;

[0110] The correction module 440 may be used to input the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal;

[0111] The mixing module 450 may be configured to mix the modified vocal signal with the accompaniment signal to obtain modified audio.

[0112] The specific details of each module in the above audio correction device have been described in detail in the corresponding audio correction method, so they will not be repeated here.

[0113] It should be noted that, although several modules or units of the equipment for action execution are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present application, the features and functions of two or more modules or units described above can be embodied in one module or unit. On the contrary, the features and functions of one module or unit described above can be further divided into being embodied by multiple modules or units.

[0114] In addition, although the steps of the method in the present application are described in a specific order in the drawings, this does not require or imply that the steps must be performed in this specific order, or that all the steps shown must be performed to achieve the desired results. Additionally or alternatively, some steps may be omitted, multiple steps may be combined into one step, and / or one step may be decomposed into multiple steps, etc.

[0115] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a mobile terminal, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0116] In an exemplary embodiment of the present application, an electronic device capable of implementing the above method is also provided.

[0117] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as a system, method or program product. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: a complete hardware implementation, a complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software, which may be collectively referred to herein as a "circuit", "module" or "system".

[0118] Refer to the following Figure 5 The electronic device 500 according to this embodiment of the present invention is described. Figure 5 The electronic device 500 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0119] like Figure 5As shown, the electronic device 500 is in the form of a general computing device. The components of the electronic device 500 may include but are not limited to: at least one processing unit 510, at least one storage unit 520, and a bus 530 connecting different system components (including the storage unit 520 and the processing unit 510).

[0120] The storage unit stores program codes, which can be executed by the processing unit 510, so that the processing unit 510 performs the steps according to various exemplary embodiments of the present invention described in the above “Exemplary Method” section of this specification. For example, the processing unit 510 can perform the following steps: Figure 1 Follow the steps shown in .

[0121] The storage unit 520 may include a readable medium in the form of a volatile storage unit, such as a random access storage unit (RAM) 5201 and / or a cache storage unit 5202 , and may further include a read-only storage unit (ROM) 5203 .

[0122] The storage unit 520 may also include a program / utility 5204 having a set (at least one) of program modules 5205, such program modules 5205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0123] Bus 530 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0124] The electronic device 500 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a client to interact with the electronic device 500, and / or communicate with any device that enables the electronic device 500 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 550, and may also include a display unit 540 connected to the input / output (I / O) interface 550. In addition, the electronic device 500 may also communicate with one or more networks (e.g., a local area network (LAN), a wide area network (WAN), and / or a public network, such as the Internet) via a network adapter 560. As shown, the network adapter 560 communicates with other modules of the electronic device 500 via a bus 530. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 500, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0125] Through the description of the above implementation methods, it is easy for those skilled in the art to understand that the example implementation methods described here can be implemented by software or by combining software with necessary hardware. Therefore, the technical solution according to the implementation methods of the present application can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the implementation methods of the present application.

[0126] In the exemplary embodiments of the present application, reference is made to Figure 6 As shown, a computer-readable storage medium is also provided, on which a program product capable of implementing the above method of this specification is stored. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps according to various exemplary implementations of the present invention described in the above "Exemplary Method" section of this specification.

[0127] refer to Figure 6As shown, a program product 600 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0128] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0129] Computer readable signal media may include data signals propagated in baseband or as part of a carrier wave, in which readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Readable signal media may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0130] The program code embodied on the readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the foregoing.

[0131] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the client computing device, partially on the client device, as a separate software package, partially on the client computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device may be connected to the client computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0132] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to an exemplary embodiment of the present invention, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0133] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any variations, uses or adaptations of the present application, which follow the general principles of the present application and include common knowledge or customary technical means in the art that are not disclosed in the present application. The specification and embodiments are to be regarded as exemplary only, and the true scope and spirit of the present application are indicated by the claims.

Claims

1. An audio correction method, characterized in that: include: Separate the audio into vocal signal and accompaniment signal; Extracting spectrum information of the audio; Inputting the spectrum information into a model for evaluating the harmony between human voice and accompaniment to obtain the harmony between human voice and accompaniment in the audio, wherein the harmony reflects the degree of deviation of human voice pitch when the human voice and accompaniment in the audio are coordinated; Obtain the type of music from which the audio is sourced; Determining the exact score of the harmony level according to the music type; When the accuracy score is greater than a predetermined threshold, inputting the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal; The modified vocal signal is mixed with the accompaniment signal to obtain modified audio.

2. The method according to claim 1, characterized in that The step of separating the audio into a vocal signal and an accompaniment signal comprises: The audio is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model.

3. The method according to claim 2, characterized in that The step of inputting the audio into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model comprises: Obtaining a waveform graph of the audio, and determining a waveform compression coefficient according to the number of troughs or peaks in the waveform graph; Compressing the height of the waveform according to the compression coefficient to obtain a wave to be separated, and outputting the audio to be separated according to the wave to be separated; The audio to be separated is input into a vocal accompaniment separation model to obtain a vocal signal and an accompaniment signal output by the vocal accompaniment separation model.

4. The method according to claim 1, characterized in that The extracting the spectrum information of the audio includes: Inputting the human voice signal into a trajectory prediction model to obtain a human voice fundamental frequency trajectory; Determining the segmentation points of the audio based on the fundamental frequency trajectory of the human voice according to a predetermined fundamental frequency magnitude change rule; Splitting the audio into a plurality of audio segments according to the segmentation points; Extracting the spectral information of the plurality of audio segments.

5. The method according to claim 4, characterized in that The step of determining the segmentation point of the audio based on the fundamental frequency trajectory of the human voice according to a predetermined fundamental frequency magnitude change rule comprises: The segmentation points of the audio for breathing and word change are determined based on the fundamental frequency trajectory of the human voice and according to the fundamental frequency size change rules during breathing and word change.

6. The method according to claim 4, characterized in that The human voice and accompaniment harmony evaluation model includes a CNN model and an RNN model connected to the CNN model; the step of inputting the spectrum information into the human voice and accompaniment harmony evaluation model to obtain the harmony between the human voice and accompaniment in the audio includes: Inputting the spectrum information of the plurality of audio segments into the CNN model respectively, and obtaining the acoustic features corresponding to each of the audio segments output by the CNN model; The acoustic features are input into the RNN model to obtain the harmony degree between the human voice and the audio corresponding to each of the audio segments output by the RNN model.

7. An audio correction device, characterized in that: include: A separation module, used for separating the audio into a vocal signal and an accompaniment signal; An extraction module, used for extracting spectrum information of the audio; An evaluation module, used for inputting the spectrum information into a harmony evaluation model of human voice and accompaniment to obtain the harmony between human voice and accompaniment in the audio, wherein the harmony reflects the degree of deviation of human voice pitch when the human voice and accompaniment in the audio are coordinated; A correction module, used for obtaining the type of music from which the audio is derived; and determining an accurate score of the harmony degree according to the type of music; When the accuracy score is greater than a predetermined threshold, inputting the human voice signal and the harmony degree into a correction model to obtain a corrected human voice signal; The mixing module is used to mix the modified vocal signal with the accompaniment signal to obtain modified audio.

8. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the method according to any one of claims 1 to 6 is implemented.

9. An electronic device, characterized in that: include: processor; as well as A memory for storing program instructions of the processor; wherein the processor is configured to perform the method of any one of claims 1-6 by executing the program instructions.

Images