Automatic tuning method, device, equipment and computer medium based on network model

Abstract

The embodiments of the present disclosure disclose an automatic tuning method, device, equipment and computer medium based on a network model. A specific implementation of the method includes: controlling an associated player to play background music, and collecting the target user's voice and the music audio of the background music in real time; performing spectrum analysis on the music audio to extract music feature information; performing pitch feature extraction processing on the voice to generate voice pitch features; inputting the voice pitch features and the music feature information into a pre-trained automatic tuning network model to obtain a voice adjustment pitch; adjusting the voice pitch according to the voice adjustment pitch, and outputting the adjusted voice. This implementation does not need to rely on hardware to ensure the tuning effect, improves the audio processing efficiency, and improves the tuning effect for the human voice.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the computer field, and in particular to an automatic tuning method, apparatus, device, and computer medium based on a network model. Background Art

[0002] The current implementation scheme on the market is to use an external sound card to achieve real-time audio adjustment. Through the built-in digital signal processing (DSP) chip or through software connected to the computer, some established algorithms are integrated, including equalizer, compressor, reverberation algorithm, etc., and the audio atmosphere and sound effects of different scenes can be adjusted and simulated by adjusting given parameters.

[0003] However, the above method usually has the following technical problems:

[0004] It is necessary to rely on relatively high-end hardware to ensure the tuning and adjustment effects and low latency of audio processing; usually it can only perform noise reduction and audio environment beautification on the original vocals, but it is impossible to bring the vocals closer to the background music, and the tuning effect is not obvious.

[0005] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known in this country to a person of ordinary skill in the art. Summary of the invention

[0006] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0007] Some embodiments of the present disclosure propose an automatic tuning method, device, electronic device, and computer-readable medium based on a network model to solve one or more of the technical problems mentioned in the above background technology section.

[0008] In a first aspect, some embodiments of the present disclosure provide an automatic tuning method based on a network model, the method comprising: controlling an associated player to play background music, and collecting the voice of a target user and the music audio of the background music in real time; performing spectral analysis on the music audio to extract music feature information, wherein the music feature information comprises: music pitch features and emotion expression features; performing pitch feature extraction processing on the voice to generate voice pitch features; inputting the voice pitch features and the music feature information into a pre-trained automatic tuning network model to obtain a voice adjustment pitch; adjusting the pitch of the voice according to the voice adjustment pitch, and controlling the player to play the adjusted voice.

[0009] In a second aspect, some embodiments of the present disclosure provide an automatic tuning device based on a network model, the device comprising: a control unit, configured to control an associated player to play background music, and to collect the voice of a target user and the music audio of the background music in real time; an analysis unit, configured to perform spectral analysis on the music audio to extract music feature information; an extraction unit, configured to perform pitch feature extraction processing on the voice to generate a voice pitch feature; an input unit, configured to input the voice pitch feature and the music feature information into a pre-trained automatic tuning network model to obtain a voice adjustment pitch; an adjustment unit, configured to adjust the pitch of the voice according to the voice adjustment pitch, and output the adjusted voice.

[0010] In a third aspect, some embodiments of the present disclosure provide an electronic device comprising: one or more processors; a storage device on which one or more programs are stored, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the above-mentioned first aspect.

[0011] In a fourth aspect, some embodiments of the present disclosure provide a computer-readable medium having a computer program stored thereon, wherein when the program is executed by a processor, the method described in any implementation manner of the above-mentioned first aspect is implemented.

[0012] The above-mentioned embodiments of the present disclosure have the following beneficial effects: through the automatic tuning method based on the network model of some embodiments of the present disclosure, there is no need to rely on hardware to ensure the tuning effect, the audio processing efficiency is improved, and the tuning effect for the human voice is improved. Specifically, the reason why it is necessary to rely on relatively high-end hardware to ensure the tuning adjustment effect and the low delay of audio processing and the tuning effect is not obvious is that it is necessary to rely on relatively high-end hardware to ensure the tuning adjustment effect and the low delay of audio processing; usually, only the original human voice can be processed by noise reduction, audio environment effect beautification, etc., and the human voice cannot be approached to the background music, and the tuning effect is not obvious. Based on this, the automatic tuning method based on the network model of some embodiments of the present disclosure, first, controls the associated player to play background music, and collects the target user's voice and the music audio of the background music in real time. Thereby, it is convenient to adjust the pitch of the user's voice according to the background music. Secondly, the music audio is subjected to spectrum analysis to extract music feature information. Then, the voice is subjected to pitch feature extraction processing to generate voice pitch features. Thereby, it is convenient to adjust the voice pitch features according to the music feature information. Then, the voice pitch feature and the music feature information are input into a pre-trained automatic tuning network model to obtain a voice adjustment pitch. Thus, the voice pitch can be adjusted by the pre-trained automatic tuning network model. Finally, according to the voice adjustment pitch, the pitch of the voice is adjusted, and the player is controlled to play the adjusted voice. There is no need to rely on hardware to ensure the tuning effect, which improves the audio processing efficiency and the tuning effect for the human voice. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and that components and elements are not necessarily drawn to scale.

[0014] Figure 1 is a flow chart of some embodiments of the automatic tuning method based on the network model according to the present disclosure;

[0015] Figure 2 is a schematic diagram of the structure of some embodiments of the automatic tuning device based on the network model according to the present disclosure;

[0016] Figure 3 It is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. DETAILED DESCRIPTION

[0017] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although certain embodiments of the present disclosure are shown in the accompanying drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. On the contrary, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0018] It should also be noted that, for ease of description, only the parts related to the invention are shown in the drawings. In the absence of conflict, the embodiments and features in the embodiments of the present disclosure can be combined with each other.

[0019] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0020] It should be noted that the modifications of "one" and "plurality" mentioned in the present disclosure are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0021] The names of the messages or information exchanged between multiple devices in the embodiments of the present disclosure are only used for illustrative purposes and are not used to limit the scope of these messages or information.

[0022] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0023] Figure 1 1 is a flow chart of some embodiments of the automatic tuning method based on a network model according to the present disclosure. 100 of some embodiments of the automatic tuning method based on a network model according to the present disclosure is shown. The automatic tuning method based on a network model comprises the following steps:

[0024] Step 101, controlling the associated player to play background music, and collecting the target user's voice and the music audio of the background music in real time.

[0025] In some embodiments, the execution subject (e.g., computing device) of the automatic tuning method based on the network model can control the associated player to play background music, and collect the target user's voice and the music audio of the background music in real time. The player can be a music player that is connected to the execution subject in communication. The target user can refer to the anchor. That is, the target user's voice and the music of the background music can be collected in real time through a microphone.

[0026] Step 102: Perform spectrum analysis on the music audio to extract music feature information.

[0027] In some embodiments, the execution subject may perform spectrum analysis on the music audio to extract music feature information. The music audio may be subjected to spectrum analysis by a spectrum analyzer to extract music feature information. Frequency analysis: In the time domain, the signal may appear very complex and difficult to directly understand its composition. In the frequency domain, the signal may be decomposed into components of different frequencies. The music feature information includes: music tone features and emotional expression features.

[0028] Step 103: Perform tone feature extraction processing on the speech to generate speech tone features.

[0029] In some embodiments, the above-mentioned execution subject can perform pitch feature extraction processing on the speech to generate speech pitch features. The pitch feature extraction processing on the speech can be performed by a fast Fourier transform (FFT) algorithm, a discrete wavelet transform algorithm, or a pitch detection method based on machine learning to generate speech pitch features. The fast Fourier transform (FFT) algorithm can convert the audio signal in the time domain into a frequency domain representation, and determine the pitch by analyzing the peak position on the spectrum graph. Common methods include techniques based on spectral peaks (SpectraPeaks) or fundamental frequency estimation (PitchEstimation). The FFT algorithm can efficiently calculate the spectrum and is suitable for real-time pitch detection. The discrete wavelet transform algorithm (DWT) can convert the audio signal into wavelet coefficients of different frequencies, and determine the pitch by analyzing the amplitude and position of the wavelet coefficients. The DWT algorithm can provide better frequency resolution than FFT and has better effects on some complex audio signals. The pitch detection algorithm based on machine learning learns the patterns of pitch features by training models, and uses these patterns to predict pitch. Common methods include support vector machine (SVM), random forest (RandomForest) and neural network (NeuraNetwork).

[0030] Optionally, obtain a training sample dataset.

[0031] In some embodiments, the execution subject may obtain a training sample data set, wherein the training sample data in the training sample data set includes: music library audio and audio tags, and the audio tags include: emotion classification tags, male and female classification tags, and tone classification tags.

[0032] Optionally, target training sample data is selected from the training sample data set.

[0033] In some embodiments, the execution subject may select target training sample data from the training sample data set, or may randomly select a training sample data from the training sample data set as the target training sample data.

[0034] Optionally, the music library audio included in the target training sample data is segmented and cut into pieces to obtain an audio frame sequence.

[0035] In some embodiments, the execution subject may slice and cut the music library audio included in the target training sample data to obtain an audio frame sequence. The audio is sliced ​​and cut into several audio frames (10ms) using FFmpeg technology.

[0036] Optionally, the audio frame sequence is subjected to Fourier transform to obtain an audio frame sequence after Fourier transform as a target audio frame sequence.

[0037] In some embodiments, the execution entity may perform Fourier transform on the audio frame sequence to obtain an audio frame sequence after Fourier transform as a target audio frame sequence.

[0038] Optionally, the target audio frame sequence is input into a frequency domain time series feature extraction network included in an initial automatic tuning network model to obtain frequency domain time series feature information.

[0039] In some embodiments, the above-mentioned execution entity can input the target audio frame sequence into the frequency domain time series feature extraction network included in the initial automatic tuning network model to obtain frequency domain time series feature information. The initial automatic tuning network model can be an untrained neural network model for adjusting audio pitch features. For example, the automatic tuning network model can be a deep neural network model constructed by combining convolutional neural network (CNN) and recurrent neural network (RNN) layers. The frequency domain time series feature extraction network can be a convolutional neural network (CNN), which is used to extract frequency domain features (such as frequency components, spectral line shapes and spectrum distribution in audio, etc.) and time series features (rhythm, pitch changes in audio, etc.) as subsequent modeling input parameters. Frequency domain time series feature information may include: frequency domain features, time series features.

[0040] Optionally, the target audio frame sequence is input into a spectrum-based high-order feature extraction network included in an initial automatic tuning network model to obtain audio high-order feature information.

[0041] In some embodiments, the execution subject may input the target audio frame sequence into a spectrum-based high-order feature extraction network included in the initial automatic tuning network model to obtain audio high-price feature information. The audio high-price feature information includes: emotional expression features and music style features. The spectrum-based high-order feature extraction network may refer to a Gaussian mixture model (GMM), which extracts features such as emotional expression and music style of the audio signal.

[0042] Optionally, the frequency domain time series feature information and the audio high price feature information are input into a fully connected layer included in an initial automatic tuning network model to obtain a pitch adjustment result.

[0043] In some embodiments, the execution entity may input the frequency domain time series feature information and the audio high price feature information into a fully connected layer included in the initial automatic tuning network model to obtain a pitch adjustment result. The pitch adjustment result may refer to the adjusted pitch.

[0044] Optionally, an audio loss value between the pitch adjustment result and a corresponding audio tag is determined.

[0045] In some embodiments, the execution subject may determine an audio loss value between the pitch adjustment result and the corresponding audio tag. The audio loss value between the pitch adjustment result and the corresponding audio tag may be determined by a preset loss function. The preset loss function may be a hinge loss function or a cross entropy loss function.

[0046] Optionally, in response to determining that the audio loss value is less than or equal to a preset loss value, the initial automatic tuning network model is determined as a trained automatic tuning network model.

[0047] In some embodiments, the execution entity may determine the initial automatic tuning network model as a trained automatic tuning network model in response to determining that the audio loss value is less than or equal to a preset loss value.

[0048] Optionally, in response to determining that the audio loss value is greater than a preset loss value, the network parameters of the initial automatic tuning network model are adjusted, target training sample data are selected from unselected training sample data, the adjusted initial automatic tuning network model is used as the initial automatic tuning network model, and the initial automatic tuning network model is trained again.

[0049] In some embodiments, the above-mentioned execution entity can adjust the network parameters of the initial automatic tuning network model in response to determining that the audio loss value is greater than a preset loss value, select target training sample data from each unselected training sample data, use the adjusted initial automatic tuning network model as the initial automatic tuning network model, and perform model training on the initial automatic tuning network model again.

[0050] Therefore, for audio live broadcast scenarios, by analyzing the pitch of the background music and the host's voice, the host's voice can be automatically adjusted to improve the audio performance. The deep learning model combined with real-time signal processing technology can intelligently adjust the voice pitch to make it blend and coordinate with the background music, improving the quality of audio live broadcast.

[0051] Step 104: input the speech pitch feature and the music feature information into a pre-trained automatic tuning network model to obtain a speech adjustment pitch.

[0052] In some embodiments, the above-mentioned execution subject can input the voice pitch feature and the music feature information into a pre-trained automatic tuning network model to obtain the voice adjustment pitch. The automatic tuning network model can be a pre-trained neural network model that takes the voice pitch feature and the music feature information as input and takes the voice adjustment pitch as output. The automatic tuning network model can be a deep neural network model constructed by combining convolutional neural network (CNN) and recurrent neural network (RNN) layers. The voice adjustment pitch can be a tone adjustment parameter adjusted corresponding to the voice pitch feature.

[0053] Step 105, adjusting the pitch of the voice according to the voice, adjusting the pitch of the voice, and outputting the adjusted voice.

[0054] In some embodiments, the execution subject can adjust the pitch according to the voice, adjust the pitch of the voice, and control the player to play the adjusted voice. The pitch of the voice can be adjusted by an associated pitch controller, and the adjusted voice can be output. That is, the adjusted voice can be transmitted to other user terminals.

[0055] Optionally, in response to receiving a voice changing instruction, the voice of the target user is collected as the audio to be changed.

[0056] In some embodiments, the execution subject may collect the target user's voice as the audio to be changed in response to receiving the voice change instruction. The voice change instruction may refer to an instruction to change the target user's voice. The voice change instruction may carry the tone style parameters after the voice change.

[0057] Optionally, the audio to be changed is processed into frames to generate an audio frame set.

[0058] In some embodiments, the execution subject may perform frame processing on the audio to be changed to generate an audio frame set. The execution subject may perform frame processing on the audio to be changed with a preset frame length to generate an audio frame set.

[0059] Optionally, clustering processing is performed on the audio frame set to obtain an audio frame group sequence set.

[0060] In some embodiments, the execution subject may perform clustering processing on the audio frame set to obtain an audio frame group sequence set. In practice, the execution subject may perform clustering processing on the audio frame set according to the audio frames corresponding to the words and phrases to generate an audio frame sequence set. That is, the audio frame sequence may represent at least one word.

[0061] Optionally, for each audio frame group in the audio frame group sequence set, the following processing steps are performed:

[0062] In the first step, according to the accent identifier of the target user, a speech recognition model corresponding to the accent identifier is selected from a preset speech recognition model set as a target speech recognition model. The accent identifier may represent the user's regional accent identifier. One speech recognition model corresponds to one accent identifier. The speech recognition model may refer to a neural network model for speech-to-text recognition. For example, the speech recognition model may be a Transformer-based speech recognition (ASR) model.

[0063] In the second step, the audio frame group is input into the target speech recognition model to obtain a text recognition result group corresponding to the audio frame group sequence, wherein the text recognition result in the text recognition result group includes: at least one recognized word corresponding to the audio frame group. The speech recognition model is also used to output each word (text recognition result) that may correspond to the audio frame sequence.

[0064] The third step is to send the text recognition result group to the user terminal of the target user to receive the text recognition result selection information corresponding to the text recognition result group submitted by the user, and determine the target text recognition result. The text recognition result selection information may refer to the user's selection information for a certain text recognition result in the text recognition result group. The target text recognition result may refer to the text recognition result corresponding to the text recognition result selection information. The user terminal may refer to a terminal with a display function.

[0065] Optionally, text-to-speech processing is performed on each determined target text recognition result to obtain synthesized speech.

[0066] In some embodiments, the execution subject may perform text-to-speech processing on each of the determined target text recognition results to obtain synthesized speech. That is, first, each of the determined target text recognition results may be sequentially combined into text; then, the text may be synthesized into speech through text-to-speech technology to obtain synthesized speech.

[0067] Optionally, a tone style corresponding to the voice changing instruction is determined.

[0068] In some embodiments, the execution subject may determine the tone style corresponding to the voice change instruction, that is, the tone style corresponding to the tone style parameter included in the voice change instruction.

[0069] Optionally, a pitch tuning network model corresponding to the pitch style is selected from a preset pitch tuning network model set as the target pitch tuning network model.

[0070] In some embodiments, the above-mentioned execution subject can select a pitch tuning network model corresponding to the pitch style from a preset pitch tuning network model set as a target pitch tuning network model. One pitch tuning network model corresponds to one pitch style. The pitch tuning network model can be a pre-trained neural network model that takes speech pitch features as input and takes pitch tuning results as output. For example, the pitch tuning network model can be a deep neural network model constructed by combining convolutional neural network (CNN) and recurrent neural network (RNN) layers.

[0071] Optionally, the synthesized speech is subjected to a tone feature extraction process to generate a synthesized speech tone feature.

[0072] In some embodiments, the execution subject may perform pitch feature extraction processing on the synthesized speech to generate the pitch feature of the synthesized speech. The pitch feature extraction processing may be performed on the synthesized speech to generate the pitch feature of the synthesized speech by a fast Fourier transform (FFT) algorithm, a discrete wavelet transform algorithm, or a pitch detection method based on machine learning.

[0073] Optionally, the synthesized speech pitch feature is input into the target pitch tuning network model to obtain a pitch tuning result.

[0074] In some embodiments, the execution entity may input the synthesized speech pitch feature into the target pitch tuning network model to obtain a pitch tuning result.

[0075] Optionally, according to the pitch tuning result, the synthesized speech is tuned to generate an adjusted speech, and the player is controlled to play the adjusted speech.

[0076] In some embodiments, the execution subject may perform tuning processing on the synthesized speech according to the pitch tuning result to generate an adjusted speech, and control the player to play the adjusted speech. The synthesized speech may be tuned by an associated pitch controller to generate an adjusted speech.

[0077] Thus, a relatively accurate text recognition result can be selected according to the user's selection of the text recognition result, thereby achieving the recognition accuracy for different accents.

[0078] Optionally, the speech recognition model set can be trained by the following steps:

[0079] The first step is to obtain a set of tone training sample data sets, wherein one speech sample data set corresponds to one accent identifier.

[0080] In the second step, for each speech sample data set in the speech sample data set, the following processing steps are performed:

[0081] 1. Select target speech sample data from the speech sample data group. A speech sample data may be randomly selected from the speech sample data group as the target speech sample data.

[0082] 2. Determine the initial speech recognition model. That is, determine the initial model of the untrained speech recognition model.

[0083] 3. Input the target speech sample data into the initial speech recognition model to obtain an initial speech recognition result.

[0084] 4. Determine the loss value between the initial speech recognition result and the corresponding sample label. That is, the loss value between the initial speech recognition result and the corresponding sample label can be determined by a preset loss function. The preset loss function can be a hinge loss function or a cross entropy loss function.

[0085] 5. In response to determining that the loss value is less than or equal to a preset sample loss value, the initial speech recognition model is determined as the speech recognition model.

[0086] The third step is to determine the determined speech recognition models as a speech recognition model set.

[0087] Optionally, the pitch tuning network model set can be trained by the following steps:

[0088] The first step is to obtain a set of tone training sample data sets, wherein one tone training sample data set corresponds to one tone style, and the tone training sample data includes: audio tone features and audio sample labels, and the audio sample labels include: emotion classification labels, male and female classification labels, and tone classification labels.

[0089] In the second step, for each tone training sample data set in the tone training sample data set set, the following processing steps are performed:

[0090] 1. Select target pitch training sample data from the pitch training sample data group. That is, a pitch training sample data can be randomly selected from the pitch training sample data group as the target pitch training sample data.

[0091] 2. Determine an initial pitch tuning network model. That is, determine an initial model of an untrained pitch tuning network model.

[0092] 3. Input the audio pitch features included in the target pitch training sample data into the initial pitch tuning network model to obtain an initial pitch tuning result.

[0093] 4. Determine an audio loss value between the initial pitch tuning result and the corresponding audio sample label. That is, the audio loss value between the initial pitch tuning result and the corresponding audio sample label can be determined by a preset loss function. The preset loss function can be a hinge loss function or a cross entropy loss function.

[0094] 5. In response to determining that the audio loss value is less than or equal to a preset audio sample loss value, determining the initial pitch tuning network model as the pitch tuning network model.

[0095] The third step is to determine the determined pitch tuning network models as a pitch tuning network model set.

[0096] Further references Figure 2 As an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of an automatic tuning device based on a network model. These embodiments of the automatic tuning device based on a network model are similar to Figure 1 Corresponding to the method embodiments shown, the automatic tuning device based on the network model can be specifically applied to various electronic devices.

[0097] like Figure 2As shown, some embodiments of the automatic tuning device 200 based on the network model include: a control unit 201, an analysis unit 202, an extraction unit 203, an input unit 204 and an adjustment unit 205. The control unit 201 is configured to control the associated player to play background music, and collect the target user's voice and the music audio of the background music in real time; the analysis unit 202 is configured to perform spectrum analysis on the music audio to extract music feature information, wherein the music feature information includes: music pitch features and emotional expression features; the extraction unit 203 is configured to perform pitch feature extraction processing on the voice to generate voice pitch features; the input unit 204 is configured to input the voice pitch features and the music feature information into the pre-trained automatic tuning network model to obtain the voice adjustment pitch; the adjustment unit 205 is configured to adjust the pitch of the voice according to the voice adjustment pitch, and output the adjusted voice.

[0098] It can be understood that the various units recorded in the automatic tuning device 300 based on the network model are similar to those in the reference Figure 1 Therefore, the operations, features and beneficial effects described above for the method are also applicable to the automatic tuning device 300 based on the network model and the units contained therein, and will not be described in detail here.

[0099] Reference below Figure 3 , which shows a schematic diagram of the structure of an electronic device 300 (e.g., a computing device) suitable for implementing some embodiments of the present disclosure. The electronic devices in some embodiments of the present disclosure may include, but are not limited to, mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 3 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0100] like Figure 3 As shown, the electronic device 300 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 301, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 302 or a program loaded from a storage device 308 into a random access memory (RAM) 303. In the RAM 303, various programs and data required for the operation of the electronic device 300 are also stored. The processing device 301, the ROM 302, and the RAM 303 are connected to each other via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0101] Typically, the following devices may be connected to the I / O interface 305: input devices 306 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 307 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 308 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 309. The communication devices 309 may allow the electronic device 300 to communicate with other devices wirelessly or by wire to exchange data. Although Figure 3 The electronic device 300 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead. Figure 3 Each block shown in the figure may represent one device, or may represent multiple devices as required.

[0102] In particular, according to some embodiments of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, some embodiments of the present disclosure include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In some such embodiments, the computer program can be downloaded and installed from the network through the communication device 309, or installed from the storage device 308, or installed from the ROM 302. When the computer program is executed by the processing device 301, the above-mentioned functions defined in the method of some embodiments of the present disclosure are executed.

[0103] It should be noted that the computer-readable medium recorded in some embodiments of the present disclosure may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In some embodiments of the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In some embodiments of the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a computer-readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0104] In some embodiments, the client and the server may communicate using any currently known or future developed network protocol such as HTTP (HyperText Transfer Protocol), and may be interconnected with any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network ("LAN"), a wide area network ("WAN"), an internet (e.g., the Internet), and a peer-to-peer network (e.g., an ad hoc peer-to-peer network), as well as any currently known or future developed network.

[0105] The computer-readable medium may be included in the electronic device; or it may exist independently without being installed in the electronic device. The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device: controls the associated player to play background music, and collects the target user's voice and the music audio of the background music in real time; performs spectrum analysis on the music audio to extract music feature information, wherein the music feature information includes: music pitch features and emotional expression features; performs pitch feature extraction processing on the voice to generate voice pitch features; inputs the voice pitch features and the music feature information into a pre-trained automatic tuning network model to obtain a voice adjustment pitch; adjusts the pitch of the voice according to the voice adjustment pitch, and controls the player to play the adjusted voice.

[0106] Computer program code for performing the operations of some embodiments of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0107] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0108] The units described in some embodiments of the present disclosure may be implemented by software or hardware. The described units may also be provided in a processor, for example, may be described as: a processor comprising: a control unit, an analysis unit, an extraction unit, an input unit, and an adjustment unit. The names of these units do not, in some cases, constitute limitations on the units themselves, for example, the analysis unit may also be described as "a unit for performing spectrum analysis on the music audio to extract music feature information".

[0109] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0110] The above descriptions are only some preferred embodiments of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above-mentioned technical features, but should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the above-mentioned features are replaced with the technical features with similar functions disclosed in the embodiments of the present disclosure (but not limited to) and the technical solutions formed.

Claims

1. An automatic tuning method based on a network model, comprising: Control the associated player to play background music, and collect the target user's voice and the music audio of the background music in real time; Performing spectrum analysis on the music audio to extract music feature information, wherein the music feature information includes: music tone features and emotion expression features; Performing pitch feature extraction processing on the speech to generate speech pitch features; Obtain a training sample data set, wherein the training sample data in the training sample data set includes: music library audio and audio tags, and the audio tags include: emotion classification tags, male and female classification tags, and tone classification tags; Selecting target training sample data from the training sample data set; Slice and cut the music library audio included in the target training sample data to obtain an audio frame sequence; Performing Fourier transform on the audio frame sequence to obtain an audio frame sequence after Fourier transform as a target audio frame sequence; Inputting the target audio frame sequence into a frequency domain temporal feature extraction network included in the initial automatic tuning network model to obtain frequency domain temporal feature information; Inputting the target audio frame sequence into a spectrum-based high-order feature extraction network included in the initial automatic tuning network model to obtain audio high-price feature information, wherein the audio high-price feature information includes: emotional expression features and music style features; Inputting the frequency domain time series feature information and the audio high price feature information into a fully connected layer included in the initial automatic tuning network model to obtain a pitch adjustment result; determining an audio loss value between the pitch adjustment result and a corresponding audio tag; In response to determining that the audio loss value is less than or equal to a preset loss value, determining the initial automatic tuning network model as a trained automatic tuning network model; Inputting the speech pitch feature and the music feature information into a pre-trained automatic tuning network model to obtain a speech adjustment pitch; The pitch of the speech is adjusted according to the speech, the pitch of the speech is adjusted, and the adjusted speech is output.

2. The method according to claim 1, wherein: The method further comprises: In response to determining that the audio loss value is greater than a preset loss value, the network parameters of the initial automatic tuning network model are adjusted, target training sample data are selected from the unselected training sample data, the adjusted initial automatic tuning network model is used as the initial automatic tuning network model, and the initial automatic tuning network model is trained again.

3. An automatic tuning device based on a network model, applied to the method according to claim 1, comprising: A control unit, configured to control an associated player to play background music, and to collect the target user's voice and the music audio of the background music in real time; An analysis unit is configured to perform spectrum analysis on the music audio to extract music feature information, wherein the music feature information includes: music tone features and emotion expression features; An extraction unit, configured to perform a tone feature extraction process on the speech to generate a speech tone feature; An input unit, configured to input the voice pitch feature and the music feature information into a pre-trained automatic tuning network model to obtain a voice adjustment pitch; The adjusting unit is configured to adjust the pitch according to the voice, adjust the pitch of the voice, and output the adjusted voice.

4. An electronic device comprising: one or more processors; a storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1-2.

5. A computer readable medium having a computer program stored thereon, wherein: When the program is executed by a processor, the method according to any one of claims 1 to 2 is implemented.

Images