A method for generating presets for multi-model integrated effects based on deep learning

Abstract

The present invention discloses a method for generating presets for a multi-model integrated effects processor based on deep learning, wherein a user uploads a song to a host computer; the host computer intercepts a segment of the song from which a timbre preset needs to be extracted, and then uploads the segment to a cloud server; the deep learning model in the cloud server infers the timbre preset at the guitar playing part based on the segment, and generates a timbre preset file and returns it to the host computer; the host computer downloads the generated timbre preset file to the integrated effects processor, completing the generation of the entire timbre preset. The method for generating presets for a multi-model integrated effects processor based on deep learning provided by the present invention simplifies the cumbersome production and debugging process of timbre presets in traditional integrated effects processors, and helps users to make timbre presets better and faster.

Description

Technical Field

[0001] The present invention relates to the field of music technology, and in particular discloses a method for generating presets for a multi-model integrated effects device based on deep learning. Background Art

[0002] As a branch of guitar instruments, the main difference between electric guitar and traditional acoustic guitar or classical guitar is that acoustic guitar emphasizes restoring the original sound of the instrument as much as possible, while electric guitar often uses various special instrument speakers and effects to change its original sound, thereby creating a unique sound suitable for different styles. This is one of the reasons why electric guitar is widely used in various styles of modern music.

[0003] Traditional electric guitar audio processing equipment is mainly based on analog circuit devices, such as tube speakers and single-block effects. This type of equipment is not only expensive and bulky, but each device also needs to be powered by a professional power supply. Each device also needs to be connected and routed with professional audio cables to form a complete effect chain and obtain the final output. If you need to quickly change the signal routing structure or quickly switch a parameter in a certain device, you may need to use multiple identical devices. The debugging, maintenance and transportation of such a set of sound systems have long been a difficult problem that musicians need to face.

[0004] With the continuous development of technologies such as digital signal processing and analog circuit modeling, analog circuit devices such as tube speakers and single-block effects that were originally needed in the process of shaping the tone of electric guitars have gradually been replaced by digital effects devices, among which the digital integrated effects device (hereinafter referred to as integrated effects device) is the representative. Integrated effects devices can model the target analog circuit devices through algorithms, thereby restoring the tone of the target analog circuit devices. More importantly, the call, parameter configuration, and signal routing of different effect processing modules can all be realized and managed through tone presets inside the integrated effects device. Musicians only need to carry one integrated effects device to achieve the tone and functions that originally required a large number of analog circuit devices to achieve, which greatly simplifies the difficulty of debugging, maintaining, and transporting the tone system, and also reduces the entry threshold for equipment for amateur electric guitar enthusiasts or beginners.

[0005] While providing convenience to users, multi-effects processors also face some new problems. First, the algorithms in multi-effects processors are usually modeled after real analog devices, and their timbre characteristics and parameter debugging techniques often follow the real analog devices, which means that selecting and debugging an effect module requires users to have a lot of real device debugging experience. Secondly, a current multi-effects processor usually integrates hundreds or thousands of effect module algorithms, each of which has several to dozens of control parameters. Different algorithms can be freely arranged and combined, and there are also multiple options for signal chain routing. This makes it difficult for amateurs or users who are not familiar with multi-effects processors to complete the production of timbre presets according to their own timbre needs, and thus obtain the timbre they want in the arrangement of a large number of algorithms and parameters. Summary of the invention

[0006] The present invention provides a multi-model integrated effects device preset generation method based on deep learning, aiming to solve at least one of the above-mentioned problems.

[0007] The present invention relates to a method for generating presets of a multi-model integrated effects device based on deep learning, comprising the following steps:

[0008] The user end uploads songs to the host computer;

[0009] The host computer intercepts the segment of the song that needs to extract the preset sound, and then uploads the segment to the cloud server;

[0010] The deep learning model in the cloud server infers the timbre preset of the guitar playing part based on the clip, and generates a timbre preset file and returns it to the host computer;

[0011] The host computer downloads the generated tone preset file to the integrated effects processor to complete the entire tone preset generation.

[0012] Furthermore, the deep learning model in the cloud server infers the timbre preset of the guitar playing part according to the clip, and generates a timbre preset file and returns it to the host computer, including the following steps:

[0013] The music source separation module separates and extracts the tracks of different instruments in the clip and outputs guitar track split files. The tracks include vocals, bass, keyboard, drums, lead guitar, rhythm guitar and other tracks. The guitar track split files include lead guitar other tracks and rhythm guitar tracks.

[0014] The timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type of the guitar track, and outputs a timbre type label;

[0015] The speaker parameter estimation module narrows down the selection range of the speaker simulation algorithm according to the timbre type label, inputs the guitar track file into the second classification neural network for speaker simulation algorithm type recognition and parameter estimation, and outputs the type and parameters of the speaker simulation algorithm to match the guitar timbre in the given track;

[0016] The speaker frequency response matching module generates a first timbre preset file with only the cabinet head and cabinet algorithm according to the parameter estimation result and returns it to the user terminal, obtains the output audio of the performance content loaded by the user terminal with the timbre preset file, and uses the target track audio as a reference to match the overall frequency response of the target timbre with the estimated timbre, so as to further improve the similarity of the timbre;

[0017] The music feature extraction module extracts feature values ​​related to the timbre preset in the audio file uploaded by the user terminal;

[0018] The music style classification module takes the audio file as input, uses the third classification neural network for music style classification to determine the type of music style to which the audio belongs, and outputs a music style label;

[0019] The performance content classification module takes the guitar track file as input, uses the fourth neural network for performance content classification to determine the performance content type of the guitar track, and outputs a performance content type label;

[0020] The preset similarity matching module takes the music style tag, the performance content tag, and the timbre type tag as input, matches the preset files matching the given tags in a preset database having a large number of timbre presets, selects a second timbre preset file that best matches the target timbre based on the guitar track audio file, and determines the parameters of other effect modules;

[0021] All algorithm module information and parameters are aggregated into a third timbre preset file and returned to the user's host computer to complete the generation of the timbre preset.

[0022] Furthermore, the music source separation module separates and extracts the tracks of different instruments in the clip, and outputs the guitar track split file, including:

[0023] Perform short-time Fourier transform on the mixed audio to obtain the mixed audio spectrum X;

[0024] Extract the eigenvalue Z of the mixed audio spectrum X, and transform the eigenvalue Z into a transition eigenvalue Q using time series modeling;

[0025] According to the transition characteristic value Q, the time-frequency masking coefficient M is calculated by using masking estimation;

[0026] Multiply the mixed spectrum X by the time-frequency masking coefficient M to obtain the sub-track spectrum S;

[0027] Perform short-time inverse Fourier transform on the sub-track spectrum to obtain the sub-track audio.

[0028] Further, in the step of performing short-time Fourier transform on the mixed audio to obtain the mixed spectrum X, the mixed spectrum X is a spectrum composed of complex numbers, which is expressed as:

[0029]

[0030] Among them, X is the mixed frequency spectrum, F is the frequency dimension, and T is the time dimension. is a plural set.

[0031] Further, the steps of extracting the eigenvalue Z of the mixed spectrum X and transforming the eigenvalue Z into the transition eigenvalue Q by using time series modeling include:

[0032] The complex spectrum According to the given sub-band frequency configuration f i , divided into K sub-band spectrum Where i = 1,…,K; f i is the target frequency band, B i f i The corresponding sub-band spectrum;

[0033] Each sub-band spectrum B i The real and imaginary parts are concatenated and input into the band feature extraction module. The band feature extraction module consists of a layer normalization and a fully connected layer. The band feature extraction module will output the sub-band spectrum features composed of real numbers. Among them, Z i is the sub-band spectrum feature; is a set of real numbers; N is the feature dimension;

[0034] All K sub-band spectrum features Merge into a full-band eigenvalue And output to the next link.

[0035] Further, the steps of extracting the eigenvalue Z of the mixed spectrum X and transforming the eigenvalue Z into the transition eigenvalue Q by using time series modeling include:

[0036] For time series, a set of RNNs are used to perform feature analysis on the time series dimension T. Modeling; for the frequency band, a set of RNNs is used to model the feature values ​​on the frequency band dimension K Modeling is performed, where Z i is the eigenvalue of the full frequency band; It is the full-band eigenvalue after time series modeling;

[0037] The output of RNN is obtained by linking the input with the output residual of the fully connected layer. The final output of time-frequency modeling is defined as Where Q is the transition characteristic value.

[0038] Further, the step of calculating the time-frequency masking coefficient M by using masking estimation according to the transition characteristic value Q includes:

[0039] Transform the feature value Divide into K groups of sub-band features i=1,..,K; each set of sub-band features has the same frequency division configuration as the previous feature extraction module, where Q i is the sub-band characteristic;

[0040] Each set of sub-band features is input into layer normalization and multi-layer perceptron to obtain its corresponding sub-band time-frequency masking coefficient i=1,…,K; where M i is the sub-band time-frequency masking coefficient, C is;

[0041] All sub-band time-frequency masking coefficients M i Splice them together to get the full-band time-frequency masking coefficient M∈C F×T ; Where M is the time-frequency masking coefficient and F is the frequency dimension;

[0042] The full-band time-frequency masking coefficient M∈C F×T Mixed spectrum with input Multiply to get the final target spectrum Perform ISTFT on the target spectrum to obtain the target audio, where X is the mixed spectrum and S is the standard spectrum.

[0043] Furthermore, the timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type of the guitar track, and in the step of outputting the timbre type label, divides the audio in the data set into segments of set time, extracts audio features from the segments, and extracts the audio features as Mel spectra. The conversion relationship between Mel spectra and frequency is:

[0044]

[0045] Among them, Mef(f) is the Mel spectrum and f is the frequency.

[0046] Furthermore, the timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type to which the guitar track belongs, and in the step of outputting the timbre type label, the Mel filter bank is a group of triangular filters on the Mel frequency scale, and the calculation formula of the Mel filter bank is:

[0047]

[0048] Among them, Center(m) is the center frequency of the triangular filter group, which is determined by the number of triangular filters; Center(m+1) is the center frequency corresponding to the m+1 filter; Center(m-1) is the center frequency corresponding to the m-1 filter; k is the FFT spectrum number, and m is the filter number.

[0049] Furthermore, the timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type to which the guitar track belongs, and in the step of outputting the timbre type label, convolves each triangular filter with the energy spectrum of the Mel spectrum to obtain a set of coefficients corresponding to the triangular filter, and then takes the logarithm of the coefficient to obtain the coefficient S(m) used to calculate the starting intensity envelope:

[0050]

[0051] Where S(m) is the coefficient of the initial intensity envelope; H m (k) is the Mel filter bank, Mel(k) is the Mel spectrum, and * is the convolution operation.

[0052] The beneficial effects achieved by the present invention are:

[0053] The present invention provides a method for generating presets for a multi-model integrated effects processor based on deep learning, wherein a user uploads a song to a host computer through a user terminal; the host computer intercepts a segment of the song from which a timbre preset needs to be extracted, and then uploads the segment to a cloud server; the deep learning model in the cloud server infers the timbre preset at the guitar playing part based on the segment, and generates a timbre preset file and returns it to the host computer; the host computer downloads the generated timbre preset file to the integrated effects processor to complete the generation of the entire timbre preset. The method for generating presets for a multi-model integrated effects processor based on deep learning provided by the present invention simplifies the cumbersome production and debugging process of timbre presets in traditional integrated effects processors, and helps users to make timbre presets better and faster. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] Figure 1 A schematic diagram of the timbre preset generation process of the present invention;

[0055] Figure 2 A diagram of the structure of the algorithm module preset for the present invention;

[0056] Figure 3 It is a schematic diagram of the flow of the music source separation module of the present invention;

[0057] Figure 4 It is a schematic diagram of the frequency division feature extraction process of the present invention;

[0058] Figure 5 This is a schematic diagram of the time-frequency modeling process of the present invention;

[0059] Figure 6 A schematic diagram of a process of mask estimation of the present invention;

[0060] Figure 7 Schematic diagram of the model structure of the timbre type classification module of the present invention;

[0061] Figure 8 It is a schematic diagram of the structure of the model of the present invention;

[0062] Fig. 9 This is a schematic diagram of the flow of the speaker frequency response matching module of the present invention;

[0063] Fig.10 It is a flow chart of the BPM detection algorithm of the present invention;

[0064] Fig.11 It is a flowchart of the modulation detection algorithm of the present invention. DETAILED DESCRIPTION

[0065] In order to better understand the above technical solution, the above technical solution will be described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0066] like Figures 1 to 11 As shown, the first embodiment of the present invention proposes a method for generating a preset for a multi-model integrated effects device based on deep learning, comprising the following steps:

[0067] Step S100: The user terminal uploads songs to the host computer.

[0068] The user uploads the song in the host computer control software of the integrated effects processor.

[0069] Step S200: The host computer intercepts the segment of the song that needs to extract the preset timbre, and then uploads the segment to the cloud server.

[0070] The host computer intercepts the segment that needs to extract the preset tone, and uploads the intercepted audio to the cloud server.

[0071] Step S300: The deep learning model in the cloud server infers the timbre preset of the guitar playing part based on the clip, and generates a timbre preset file and returns it to the host computer.

[0072] The deep learning model in the cloud server will infer the tone preset that is closest to the guitar playing part in the audio uploaded by the user, and generate a tone preset file and return it to the host computer software.

[0073] Step S400: The host computer downloads the generated timbre preset file to the integrated effects processor, completing the generation of the entire timbre preset.

[0074] The upper computer software downloads the generated tone preset file to the integrated effects processor via the Bluetooth protocol, completing the entire tone preset generation process.

[0075] Further, see Figures 1 to 11 In the method for generating presets for multi-model integrated effects based on deep learning provided in this embodiment, when generating a preset timbre file, the multi-model integrated effects preset generation algorithm based on deep learning used in the cloud server is composed of multiple sub-modules, and step S300 includes:

[0076] Step S310, the music source separation module separates and extracts the audio tracks of different instruments in the clip, and outputs a guitar track split file, the audio tracks include vocals, bass, keyboard, drums, lead guitar, rhythm guitar and other audio tracks, and the guitar track split file includes lead guitar other audio tracks and rhythm guitar audio tracks.

[0077] The music source separation module will separate and extract the audio tracks of different instruments from the audio files uploaded by the user, including vocals, bass, keyboards, drums, lead guitar, rhythm guitar and other tracks, and output the split-track audio files of the lead guitar and rhythm guitar tracks as results to subsequent modules.

[0078] Step S320: The timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type to which the guitar track belongs, and outputs a timbre type label.

[0079] The tone type classification module will separate and extract the audio tracks of different instruments from the audio files uploaded by the user, including vocals, bass, keyboards, drums, lead guitar, rhythm guitar and other tracks, and output the split-track audio files of the lead guitar and rhythm guitar tracks as results to subsequent modules.

[0080] Step S330, the speaker parameter estimation module narrows down the selection range of the speaker simulation algorithm according to the timbre type label, inputs the guitar track file into the second classification neural network for speaker simulation algorithm type recognition and parameter estimation, and outputs the type and parameters of the speaker simulation algorithm to match the guitar timbre in the given track.

[0081] The speaker parameter estimation module will first narrow down the selection range of speaker simulation algorithms based on the timbre type labels output by the timbre type classification module. Then the guitar track files extracted by the music source separation module will be input into the neural network for speaker simulation algorithm type identification and parameter estimation. Finally, the speaker parameter estimation module will output the type and parameters of the speaker simulation algorithm to match the guitar timbre in the given track.

[0082] Step S340, the speaker frequency response matching module generates a first timbre preset file with only the amp head and cabinet algorithm according to the result of parameter estimation and returns it to the user terminal, obtains the output audio of the performance content loaded by the user terminal with the timbre preset file, and uses the target track audio as a reference to match the overall frequency response of the target timbre with the estimated timbre, so as to further improve the similarity of the timbre.

[0083] The speaker frequency response matching module will generate a preset file with only the head and cabinet algorithm based on the results of the speaker parameter estimation module and return it to the user. After the user loads the preset file and plays a piece of content, the system will obtain its output audio and use the target track audio as a reference to perform the EQ Match algorithm to match the overall frequency response of the target timbre and the estimated timbre to further improve the similarity of the timbre. After completing this step, the parameters of the head, cabinet, and EQ (Equalize) modules are determined.

[0084] Step S350: The music feature extraction module extracts feature values ​​related to the timbre preset in the audio file uploaded by the user terminal.

[0085] The music feature extraction module will extract the feature values ​​related to the preset timbre from the audio files uploaded by the user, including music speed (BPM, beats per minute), music mode, left and right channel audio correlation coefficient, stereo sound image, etc. These music feature values ​​will be used as the numerical output of the time, mixing ratio and other parameters in the peripheral effect modules such as delay and modulation.

[0086] Step S360: The music style classification module takes the audio file as input, uses the third classification neural network for music style classification to determine the music style type to which the audio belongs, and outputs a music style label.

[0087] The music style classification module will take the audio file uploaded by the user as input and use the music style classification neural network to determine the music style type of the audio. The music style types include: blues, classical, country, folk, funk, jazz, metal, pop, punk, hard rock, etc. The music style label will be output as the result to the subsequent modules.

[0088] Step S370: The performance content classification module takes the guitar track file as input, uses the fourth neural network for performance content classification to determine the performance content type of the guitar track, and outputs a performance content type label.

[0089] The performance content classification module will use the guitar track files extracted in the music source separation module as input, and use the performance content classification neural network to determine the performance content type of the guitar track, including chord strumming, chord decomposition, riff rhythm, lead solo, etc. The performance content type label will be output as a result to the subsequent modules.

[0090] Step S380, the preset similarity matching module takes the music style tag, the performance content tag, and the timbre type tag as input, matches the preset files that match the given tags in the preset database with a large number of timbre presets, and selects a second timbre preset file that best matches the target timbre based on the guitar track audio file, and determines the parameters of other effect modules.

[0091] The preset similarity matching module takes the music style label output from the music style classification module, the performance content label output from the performance content classification module, and the timbre type label output from the timbre type classification module as input. It matches the preset files that match the given labels in the preset database with a large number of timbre presets, and selects a preset file that best matches the target timbre based on the guitar split track audio file output from the music source separation module. After completing this step, the parameters of other effect modules are determined.

[0092] Step S390: Aggregate all algorithm module information and parameters into a third timbre preset file and return it to the user's host computer, completing the generation of the timbre preset.

[0093] Finally, all algorithm module information and parameters are summarized into a preset file and returned to the user's host computer software, completing the tone preset generation process.

[0094] Preferably, see Figures 1 to 11 In the method for generating presets of multi-model integrated effects devices based on deep learning provided in this embodiment, step S310 includes:

[0095] Step S311: Perform short-time Fourier transform on the mixed audio to obtain the mixed audio spectrum X.

[0096] The music source separation module will separate and extract the audio tracks of different instruments from the audio files uploaded by the user, including vocals, bass, keyboards, drums, lead guitar, rhythm guitar and other tracks, and output the split-track audio files of the lead guitar and rhythm guitar tracks as results to subsequent modules.

[0097] The music source separation module is implemented using a neural network and is composed of a feature extraction module, a time-frequency modeling module, and a masking estimation module. First, the mixed audio is subjected to a short-time Fourier transform (STFT) to obtain the mixed audio spectrum.

[0098] Step S312: extract the eigenvalue Z of the mixed spectrum X, and transform the eigenvalue Z into a transition eigenvalue Q by using time series modeling.

[0099] The feature extraction module extracts the eigenvalue Z of the mixed frequency spectrum, and the eigenvalue Z is transformed into the transition eigenvalue Q using time series modeling.

[0100] Step S313: Calculate the time-frequency masking coefficient M based on the transition characteristic value Q using masking estimation.

[0101] The time-frequency modeling module transforms the eigenvalue Q, and the masking estimation module calculates the time-frequency masking coefficient M based on the transformed eigenvalue Q using masking estimation.

[0102] Step S314: multiply the mixed audio spectrum X by the time-frequency masking coefficient M to obtain the sub-track spectrum S.

[0103] Then, the mixed spectrum X is multiplied by the time-frequency masking coefficient M to obtain the sub-track spectrum S.

[0104] Step S315: perform inverse short-time Fourier transform on the sub-track spectrum to obtain the sub-track audio.

[0105] Finally, perform an inverse short-time Fourier transform (ISTFT) on the sub-track spectrum to obtain the sub-track audio.

[0106] Further, see Figures 1 to 11 In the method for generating presets for multi-model integrated effects based on deep learning provided in this embodiment, in step S311, the mixed audio spectrum X is a spectrum composed of complex numbers, which is expressed as:

[0107]

[0108] In formula (1), X is the mixed frequency spectrum, F is the frequency dimension, and T is the time dimension. is a plural set.

[0109] Further, see Figures 1 to 11 In the method for generating presets of multi-model integrated effects devices based on deep learning provided in this embodiment, step S312 includes:

[0110] Step S3121: complex spectrum According to the given sub-band frequency configuration f i , divided into K sub-band spectrum Where i = 1,…,K; f i is the target frequency band, B i f i The corresponding sub-band spectrum.

[0111] Step S3122: each sub-band spectrum B i The real and imaginary parts are concatenated and input into the band feature extraction module. The band feature extraction module consists of a layer normalization and a fully connected layer. The band feature extraction module will output the sub-band spectrum features composed of real numbers. Among them, Z i is the sub-band spectrum feature; is a set of real numbers; N is the feature dimension.

[0112] Step S3123: All K sub-band spectrum features Merge into a full-band eigenvalue And output to the next link.

[0113] Preferably, see Figures 1 to 11 In the method for generating presets of multi-model integrated effects devices based on deep learning provided in this embodiment, step S312 includes:

[0114] Step S3124: For the time series, use a set of RNNs to perform feature value Modeling; for the frequency band, a set of RNNs is used to model the feature values ​​on the frequency band dimension K Modeling is performed, where Z i is the eigenvalue of the full frequency band; It is the full-band feature value after time series modeling.

[0115] Step S3125: Link the input to the output residual of the fully connected layer (FC) to obtain the output of the RNN. The final output of the time-frequency modeling is defined as Where Q is the transition characteristic value.

[0116] Further, see Figures 1 to 11 In the method for generating presets of multi-model integrated effects devices based on deep learning provided in this embodiment, step S313 includes:

[0117] Step S3131: Transform the characteristic value Divide into K groups of sub-band features i=1,..,K; each set of sub-band features has the same frequency division configuration as the previous feature extraction module, where Q i is the sub-band feature.

[0118] Step S3132: Input each set of sub-band features into layer normalization and multi-layer perceptron to obtain its corresponding sub-band time-frequency masking coefficient. i=1,…,K; where M i is the subband time-frequency masking coefficient, and C is a complex number set.

[0119] Step S3133: Set all sub-band time-frequency masking coefficients M i Splice them together to get the full-band time-frequency masking coefficient M∈C F×T ; Where M is the time-frequency masking coefficient and F is the frequency dimension.

[0120] Step S3134: The time-frequency masking coefficient M∈C F×T Mixed spectrum with input Multiply to get the final target spectrum Perform ISTFT on the target spectrum to obtain the target audio, where X is the mixed spectrum and S is the standard spectrum.

[0121] Preferably, see Figures 1 to 11 In the multi-model integrated effects preset generation method based on deep learning provided in this embodiment, in step S320, the audio in the data set is divided into segments of set time, and audio features are extracted from the segments. The extracted audio features are Mel spectra, and the conversion relationship between Mel spectra and frequency is:

[0122]

[0123] In formula (2), Mef(f) is the Mel spectrum, and f is the frequency.

[0124] Mel spectrum is a frequency scale based on human auditory distribution. In the Mel frequency domain, human perception of pitch is linear. For example, if the Mel frequency of two speech segments differs by two times, the human ear will hear the pitch of the two segments differ by two times.

[0125] Further, see Figures 1 to 11 In the multi-model integrated effects preset generation method based on deep learning provided in this embodiment, in step S320, the Mel filter bank is a group of triangular filters on the Mel frequency scale, and the calculation formula of the Mel filter bank is:

[0126]

[0127] In formula (3), Center(m) is the center frequency of the triangular filter group, which is determined by the number of triangular filters; Center(m+1) is the center frequency corresponding to the m+1 filter; Center(m-1) is the center frequency corresponding to the m-1 filter; k is the FFT spectrum index, and m is the filter index.

[0128] The Mel filter bank is a set of triangular filters on the Mel frequency scale. The passband bandwidth of the Mel filter bank is uniform and evenly distributed on the Mel spectrum with 50% overlap.

[0129] Further, see Figures 1 to 11 In the multi-model integrated effector preset generation method based on deep learning provided in this embodiment, in step S320, each triangular filter is convolved with the energy spectrum of the Mel spectrum to obtain a set of coefficients corresponding to the triangular filter, and then the logarithm is taken to obtain the coefficient S(m) for calculating the starting intensity envelope:

[0130]

[0131] In formula (4), S(m) is the coefficient of the initial intensity envelope; H m (k) is the Mel filter bank, Mel(k) is the Mel spectrum, and * is the convolution operation.

[0132] The previously extracted logarithmic Mel spectrum is input into the neural network for classification in the form of an image. The classification network consists of three layers of convolutional neural networks (CNNs), and each layer of CNN consists of a two-dimensional convolution layer, an activation function, a two-dimensional maximum pooling layer, and a Dropout layer. Finally, it is input into a module consisting of a Dense layer and a Dropout layer, and the classification result is output by the Softmax layer.

[0133] The same classification network will be used in the subsequent music style classification module and performance content classification module, but the input training data and output classification labels will be different.

[0134] Further, see Figures 1 to 11 In the method for generating presets of multi-model integrated effects based on deep learning provided by this embodiment, in step S330, the speaker parameter estimation module first narrows down the selection range of the speaker simulation algorithm according to the timbre type label output by the timbre type classification module. Then, the guitar track file extracted from the music source separation module is input into the neural network for speaker simulation algorithm type identification and parameter estimation. Finally, this module outputs the type and parameters of the speaker simulation algorithm to match the guitar timbre in the given track.

[0135] The training dataset consists of dry sounds from the IDMT-SMT-Audio-Effects dataset. It consists of recordings (wav-44.1kHz, 16-bit, mono) of monophonic (624 single notes) and chords (420 intervals and chords) from 2 different electric guitars. Each has two pickup settings and up to 3 picking styles. The monophonic recordings cover the common pitch range of a 6-string electric guitar, while the polyphonic samples are generated by mixing monophonic recordings. All samples are 2 seconds long. The monophonic recordings require the background noise to be removed before the note starts.

[0136] The wet sound was created using the 80 amp and cabinet modeling algorithms in this multi-effects processor, where the amp and cabinet pairing followed the default factory settings. All amp modeling knobs can be summarized as Gain, Bass, Middle, Treble, Presence, and the other knobs were set to default. 10,000 random samples were generated for each modeling algorithm, resulting in a total of 260,000 samples (130,000 single notes, 130,000 chords), equivalent to about 140 hours. Finally, both the unprocessed input and processed output were normalized to -6dBFS.

[0137] The schematic diagram of the model structure is as follows: Figure 8 As shown in the figure, it consists of 2 convolutional layers and 3 fully connected layers, and each hidden layer has a batch normalization layer.

[0138] The input features of the model are Mel power spectra, extracted from the audio in 23ms Hann windows with an overlap of 50%, using a total of 128 Mel bands in the range of 0-22050Hz. For a given 2s audio input, feature extraction produces a Tx128 output (T=87).

[0139] Further, see Figures 1 to 11 In the multi-model integrated effects preset generation method based on deep learning provided in this embodiment, in step S340, the speaker frequency response matching module will match the frequency response difference between the target track audio and the speaker parameter estimation result to further improve the similarity of the overall timbre. First, read the data of the source audio and the target audio, and obtain the frequency response difference between the two based on the power spectral density of the two. Then, perform a peak search on the frequency response difference, confirm the initial parameters of the filter group based on the peak information, and finally, use the gradient descent and back propagation algorithm to update the parameters of the filter group to minimize the frequency response difference between the source audio and the target audio. The flow chart of the speaker frequency response matching module is shown in the figure. Fig. 9 shown.

[0140] The source audio and the target audio are read separately, and 4 seconds of data are read backward every 0.5 seconds, and the data is accumulated and saved in a 4-second buffer until all the data is read. After the data is read, the norm of the source buffer and the target buffer is calculated, and the source buffer is multiplied by a gain coefficient to increase it to the same volume as the target buffer.

[0141] The Welch algorithm is used to calculate the spectrum difference between the source audio and the target audio. After the spectrum is calculated, it is converted to the decibel domain, some noise is smoothed to remove it, and then it is interpolated to the frequency value of 1024 points in logarithmic coordinates, and then smoothed to obtain the final spectrum that needs to be fitted.

[0142] For the smoothed spectrum, first subtract its mean value to make it evenly distributed on both the positive and negative sides, so as to find more peaks. Then search for the spectrum peaks, find the maximum value greater than 0.1 and the minimum value less than -0.1. If the number of peaks found is greater than the set maximum filter number of 24, first set a series of thresholds, filter out the peaks less than the thresholds one by one, and determine whether the number of remaining peaks is less than 24. If the number of peaks after filtering is still greater than 24, merge the adjacent peaks of the same sign according to the principle of minimum error. For two adjacent peaks, if the result of linear interpolation is slightly different from the actual spectrum, the two peaks can be merged. The merging method is to merge the peak frequencies into the geometric mean of the two peak frequencies, and merge the peak gains into the arithmetic mean of their decibel values. If the number of peaks is still large, only the 24 peaks with the largest gain values ​​are retained. Finally, the peaks are sorted in ascending order of frequency.

[0143] In this way, 24 peaking filters are preliminarily found, where the frequency corresponding to the peak is initialized as the center frequency of the filter, the gain corresponding to the peak is initialized as the gain of the filter, and the geometric mean of the frequencies of two adjacent peaks is initialized as the bandwidth of each peak, and then the quality factor Q value of the filter is calculated. Finally, calculate whether the gain of the actual spectrum is consistent with the transfer function to be fitted, and correct the gain of the filter. Use the sigmoid function to limit the value range of the center frequency fc to (0, fs / 2), where fs is the sampling rate, and limit the Q value to (0.1, 20) to prevent the center frequency fc and Q from exceeding the value range during back propagation.

[0144] The gradient descent algorithm is used to make the transfer functions of the above 24 Peaking filters closer to the transfer function to be fitted.

[0145] First is the forward propagation. The numerator and denominator coefficients of each filter can be expressed as a function of fc, Q, and Gain, and then the transfer function can be expressed as:

[0146] P k (f; fc k ,Q k ,Gain k ) (5)

[0147] In formula (5), fc is the center frequency, Q is the quality factor, and Gain is the gain.

[0148] The transfer functions of the filters are cascaded and multiplied, and the total transfer function in the decibel domain can be calculated as:

[0149]

[0150] In formula (6), fc is the center frequency, Q is the quality factor, and Gain is the gain.

[0151] Finally, the transfer function at different frequencies is compared with the P to be fitted. i , calculate the mean-square error (MSE) loss:

[0152]

[0153] In formula (7), P f is the current filter, P i is the target filter.

[0154] At the same time, a regularization term is added to the loss function to make the values ​​of fc, Q, and Gain more in line with the expected range. Then, the partial derivative of each filter parameter fc, Q, and Gain with respect to loss is calculated, and the parameters of each filter are updated by back propagation. For fc:

[0155]

[0156] In formula (8), α is the learning rate and loss is the loss function.

[0157] After 300 rounds of updates, the filter parameters can converge to a value very close to the target transfer function.

[0158] First, the importance of the filter is defined as:

[0159]

[0160] In formula (9), fc is the center frequency, Q is the quality factor, and Gain is the gain.

[0161] The importance of a filter indicates how much of an effect the filter has on the frequency response. Sorting filters by importance makes it easier to automatically select more important filters when fewer filters are available.

[0162] Then, calculate the filter parameters and save them. Here, all filter gains are multiplied by -1.0, -0.5, 0.0, 0.5, 1.0, 1.5, and 2.0, respectively, so that you can choose the degree of closeness or distance from the target frequency response as needed.

[0163] Finally, match the volume of the source buffer after all filters are applied to the volume of the target buffer. Since the transfer function has been de-averaged before and the volume may change after the filter is added, the volume of the source buffer needs to be multiplied by a coefficient again to match it with the target buffer.

[0164] The calculated filter parameters can be used to process the input signal to make it closer to the target frequency response. During processing, the number of filters used and the gain of the filters can be adjusted as needed. The fewer the number of filters, the smoother the spectrum and the less detail. Using too few filters may lead to inaccurate transfer functions. The greater the gain of the filter, the more it will shift in the direction of the target frequency response. When the filter gain is negative, the output can be more distinguished from the target frequency response.

[0165] Preferably, see Figures 1 to 11 In the method for generating presets of multi-model integrated effects based on deep learning provided in this embodiment, in step S350, the music feature extraction module will extract feature values ​​related to the preset timbre from the audio file uploaded by the user, and the feature values ​​include: music speed (BPM, beats per minute), music mode, left and right channel audio correlation coefficient, stereo sound image, etc. These music feature values ​​will be used as numerical outputs of parameters such as time and mixing ratio in peripheral effect modules such as delay and modulation.

[0166] The BPM detection algorithm determines the tempo by calculating the autocorrelation function of the starting intensity envelope of the music, combining the actual recording duration and the given beat type. The algorithm is mainly divided into four steps:

[0167] The first step is audio feature extraction. In order to make the final measured result accurate, it is necessary to select an audio feature that can better reflect the speed characteristics of the music as the input parameter of the system. The logarithmic Mel spectrum has a good performance when used for speed analysis, so the audio feature used in the algorithm analysis is the logarithmic Mel spectrum.

[0168] The second step is to calculate the start intensity envelope. The start intensity envelope is an abstract feature used to describe the starting point of the notes and percussion instruments in the music. The sounds in the music generally last for a period of time, and the sounds during the duration are not of concern to this algorithm because they do not have the particularity related to speed. Therefore, the algorithm removes these interfering factors by calculating the start intensity envelope.

[0169] The third step is to calculate the autocorrelation function. According to the obtained initial intensity envelope, the autocorrelation function is calculated.

[0170] The fourth step is to calculate the candidate speed. According to the actual recording time and the set beat type, assuming that the input audio contains 1 / 2 / 4 / 8 / 16... bars, a set of speeds can be obtained. Among them, there will be 2 to 3 candidate speeds between 40BPM and 250BPM, and their corresponding autocorrelation function values ​​are compared to determine the actual speed.

[0171] The starting intensity envelope reflects the degree of change between each frame and the frame that precedes it by a certain time (usually the previous frame) based on a certain feature. The length of one frame in this algorithm is 1024 points, and the step length is 128 points. The method for calculating the starting intensity envelope in this algorithm is that the starting intensity envelope at a certain time t is obtained by the following formula:

[0172] ΔS(m,t)=max(S(m,t)-S(m,t-lag),0) (10)

[0173]

[0174] In formulas (10) and (11), t represents time, S represents the feature used, and lag is the distance for calculating the starting intensity envelope, which is 1 frame in this algorithm.

[0175] In this algorithm, it is the logarithmic Mel spectrum of the music. In order to accurately obtain the position information of the sound start that can reflect the speed, the logarithmic Mel spectrum of each point is subtracted from the value of the previous point, and the larger value between it and 0 is taken, and then averaged on the frequency axis. In this way, the degree to which the characteristics of a point have become larger than those of the previous point can be evaluated, while the points that have become smaller can be ignored.

[0176] Calculate the autocorrelation function of the onset intensity envelope. The autocorrelation of 8192 points corresponds to a recording length of 23.8 seconds, which is sufficient to determine the speed.

[0177] When measuring the tempo of a piece of music with varying tempo, or when the tempo needs to be displayed in real time, local autocorrelation is required.

[0178] Based on the recording duration and the set beat type, a set of alternative tempos can be calculated.

[0179] The position of the corresponding autocorrelation function can be calculated based on the speed, and the speed with the maximum autocorrelation function is taken as the final speed.

[0180] The mode detection algorithm can detect the musical mode of the input audio. The basic idea of ​​the algorithm is to obtain the frequency domain signal corresponding to the musical sound through the constant Q transform (CQT), and then superimpose different octaves to obtain the chromaticity spectrum (Chroma), and determine the mode by the distribution of the chromaticity spectrum. The mode can be directly matched with the template, or the chromaticity spectrum can be used as a feature input for machine learning (hidden Markov, dynamic Bayesian network) or deep learning to obtain a mapping relationship. Among them, the template matching method can output a result in each frame, but it does not have the timing of judging the mode, and it will be unstable and have many errors. The machine learning and deep learning methods can obtain highly accurate results, but complete data needs to be input to judge the mode. This algorithm explains the implementation of the template matching method and the hidden Markov network.

[0181] The process of the mode detection algorithm is as follows:

[0182] 1. Downsample the input signal to 11025Hz to reduce the amount of calculation.

[0183] 2. Use low-pass filtering to suppress spectrum aliasing under low sampling, and at the same time suppress the harmonic components when playing the guitar to reduce the interference of non-homophone components caused by 3rd, 5th, and 6th order harmonics.

[0184] 3. Frame the signal and pass it through a Hamming window to suppress spectrum leakage.

[0185] 4. Calculate the fast Fourier transform.

[0186] 5. Calculate the constant Q transform.

[0187] 6. Adjust the standard pitch of the music to 440Hz.

[0188] 7. Superimpose the frequency grids of homophones in the spectrum obtained by CQT (Constant-Q Transform) to obtain Chroma, and then superimpose the frequency grids below 220Hz to obtain Bass-Chroma. Combine these two 12-dimensional vectors into a 24-dimensional vector as the observation state.

[0189] 8. Use template matching to convert the observed state into an implicit state, i.e. the probability distribution of the modulation.

[0190] 9. Decode the hidden state using HMM (Hidden Markov Model) to get the modulation mode.

[0191] In the discrete Fourier transform, the period of each frequency grid rotation factor is an integer fraction of the frame length. This period distribution makes the frequency grids distributed in an arithmetic progression with an interval equal to the frame frequency. This interval is also the frequency resolution of the DFT (Discrete Fourier Transform). The quality factor Q of the kth frequency grid is k :

[0192]

[0193] In formula (12), δf k is the filter bandwidth, that is, the range of half a frequency grid before and after it, which is equal to the frequency resolution in DFT.

[0194] Therefore, in DFT, Q is different for different frequency bins. Since the Q value of the actual note distribution (geometric distribution) can be considered a constant, the frequency distribution of DFT does not match the note distribution, which is manifested in that DFT has insufficient resolution in bass and excessive resolution in treble.

[0195] The constant Q transformation is to make Q a constant, then δf k =f k / Q. If the kth cq Δf of each frequency grid k =δf k but:

[0196]

[0197] In formula (13), fs is the sampling rate and Q is the quality factor.

[0198] At this time, the spectrum resolution is proportional to the frequency, which conforms to the law of note distribution. It can be found that each frequency grid in CQT has a different N k , with decimal places. To do this, it is necessary to extract the length N of each frequency grid from the frame. k The data is cut from the middle by a Hamming window; at the same time, the rotation factor frequency needs to be changed:

[0199]

[0200] In formula (14), fs is the sampling rate, N k is the length corresponding to the frequency grid.

[0201] Thus, the calculation formula of CQT can be obtained:

[0202]

[0203] In formula (15), N k is the frequency grid length, w is the Hamming window, and x is the input signal.

[0204] Chroma is a characteristic of the energy of the twelve notes in a frame obtained by superimposing the frequencies of the same note names in different octaves. Chroma is very suitable for analyzing modes. Since the tonic is a very important part of the mode and the harmonics have a great interference in the judgment of the tonic, this algorithm calculates a Bass Chroma for the notes below A3 to judge the tonic, and then calculates a Chroma for all the notes to judge the notes within the key. When superimposing, in order to reduce harmonic interference, different weights are used for different notes. This algorithm uses Gaussian distribution.

[0205]

[0206] In formula (16), Semitone is the result of CQT, and k is the semitone pitch.

[0207] Combine Bass Chroma and Chroma into a 24-dimensional vector Chroma24. The first 12 dimensions are used to analyze the main tone, and the last 12 dimensions are used to determine other tones in the key.

[0208] For the obtained Chroma24, this algorithm calculates the inner product between it and the modulation template vector, and normalizes it to obtain the modulation probability distribution corresponding to each frame. After experiments, the modulation template vector is calculated according to the following principle:

[0209] The actual notes of the mode are used to obtain 24-dimensional note distribution information, of which the last 12 dimensions correspond to the twelve notes from C to B, the notes within the key are 1, and the others are 0. Taking C major as an example, the last 12 dimensions are [1,0,1,0,1,1,0,1,0,1,0,1]. For the first 12 dimensions, this algorithm sets the dimension corresponding to the lowest note to 1 and sets the other notes within the key to 0.3. Taking C major as an example, the first 12 dimensions are [1,0,0.3,0,0.3,0.3,0,0.3,0,0.3,0,0.3].

[0210] Compute the sum of all elements of each vector.

[0211] Divide all elements of each vector by its sum.

[0212] Therefore, this algorithm obtains the template corresponding to each mode.

[0213] Mode detection is a decoding problem. The observed state is the probability distribution of each mode calculated by the template matching step above. The algorithm used is the Viterbi algorithm. The Viterbi algorithm is a dynamic programming method for finding the optimal path of the observation sequence. Since the optimal path of the current node is independent of future nodes, the Viterbi algorithm finds the current optimal path for each node and discards the remaining paths.

[0214] Suppose there are K implicit states (debugging templates), the process is as follows:

[0215] For the first node, the optimal path probability distribution is obtained from the initial probability matrix.

[0216] Path(0,k)=π(k)*Mode(0,k) (17)

[0217] In formula (17), π is the initial probability distribution, k is the number of implicit states, and Mode is the tuning template.

[0218] Then normalize Path(0,k) so that the sum for k is 0.

[0219] For subsequent nodes, each state will get K probabilities from the transition probability matrix, indicating the probability of each current path if the state is the new state.

[0220] Path k (n,k′)=A(k′,k)*Mode(n,k) (18)

[0221] In formula (18), A is the state transfer matrix, k is the number of implicit states, and Mode is the tuning template.

[0222] Under the Markov condition, future points will not change the current optimal path, so the non-optimal path corresponding to each state can be deleted.

[0223] Path(n, k) = argmax k′ [Path k (n, k′)] (19)

[0224] In formula (19), argmax is the maximum value, and Path(n,k) is the optimal path. Thus, the complete optimal path Path(n,k) can be calculated.

[0225] The probability distribution of the last frame can determine the optimal path of the entire input. The backward algorithm starts from the last frame and finds the state corresponding to each node of the optimal path forward.

[0226] H(N)=argmax k [Path(N, k)] (20)

[0227] H(n)=path(n,H(n+1)) (21)

[0228] In formulas (20) and (21), argmax is the maximum value. Thus, the output is obtained.

[0229] Preferably, see Figures 1 to 11 In the multi-model integrated effects preset generation method based on deep learning provided in this embodiment, in step S360, the music style classification module will take the audio file uploaded by the user as input, and use the music style classification neural network to determine the music style type of the audio. The music style types include: blues, classical, country, folk, funk, jazz, metal, pop, punk, hard rock, etc. The music style label will be output as a result to the subsequent module.

[0230] Further, see Figures 1 to 11In the multi-model integrated effects preset generation method based on deep learning provided in this embodiment, in step S370, the performance content classification module will use the guitar track file extracted from the music source separation module as input, and use the performance content classification neural network to determine the performance content type of the guitar track. The performance content type includes chord strumming, chord decomposition, riff rhythm, lead solo, etc. The performance content type label will be output to the subsequent module as a result.

[0231] Preferably, see Figures 1 to 11 In the multi-model integrated effects preset generation method based on deep learning provided in this embodiment, in step S380, the preset similarity matching module takes the music style label output from the music style classification module, the performance content label output from the performance content classification module, and the timbre type label output from the timbre type classification module as input. The preset database with a large number of timbre presets matches the preset files that match the given labels, and selects a preset file that best matches the target timbre based on the guitar track audio file output from the music source separation module. After completing this step, the parameters of other effect modules are determined.

[0232] The sound preset database is created by professional musicians and contains a large number of sound presets of different styles. The fields in the database include:

[0233] Song information corresponding to the preset: song title, album name, band name, musician name, guitar track, song section, and song time range corresponding to the preset tone.

[0234] Equipment information for making presets: guitar, pickups, strings, picks, cables, power supplies, headphones, sound cards.

[0235] Preset Demo audio information: system, DAW (Digital Audio Workstation), file name, file information (format, accuracy, sampling rate, stereo, etc.).

[0236] Preset information of the preset file: preset name, device name, firmware version, software version, preset content (module, algorithm, parameter value, etc.).

[0237] Preset Tag information: including music style, tone type, band musician information, amplifier information, etc.

[0238] The timbre type classification module, music style classification module and performance content classification module will output various classification labels. By searching and comparing these labels with the labels in the timbre database according to specific weights, a preset with the highest similarity can be obtained. The modules and their parameters other than the amp and cabinet in this preset are extracted and placed in the effect chain of the target preset, and the production of the target preset is completed.

[0239] The multi-model integrated effects preset generation method based on deep learning provided in this embodiment, compared with the prior art, uploads songs to the host computer through the user end; the host computer intercepts the segment of the song that needs to extract the timbre preset, and then uploads the segment to the cloud server; the deep learning model in the cloud server infers the timbre preset at the guitar playing part based on the segment, and generates a timbre preset file and returns it to the host computer; the host computer downloads the generated timbre preset file to the integrated effects, completing the entire timbre preset generation. The multi-model integrated effects preset generation method based on deep learning provided in this embodiment simplifies the cumbersome timbre preset production and debugging process in traditional integrated effects, and helps users to make timbre presets better and faster.

[0240] Although preferred embodiments of the present invention have been described, additional changes and modifications may be made to these embodiments by those skilled in the art once the basic inventive concepts are known. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present invention. Obviously, those skilled in the art may make various changes and modifications to the present invention without departing from the spirit and scope of the present invention. Thus, if these modifications and variations of the present invention fall within the scope of the claims of the present invention and their equivalents, the present invention is also intended to include these modifications and variations.

Claims

1. A method for generating presets for multi-model integrated effects devices based on deep learning, characterized in that: The following steps are involved: The user end uploads songs to the host computer; The host computer intercepts the segment of the song from which the preset timbre needs to be extracted, and then uploads the segment to the cloud server; The deep learning model in the cloud server infers the timbre preset of the guitar playing part according to the clip, and generates a timbre preset file and returns it to the host computer; The host computer downloads the generated timbre preset file to the integrated effects processor to complete the generation of the entire timbre preset; The deep learning model in the cloud server infers the timbre preset of the guitar playing part according to the clip, and generates a timbre preset file and returns it to the host computer, including: The music source separation module separates and extracts the tracks of different instruments in the clip, and outputs a guitar track file, wherein the tracks include vocals, bass, keyboard, drums, lead guitar and rhythm guitar, and the guitar track file includes the lead guitar track and the rhythm guitar track; The timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type of the guitar track, and outputs a timbre type label; The speaker parameter estimation module narrows down the selection range of the speaker simulation algorithm according to the timbre type label, inputs the guitar track file into a second classification neural network for speaker simulation algorithm type recognition and parameter estimation, and outputs the type and parameters of the speaker simulation algorithm to match the guitar timbre in a given track; The speaker frequency response matching module generates a first timbre preset file with only the amp head and cabinet algorithm according to the parameter estimation result and returns it to the user terminal, obtains the output audio of the performance content loaded by the user terminal with the timbre preset file, and uses the target track audio as a reference to match the overall frequency response of the target timbre with the estimated timbre, so as to further improve the similarity of the timbre; The music feature extraction module extracts feature values ​​related to the timbre preset in the audio file uploaded by the user terminal; The music style classification module takes the audio file as input, uses the third classification neural network for music style classification to determine the type of music style to which the audio belongs, and outputs a music style label; The performance content classification module takes the guitar track file as input, uses a fourth neural network for performance content classification to determine the performance content type of the guitar track, and outputs a performance content type label; The preset similarity matching module takes the music style tag, the performance content type tag, and the timbre type tag as input, matches a preset file that matches a given tag in a preset database having a large number of timbre presets, selects a second timbre preset file that best matches the target timbre according to the guitar track file, and determines the parameters of the guitar effect module; All algorithm module information and parameters are aggregated into a third timbre preset file and returned to the user's host computer to complete the generation of the timbre preset.

2. The method for generating presets for multi-model integrated effects devices based on deep learning according to claim 1, characterized in that: The music source separation module separates and extracts the tracks of different instruments in the clip, and outputs the guitar track file, including: Perform short-time Fourier transform on the mixed audio to obtain the mixed audio spectrum X; Extracting an eigenvalue Z of the mixed audio spectrum X, and transforming the eigenvalue Z into a transition eigenvalue Q by using time series modeling; According to the transition characteristic value Q, a time-frequency masking coefficient M is calculated by using masking estimation; Multiply the mixed audio spectrum X by the time-frequency masking coefficient M to obtain a sub-track spectrum S; Perform short-time inverse Fourier transform on the sub-track spectrum to obtain the sub-track audio.

3. The method for generating presets of multi-model integrated effects devices based on deep learning according to claim 2, characterized in that: In the step of performing short-time Fourier transform on the mixed audio to obtain the mixed audio spectrum X, the mixed audio spectrum X is a spectrum composed of complex numbers, which is expressed as: Among them, X is the mixed frequency spectrum, F is the frequency dimension, and T is the time dimension. is a plural set.

4. The method for generating presets of multi-model integrated effects devices based on deep learning as claimed in claim 3, characterized in that: The step of extracting the eigenvalue Z of the mixed audio spectrum X and transforming the eigenvalue Z into a transition eigenvalue Q by using time series modeling comprises: The complex spectrum According to the given sub-band frequency configuration f i , divided into K sub-band spectrum Where, i = 1,…,K; f i is the target frequency band, B i f i The corresponding sub-band spectrum; Each sub-band spectrum B i The real and imaginary parts of the sub-band spectrum are concatenated and input into the frequency band feature extraction module, which consists of a layer normalization and a fully connected layer. The frequency band feature extraction module outputs a sub-band spectrum feature composed of real numbers. Among them, Z i is the sub-band spectrum feature; is a set of real numbers; N is the feature dimension; All K sub-band spectrum features Merge into a full-band eigenvalue And output to the next link.

5. The method for generating presets of multi-model integrated effects devices based on deep learning according to claim 4, characterized in that: The step of extracting the eigenvalue Z of the mixed audio spectrum X and transforming the eigenvalue Z into a transition eigenvalue Q by using time series modeling comprises: For time series, a set of RNNs are used to perform feature analysis on the time series dimension T. Modeling; for the frequency band, a set of RNNs is used to model the feature values ​​on the frequency band dimension K Modeling is performed, where Z i is the eigenvalue of the full frequency band; It is the full-band eigenvalue after time series modeling; The output of RNN is obtained by linking the input with the output residual of the fully connected layer. The final output of time-frequency modeling is defined as Where Q is the transition characteristic value.

6. The method for generating presets of multi-model integrated effects devices based on deep learning as claimed in claim 5, characterized in that: The step of calculating the time-frequency masking coefficient M by using masking estimation according to the transition characteristic value Q comprises: Transform the feature value Divide into K groups of sub-band features i=1,..,K; each set of sub-band features has the same frequency division configuration as the previous feature extraction module, where Q i is the sub-band characteristic; Each set of sub-band features is input into layer normalization and multi-layer perceptron to obtain its corresponding sub-band time-frequency masking coefficient i=1,…,K; where M i is the subband time-frequency masking coefficient, C is a complex number set; All sub-band time-frequency masking coefficients M i Splice them together to get the full-band time-frequency masking coefficient M∈C F×T ; Where M is the time-frequency masking coefficient and F is the frequency dimension; The full-band time-frequency masking coefficient M∈C F×T Mixed spectrum with input Multiply to get the final target spectrum Perform ISTFT on the target spectrum to obtain the target audio, where X is the mixed spectrum and S is the standard spectrum.

7. The method for generating presets for multi-model integrated effects devices based on deep learning according to claim 1, characterized in that: The timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type to which the guitar track belongs, and in the step of outputting the timbre type label, divides the audio in the data set into segments of a set time, extracts audio features from the segments, and extracts the audio features as Mel spectra. The conversion relationship between the Mel spectra and the frequency is: Among them, Mef(f) is the Mel spectrum and f is the frequency.

8. The method for generating presets for multi-model integrated effects devices based on deep learning according to claim 1, characterized in that: The timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type to which the guitar track belongs, and in the step of outputting the timbre type label, the Mel filter bank is a group of triangular filters on the Mel frequency scale, and the calculation formula of the Mel filter bank is: Among them, Center(m) is the center frequency of the triangular filter group, which is determined by the number of triangular filters; Center(m+1) is the center frequency corresponding to the m+1 filter; Center(m-1) is the center frequency corresponding to the m-1 filter; k is the FFT spectrum number, and m is the filter number.

9. The method for generating presets of multi-model integrated effects devices based on deep learning according to claim 8, characterized in that: The timbre type classification module takes the guitar track file as input, uses the first classification neural network for timbre classification to determine the timbre type to which the guitar track belongs, and in the step of outputting the timbre type label, convolves each triangular filter with the energy spectrum of the Mel spectrum to obtain a set of coefficients corresponding to the triangular filter, and then takes the logarithm of the coefficients to obtain the coefficient S(m) used to calculate the starting intensity envelope: Where S(m) is the coefficient of the initial intensity envelope; H m (k) is the Mel filter bank, Mel(k) is the Mel spectrum, and * is the convolution operation.

Images