Musical instrument timbre conversion model construction method and musical instrument timbre conversion method

Abstract

An embodiment of the present invention provides a method for constructing a musical instrument timbre conversion model, comprising: converting a first sample audio vector sequence into a second sample audio vector sequence, and then converting the second sample audio vector sequence into an input sample audio vector sequence; and updating the model parameters of the model to be trained through the calculated first loss value and the first score, so as to train the musical instrument timbre conversion model; the embodiment of the present invention trains the model through the first loss value and the first branch, and the training efficiency is high; and the accuracy of the musical instrument timbre conversion model trained by the above method is improved.

Description

Technical Field

[0001] The embodiments of the present invention relate to the field of artificial intelligence technology, and in particular to a method for constructing a musical instrument timbre conversion model and a method for converting musical instrument timbre. Background Art

[0002] At present, the training of the instrument timbre conversion model based on neural networks mainly relies on the supervised method. The model training method requires the collection of a large amount of paired audio training data of the source instrument and the target instrument to be converted, and the data needs to be aligned in the time dimension. In actual application, it is costly to obtain paired aligned training data, and the quantity and quality of the collected training data cannot be guaranteed, resulting in poor instrument timbre conversion effect of the model trained by the training data collected by the supervised method. Summary of the invention

[0003] In view of this, embodiments of the present invention provide a method, apparatus, computer device, computer-readable storage medium and a method for musical instrument timbre conversion model construction, which are used to solve the problem of poor musical instrument timbre conversion effect of the model trained with training data collected using a deceleration method.

[0004] The embodiment of the present invention solves the above technical problems through the following technical solutions:

[0005] One aspect of an embodiment of the present invention provides a method for constructing a musical instrument timbre conversion model, comprising:

[0006] Collecting a plurality of first sample audio data, wherein the first sample audio data is audio data obtained based on a performance of a first type of musical instrument;

[0007] The plurality of first sample audio data are converted into a plurality of first sample audio vector sequences, and the plurality of first sample audio vector sequences are input into the model to be trained,

[0008] The plurality of first sample audio vector sequences are converted into a second sample audio vector sequence through the model to be trained, wherein the second sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of a second type of musical instrument;

[0009] Restoring each second sample audio vector sequence to a corresponding input sample audio vector sequence, wherein the input sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of the first type of musical instrument;

[0010] Comparing the input sample audio vector sequence with the corresponding first sample audio vector sequence, and obtaining a first loss value;

[0011] Based on the plurality of second sample audio vector sequences, obtaining a first score corresponding to each second sample audio vector sequence;

[0012] Based on the first score and the first loss value, updating the model parameters of the model to be trained;

[0013] The musical instrument timbre conversion model is obtained according to the updated model parameters.

[0014] Optionally, the model to be trained includes a first converter, a second converter, a first discriminator and a second discriminator, the first converter is used to convert the first sample audio vector sequence into the second sample audio vector sequence; the second converter is used to convert the third sample audio vector sequence into a fourth sample audio vector sequence, wherein the third sample audio data is a vector sequence corresponding to the audio data obtained based on the performance of the second type of musical instrument, and the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained based on the simulation of the performance of the first type of musical instrument; the first discriminator is used to discriminate whether the audio data is data obtained by simulating the performance of the first type of musical instrument; the second discriminator is used to discriminate whether the audio data is data obtained by simulating the performance of the second type of musical instrument;

[0015] The updating of the model parameters of the to-be-trained model based on the first score and the first loss value further includes:

[0016] Under the condition that the model parameters of the second converter and the second discriminator remain unchanged, obtaining a first error of each neuron in the first converter by a back propagation algorithm according to the first loss value;

[0017] updating a model parameter of the first converter according to each first error in the first converter;

[0018] The model parameters of the first discriminator are updated based on the updated model parameters of the first converter and the first score.

[0019] Optionally, obtaining the musical instrument timbre conversion model according to the updated model parameters further includes:

[0020] According to the updated model parameters, an updated model to be trained is obtained;

[0021] After updating the model parameters in the updated model to be trained a preset number of times, the instrument timbre conversion model is obtained.

[0022] Optionally, the converting the plurality of first sample audio data into a plurality of first sample audio vector sequences further includes:

[0023] Performing frame processing on each first sample audio data, and obtaining a plurality of frames of first sub-sample audio data;

[0024] Performing Fourier transform on the first sub-sample audio data of each frame to obtain an amplitude corresponding to the first sub-sample audio data of each frame;

[0025] Obtaining a linear spectrum according to the amplitude of the first sub-sample audio data of each frame; and

[0026] A weighted sum is performed on the linear spectra to obtain a first sample audio vector sequence corresponding to the first sample audio data.

[0027] Optionally, the method further comprises:

[0028] Collecting a plurality of third sample audio data, wherein the third sample audio data is audio data obtained based on performance of the second type of musical instrument;

[0029] The plurality of third sample audio data are converted into a plurality of third sample audio vector sequences, and the plurality of third sample audio vector sequences are input into the model to be trained,

[0030] The plurality of third sample audio vector sequences are converted into a fourth sample audio vector sequence through the model to be trained, wherein the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained by simulating the performance of the first type of musical instrument;

[0031] Restoring each fourth sample audio vector sequence into a corresponding fifth sample audio vector sequence, wherein the fifth sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of the second type of musical instrument;

[0032] Comparing the fifth sample audio vector sequence with the corresponding third sample audio vector sequence, and obtaining a second loss value;

[0033] Based on multiple fourth sample audio vector sequences, obtaining a second score corresponding to each fourth sample audio vector sequence;

[0034] Based on the second score and the second loss value, updating the model parameters of the model to be trained;

[0035] The musical instrument timbre conversion model is obtained according to the updated model parameters.

[0036] Optionally, updating the model parameters of the to-be-trained model based on the second score and the second loss value further includes:

[0037] Under the condition that the model parameters of the first converter and the first discriminator remain unchanged, updating the model parameters of the second converter according to the second loss value; and

[0038] The model parameters of the second discriminator are updated based on the updated model parameters of the second converter and the second score.

[0039] Another aspect of the embodiments of the present invention provides a device for constructing a musical instrument timbre conversion model, comprising:

[0040] A collection module, used for collecting a plurality of first sample audio data, wherein the first sample audio data is audio data obtained based on a performance of a first type of musical instrument;

[0041] a vector conversion module, used to convert the plurality of first sample audio data into a plurality of first sample audio vector sequences, and input the plurality of first sample audio vector sequences into a model to be trained,

[0042] A conversion module, used to convert the plurality of first sample audio vector sequences into a second sample audio vector sequence through the model to be trained, wherein the second sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of a second type of musical instrument;

[0043] A recovery module, used to restore each second sample audio vector sequence to a corresponding input sample audio vector sequence, wherein the input sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of the first type of musical instrument;

[0044] A comparison module, used to compare the input sample audio vector sequence with the corresponding first sample audio vector sequence, and obtain a first loss value;

[0045] An acquisition module, configured to acquire, based on a plurality of second sample audio vector sequences, a first score corresponding to each second sample audio vector sequence;

[0046] An updating module, configured to update the model parameters of the to-be-trained model based on the first score and the first loss value; and

[0047] The generation module is used to obtain the instrument timbre conversion model according to the updated model parameters.

[0048] Another aspect of the embodiments of the present invention provides a method for converting musical instrument timbre, comprising:

[0049] Collecting a plurality of first target audio data, wherein the first target audio data is audio data obtained based on a performance of a first type of musical instrument;

[0050] Converting the plurality of first target audio data into a plurality of first target audio vector sequences, and inputting the plurality of first target audio vector sequences into the musical instrument timbre conversion model as described above;

[0051] The multiple first target audio vector sequences are converted into corresponding multiple second target audio vector sequences through the musical instrument timbre conversion model, wherein the second target audio vector sequences are vector sequences corresponding to audio data obtained by simulating a performance of a second type of musical instrument; and

[0052] Convert multiple second target audio vector sequences into corresponding multiple second target audio data.

[0053] In order to achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the steps of the method for constructing a musical instrument timbre conversion model as described above are implemented.

[0054] In order to achieve the above-mentioned purpose, another aspect of an embodiment of the present invention provides a computer-readable storage medium, in which a computer program is stored. The computer program can be executed by at least one processor so that the at least one processor performs the steps of the method for constructing a musical instrument timbre conversion model as described above.

[0055] The musical instrument timbre conversion model construction method, device, computer equipment and computer-readable storage medium provided in the embodiments of the present invention convert a first sample audio vector sequence into a second sample audio vector sequence, and then convert the second sample audio vector sequence into an input sample audio vector sequence; update the model parameters of the model to be trained by calculating the first loss value and the first score, so as to train the musical instrument timbre conversion model; the embodiment of the present invention trains the model through the first loss value and the first branch, and the training efficiency is high; and the conversion effect of the musical instrument timbre conversion model trained by the above method is improved.

[0056] The present invention is described in detail below with reference to the accompanying drawings and specific embodiments, but is not intended to limit the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 A flowchart of the steps of the method for constructing a musical instrument timbre conversion model according to the first embodiment of the present invention;

[0058] Figure 2 A flowchart of the steps of obtaining a first sample audio vector sequence in the method for constructing a musical instrument timbre conversion model according to the first embodiment of the present invention;

[0059] Figure 3 A flowchart of the steps of updating model parameters in the method for constructing a musical instrument timbre conversion model according to the first embodiment of the present invention;

[0060] Figure 4A flowchart of the steps of model training in the method for constructing a musical instrument timbre conversion model according to the first embodiment of the present invention;

[0061] Figure 5 A flowchart of the steps of updating model parameters in the method for constructing a musical instrument timbre conversion model according to the first embodiment of the present invention;

[0062] Figure 6 A schematic diagram of a program module of a device for constructing a musical instrument timbre conversion model according to a second embodiment of the present invention;

[0063] Figure 7 Flow chart of the steps of the musical instrument timbre conversion method according to the third embodiment of the present invention;

[0064] Figure 8 Schematic diagram of the hardware structure of the computer device according to the fourth embodiment of the present invention. DETAILED DESCRIPTION

[0065] In order to make the purpose, technical scheme and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present invention.

[0066] It should be noted that the descriptions involving "first", "second", etc. in the embodiments of the present invention are only for descriptive purposes and cannot be understood as indicating or suggesting their relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In addition, the technical solutions between the various embodiments can be combined with each other, but they must be based on the ability of ordinary technicians in the field to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be deemed that such a combination of technical solutions does not exist and is not within the scope of protection required by the present invention.

[0067] In the description of the present invention, it should be understood that the numerical labels before the steps do not identify the order of executing the steps, but are only used to facilitate the description of the present invention and distinguish each step, and therefore cannot be understood as a limitation of the present invention.

[0068] Embodiment 1

[0069] See also Figure 1 , shows a flowchart of the steps of the method for constructing a musical instrument timbre conversion model according to an embodiment of the present invention. It can be understood that the flowchart in the embodiment of the method is not used to limit the order of executing the steps.

[0070] The following is an exemplary description using a computer device as the execution subject, as follows:

[0071] like Figure 1 As shown, the musical instrument timbre conversion model construction method may include steps S100 to S114, wherein:

[0072] Step S100 , collecting a plurality of first sample audio data, wherein the first sample audio data is audio data obtained based on performance of a first type of musical instrument.

[0073] In this embodiment, a plurality of first sample audio data are obtained as training sets to train the model to be trained. Compared with the supervised learning method, there is no need to annotate the first sample audio data in the time dimension, and the audio data only needs to be classified according to the instrument type corresponding to the audio data. Therefore, the method of this embodiment can greatly reduce the cost of data acquisition, and can have more training data to train the model to be trained, which provides a good basic condition for improving the performance of the model.

[0074] Step S102: convert the multiple first sample audio data into multiple first sample audio vector sequences, and input the multiple first sample audio vector sequences into the model to be trained.

[0075] To facilitate data processing for the model to be trained, please refer to Figure 2 , the step S102 of converting the multiple first sample audio data into multiple first sample audio vector sequences can further include steps S200 to S206, wherein: step S200, performing frame processing on each first sample audio data, and obtaining multiple frames of first sub-sample audio data; step S202, performing Fourier transform on each frame of first sub-sample audio data, and obtaining the amplitude corresponding to each frame of first sub-sample audio data; step S204, obtaining a linear spectrum according to the amplitude of each frame of first sub-sample audio data; and step S206, performing weighted summation on the linear spectrum, and obtaining a first sample audio vector sequence corresponding to the first sample audio data. In this embodiment, the audio data is vectorized to facilitate the data processing of the model to be trained, and effectively improve the data processing efficiency.

[0076] Step S104: converting the plurality of first sample audio vector sequences into a second sample audio vector sequence through the model to be trained, wherein the second sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating performance of a second type of musical instrument.

[0077] The first type of musical instrument and the second type of musical instrument are different musical instruments. The model to be trained involves four groups of neural networks, which are respectively used to characterize the first converter, the second converter, the first discriminator, the second discriminator, the encoder and the decoder of the transformer (self-attention) layer; wherein the first converter and the first discriminator are used to convert the audio of the first type of musical instrument into the audio of the second type of musical instrument; the second converter and the second discriminator are used to convert the audio of the second type of musical instrument into the audio of the first type of musical instrument; the encoder and the decoder are used for the conversion between audio data and audio vector sequences.

[0078] The first transformer receives a first sample audio vector sequence from a first type of musical instrument and uses an encoder in the transformer layer to convert the first sample audio vector sequence into a second sample audio vector sequence of a second type of musical instrument of the same length.

[0079] Step S106: restore each second sample audio vector sequence to a corresponding input sample audio vector sequence, wherein the input sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating performance of the first type of musical instrument.

[0080] The converted second sample audio vector sequence is received by the second converter, and the decoder in the transformer layer is used to restore the second sample audio vector sequence to the input audio vector sequence.

[0081] Step S108: compare the input sample audio vector sequence with the corresponding first sample audio vector sequence to obtain a first loss value.

[0082] In this embodiment, the input sample audio vector sequence and the corresponding first sample audio vector sequence are compared to form a supervisory signal, which is used to update the model parameters of the first converter and the first discriminator. The minimum square error, i.e., the minimum reconstruction error, between the input audio vector sequence at each moment and the corresponding first sample audio vector sequence is calculated.

[0083] Step S110: Based on the multiple second sample audio vector sequences, obtain a first score corresponding to each second sample audio vector sequence.

[0084] In this embodiment, the first discriminator identifies the second sample audio vector sequence as audio data from actual performance of the second type of musical instrument or audio data that is converted to simulate the performance of the second type of musical instrument, and outputs a score; the score is compared with the preset score to determine the second sample audio vector sequence. If the score is greater than the preset score, it is determined that the second sample audio vector sequence is from audio data actually performed by the second type of musical instrument; if the score is less than the preset score, it is determined that the second sample audio vector sequence is audio data that is converted to simulate the performance of the second type of musical instrument.

[0085] Step S112: Update the model parameters of the model to be trained based on the first score and the first loss value.

[0086] In an exemplary embodiment, the model to be trained includes a first converter, a second converter, a first discriminator and a second discriminator, the first converter is used to convert the first sample audio vector sequence into the second sample audio vector sequence; the second converter is used to convert the third sample audio vector sequence into a fourth sample audio vector sequence, wherein the third sample audio data is a vector sequence corresponding to the audio data obtained by playing the second type of musical instrument, and the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained by simulating the performance of the first type of musical instrument; the first discriminator is used to distinguish whether the audio data is data obtained by simulating the performance of the first type of musical instrument; the second discriminator is used to distinguish whether the audio data is data obtained by simulating the performance of the second type of musical instrument; in order to improve the model training efficiency and conversion effect,

[0087] See also Figure 3, the updating of the model parameters of the model to be trained based on the first score and the first loss value may also include the following steps S300 to S302, wherein: step S300, when the model parameters of the second converter and the second discriminator remain unchanged, according to the first loss value, the first error of each neuron in the first converter is obtained by the back propagation algorithm; step S302, according to each first error in the first converter, the model parameters of the first converter are updated; and step S304, based on the updated model parameters of the first converter and the first score, the model parameters of the first discriminator are updated. In this embodiment, two sets of neural network parameters are updated simultaneously during unidirectional training, that is, when the training model is capable of converting the audio data of the first type of musical instrument into the audio data of the second type of musical instrument, the model parameters of the first converter and the first discriminator are updated. The first loss value is obtained by forward propagation calculation; the first error of each neuron in the first converter is calculated by the back propagation algorithm; the derivatives of the model parameters of the first converter are calculated based on each first error and the gradient descent algorithm, and the model parameters of the first converter are updated based on each derivative.

[0088] For the first converter, its goal is to generate audio data that simulates the performance of the second type of musical instrument so that the first discriminator can output a higher first score. The goal of the first discriminator is to improve the accuracy of audio data recognition, giving high scores to audio data from real performances of the second type of musical instrument and low scores to audio data from converted simulated performances of the second type of musical instrument.

[0089] The update of the model parameters of the first converter depends on the first loss function, which can be seen in the following formula 1:

[0090]

[0091] in, represents the first loss value; E represents the error, d~M1 represents sampling from the data set M1, C w1 represents the first converter represented by 1 as a function of the neural network parameters, C w2 represents the function of the second transformer represented by w2 as the neural network parameters, The first discriminator is represented by As a function of the neural network parameter representation.

[0092] The update of the model parameters of the first discriminator depends on the second loss function, which can be seen in the following formula 2:

[0093]

[0094] Among them, E represents the error, d~M1 represents sampling from data set M1, d~M2 represents sampling from data set M2, and C w1 represents the function of the first converter characterized by w1 as the neural network parameter, The first discriminator is represented by As a function of the neural network parameter representation.

[0095] Step S114, obtaining a musical instrument timbre conversion model according to the updated model parameters.

[0096] In an exemplary embodiment, the step S114 of obtaining the musical instrument timbre conversion model according to the updated model parameters may also include the following operations: obtaining an updated model to be trained according to the updated model parameters; and obtaining the musical instrument timbre conversion model after updating the model parameters in the updated model to be trained for a preset number of times. The model converges by the preset number of iterative trainings, which can improve the accuracy of the model data processing.

[0097] In order to improve the applicability of the model, the method also includes two-way training; to improve Figure 4 , the method also includes steps S400 to S414, wherein: step S400, collecting a plurality of third sample audio data, wherein the third sample audio data is audio data obtained based on the performance of the second type of musical instrument; step S402, converting the plurality of third sample audio data into a plurality of third sample audio vector sequences, and inputting the plurality of third sample audio vector sequences into the model to be trained; step S404, converting the plurality of third sample audio vector sequences into a fourth sample audio vector sequence through the model to be trained, wherein the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained based on simulating the performance of the first type of musical instrument; step S406, Each fourth sample audio vector sequence is restored to a corresponding fifth sample audio vector sequence, wherein the fifth sample audio vector sequence is a vector sequence corresponding to the audio data obtained by simulating the performance of the second type of musical instrument; step S408, comparing the fifth sample audio vector sequence with the corresponding third sample audio vector sequence, and obtaining a second loss value; step S410, based on multiple fourth sample audio vector sequences, obtaining a second score corresponding to each fourth sample audio vector sequence; step S412, based on the second score and the second loss value, updating the model parameters of the model to be trained; and step S414, obtaining a musical instrument timbre conversion model according to the updated model parameters.

[0098] The update of the model parameters of the second converter depends on the third loss function, which can be seen in the following formula 3:

[0099]

[0100] in, represents the first loss value; E represents the error, d~M2 represents sampling from the data set M2, C w1 represents the function of the first converter represented by w1 as the neural network parameter, C w2 represents the function of the second transformer represented by w2 as the neural network parameters, The second discriminator is represented by As a function of the neural network parameter representation.

[0101] The update of the model parameters of the second discriminator depends on the fourth loss function, which can be seen in the following formula 4:

[0102]

[0103] Among them, E represents the error, d~M1 represents sampling from data set M1, d~M2 represents sampling from data set M2, and C w2 represents the function of the second transformer represented by 2 as the neural network parameter, The second discriminator is represented by As a function of the neural network parameter representation.

[0104] To improve model training efficiency and accuracy, please refer to Figure 5 , the updating of the model parameters of the model to be trained based on the second score and the second loss value can also be obtained by the following operations, wherein: step S500, updating the model parameters of the second converter according to the second loss value when the model parameters of the first converter and the first discriminator remain unchanged; and step S502, updating the model parameters of the second discriminator based on the updated model parameters of the second converter and the second score.

[0105] The embodiment of the present invention does not require paired audio training data of the source instrument and the target conversion instrument. The processing of the training data only requires the classification of the training data according to the instrument type, which overcomes the problem of high cost of obtaining paired alignment training data, so that the data can be used by the training model without a lot of processing. Compared with the traditional training data acquisition method, it saves a lot of time and cost in data preparation, and improves the applicability of the method; the instrument timbre conversion model can realize timbre conversion between multiple different types of instruments. And through the first loss function, the second loss function, the third loss function and the fourth loss function, the training model to be trained is trained, effectively improving the training efficiency and accuracy of the model.

[0106] Embodiment 2

[0107] Please continue reading Figure 6, showing a schematic diagram of the program modules of the musical instrument timbre conversion model construction device of the present invention. In this embodiment, the musical instrument timbre conversion model construction device may include or be divided into one or more program modules, one or more program modules are stored in a storage medium, and are executed by one or more processors to complete the present invention and implement the above-mentioned musical instrument timbre conversion model construction method. The program module referred to in the embodiment of the present invention refers to a series of computer program instruction segments that can perform specific functions, and is more suitable for describing the execution process of the musical instrument timbre conversion model construction device in the storage medium than the program itself. The following description will specifically introduce the functions of each program module of this embodiment:

[0108] The musical instrument timbre conversion model construction device includes: a collection module 600, a vector conversion module 602, a conversion module 604, a recovery module 606, a comparison module 608, an acquisition module 610, an update module 612 and a generation module 614; wherein:

[0109] The collection module 600 is used to collect a plurality of first sample audio data, wherein the first sample audio data is audio data obtained based on a performance of a first type of musical instrument;

[0110] A vector conversion module 602 is used to convert the plurality of first sample audio data into a plurality of first sample audio vector sequences, and input the plurality of first sample audio vector sequences into a model to be trained,

[0111] A conversion module 604 is used to convert the plurality of first sample audio vector sequences into a second sample audio vector sequence through the model to be trained, wherein the second sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of a second type of musical instrument;

[0112] A recovery module 606 is used to restore each second sample audio vector sequence to a corresponding input sample audio vector sequence, wherein the input sample audio vector sequence is a vector sequence corresponding to the audio data obtained by simulating the performance of the first type of musical instrument;

[0113] A comparison module 608, configured to compare the input sample audio vector sequence with a corresponding first sample audio vector sequence and obtain a first loss value;

[0114] An acquisition module 610 is used to acquire a first score corresponding to each second sample audio vector sequence based on multiple second sample audio vector sequences;

[0115] An updating module 612, configured to update the model parameters of the to-be-trained model based on the first score and the first loss value; and

[0116] The generation module 614 is used to obtain the instrument timbre conversion model according to the updated model parameters.

[0117] Embodiment 3

[0118] See also Figure 7 , which shows a flowchart of the steps of the musical instrument timbre conversion method according to an embodiment of the present invention. It is understood that the flowchart in the embodiment of the present method is not used to limit the order of executing the steps. The following is an exemplary description using a computer device as the execution subject, as follows:

[0119] like Figure 7 As shown, the musical instrument timbre conversion method may include steps S700 to S706, wherein:

[0120] Step S700, collecting a plurality of first target audio data, wherein the first target audio data is audio data obtained based on a performance of a first type of musical instrument;

[0121] Step S702, converting the plurality of first target audio data into a plurality of first target audio vector sequences, and inputting the plurality of first target audio vector sequences into the musical instrument timbre conversion model as described above;

[0122] Step S704, converting the plurality of first target audio vector sequences into a plurality of corresponding second target audio vector sequences through the musical instrument timbre conversion model, wherein the second target audio vector sequences are vector sequences corresponding to audio data obtained by simulating a performance of a second type of musical instrument; and

[0123] Step S706: convert the multiple second target audio vector sequences into corresponding multiple second target audio data.

[0124] Embodiment 4

[0125] See also Figure 8 , is a schematic diagram of the hardware architecture of the computer device of the fourth embodiment of the present invention. In this embodiment, the computer device 2 is a device that can automatically perform numerical calculations and / or information processing according to pre-set or stored instructions. The computer device 2 can be a rack server, a blade server, a tower server or a cabinet server (including an independent server, or a server cluster composed of multiple servers), etc. Figure 8 As shown, the computer device 2 at least includes, but is not limited to, a memory 21, a processor 22, a network interface 23, and a musical instrument timbre conversion model construction device that can be interconnected through a system bus.

[0126] In this embodiment, the memory 21 includes at least one type of computer-readable storage medium, and the readable storage medium includes flash memory, hard disk, multimedia card, card-type memory (for example, SD or DX memory, etc.), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, disk, optical disk, etc. In some embodiments, the memory 21 can be an internal storage unit of the computer device 2, such as a hard disk or memory of the computer device 2. In other embodiments, the memory 21 can also be an external storage device of the computer device 2, such as a plug-in hard disk equipped on the computer device 2, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. Of course, the memory 21 can also include both the internal storage unit of the computer device 2 and its external storage device. In this embodiment, the memory 21 is generally used to store the operating system and various application software installed on the computer device 2, such as the program code of the instrument conversion model construction device of the above embodiment. In addition, the memory 21 can also be used to temporarily store various types of data that have been output or are to be output.

[0127] The processor 22 may be a central processing unit (CPU), a controller, a microcontroller, a microprocessor, or other data processing chips in some embodiments. The processor 22 is generally used to control the overall operation of the computer device 2. In this embodiment, the processor 22 is used to run the program code stored in the memory 21 or process data, such as running the instrument conversion model construction device to implement the instrument conversion model construction method of the above embodiment.

[0128] The network interface 23 may include a wireless network interface or a wired network interface, and the network interface 23 is generally used to establish a communication connection between the computer device 2 and other electronic devices. For example, the network interface 23 is used to connect the computer device 2 to an external terminal through a network, and to establish a data transmission channel and a communication connection between the computer device 2 and the external terminal. The network may be a wireless or wired network such as an Intranet, the Internet, the Global System of Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), a 4G network, a 5G network, Bluetooth, or Wi-Fi.

[0129] It should be pointed out that Figure 8 Only computer device 2 is shown with components 21 - 23 , but it should be understood that implementation of all of the components shown is not a requirement, and more or fewer components may alternatively be implemented.

[0130] In this embodiment, the musical instrument conversion model building device stored in the memory 21 can also be divided into one or more program modules, and the one or more program modules are stored in the memory 21 and executed by one or more processors (processor 22 in this embodiment) to complete the present invention.

[0131] For example, Figure 6 The program module schematic diagram of the embodiment 1 of the instrument conversion model construction device is shown. In this embodiment, the instrument conversion model construction device can be divided into a collection module 600, a vector conversion module 602, a conversion module 604, a recovery module 606, a comparison module 608, an acquisition module 610, an update module 612 and a generation module 614. Among them, the program module referred to in the present invention refers to a series of computer program instruction segments that can complete specific functions, which is more suitable for describing the execution process of the instrument conversion model construction device in the computer device 2 than a program. The specific functions of the program modules 600-614 have been described in detail in the embodiment 2, and will not be repeated here.

[0132] Embodiment 5

[0133] The present embodiment also provides a computer-readable storage medium, such as a flash memory, a hard disk, a multimedia card, a card-type memory (e.g., an SD or DX memory, etc.), a random access memory (RAM), a static random access memory (SRAM), a read-only memory (ROM), an electrically erasable programmable read-only memory (EEPROM), a programmable read-only memory (PROM), a magnetic memory, a disk, an optical disk, a server, an App application store, etc., on which a computer program is stored, and when the program is executed by a processor, a corresponding function is realized. The computer-readable storage medium of the present embodiment is used to store a musical instrument conversion model construction device, and when executed by a processor, the musical instrument conversion model construction method of the above embodiment is realized.

[0134] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0135] Through the description of the above implementation methods, those skilled in the art can clearly understand that the above embodiment methods can be implemented by means of software plus a necessary general hardware platform, and of course by hardware, but in many cases the former is a better implementation method.

[0136] The above are only preferred embodiments of the present invention, and are not intended to limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A method for constructing a musical instrument timbre conversion model, characterized in that: include: Collecting a plurality of first sample audio data, wherein the first sample audio data is audio data obtained based on a performance of a first type of musical instrument; Convert the plurality of first sample audio data into a plurality of first sample audio vector sequences, and input the plurality of first sample audio vector sequences into a model to be trained; The plurality of first sample audio vector sequences are converted into a second sample audio vector sequence through the model to be trained, wherein the second sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of a second type of musical instrument; Restoring each second sample audio vector sequence to a corresponding input sample audio vector sequence, wherein the input sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of the first type of musical instrument; Comparing the input sample audio vector sequence with the corresponding first sample audio vector sequence, and obtaining a first loss value; Based on the plurality of second sample audio vector sequences, obtaining a first score corresponding to each second sample audio vector sequence; Based on the first score and the first loss value, updating the model parameters of the model to be trained; Obtaining a musical instrument timbre conversion model according to the updated model parameters; Wherein, the model to be trained includes a first converter, a second converter, a first discriminator and a second discriminator, the first converter is used to convert the first sample audio vector sequence into the second sample audio vector sequence; the second converter is used to convert the third sample audio vector sequence into a fourth sample audio vector sequence, wherein the third sample audio vector sequence is a vector sequence corresponding to the audio data obtained based on the performance of the second type of musical instrument, and the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained based on the simulation of the performance of the first type of musical instrument; the first discriminator is used to distinguish whether the audio data is data obtained by simulating the performance of the first type of musical instrument; the second discriminator is used to distinguish whether the audio data is data obtained by simulating the performance of the second type of musical instrument; The updating of the model parameters of the to-be-trained model based on the first score and the first loss value further includes: Under the condition that the model parameters of the second converter and the second discriminator remain unchanged, obtaining a first error of each neuron in the first converter by a back propagation algorithm according to the first loss value; updating a model parameter of the first converter according to each first error in the first converter; The model parameters of the first discriminator are updated based on the updated model parameters of the first converter and the first score.

2. The method for constructing a musical instrument timbre conversion model according to claim 1, characterized in that: The step of obtaining the musical instrument timbre conversion model according to the updated model parameters also includes: According to the updated model parameters, an updated model to be trained is obtained; After updating the model parameters in the updated model to be trained a preset number of times, the instrument timbre conversion model is obtained.

3. The method for constructing a musical instrument timbre conversion model according to claim 1, characterized in that: The converting the plurality of first sample audio data into a plurality of first sample audio vector sequences also includes: Performing frame processing on each first sample audio data, and obtaining a plurality of frames of first sub-sample audio data; Performing Fourier transform on the first sub-sample audio data of each frame to obtain an amplitude corresponding to the first sub-sample audio data of each frame; Obtaining a linear spectrum according to the amplitude of the first sub-sample audio data of each frame; and A weighted sum is performed on the linear spectra to obtain a first sample audio vector sequence corresponding to the first sample audio data.

4. The method for constructing a musical instrument timbre conversion model according to claim 2, characterized in that: The method further comprises: Collecting a plurality of third sample audio data, wherein the third sample audio data is audio data obtained based on performance of the second type of musical instrument; Convert the plurality of third sample audio data into a plurality of third sample audio vector sequences, and input the plurality of third sample audio vector sequences into the model to be trained; The plurality of third sample audio vector sequences are converted into a fourth sample audio vector sequence through the model to be trained, wherein the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained by simulating the performance of the first type of musical instrument; Restoring each fourth sample audio vector sequence into a corresponding fifth sample audio vector sequence, wherein the fifth sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of the second type of musical instrument; Comparing the fifth sample audio vector sequence with the corresponding third sample audio vector sequence, and obtaining a second loss value; Based on multiple fourth sample audio vector sequences, obtaining a second score corresponding to each fourth sample audio vector sequence; Based on the second score and the second loss value, updating the model parameters of the model to be trained; The musical instrument timbre conversion model is obtained according to the updated model parameters.

5. The method for constructing a musical instrument timbre conversion model according to claim 4, characterized in that: The updating of the model parameters of the to-be-trained model based on the second score and the second loss value further includes: Under the condition that the model parameters of the first converter and the first discriminator remain unchanged, updating the model parameters of the second converter according to the second loss value; and The model parameters of the second discriminator are updated based on the updated model parameters of the second converter and the second score.

6. A device for constructing a musical instrument timbre conversion model, characterized in that: include: A collection module, used for collecting a plurality of first sample audio data, wherein the first sample audio data is audio data obtained based on a performance of a first type of musical instrument; a vector conversion module, used to convert the plurality of first sample audio data into a plurality of first sample audio vector sequences, and input the plurality of first sample audio vector sequences into a model to be trained, A conversion module, used to convert the plurality of first sample audio vector sequences into a second sample audio vector sequence through the model to be trained, wherein the second sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of a second type of musical instrument; A recovery module, used to restore each second sample audio vector sequence to a corresponding input sample audio vector sequence, wherein the input sample audio vector sequence is a vector sequence corresponding to audio data obtained by simulating a performance of the first type of musical instrument; A comparison module, used to compare the input sample audio vector sequence with the corresponding first sample audio vector sequence, and obtain a first loss value; An acquisition module, configured to acquire, based on a plurality of second sample audio vector sequences, a first score corresponding to each second sample audio vector sequence; An updating module, configured to update the model parameters of the to-be-trained model based on the first score and the first loss value; and A generation module, used to obtain a musical instrument timbre conversion model according to the updated model parameters; Wherein, the model to be trained includes a first converter, a second converter, a first discriminator and a second discriminator, the first converter is used to convert the first sample audio vector sequence into the second sample audio vector sequence; the second converter is used to convert the third sample audio vector sequence into a fourth sample audio vector sequence, wherein the third sample audio vector sequence is a vector sequence corresponding to the audio data obtained based on the performance of the second type of musical instrument, and the fourth sample audio vector sequence is a vector sequence corresponding to the audio data obtained based on the simulation of the performance of the first type of musical instrument; the first discriminator is used to distinguish whether the audio data is data obtained by simulating the performance of the first type of musical instrument; the second discriminator is used to distinguish whether the audio data is data obtained by simulating the performance of the second type of musical instrument; The updating of the model parameters of the to-be-trained model based on the first score and the first loss value further includes: Under the condition that the model parameters of the second converter and the second discriminator remain unchanged, obtaining a first error of each neuron in the first converter by a back propagation algorithm according to the first loss value; updating a model parameter of the first converter according to each first error in the first converter; The model parameters of the first discriminator are updated based on the updated model parameters of the first converter and the first score.

7. A method for converting musical instrument timbre, characterized in that: include: Collecting a plurality of first target audio data, wherein the first target audio data is audio data obtained based on a performance of a first type of musical instrument; Converting the plurality of first target audio data into a plurality of first target audio vector sequences, and inputting the plurality of first target audio vector sequences into the musical instrument timbre conversion model as described in any one of claims 1 to 5; The multiple first target audio vector sequences are converted into corresponding multiple second target audio vector sequences through the musical instrument timbre conversion model, wherein the second target audio vector sequences are vector sequences corresponding to audio data obtained by simulating a performance of a second type of musical instrument; and Convert multiple second target audio vector sequences into corresponding multiple second target audio data.

8. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for constructing a musical instrument timbre conversion model as described in any one of claims 1 to 5 are implemented.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which can be executed by at least one processor to enable the at least one processor to perform the steps of the method for constructing a musical instrument timbre conversion model as described in any one of claims 1 to 5.

Images