Music generation method and device

Abstract

The invention discloses a music generation method and device. The method comprises the steps of obtaining tone information in first audio data in response to triggering of a music generation event, and generating at least one tone coding vector based on the tone information; obtaining rhythm information in the second audio data, and generating at least one rhythm coding vector based on the rhythm information; obtaining target lyrics, performing word segmentation processing on the target lyrics, and converting each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector; and inputting the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation large model, and generating target music according to an output result of the music generation large model. According to the scheme, the decoupling of the timbre and the rhythm can be realized, so that the timbre duplicating of any timbre on any rhythm is realized, and the free generation of music by using the duplicated timbre and any rhythm can be realized.

Description

Technical Field

[0001] The present invention relates to the field of audio technology, and in particular to a music generating method and device. Background Art

[0002] The existing song generation technology generally processes the vocals and accompaniment separately, wherein the accompaniment is generally extracted from the accompaniment in the reference song or template song, and the vocals are generally derived from the voice input. Referring to the sound spectrum of the original singer, the voice input spectrum is extended, enhanced, and speeded up as much as possible to generate a personal spectrum corresponding to the reference original singer, and then generate a cappella vocals. Finally, the accompaniment or the a cappella vocals can be output separately, or the two tracks of the accompaniment and the a cappella vocals can be merged into one output song. That is, the existing music generation technology can generally only achieve vocal optimization by adjusting the vocal spectrum, and cannot achieve the reproduction of personal timbre and the personalized customization, creation out of nothing, and free generation of songs, and the flexibility is low. Summary of the invention

[0003] The present invention provides a music generation method and device, which can not only realize the decoupling of timbre and rhythm, thereby realizing the timbre replication of any timbre in any rhythm, but also realize the free generation of music with the replicated timbre and any rhythm.

[0004] According to one aspect of the present invention, there is provided a music generation method, comprising:

[0005] In response to a music generation event being triggered, acquiring timbre information in the first audio data, and generating at least one timbre encoding vector based on the timbre information;

[0006] Acquire prosody information in the second audio data, and generate at least one prosody encoding vector based on the prosody information;

[0007] Obtain target lyrics, perform word segmentation processing on the target lyrics, and convert each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector;

[0008] The at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector are input into a pre-constructed music generation model, and the target music is generated according to the output result of the music generation model.

[0009] According to another aspect of the present invention, there is provided a music generating device, comprising:

[0010] A timbre coding vector generating module, configured to obtain timbre information in the first audio data in response to a music generation event being triggered, and to generate at least one timbre coding vector based on the timbre information;

[0011] A prosody coding vector generation module, configured to obtain prosody information in the second audio data, and generate at least one prosody coding vector based on the prosody information;

[0012] A word segmentation unit vector determination module is used to obtain target lyrics, perform word segmentation processing on the target lyrics, and convert each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector;

[0013] The target music generation module is used to input the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generate target music according to the output result of the music generation model.

[0014] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising:

[0015] at least one processor; and

[0016] a memory communicatively connected to the at least one processor; wherein,

[0017] The memory stores a computer program that can be executed by the at least one processor, and the computer program is executed by the at least one processor so that the at least one processor can execute the music generation method described in any embodiment of the present invention.

[0018] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to implement the music generation method described in any embodiment of the present invention when executed by a processor.

[0019] The music generation scheme of the embodiment of the present invention, in response to a music generation event being triggered, obtains timbre information in the first audio data, and generates at least one timbre coding vector based on the timbre information; obtains rhythm information in the second audio data, and generates at least one rhythm coding vector based on the rhythm information; obtains target lyrics, performs word segmentation processing on the target lyrics, and converts each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector; inputs the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generates target music according to the output result of the music generation model. The technical scheme provided by the embodiment of the present invention can not only realize the decoupling of timbre and rhythm, thereby realizing the timbre replication of any timbre in any rhythm, but also realize the free generation of music with the replicated timbre and any rhythm.

[0020] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present invention, nor are they intended to limit the scope of the present invention. Other features of the present invention will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings in the following description are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0022] Figure 1 A flowchart of a music generation method provided in Embodiment 1 of the present invention;

[0023] Figure 2 A flowchart of a music generation method provided in Embodiment 2 of the present invention;

[0024] Figure 3 A flowchart of a music generation method provided in Embodiment 3 of the present invention;

[0025] Figure 4 A schematic diagram of the structure of a music generating device provided in Embodiment 4 of the present invention;

[0026] Figure 5 It is a schematic diagram of the structure of an electronic device for implementing the music generating method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0027] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0028] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0029] Embodiment 1

[0030] Figure 1 This is a flowchart of a music generation method provided in Embodiment 1 of the present invention. This embodiment is applicable to the case of generating music. The method can be executed by a music generation device. The music generation device can be implemented in the form of hardware and / or software. The music generation device can be configured in an electronic device. Figure 1 As shown, the method includes:

[0031] S110. In response to a music generation event being triggered, acquiring timbre information in the first audio data, and generating at least one timbre coding vector based on the timbre information.

[0032] Exemplarily, when a music generation request input by a user is received, it is determined that a music generation event is triggered. In response to the music generation event being triggered, the first audio data is obtained. Among them, the first audio data is the sound whose timbre needs to be reproduced, which can be a human voice or a musical instrument. The first audio data can be the sound of any content, for example, it can be a sentence spoken by the user, or it can be the sound of preset content, such as the sound of preset content read by the user, or it can be a piece of music specified by the user or selected from a music database. The first audio data can be sound information of any length, or it can be sound information of a preset length, or it can be a sound of a preset length (such as 6S or 10S) intercepted as the first audio data after the user inputs sound information not less than a preset length. It should be noted that the embodiment of the present invention does not limit the form, content and length of the first audio data.

[0033] In the embodiment of the present invention, timbre information is extracted from the first audio data, wherein timbre refers to the fact that different sounds always have distinctive characteristics in terms of waveform, and different objects vibrate with different timbre characteristics. Timbre is also called timbre quality, and different vibrations can be combined into different sounds, such as each musical instrument, the vocal cords of different people, and all other vibrating objects can produce different sounds with their own characteristics. When an object vibrates, it will emit a fundamental tone, and at the same time, its various parts also have complex vibrations, and the sounds produced by the vibrations of each part are combined into overtones. In addition to a 'fundamental tone', the sound naturally adds many different 'frequencies' (the number of times a vibrating object vibrates in 1 second) and 'interweaving' with overtones, which determines different timbres, so that people can distinguish different sounds after listening. Therefore, sound contains three aspects: 1 / the amplitude of the sound, that is, the intensity or amplitude of the audio; 2 / the frequency of the sound, that is, the frequency of the audio or the number of changes per second; 3 / the timbre of the sound, which is determined by the overtones. Optionally, the acquisition of timbre information in the first audio data includes: normalizing the first audio data in terms of velocity and / or pitch, and extracting timbre information from the first audio data after normalization. Exemplarily, the first audio data is normalized, wherein the normalization includes at least one of velocity normalization and pitch (Pitch, "fundamental tone") normalization, so that only timbre information is retained in the first audio data, thereby being able to extract timbre information from the first audio data after normalization. Velocity normalization is to normalize the sounding interval between two notes by stretching or compressing to improve the timbre extraction effect and eliminate the influence of speech speed; pitch normalization is to normalize the loudness and / or tone (frequency) of the "fundamental tone" to better extract timbre features other than pitch ("overtones").

[0034] In an embodiment of the present invention, at least one layer of timbre coding vector is generated based on the timbre information. Exemplarily, the timbre information can be encoded as a whole into at least one layer of global vector by an encoder, and the global vector can be used as the timbre coding vector, or a portion of the timbre of a preset duration can be intercepted from the timbre information, and the intercepted portion of the timbre of the preset length can be encoded into at least one layer of timbre coding vector by an encoder. The encoder can include a MERT encoder and a Mel encoder.

[0035] S120: Acquire prosody information in the second audio data, and generate at least one prosody encoding vector based on the prosody information.

[0036] In an embodiment of the present invention, the second audio data is obtained, and the rhythmic information in the second audio data is determined. The second audio data is the sound that needs to remove the timbre, which can be a human voice or a musical instrument. For example, the second audio data can be a reference music input by the user (that is, a song containing human voice and accompaniment), or it can be a music of some dimensions separated from the reference music input by the user, such as the human voice part or the accompaniment part in the song. Optionally, the rhythmic information in the second audio data is obtained, including: performing at least one adjustment method of random pitch perturbation, formant adjustment and equalizer adjustment on the second audio data, and extracting the rhythmic information from the adjusted second audio data. Exemplarily, the second audio data is subjected to perturbation processing to obtain the rhythmic information in the second audio data. The perturbation processing includes at least one adjustment method of random pitch perturbation (Pitch, "base tone"), formant adjustment, and equalizer adjustment. The random pitch perturbation includes random perturbation of amplitude (loudness) and / or frequency (pitch); the formant adjustment includes the position of the formant distribution in the spectrum; the equalizer adjustment includes the proportion of audio energy of different frequencies. After adjusting the second audio data by the above-mentioned random disturbance and / or preset adjustment method, the timbre information in the second audio data can be removed and the rhythm information therein can be retained.

[0037] In an embodiment of the present invention, at least one prosody coding vector of at least one layer is generated based on the prosody information in the second audio data. Exemplarily, the prosody information can be encoded as a whole into at least one global vector of at least one layer by an encoder, and the global vector can be used as the prosody coding vector, or a partial prosody of a preset duration can be intercepted from the prosody information, and the intercepted partial prosody of the preset length can be encoded into at least one prosody coding vector of at least one layer by an encoder. The encoder can include a MERT encoder and a Mel encoder.

[0038] S130, obtaining target lyrics, performing word segmentation processing on the target lyrics, and converting each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector.

[0039] In an embodiment of the present invention, target lyrics are obtained, wherein the target lyrics may be a section of lyrics input by the user in text, or a section of lyrics input by the user in voice, or lyrics consisting of subtitle information or voice information in the input video. The target lyrics may be any lyrics content, and the target lyrics may be content of any length. It should be noted that the embodiment of the present invention does not limit the content and length of the target lyrics. The target lyrics are segmented based on a preset word segmentation algorithm, and the target lyrics are divided into a plurality of word segmentation units, wherein the word segmentation unit may be each character, each word, or each sentence. Each word segmentation unit in the target lyrics is encoded based on an encoder, and a corresponding word segmentation unit vector of at least one layer is generated. Among them, the more layers of the word segmentation unit vector, the more it can represent the richer information in the target lyrics, such as emotional information.

[0040] S140, inputting the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generating target music according to the output result of the music generation model.

[0041] In an embodiment of the present invention, at least one timbre coding vector, at least one rhythm coding vector and a word segmentation unit vector corresponding to each word segmentation unit are input into a pre-constructed music generation model, so that the music generation model outputs a feature vector corresponding to the music according to the input information, and converts the feature vector into target music based on the music generation module. The music generation model is a pre-trained machine learning model, for example, the music generation model is any open source model, such as GPT.

[0042] The music generation method of the embodiment of the present invention, in response to a music generation event being triggered, obtains timbre information in the first audio data, and generates at least one timbre coding vector based on the timbre information; obtains rhythm information in the second audio data, and generates at least one rhythm coding vector based on the rhythm information; obtains target lyrics, performs word segmentation processing on the target lyrics, and converts each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector; inputs the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generates target music according to the output result of the music generation model. Through the technical solution provided by the embodiment of the present invention, not only can the decoupling of timbre and rhythm be achieved, thereby achieving the timbre replication of any timbre in any rhythm, but also the free generation of music with the replicated timbre and any rhythm can be achieved.

[0043] Embodiment 2

[0044] Figure 2 A flowchart of a music generation method provided in Embodiment 2 of the present invention is shown in FIG. Figure 2 As shown, the method includes:

[0045] S210. When a general music generation request input by a user is received, it is determined that a music generation event is triggered, and timbre information in the first audio data is obtained, and at least one timbre encoding vector is generated based on the timbre information.

[0046] S220: Acquire at least one target reference music, use at least one target reference music as second audio data, and extract rhythm information from the second audio data.

[0047] In an embodiment of the present invention, when a user wants to generate a certain type of music with the same rhythm as the accompaniment part and / or the vocal part, a general music generation request is input. Wherein, the general music generation request may include any generation request of the accompaniment music generation request, the a cappella music generation request and the song music generation request. When the general music generation request input by the user is received, it is determined that the music generation event is triggered. In response to the music generation event being triggered, at least one target reference music of a preset duration (such as 5 seconds or 10 seconds) matching the general music generation request is obtained. Wherein, at least one target reference music is music containing at least one different element, feature, style, and dimension. Exemplarily, when the general music generation request is a cappella music generation request, the target reference music may be a cappella music containing only vocals; when the general music generation request is an accompaniment music generation request, the target reference music may be an accompaniment music containing only accompaniment; when the general music generation request is a song music generation request, the target reference music may be a song music containing vocals and accompaniment. At least one target reference music is used as the second audio data, and rhythm information is extracted from the second audio data.

[0048] S230 . Encode the prosody information through an encoder to generate at least one prosody coding vector.

[0049] S240, obtaining target lyrics, performing word segmentation processing on the target lyrics, and converting each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector.

[0050] S250, inputting the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generating target music according to the output result of the music generation model.

[0051] In an embodiment of the present invention, when the general music generation request is a song music generation request, the music generation model is a song generation model, that is, at least one timbre coding vector, at least one rhythm coding vector generated based on the rhythm information of at least one target reference music, and the word segmentation unit vector corresponding to the target lyrics are input into the song generation model, and the target song music containing the target lyrics and replicating the timbre of the first audio data is generated according to the song generation model, wherein the rhythm of the target song music is the same or similar to the accompaniment rhythm and vocal rhythm of the target reference music. When the general music generation request is a cappella music generation request, the music generation model is a cappella generation model, that is, at least one timbre coding vector, at least one rhythm coding vector generated based on the rhythm information of at least one target reference music, and the word segmentation unit vector corresponding to the target lyrics are input into the cappella generation model, and the target cappella music containing the target lyrics and replicating the timbre of the first audio data is generated according to the cappella generation model, wherein the rhythm of the target cappella music is the same or similar to the vocal rhythm of the target reference music. When the general music generation request is an accompaniment music generation request, the music generation model is an accompaniment generation model, that is, at least one timbre coding vector, at least one rhythm coding vector generated based on the rhythm information of at least one target reference music, and the word segmentation unit vector corresponding to the target lyrics are input into the accompaniment generation model, and the accompaniment generation model is used to generate the target accompaniment music with the timbre of the first audio data, wherein the target accompaniment music has the same or similar accompaniment rhythm as the target reference music. It should be noted that the song generation model, the a cappella generation model, and the accompaniment generation model are the same model, and by controlling the content of the coding vector input to the model, the model can output coding vectors of different data types.

[0052] The technical solution provided by the embodiment of the present invention can not only realize the decoupling of timbre and rhythm, thereby realizing the timbre replication of any timbre in any rhythm, but also realize the free generation of music with the replicated timbre and any rhythm.

[0053] Embodiment 3

[0054] Figure 3 A flowchart of a music generation method provided in Embodiment 3 of the present invention is shown in FIG. Figure 3 As shown, the method includes:

[0055] S310. When a personalized music generation request input by a user is received, it is determined that a music generation event is triggered, and timbre information in the first audio data is obtained, and at least one timbre coding vector is generated based on the timbre information.

[0056] Exemplarily, the personalized music generation request may include any one or more of a music generation request containing a cappella rhythm, a music generation request containing an accompaniment rhythm, or a music generation request containing a song rhythm. Optionally, the personalized music generation request may also be a music generation request containing a more detailed and personalized rhythm specified by the user, for example, a music generation request containing a cappella rhythm for men (bass, baritone, tenor), a music generation request containing a cappella rhythm for women (altina, mezzo-soprano, soprano), a music generation request containing a cappella rhythm for one or more characters, a music generation request containing a cappella rhythm for various languages ​​(such as Chinese, dialects, English, etc.), a music generation request containing an accompaniment rhythm for one or more instruments (such as piano), a music generation request containing a solo solo rhythm, and a music generation request containing a chorus multi-instrument ensemble rhythm. When the personalized music generation request input by the user is received, it is determined that the music generation event is triggered, and the timbre information in the first audio data is obtained, and at least one timbre coding vector is generated based on the timbre information.

[0057] S320. Obtain at least one target reference music, and extract a target reference music portion of at least one dimension that matches the personalized music generation request from at least one of the target reference music, and use the target reference music portion of at least one dimension as the second audio data.

[0058] When a personalized music generation request input by a user is received, at least one target reference music is obtained, and a target reference music part of at least one dimension matching the personalized music generation request is extracted from the at least one target reference music, and the target reference music part of at least one dimension is used as the second audio data. Exemplarily, at least one target reference music is analyzed, and target reference music parts of two dimensions of accompaniment part and vocal part are extracted from the target reference music. Among them, a variety of methods can be used to extract the audio data of the vocal part and the audio data of the accompaniment part from the audio of the target reference music. For example, the target reference music can be Fourier transformed to obtain a mixed amplitude spectrum and a mixed phase spectrum of the accompaniment signal and the vocal signal, and then the amplitude spectrum of the vocal and accompaniment sound is separated using a trained separation model (which can be a model built based on a deep neural network). The amplitude spectrum and the mixed phase spectrum of the separated vocal and accompaniment sound are subjected to inverse Fourier transform to obtain the audio data of the vocal part and the audio data of the accompaniment part. Exemplarily again, the audio data of the vocal part and the audio data of the accompaniment part in the target reference music can be extracted by audio filtering. As another example, the audio of the target reference music may be copied to two audio tracks, and the high-frequency vocal signal and the low-frequency accompaniment signal may be filtered out respectively to separate the vocal signal from the accompaniment signal.

[0059] In an embodiment of the present invention, if the personalized music generation request is a music generation request containing male a cappella rhythm, the target reference music is analyzed and the male voice part is extracted from the target reference music; if the personalized music generation request is a music generation request containing accompaniment rhythms of multiple instruments, the target reference music is analyzed and the accompaniment part of each instrument is extracted from the target reference music. It should be noted that the embodiment of the present invention does not limit the number of dimensions of the target reference music part that matches the personalized music generation request.

[0060] S330. For the target reference music part of each dimension in the second audio data, extract prosody information from the target reference music part, and encode the prosody through an encoder to generate at least one prosody encoding vector.

[0061] In an embodiment of the present invention, prosody information is extracted from the target reference music portion of each dimension, and the prosody information extracted from the target reference music portion of each dimension is encoded by an encoder to generate at least one corresponding prosody encoding vector.

[0062] S340, obtaining target lyrics, performing word segmentation processing on the target lyrics, and converting each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector.

[0063] S350, inputting the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generating target music according to the output result of the music generation model.

[0064] The technical solution provided by the embodiment of the present invention can not only achieve the decoupling of timbre and rhythm, thereby realizing the timbre replication of any timbre at any rhythm, but also realize the free generation of music with the replicated timbre and any rhythm. It has high flexibility, greatly improves the diversity of the generated target music, and meets the user's demand for personalized music generation.

[0065] In some embodiments, before inputting the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation large model, it also includes: obtaining a music training sample set; wherein the music training sample set contains at least two groups of music training samples, each group of music training samples includes fitting music, at least one reference timbre, at least one reference rhythm and at least one sample lyrics; respectively encoding the fitting music, the reference timbre, the reference rhythm and the sample lyrics in each group of the music training samples to generate at least one corresponding sample coding vector; training the preset large model based on the sample coding vector to generate a music generation large model.

[0066] In an embodiment of the present invention, a music training sample set is obtained, wherein the music training sample set includes at least two groups of music training samples, each group of music training samples includes fitting music, at least one reference timbre, at least one reference rhythm and at least one sample lyrics. Exemplarily, a certain mature music can be used as the fitting music in a certain group of music training samples, the timbre information extracted from the mature music can be used as the reference timbre, the rhythm information extracted from the mature music can be used as the reference rhythm, and any music lyrics in the mature music can be used as the sample lyrics in the group of music training samples. Based on the fitting music, reference timbre, reference rhythm and sample lyrics in each group of music training samples, at least one corresponding sample coding vector is generated, that is, at least one corresponding sample coding vector is generated based on the fitting music in each group of music training samples, at least one corresponding sample coding vector is generated based on the reference timbre in each group of music training samples, at least one corresponding sample coding vector is generated based on the reference rhythm in each group of music training samples, and at least one corresponding sample coding vector is generated based on the sample lyrics in each group of music training samples. Wherein, the sample coding vector is a coding vector of at least one layer. The preset big model is trained based on the sample coding vector to generate a music generation big model, wherein the preset big model can be any open source big model, such as GPT.

[0067] Optionally, the method for obtaining the reference timbre and reference rhythm in each group of music training samples includes: obtaining a first sample reference music, and obtaining a second sample reference music of the same type or the same singer as the first sample reference music; determining the reference rhythm based on the first sample reference music, and determining the reference timbre based on the second sample reference music. Exemplarily, a mature music can be used as the first sample reference music, and the reference rhythm of at least part of the music in the first sample reference music is obtained, and the reference rhythm is used as the reference rhythm in a group of music training samples. The second sample reference music of the same type or the same singer as the first sample reference music is obtained, and the timbre information in the second sample reference music is used as the reference timbre. For example, the voice data of the music singer of the first sample reference music in other occasions can be used as the timbre reference data, or the singing data of the music singer of the first sample reference music outside the first sample reference music can be used as the timbre reference data, or the audio of the same instrument played in other occasions can be used as the timbre reference data.

[0068] Optionally, determining the reference rhythm based on the first sample reference music includes: extracting the first sample reference music part of at least one dimension from the first sample reference music; extracting the reference rhythm from the first sample reference music part of all dimensions or the first part of dimensions. Exemplarily, the first sample reference music is split into a preset number of dimensions, such as splitting the first sample reference music part of at least one dimension (such as the sample reference music part of two dimensions of the accompaniment part and the vocal part) from the first sample reference music. Extract the reference rhythm from the first sample reference music part of all dimensions, for example, respectively extract the reference rhythm from the accompaniment part of the first sample reference music and extract the reference rhythm from the vocal part of the first sample reference music. Optionally, the reference rhythm can also be extracted from the first sample reference music part of the first part of dimensions, for example, extract the reference rhythm only from the accompaniment part of the first sample reference music, or extract the reference rhythm only from the vocal part of the first sample reference music.

[0069] Optionally, the method for obtaining the reference timbre and reference rhythm in each group of music training samples includes: obtaining a third sample reference music, and extracting a second sample reference music part of at least one dimension from the third sample reference music; extracting a reference rhythm from the second sample reference music part of all dimensions or the first part of dimensions, and extracting a reference timbre from the second sample reference music part of all dimensions or the second part of dimensions. Exemplarily, a mature music can be used as the third sample reference music, and the third sample reference music is split into a preset number of dimensions, such as splitting a second sample reference music part of at least one dimension (such as a sample reference music part of two dimensions of an accompaniment part and a vocal part) from the third sample reference music. Extract the reference rhythm from the second sample reference music part of all dimensions, for example, extract the reference rhythm from the accompaniment part of the third sample reference music and extract the reference rhythm from the vocal part of the third sample reference music. Optionally, the reference rhythm can also be extracted from the third sample reference music part of the first part of dimensions, for example, extract the reference rhythm only from the accompaniment part of the third sample reference music, or extract the reference rhythm only from the vocal part of the third sample reference music. The reference timbre is extracted from the second sample reference music part of all dimensions, for example, the reference timbre is extracted from the accompaniment part of the third sample reference music and the reference timbre is extracted from the vocal part of the third sample reference music. Optionally, the reference timbre can also be extracted from the third sample reference music part of the second part dimension, for example, the reference timbre is extracted only from the accompaniment part of the third sample reference music, or the reference timbre is extracted only from the vocal part of the third sample reference music.

[0070] Embodiment 4

[0071] Figure 4This is a schematic diagram of the structure of a music generating device provided by Embodiment 4 of the present invention. Figure 4 As shown, the device comprises:

[0072] The timbre coding vector generating module 410 is used to obtain the timbre information in the first audio data in response to the music generation event being triggered, and generate at least one timbre coding vector based on the timbre information;

[0073] A prosody coding vector generating module 420, configured to obtain prosody information in the second audio data, and generate at least one prosody coding vector based on the prosody information;

[0074] A word segmentation unit vector determination module 430 is used to obtain target lyrics, perform word segmentation processing on the target lyrics, and convert each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector;

[0075] The target music generation module 440 is used to input the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generate target music according to the output result of the music generation model.

[0076] Optionally, in response to a music generation event being triggered, comprising:

[0077] When receiving a general music generation request input by a user, determining that a music generation event is triggered;

[0078] Correspondingly, the rhythm coding vector generation module is used to:

[0079] Acquire at least one target reference music, use at least one target reference music as second audio data, and extract rhythm information from the second audio data;

[0080] The prosody information is encoded by an encoder to generate at least one prosody encoding vector.

[0081] Optionally, in response to a music generation event being triggered, comprising:

[0082] When receiving a personalized music generation request input by a user, determining that a music generation event is triggered;

[0083] Correspondingly, the rhythm coding vector generation module is used to:

[0084] Acquire at least one target reference music, and extract a target reference music portion of at least one dimension matching the personalized music generation request from the at least one target reference music, and use the target reference music portion of at least one dimension as second audio data;

[0085] For the target reference music portion of each dimension in the second audio data, prosody information is extracted from the target reference music portion, and the prosody is encoded by an encoder to generate at least one prosody encoding vector.

[0086] Optional, timbre coding vector generation module, used to:

[0087] The first audio data is subjected to speed and / or pitch normalization processing, and timbre information is extracted from the first audio data after the normalization processing.

[0088] Optionally, a prosody coding vector generation module is used to:

[0089] The second audio data is adjusted by at least one of random pitch perturbation, formant adjustment and equalizer adjustment, and rhythm information is extracted from the adjusted second audio data.

[0090] Optionally, the device further comprises:

[0091] A music training sample set acquisition module is used to acquire a music training sample set before inputting the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-built music generation model; wherein the music training sample set includes at least two groups of music training samples, each group of music training samples includes fitting music, at least one reference timbre, at least one reference rhythm and at least one sample lyrics;

[0092] A sample coding vector generation module, used to encode the fitting music, the reference timbre, the reference rhythm and the sample lyrics in each group of the music training samples, and generate at least one corresponding sample coding vector;

[0093] The music generation big model generation module is used to train the preset big model based on the sample coding vector to generate the music generation big model.

[0094] Optionally, the method for obtaining the reference timbre and reference rhythm in each group of music training samples includes:

[0095] Acquire a first sample reference music, and acquire a second sample reference music of the same type or by the same singer as the first sample reference music;

[0096] A reference rhythm is determined based on the first sample reference music, and a reference timbre is determined based on the second sample reference music.

[0097] Optionally, determining a reference rhythm based on the first sample reference music includes:

[0098] extracting a first sample reference music portion of at least one dimension from the first sample reference music;

[0099] A reference rhythm is extracted from the first sample reference music portion in all dimensions or in the first partial dimensions.

[0100] Optionally, the method for obtaining the reference timbre and reference rhythm in each group of music training samples includes:

[0101] Acquire a third sample reference music, and extract at least one dimension of a second sample reference music portion from the third sample reference music;

[0102] A reference rhythm is extracted from the second sample reference music portion in all dimensions or in the first partial dimensions, and a reference timbre is extracted from the second sample reference music portion in all dimensions or in the second partial dimensions.

[0103] The music generating device provided in the embodiment of the present invention can execute the music generating method provided in any embodiment of the present invention, and has the functional modules and beneficial effects corresponding to the execution method.

[0104] Embodiment 5

[0105] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices (such as helmets, glasses, watches, etc.) and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present invention described and / or required herein.

[0106] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0107] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0108] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microcontroller, etc. The processor 11 performs the various methods and processes described above, such as a music generation method.

[0109] In some embodiments, the music generation method may be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the music generation method described above may be performed. Alternatively, in other embodiments, the processor 11 may be configured to perform the music generation method in any other appropriate manner (e.g., by means of firmware).

[0110] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] Computer programs for implementing the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0112] In the context of the present invention, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0113] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0114] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0115] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0116] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps described in the present invention can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of the present invention can be achieved, and this document does not limit this.

[0117] The above specific implementations do not constitute a limitation on the protection scope of the present invention. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present invention should be included in the protection scope of the present invention.

Claims

1. A music generation method, characterized in that: include: In response to a music generation event being triggered, acquiring timbre information in the first audio data, and generating at least one timbre encoding vector based on the timbre information; Acquire prosody information in the second audio data, and generate at least one prosody encoding vector based on the prosody information; Obtain target lyrics, perform word segmentation processing on the target lyrics, and convert each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector; The at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector are input into a pre-constructed music generation model, and the target music is generated according to the output result of the music generation model.

2. The method according to claim 1, characterized in that: In response to a music generation event being triggered, including: When receiving a general music generation request input by a user, determining that a music generation event is triggered; Accordingly, obtaining the prosody information in the second audio data and generating at least one prosody coding vector based on the prosody information includes: Acquire at least one target reference music, use at least one target reference music as second audio data, and extract rhythm information from the second audio data; The prosody information is encoded by an encoder to generate at least one prosody encoding vector.

3. The method according to claim 1, characterized in that In response to a music generation event being triggered, including: When receiving a personalized music generation request input by a user, determining that a music generation event is triggered; Accordingly, obtaining the prosody information in the second audio data and generating at least one prosody coding vector based on the prosody information includes: Acquire at least one target reference music, and extract a target reference music portion of at least one dimension matching the personalized music generation request from the at least one target reference music, and use the target reference music portion of at least one dimension as second audio data; For the target reference music portion of each dimension in the second audio data, prosody information is extracted from the target reference music portion, and the prosody is encoded by an encoder to generate at least one prosody encoding vector.

4. The method according to claim 1, characterized in that: The step of obtaining the timbre information in the first audio data includes: The first audio data is subjected to speed and / or pitch normalization processing, and timbre information is extracted from the first audio data after the normalization processing.

5. The method according to claim 1, characterized in that: The step of obtaining the prosody information in the second audio data comprises: The second audio data is adjusted by at least one of random pitch perturbation, formant adjustment and equalizer adjustment, and rhythm information is extracted from the adjusted second audio data.

6. The method according to claim 1, characterized in that: Before inputting the at least one timbre coding vector, the at least one prosody coding vector and the word segmentation unit vector into the pre-constructed music generation model, the method further includes: Acquire a music training sample set; wherein the music training sample set includes at least two groups of music training samples, each group of music training samples includes fitting music, at least one reference timbre, at least one reference rhythm and at least one sample lyrics; Encoding the fitting music, the reference timbre, the reference rhythm and the sample lyrics in each group of the music training samples respectively to generate at least one corresponding sample encoding vector; The preset large model is trained based on the sample coding vector to generate a music generation large model.

7. The method according to claim 6, characterized in that The method for obtaining the reference timbre and reference rhythm in each group of music training samples includes: Acquire a first sample reference music, and acquire a second sample reference music of the same type or by the same singer as the first sample reference music; A reference rhythm is determined based on the first sample reference music, and a reference timbre is determined based on the second sample reference music.

8. The method according to claim 7, characterized in that Determining a reference rhythm based on the first sample reference music includes: extracting a first sample reference music portion of at least one dimension from the first sample reference music; A reference rhythm is extracted from the first sample reference music portion in all dimensions or in the first partial dimensions.

9. The method according to claim 6, characterized in that The method for obtaining the reference timbre and reference rhythm in each group of music training samples includes: Acquire a third sample reference music, and extract at least one dimension of a second sample reference music portion from the third sample reference music; A reference rhythm is extracted from the second sample reference music portion in all dimensions or in the first partial dimensions, and a reference timbre is extracted from the second sample reference music portion in all dimensions or in the second partial dimensions.

10. A music generating device, characterized in that: include: A timbre coding vector generating module, configured to obtain timbre information in the first audio data in response to a music generation event being triggered, and to generate at least one timbre coding vector based on the timbre information; A prosody coding vector generation module, configured to obtain prosody information in the second audio data, and generate at least one prosody coding vector based on the prosody information; A word segmentation unit vector determination module is used to obtain target lyrics, perform word segmentation processing on the target lyrics, and convert each word segmentation unit in the target lyrics into a corresponding word segmentation unit vector; The target music generation module is used to input the at least one timbre coding vector, the at least one rhythm coding vector and the word segmentation unit vector into a pre-constructed music generation model, and generate target music according to the output result of the music generation model.

Images