Music conversion method, related device, equipment and storage medium

Abstract

The invention discloses a music conversion method, a related device, equipment and a storage medium, and the method comprises the steps: extracting a first audio feature of to-be-converted music and a first sub-feature related to a plurality of music components based on the audio data of the to-be-converted music, and extracting a second audio feature of the to-be-converted music based on the audio data of reference music; extracting a second audio feature of the reference music and a second sub-feature about a plurality of music components; constructing sub-feature pairs belonging to different music components based on the music components of the first sub-features and the music components of the second sub-features; wherein the sub-feature pair of any music component represents a feature mapping relationship of a second sub-feature belonging to the music component with respect to the to-be-converted music; and generating target music based on the sub-feature pair, the first audio feature and the second audio feature. According to the scheme, automatic conversion of music styles can be realized, and the accuracy of music conversion is improved.

Description

Technical Field

[0001] The present application relates to the field of audio processing technology, and in particular to a music conversion method and related devices, equipment and storage media. Background Art

[0002] As users' demands for music become increasingly diversified, different users expect to choose a version of the same song that satisfies their favorite style.

[0003] In the prior art, the conversion of music styles is usually achieved by re-interpreting songs based on professional mixers or bands, or by performing spectrum migration through the spectrum signals of the music to be converted. However, professional mixers or bands require high interpretation costs, and due to the complexity of the music signal, the target music effect obtained after spectrum migration is poor, which cannot meet the user's needs for different music styles. In view of this, how to achieve automatic conversion of music styles and improve the accuracy of music conversion has become an urgent problem to be solved. Summary of the invention

[0004] The main technical problem solved by the present application is to provide a music conversion method and related devices, equipment and storage media, which can realize the automatic conversion of music styles and improve the accuracy of music conversion.

[0005] In order to solve the above technical problems, the first aspect of the present application provides a music conversion method, including: based on the audio data of the music to be converted, extracting the first audio feature of the music to be converted and the first sub-feature about several music components, and based on the audio data of the reference music, extracting the second audio feature of the reference music and the second sub-feature about several music components; based on the music components of each first sub-feature and the music components of each second sub-feature, constructing sub-feature pairs belonging to different music components; wherein the sub-feature pair of any music component represents: the feature mapping relationship between the second sub-feature belonging to the music component and the music to be converted; based on the sub-feature pair, the first audio feature and the second audio feature, generating the target music.

[0006] In order to solve the above-mentioned technical problems, the second aspect of the present application provides a music conversion device, including: an extraction module, a construction module and a generation module, the extraction module is used to extract the first audio feature of the music to be converted and the first sub-features about several music components based on the audio data of the music to be converted, and to extract the second audio feature of the reference music and the second sub-features about several music components based on the audio data of the reference music; the construction module is used to construct sub-feature pairs belonging to different music components based on the music components of each first sub-feature and the music components of each second sub-feature; wherein the sub-feature pair of any music component represents: the feature mapping relationship between the second sub-feature belonging to the music component and the music to be converted; the generation module is used to generate the target music based on the sub-feature pairs, the first audio feature and the second audio feature.

[0007] In order to solve the above technical problems, the third aspect of the present application provides an electronic device, including a memory and a processor coupled to each other, the memory storing program instructions, and the processor being used to execute the program instructions to implement the music conversion method in the above first aspect.

[0008] In order to solve the above technical problems, the fourth aspect of the present application provides a computer-readable storage medium storing program instructions that can be executed by a processor, and the program instructions are used to implement the music conversion method described in the first aspect above.

[0009] The above scheme extracts the first audio feature of the music to be converted and the first sub-features about several music components based on the audio data of the music to be converted, and extracts the second audio feature of the reference music and the second sub-features about several music components based on the audio data of the reference music, and constructs sub-feature pairs belonging to different music components based on the music components of each first sub-feature and each second sub-feature, and the sub-feature pairs of any music component represent: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted, and generates the target music based on the sub-feature pairs, the first audio feature and the second audio feature. Therefore, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be realized, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, so that the feature matching relationship between the music to be converted and the reference music can be retained as much as possible at the music component level, and the fine-grainedness of the target music generation can be improved. For music with complex audio signals, the loss of detail data during the conversion process can also be reduced as much as possible. Therefore, the automatic conversion of music styles can be realized, and the accuracy of music conversion can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] Figure 1 It is a flowchart of an embodiment of the music conversion method of the present application; Figure 2It is a schematic diagram of a framework of an embodiment of a music conversion model in the music conversion method of the present application; Figure 3 It is a schematic diagram of the framework of an embodiment of the music conversion device of the present application; Figure 4 It is a schematic diagram of the framework of an embodiment of the electronic device of the present application; Figure 5 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium of the present application. DETAILED DESCRIPTION

[0011] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0012] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0013] If the technical solution of this application involves personal information, the product using the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using the technical solution of this application has obtained the individual's separate consent before processing the sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, clear and prominent signs are set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that he or she agrees to the collection of his or her personal information; or on the device that processes personal information, the personal information processing rules are notified by obvious signs / information, and the individual's authorization is obtained through pop-up information or by asking the individual to upload his or her personal information; among them, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.

[0014] The terms "system" and "network" are often used interchangeably in this article. The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the fragment " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0015] See also Figure 1 , Figure 1 This is a flowchart of an embodiment of the music conversion method of the present application. Specifically, it may include the following steps: Step S10: Based on the audio data of the music to be converted, extract the first audio feature of the music to be converted and the first sub-features about several music components, and based on the audio data of the reference music, extract the second audio feature of the reference music and the second sub-features about several music components.

[0016] In the disclosed embodiments, the specific types of the music to be converted and the reference music are not limited in this application. For example, the music may be composed of multi-instrument audio and single vocal audio, music may be composed of multi-instrument audio, music may be composed of multi-voice audio, etc. Specifically, the music to be converted may be folk-style song A, and the reference music may be rock-style song B.

[0017] In one implementation scenario, the first audio feature of the music to be converted may reflect the overall properties of the music to be converted, such as the style, rhythm, audio content, etc. of the music to be converted, and the second audio feature of the reference music may reflect the overall properties of the reference music, such as the style, rhythm, audio content, etc. of the reference music.

[0018] In a specific implementation scenario, the first audio feature and the second audio feature can be implemented based on Mel-frequency cepstral coefficients, spectrograms, Mel-spectrum and other technologies. For example, for each frame of audio data, the MFCC feature is calculated. MFCC provides an effective audio feature representation by simulating the auditory characteristics of the human ear. The calculation process includes: applying a discrete Fourier transform (DFT) to obtain the spectrum of the frame, mapping the spectrum to the Mel scale, and using a set of Mel filters. Take the logarithm of the output of each Mel filter, apply a discrete cosine transform (DCT) to the logarithmic Mel spectrum, and obtain the MFCC feature as the audio feature of the target speech segment. It should be noted that the extraction method of the first audio feature and the second audio feature is not limited in this application. For details, please refer to the technical details of the above-mentioned technical method. For the sake of brevity, it will not be repeated here.

[0019] In one implementation scenario, a music component represents the audio components of music, such as drums, bass, piano, vocals, electronic mixing, etc. The timbre, loudness, frequency, etc. of each music component are different. The melody, content, etc. of the same music component in different music are not the same. Different styles of music contain different types of music components. For example, folk-style song A contains guitar, bass, tambourine, harmonica, maracas, vocals, etc., and rock-style song B contains electronic guitar, bass, drums, electronic mixing, etc.

[0020] In a specific implementation scenario, target sub-audios of several music components in target music are obtained, and target sub-features are encoded based on the target sub-audios. Specifically, when the target music is music to be converted, the target sub-feature is the first sub-feature, and when the target music is reference music, the target sub-feature is the second sub-feature.

[0021] In a specific implementation scenario, based on the audio information of the target music, the target sub-audio of each type of music component in the target music is obtained. For example, based on the timbre, loudness, frequency, etc. in the audio information, the target sub-audio belonging to each instrument is obtained, or the target sub-audio belonging to each singer is obtained. As a possible implementation method, a music component separation model can be pre-trained, and the music component separation model can include but is not limited to the network model of the Encoder-Decoder architecture. In order to ensure the generation accuracy of the music component separation model as much as possible, sample music composed of multiple sample sub-audio can be collected, and the real music components can be annotated on the sample sub-audio. On this basis, the sample music can be processed based on the music component separation model to obtain the predicted sub-audio of the sample music and the predicted music components of each predicted sub-audio, so that the network parameters of the music component separation model can be adjusted based on the difference between the sample sub-audio and the predicted sub-audio, and the difference between the real music component and the predicted music component, until the music component separation model training converges, and the target music can be processed based on the converged music component separation model to obtain the target sub-audio of each music component. It should be noted that the specific processing process of the music component separation model can refer to the technical details such as the network model of the Encoder-Decoder architecture, which will not be repeated here.

[0022] In another specific implementation scenario, the target sub-features are obtained by encoding the target music using a sub-component encoder. The specific network structure of the sub-component encoder is not limited in this application, such as a conditional similarity network structure.

[0023] In a specific implementation scenario, the training process of the sub-component encoder includes: obtaining sample music containing several music component sample sub-audios, and each sample sub-audio is labeled with a sample music component, obtaining positive sample audio and negative sample audio about the sample music component, the positive sample audio is consistent with the sample music component of the sample audio, and the negative sample audio is inconsistent with the sample music component of the sample audio. For example, the positive sample audio is audio of the same instrument type in different songs, and the negative sample audio is audio of different instrument types in the same song, based on the sub-component encoder, the sample sub-audio, positive sample audio, and negative sample audio about the sample music component are respectively encoded to obtain sample sub-audio features, positive sample audio features, and negative sample audio features, and the differences between the sample sub-audio features, positive sample audio features, and negative sample audio features are measured based on a triplet loss function to adjust the network parameters of the sub-component encoder. The above scheme trains the sub-component encoder through the triplet loss function and optimizes the sub-component encoder by defining a triplet, so that the sample sub-audios of the same music component are closer in the feature space, and the sample sub-audios of different music components are farther away from each other. This can improve the accuracy of feature encoding of target sub-audios of different music components, thereby improving the effect of music conversion.

[0024] It should be noted that the method for extracting the first sub-feature and the method for extracting the second sub-feature may be the same or different, and is not limited in the present application.

[0025] In one implementation scenario, before extracting the first audio feature of the music to be converted and the first sub-features of several musical components based on the audio data of the music to be converted, a standard instrument coding library is obtained, which contains standard codes of each instrument on the basic pitch. The basic pitch is also called the native pitch, which refers to the seven pitches with independent names in the musical system. These pitches correspond to the sounds made by the white keys on the piano. The basic pitch is represented by the seven letters "C, D, E, F, G, A, B" or "do, re, mi, fa, sol, la, si", which are called the "note names" of the basic pitch. It can be understood that the audio information on the basic pitch of different instruments or different singers will be different, specifically the difference in loudness, timbre, frequency, etc., while the audio information on the basic pitch of the same instrument type is relatively similar. Therefore, each standard code in the standard instrument coding library can reflect the basic audio information of a type of instrument and is not affected by melody, rhythm, etc.

[0026] It should be noted that the method for obtaining the standard encoding of basic pitches of various musical instruments is not limited in this application.

[0027] In a specific implementation scenario, when the music component of the first sub-feature belongs to a musical instrument, after extracting several first sub-features about the music to be converted and before constructing sub-feature pairs belonging to different music components, a first standard code consistent with the music component of the first sub-feature in the standard instrument code library is selected, and the fused code obtained by fusing the first sub-feature with the first standard code is used as the updated first sub-feature. The above scheme embeds the standard code about the musical instrument component into the first sub-feature of the music to be converted to obtain the updated first sub-feature, which can improve the specificity of the sub-features of different music components, and for music with complex audio signals, it can also minimize the loss of detail data during the conversion process, so it can further improve the accuracy of music conversion.

[0028] In a specific implementation scenario, the first sub-feature is merged with the first standard code by feature concatenation or other methods to obtain an updated first sub-feature.

[0029] In a specific implementation scenario, when the music component of the second sub-feature belongs to a musical instrument, after extracting several second sub-features about the reference music and before constructing sub-feature pairs belonging to different music components, a second standard code consistent with the music component of the second sub-feature in the standard instrument code library is selected, and the fused code obtained by fusing the second sub-feature with the second standard code is used as the updated second sub-feature. The above scheme embeds the standard code about the musical instrument component into the second sub-feature of the music to be converted to obtain the updated second sub-feature, which can improve the specificity of the sub-features of different music components, and for music with complex audio signals, it can also minimize the loss of detail data during the conversion process, so it can further improve the accuracy of music conversion.

[0030] In a specific implementation scenario, the second sub-feature is merged with the second standard code by feature concatenation or other methods to obtain an updated second sub-feature.

[0031] Step S20: constructing sub-feature pairs belonging to different music components based on the music components of each first sub-feature and the music components of each second sub-feature.

[0032] In the disclosed embodiment, the sub-feature pair of any music component represents: the feature mapping relationship of the second sub-feature belonging to the music component to the music to be converted. For example, the feature mapping relationship of the second sub-feature belonging to the guitar to the music to be converted, and the sub-feature pair can represent the sub-feature conversion logic at the music component level.

[0033] In one implementation scenario, when there is a first sub-feature that is consistent with the musical component of the second sub-feature, a sub-feature pair is constructed based on the second sub-feature and the first sub-feature that is consistent with the musical component of the second sub-feature. For example, there is a second sub-feature A whose musical component is guitar, and there is a first sub-feature B whose musical component is guitar, and a sub-feature pair (A, B) of guitar is constructed.

[0034] In an implementation scenario, when there is no first sub-feature that is consistent with the musical component of the second sub-feature, a sub-feature pair is constructed based on the second sub-feature and the zero vector feature. For example, there is a second sub-feature C whose musical component is piano, but there is no first sub-feature whose musical component is piano, the zero feature vector is set to M, and the piano sub-feature pair (C, M) is constructed.

[0035] In a specific implementation scenario, when there is no second sub-feature that is consistent with the music component of the first sub-feature, no sub-feature pair is constructed for the music component. The above scheme can generate sub-feature pairs based on the music component attributes of the reference music as much as possible, thereby improving the accuracy of target music generation.

[0036] Step S30: Generate target music based on the sub-feature pairs, the first audio feature and the second audio feature.

[0037] In one implementation scenario, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be achieved, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, which can preserve the feature matching relationship between the music to be converted and the reference music as much as possible at the music component level, improve the fine-grainedness of the target music generation, and for music with complex audio signals, can also reduce the loss of detail data in the conversion process as much as possible. Therefore, the automatic conversion of music styles can be achieved and the accuracy of music conversion can be improved.

[0038] In an implementation scenario, based on each sub-feature pair, the first audio feature and the second audio feature, audio hidden features of the sub-feature pairs belonging to each music component are generated respectively. Specifically, the audio hidden features of the sub-feature pairs are vectors, matrices, etc., which are not limited in this application. Based on each audio hidden feature, the first audio feature and the second audio feature, the target music is decoded. In the above scheme, the audio hidden features of the sub-feature pairs can characterize the encoding features of each music component with respect to the music to be converted and the reference music, and can retain the feature matching relationship between the music to be converted and the reference music as much as possible at the music component level, improve the granularity of the target music generation, and for music with complex audio signals, can also minimize the loss of detail data during the conversion process. Therefore, the target music generated based on each audio hidden feature, the first audio feature and the second audio feature can improve the accuracy of music generation.

[0039] In a specific implementation scenario, before generating audio hidden features of sub-feature pairs belonging to each music component based on each sub-feature pair, the first audio feature and the second audio feature, a first weight, a second weight and a third weight are obtained based on the music component to which the sub-feature pair belongs, the first weight is not less than the sum of the values ​​of the second weight and the third weight, the first weight represents the importance of the sub-feature pair in the audio hidden layer feature generation stage, the second weight represents the importance of the first audio feature in the audio hidden layer feature generation stage, and the third weight represents the importance of the second audio feature in the audio hidden layer feature generation stage. In the above scheme, the first weight is not less than the sum of the values ​​of the second weight and the third weight, which represents that the importance of the sub-feature pairs of each music component is more emphasized in the audio hidden layer feature generation stage, which improves the granularity of the target music generation and can also minimize the loss of detail data in the conversion process for music with complex audio signals.

[0040] It should be noted that the method for obtaining the weight is not limited in this application, for example, a fixed value set by a technician, a dynamic value dynamically adjusted based on the music type and music component type, etc.

[0041] In a specific implementation scenario, based on the first weight, the second weight, and the third weight corresponding to the music component to which the sub-feature pair belongs, the sub-feature pair, the first audio feature, and the second audio feature are weighted respectively to obtain the weighted sub-feature pair, the first audio feature, and the second audio feature, and based on the weighted sub-feature pair, the first audio feature, and the second audio feature, the audio hidden layer features of the sub-feature pair are processed. In the above scheme, the audio hidden layer features of the sub-feature pair can characterize the encoding features of each music component with respect to the music to be converted and the reference music, and can retain the feature matching relationship between the music to be converted and the reference music as much as possible at the music component level, improve the granularity of the target music generation, and for music with complex audio signals, can also minimize the loss of detail data during the conversion process. Therefore, the target music generated based on each audio hidden layer feature, the first audio feature, and the second audio feature can improve the accuracy of music generation.

[0042] The above scheme extracts the first audio feature of the music to be converted and the first sub-features about several music components based on the audio data of the music to be converted, and extracts the second audio feature of the reference music and the second sub-features about several music components based on the audio data of the reference music, and constructs sub-feature pairs belonging to different music components based on the music components of each first sub-feature and each second sub-feature, and the sub-feature pairs of any music component represent: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted, and generates the target music based on the sub-feature pairs, the first audio feature and the second audio feature. Therefore, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be realized, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, so that the feature matching relationship between the music to be converted and the reference music can be retained as much as possible at the music component level, and the fine-grainedness of the target music generation can be improved. For music with complex audio signals, the loss of detail data during the conversion process can also be reduced as much as possible. Therefore, the automatic conversion of music styles can be realized, and the accuracy of music conversion can be improved.

[0043] See also Figure 2 , Figure 2 2 is a schematic diagram of a framework of an embodiment of a music conversion model 20 in the music conversion method of the present application. In a specific implementation scenario, the target music is generated by the music conversion model 20 based on the input music to be converted and the reference music, such as Figure 2 As shown, the music conversion model 20 includes a sub-component encoding network 211, a feature fusion network 220, a sub-feature pair construction network 230, a sub-component decoding network 240, a global decoding network 250 and a feature extraction network 212. The sub-component encoding network 211 is used to extract the first sub-features of the music to be converted about several music components and the second sub-features of the reference music about several music components. The feature extraction network 212 is used to extract the first audio feature of the music to be converted and the second audio feature of the reference music. The feature fusion network 220 is used to fuse the first sub-feature with the first standard encoding to obtain an updated first sub-feature, and to fuse the second sub-feature with the second standard encoding to obtain an updated second sub-feature. The sub-feature pair construction network 230 is used to construct sub-feature pairs of different music components based on the first sub-feature and the second sub-feature. The sub-component decoding network 240 is used to generate audio hidden layer features of the sub-feature pairs belonging to each music component based on the sub-feature pairs, the first audio features, and the second audio features. The global decoding network 250 is used to decode the target music based on each audio hidden layer feature, the first audio feature, and the second audio feature.

[0044] In a specific implementation scenario, the sub-component decoding network 240 is a shallow decoder that focuses on the music component migration task. The sub-component decoding network 240 weights the sub-feature pair, the first audio feature, and the second audio feature based on the first weight, the second weight, and the third weight corresponding to the music component to which the sub-feature pair belongs, respectively, to obtain the weighted sub-feature pair, the first audio feature, and the second audio feature. Based on the weighted sub-feature pair, the first audio feature, and the second audio feature, the audio hidden layer features of the sub-feature pair are processed. Specifically, the weighted processing can be achieved through a gated weighted structure. The global decoding network 250 focuses on the overall style migration task, which can constrain the scope of data processing effects and improve the accuracy of target music generation.

[0045] It should be noted that the specific network structures of the sub-component encoding network 211, the feature fusion network 220, the sub-feature pair construction network 230, the sub-component decoding network 240, the global decoding network 250 and the feature extraction network 212 are not limited in this application. The specific implementation method can be found in the aforementioned embodiment. For the sake of brevity, it will not be repeated here.

[0046] In the above scheme, the music conversion model 20 extracts the first audio feature of the music to be converted and the first sub-feature about several music components based on the audio data of the music to be converted, and extracts the second audio feature of the reference music and the second sub-feature about several music components based on the audio data of the reference music, and constructs sub-feature pairs belonging to different music components based on the music components of each first sub-feature and each second sub-feature, and the sub-feature pair of any music component represents: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted, and the target music is generated based on the sub-feature pair, the first audio feature and the second audio feature. Therefore, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be realized, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, so that the feature matching relationship between the music to be converted and the reference music can be retained as much as possible at the music component level, and the fine-grainedness of the target music generation can be improved. For music with complex audio signals, the loss of detail data in the conversion process can also be reduced as much as possible. Therefore, the automatic conversion of music styles can be realized, and the accuracy of music conversion can be improved.

[0047] See also Figure 3 , Figure 3 Schematic diagram of the framework of an embodiment of the music conversion device 30 of the present application. Figure 3As shown, the music conversion device 30 includes: an extraction module 31, a construction module 32 and a generation module 33, the extraction module 31 is used to extract the first audio feature of the music to be converted and the first sub-feature about several music components based on the audio data of the music to be converted, and to extract the second audio feature of the reference music and the second sub-feature about several music components based on the audio data of the reference music; the construction module 32 is used to construct sub-feature pairs belonging to different music components based on the music components of each first sub-feature and the music components of each second sub-feature; wherein the sub-feature pair of any music component represents: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted; the generation module 33 is used to generate the target music based on the sub-feature pairs, the first audio feature and the second audio feature.

[0048] In the above scheme, the music conversion device 30 extracts the first audio feature of the music to be converted and the first sub-feature about several music components based on the audio data of the music to be converted, and extracts the second audio feature of the reference music and the second sub-feature about several music components based on the audio data of the reference music, and constructs sub-feature pairs belonging to different music components based on the music components of each first sub-feature and each second sub-feature, and the sub-feature pair of any music component represents: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted, and the target music is generated based on the sub-feature pair, the first audio feature and the second audio feature. Therefore, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be realized, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, so that the feature matching relationship between the music to be converted and the reference music can be retained as much as possible at the music component level, and the fine-grainedness of the target music generation can be improved. For music with complex audio signals, the loss of detail data during the conversion process can also be reduced as much as possible. Therefore, the automatic conversion of music styles can be realized, and the accuracy of music conversion can be improved.

[0049] In some disclosed embodiments, the generation module 33 also includes a hidden feature generation module (not shown), which is used to generate audio hidden features of sub-feature pairs belonging to each music component based on each sub-feature pair, the first audio feature and the second audio feature; the generation module 33 also includes an audio feature decoding module (not shown), which is used to decode the target music based on each audio hidden feature, the first audio feature and the second audio feature.

[0050] In some disclosed embodiments, before generating audio hidden features of sub-feature pairs belonging to each music component based on each sub-feature pair, the first audio feature and the second audio feature, the music conversion device 30 also includes a weight acquisition module (not shown) for acquiring a first weight, a second weight and a third weight based on the music component to which the sub-feature pair belongs; wherein the first weight is not less than the sum of the values ​​of the second weight and the third weight, the first weight represents the importance of the sub-feature pair in the audio hidden layer feature generation stage, the second weight represents the importance of the first audio feature in the audio hidden layer feature generation stage, and the third weight represents the importance of the second audio feature in the audio hidden layer feature generation stage; the hidden layer feature generation module (not shown) also includes a weighting module (not shown) for weighting the sub-feature pair, the first audio feature and the second audio feature based on the first weight, the second weight and the third weight corresponding to the music component to which the sub-feature pair belongs, respectively, to obtain weighted sub-feature pairs, first audio features and second audio features; the hidden layer feature generation module (not shown) also includes a processing module (not shown) for processing the audio hidden layer features of the sub-feature pairs based on the weighted sub-feature pairs, the first audio features and the second audio features.

[0051] In some disclosed embodiments, the construction module 32 also includes a first construction module (not shown) for constructing a sub-feature pair based on the second sub-feature and the first sub-feature consistent with the second sub-feature musical component in response to the presence of the first sub-feature consistent with the musical component of the second sub-feature; the construction module 32 also includes a second construction module (not shown) for constructing a sub-feature pair based on the second sub-feature and the zero vector feature in response to the absence of the first sub-feature consistent with the musical component of the second sub-feature.

[0052] In some disclosed embodiments, the music conversion device also includes a sub-audio acquisition module (not shown) for acquiring target sub-audio of several music components in the target music; the music conversion device also includes a sub-feature encoding module (not shown) for encoding target sub-features based on the target sub-audio; wherein, when the target music is the music to be converted, the target sub-feature is the first sub-feature, and when the target music is the reference music, the target sub-feature is the second sub-feature.

[0053] In some disclosed embodiments, the target sub-features are obtained by encoding the target music by a sub-component encoder, and the music conversion device 30 also includes a sub-component encoder training module (not shown) for obtaining sample music containing several music component sample sub-audios; wherein each sample sub-audio is labeled with a sample music component; positive sample audio and negative sample audio regarding the sample music component are obtained; wherein the positive sample audio is consistent with the sample music component of the sample audio, and the negative sample audio is inconsistent with the sample music component of the sample audio; based on the sub-component encoder, the sample sub-audio, positive sample audio, and negative sample audio regarding the sample music component are respectively encoded to obtain sample sub-audio features, positive sample audio features, and negative sample audio features; based on the triplet loss function, the differences among the sample sub-audio features, positive sample audio features, and negative sample audio features are measured to adjust the network parameters of the sub-component encoder.

[0054] In some disclosed embodiments, before extracting the first audio feature of the music to be converted and the first sub-features about several music components based on the audio data of the music to be converted, the music conversion device 30 also includes a coding library acquisition module (not shown) for acquiring a standard instrument coding library; wherein the standard instrument coding library contains standard codes of each instrument about the basic pitch; in the case where the music component of the first sub-feature belongs to the instrument, after extracting several first sub-features about the music to be converted and before constructing sub-feature pairs belonging to different music components, the music conversion device 30 also includes a first updating module (not shown) for selecting a first standard code consistent with the first sub-feature music component in the standard instrument coding library; and using a fused code obtained by fusing the first sub-feature with the first standard code as the updated first sub-feature; in the case where the music component of the second sub-feature belongs to the instrument, after extracting several second sub-features about the reference music and before constructing sub-feature pairs belonging to different music components, the music conversion device 30 also includes a second updating module (not shown) for selecting a second standard code consistent with the second sub-feature music component in the standard instrument coding library; and using a fused code obtained by fusing the second sub-feature with the second standard code as the updated second sub-feature.

[0055] See also Figure 4 , Figure 4 4 is a schematic diagram of a framework of an embodiment of an electronic device 40 of the present application. The electronic device 40 includes a memory 41 and a processor 42. The memory 41 stores program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-mentioned music conversion method embodiments. For details, please refer to the aforementioned disclosed embodiments, which will not be repeated here. The electronic device 40 may specifically include but is not limited to: a server, a smart phone, a laptop computer, a tablet computer, a self-service machine, etc., which are not limited here.

[0056] Specifically, the processor 42 is used to control itself and the memory 41 to implement the steps in any of the above-mentioned music conversion method embodiments. The processor 42 can also be called a CPU (Central Processing Unit). The processor 42 may be an integrated circuit chip with signal processing capabilities. The processor 42 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 42 can be implemented by an integrated circuit chip.

[0057] In the above scheme, the electronic device 40 extracts the first audio feature of the music to be converted and the first sub-feature about several music components based on the audio data of the music to be converted, and extracts the second audio feature of the reference music and the second sub-feature about several music components based on the audio data of the reference music, and constructs sub-feature pairs belonging to different music components based on the music components of each first sub-feature and each second sub-feature, and the sub-feature pair of any music component represents: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted, and generates the target music based on the sub-feature pair, the first audio feature and the second audio feature. Therefore, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be realized, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, so that the feature matching relationship between the music to be converted and the reference music can be retained as much as possible at the music component level, and the fine-grainedness of the target music generation can be improved, and for music with complex audio signals, the loss of detail data during the conversion process can be reduced as much as possible. Therefore, the automatic conversion of music styles can be realized, and the accuracy of music conversion can be improved.

[0058] See also Figure 5 , Figure 5 1 is a schematic diagram of a framework of an embodiment of a computer-readable storage medium 50 of the present application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor, and the program instructions 51 are used to implement the steps in any of the above-mentioned music conversion method embodiments.

[0059] In the above scheme, the computer-readable storage medium 50 extracts the first audio feature of the music to be converted and the first sub-feature about several music components based on the audio data of the music to be converted, and extracts the second audio feature of the reference music and the second sub-feature about several music components based on the audio data of the reference music, and constructs sub-feature pairs belonging to different music components based on the music components of each first sub-feature and each second sub-feature, and the sub-feature pair of any music component represents: the feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted, and generates the target music based on the sub-feature pair, the first audio feature and the second audio feature. Therefore, based on the audio data of the music to be converted and the audio data of the reference music, the automatic conversion of the music of the desired style can be realized, and the sub-feature pairs belonging to different music components are used as the feature data for generating the target music, so that the feature matching relationship between the music to be converted and the reference music can be retained as much as possible at the music component level, and the fine-grainedness of the target music generation can be improved, and for music with complex audio signals, the loss of detail data in the conversion process can be reduced as much as possible. Therefore, the automatic conversion of music styles can be realized, and the accuracy of music conversion can be improved.

[0060] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0061] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0062] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0063] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0064] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0065] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

Claims

1. A music conversion method, characterized in that: include: Extracting a first audio feature of the music to be converted and a first sub-feature of a plurality of music components based on the audio data of the music to be converted, and extracting a second audio feature of the reference music and a second sub-feature of a plurality of music components based on the audio data of the reference music; Based on the music components of each of the first sub-features and the music components of each of the second sub-features, sub-feature pairs belonging to different music components are constructed; wherein the sub-feature pair of any of the music components represents: a feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted; Target music is generated based on the sub-feature pairs, the first audio feature, and the second audio feature.

2. The method according to claim 1, characterized in that: The generating target music based on the sub-feature pair, the first audio feature and the second audio feature comprises: Based on each of the sub-feature pairs, the first audio feature, and the second audio feature, respectively generate audio hidden layer features of the sub-feature pairs belonging to each of the music components; The target music is decoded based on the respective audio hidden layer features, the first audio features and the second audio features.

3. The method according to claim 2, characterized in that Before respectively generating audio hidden layer features of the sub-feature pairs belonging to each of the music components based on each of the sub-feature pairs, the first audio feature, and the second audio feature, the method further includes: Based on the music component to which the sub-feature pair belongs, a first weight, a second weight and a third weight are obtained; wherein the first weight is not less than the sum of the values ​​of the second weight and the third weight, the first weight represents the importance of the sub-feature pair in the audio hidden layer feature generation stage, the second weight represents the importance of the first audio feature in the audio hidden layer feature generation stage, and the third weight represents the importance of the second audio feature in the audio hidden layer feature generation stage; The step of generating audio hidden layer features of the sub-feature pairs belonging to each of the music components based on each of the sub-feature pairs, the first audio feature, and the second audio feature comprises: Based on the first weight, the second weight, and the third weight corresponding to the music component to which the sub-feature pair belongs, weighting the sub-feature pair, the first audio feature, and the second audio feature respectively to obtain the weighted sub-feature pair, the first audio feature, and the second audio feature; Based on the weighted sub-feature pair, the first audio feature, and the second audio feature, audio hidden layer features of the sub-feature pair are obtained by processing.

4. The method according to claim 1, characterized in that The constructing sub-feature pairs belonging to different music components based on the music components of each of the first sub-features and the music components of each of the second sub-features comprises: In response to the presence of the first sub-feature being consistent with the musical component of the second sub-feature, constructing the sub-feature pair based on the second sub-feature and the first sub-feature being consistent with the musical component of the second sub-feature; In response to the absence of the first sub-feature that is consistent with the musical component of the second sub-feature, the sub-feature pair is constructed based on the second sub-feature and a zero-vector feature.

5. The method according to claim 1, characterized in that The method further comprises: Obtain target sub-audios of several music components in target music; Based on the target sub-audio, encoding obtains a target sub-feature; Wherein, when the target music is the music to be converted, the target sub-feature is the first sub-feature; and when the target music is the reference music, the target sub-feature is the second sub-feature.

6. The method according to claim 5, characterized in that The target sub-feature is obtained by encoding the target music with a sub-component encoder, and the training process of the sub-component encoder includes: Obtaining sample music containing a plurality of music component sample sub-audios, wherein each sample sub-audio is labeled with a sample music component; Acquire a positive sample audio and a negative sample audio about the sample music component; wherein the positive sample audio is consistent with the sample music component of the sample audio, and the negative sample audio is inconsistent with the sample music component of the sample audio; Based on the sub-component encoder, the sample sub-audio, positive sample audio, and negative sample audio of the sample music component are respectively encoded to obtain sample sub-audio features, positive sample audio features, and negative sample audio features; The differences among the sample sub-audio features, the positive sample audio features, and the negative sample audio features are measured based on a triplet loss function to adjust the network parameters of the sub-component encoder.

7. The method according to claim 1, characterized in that Before extracting the first audio feature of the music to be converted and the first sub-features about a plurality of music components based on the audio data of the music to be converted, the method further comprises: Obtain a standard musical instrument code library; wherein the standard musical instrument code library contains standard codes of basic pitches of various musical instruments; In the case where the music component of the first sub-feature belongs to a musical instrument, after extracting a plurality of first sub-features about the music to be converted and before constructing sub-feature pairs belonging to different music components, the method further includes: Selecting a first standard code in the standard instrument code library that is consistent with the first sub-feature music component; The fused code obtained by fusing the first sub-feature with the first standard code is used as the updated first sub-feature; In the case where the music component of the second sub-feature belongs to a musical instrument, after extracting a plurality of second sub-features about the reference music and before constructing sub-feature pairs belonging to different music components, the method further includes: Selecting a second standard code in the standard instrument code library that is consistent with the second sub-characteristic music component; The fused code obtained by fusing the second sub-feature with the second standard code is used as the updated second sub-feature.

8. A music conversion device, characterized in that: include: An extraction module, configured to extract a first audio feature of the music to be converted and first sub-features about a plurality of music components based on the audio data of the music to be converted, and to extract a second audio feature of the reference music and second sub-features about a plurality of music components based on the audio data of the reference music; A construction module, configured to construct sub-feature pairs belonging to different music components based on the music components of each of the first sub-features and the music components of each of the second sub-features; wherein the sub-feature pair of any of the music components represents: a feature mapping relationship of the second sub-feature belonging to the music component with respect to the music to be converted; A generating module is used to generate target music based on the sub-feature pair, the first audio feature and the second audio feature.

9. An electronic device, characterized in that: It comprises a memory and a processor coupled to each other, wherein the memory stores program instructions, and the processor is used to execute the program instructions to implement the music conversion method described in any one of claims 1 to 7.

10. A computer-readable storage medium having program instructions stored thereon, characterized in that: When the program instructions are executed by a processor, the music conversion method described in any one of claims 1 to 7 is implemented.

Images