Audio processing method and device, electronic equipment and storage medium

Abstract

Embodiments of the invention provide an audio processing method and apparatus, an electronic device and a storage medium. The method comprises the steps of obtaining first music data and a processing instruction in a text form associated with the first music data; extracting a first chord proceeding feature and an audio feature of the first music data and a text feature of the processing instruction through a music processing model; processing the audio features through the music processing model according to the first chord proceeding features and the text features to generate second music data; a similarity between a first chord progression feature of the first music data and a second chord progression feature of the second music data is greater than a similarity threshold.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an audio processing method, device, electronic device and storage medium. Background Art

[0002] Existing technologies can process audio data, such as adding a kitten's cry to a piece of audio data of a bird's cry. However, the existing technology focuses on processing speech rather than music. After the music is processed by the existing technology, the harmony and consistency of the music before and after the processing are low. Summary of the invention

[0003] The embodiments of the present disclosure provide an audio processing method, device, electronic device and storage medium, which can process music data and ensure that the music data before and after processing have high harmony and consistency in musicality.

[0004] In a first aspect, an embodiment of the present disclosure provides an audio processing method, including:

[0005] acquiring first music data and a processing instruction in text form associated with the first music data;

[0006] Extracting the first chord progression feature and the audio feature of the first music data and the text feature of the processing instruction through the music processing model;

[0007] Through the music processing model, the audio feature is processed according to the first chord progression feature and the text feature to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

[0008] In a second aspect, an embodiment of the present disclosure provides an audio processing device, including:

[0009] an acquisition unit, configured to acquire first music data and a processing instruction in text form associated with the first music data;

[0010] an extraction unit, configured to extract, by means of a music processing model, a first chord progression feature and an audio feature of the first music data, and a text feature of the processing instruction;

[0011] A generating unit is used to process the audio feature according to the first chord progression feature and the text feature through the music processing model to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

[0012] In a third aspect, an embodiment of the present disclosure provides an electronic device, comprising: a processor; and a memory configured to store computer-executable instructions, wherein the computer-executable instructions, when executed, enable the processor to implement the steps of the method described in the first aspect above.

[0013] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions implement the steps of the method described in the first aspect when executed by a processor.

[0014] In one or more embodiments of the present disclosure, first music data and a processing instruction in text form associated with the first music data are obtained, and a first chord progression feature and an audio feature of the first music data and a text feature of the processing instruction are extracted through a music processing model, and the audio feature is processed according to the first chord progression feature and the text feature through the music processing model to generate second music data, wherein the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold. It can be seen that since the music processing model can extract the first chord progression feature of the first music data and generate the second music data based on the extracted first chord progression feature, when the first music data is processed to generate the second music data, the chord progression features of the first music data and the second music data can be ensured to be relatively consistent, so that the first music data and the second music data have a higher harmony and consistency in terms of musicality. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate one or more embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in the present disclosure. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative labor.

[0016] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure;

[0017] Figure 2 A schematic diagram of the application principle of a music processing model provided by an embodiment of the present disclosure;

[0018] Figure 3 A schematic diagram of a process for training a music processing model provided by an embodiment of the present disclosure;

[0019] Figure 4A schematic diagram of the training principle of a music processing model provided in one embodiment of the present disclosure;

[0020] Figure 5 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present disclosure;

[0021] Figure 6 A schematic diagram of the structure of an electronic device provided in one embodiment of the present disclosure. DETAILED DESCRIPTION

[0022] In order to enable those skilled in the art to better understand the technical solutions in one or more embodiments of the present disclosure, the technical solutions in one or more embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in one or more embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. Based on one or more embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the protection scope of the present disclosure.

[0023] The disclosed embodiment provides an audio processing method, which can process music data and ensure that the music data before and after the processing have high harmony and consistency in terms of musicality.

[0024] Figure 1 A flowchart of an audio processing method provided by an embodiment of the present disclosure is shown in FIG. Figure 1 As shown, the process includes:

[0025] Step S102, obtaining first music data and a processing instruction in text form associated with the first music data;

[0026] Step S104, extracting the first chord progression feature and the audio feature of the first music data and the text feature of the processing instruction through the music processing model;

[0027] Step S106, using a music processing model, the audio feature is processed according to the first chord progression feature and the text feature to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

[0028] In this embodiment, first music data and a processing instruction in text form associated with the first music data are obtained, and the first chord progression feature and audio feature of the first music data and the text feature of the processing instruction are extracted through a music processing model, and the audio feature is processed according to the first chord progression feature and the text feature through the music processing model to generate second music data, wherein the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold. It can be seen that since the music processing model can extract the first chord progression feature of the first music data and generate the second music data based on the extracted first chord progression feature, when the first music data is processed to generate the second music data, the chord progression features of the first music data and the second music data can be ensured to be relatively consistent, so that the first music data and the second music data have a higher harmony and consistency in terms of musicality.

[0029] Below Figure 1 The process is described in detail.

[0030] In the above step S102, the first music data and the processing instructions in the form of text associated with the first music data are obtained. The first music data is the music data before processing. The first music data and the processing instructions in the form of text associated with the first music data can be obtained by user input. The user processes the first music data by inputting the first music data and the processing instructions. The processing instructions are used to perform at least one of the following processing on the first music data: adding instrument tracks to the first music data, deleting the existing instrument tracks in the first music data, modifying the music style of the first music data, and modifying the music emotion of the first music data. Among them, the music style refers to the style of the music melody, and the music style can be, for example: fast-paced, slow-paced, aggressive, high-pitched, soothing, low-pitched, etc. The music emotion refers to the feeling of the mood that the music brings to the audience, and the music style can be, for example: happy, cheerful, sad, melancholy, calm, etc.

[0031] Take the first music data as an example of a piece of pure music with piano and bass. In one example, the processing instruction is: "Add guitar", then the processing instruction is used to remix the first music data, and add a guitar track to the first music data. In another example, the processing instruction is: "Remove bass", then the processing instruction is used to remix the first music data, and delete the existing bass track in the first music data. In another example, the processing instruction is: "Add guitar, remove bass, and process it into a fast-paced style", then the processing instruction is used to remix the first music data, add a guitar track and remove the bass track, and process the first music data into fast-paced music data. In another example, the processing instruction is: "Add guitar, process it into sad music", then the processing instruction is used to remix the first music data, add a guitar track, and process the first music data into music data that makes the audience feel sad.

[0032] In one example, a user can upload first music data in a terminal device such as a mobile phone, and input a processing instruction in text form in a manner similar to inputting a chat message, thereby obtaining the first music data and the processing instruction. In another example, a user can upload first music data in a terminal device such as a mobile phone, and input a processing instruction in voice form in a manner similar to inputting a chat message, thereby obtaining the first music data and the processing instruction in voice form, and then converting the processing instruction in voice form into a processing instruction in text form. After processing and obtaining the second music data, the terminal device can play the second music data, thereby realizing user-interactive music data processing.

[0033] In the above step S102, the first music data and the processing instruction in text form are also input into the music processing model. The music processing model is a pre-trained model, and can process the first music data according to the processing instruction in text form to obtain second music data matching the processing instruction. The second music data can match the processing content indicated by the processing instruction. For example, in the above example, the first music data is remixed according to the processing instruction in text form, a guitar track is added, and the first music data is processed into second music data that makes the audience feel sad.

[0034] In the above step S104, the chord progression feature of the first music data is extracted as the first chord progression feature through the music processing model, the audio feature of the first music data is extracted, and the text feature of the processing instruction is extracted.

[0035] Figure 2 A schematic diagram of the application principle of a music processing model provided in an embodiment of the present disclosure is shown in FIG. Figure 2As shown, the music processing model includes a pre-processing module, and the pre-processing module includes a chord progression feature extraction unit, an audio feature extraction unit, and a text feature extraction unit. The first music data is input into the chord progression feature extraction unit and the audio feature extraction unit respectively, and the first music data is processed by the chord progression feature extraction unit to extract the chord progression feature of the first music data as the first chord progression feature, and the first music data is processed by the audio feature extraction unit to extract the audio feature of the first music data. The first chord progression feature is used to represent the chord type information of the first music data at different time points, and the first chord progression feature can be represented by a matrix, and the matrix implicitly represents the chord type information of the first music data at different time points. The audio feature is used to represent the pitch information, music duration information, rhythm information, audio frequency information, etc. of the first music data, and the audio feature can be represented by a matrix, and the matrix implicitly represents the pitch information, music duration information, rhythm information, audio frequency information, etc. of the first music data. The processing instruction is input into the text feature extraction unit, and the processing instruction is processed by the text feature extraction unit to extract the text feature of the processing instruction. The text feature at least includes the semantic information of the processing instruction.

[0036] In the above step S106, the audio features are processed according to the first chord progression features and the text features through the music processing model to generate the second music data. Since the first chord progression features of the first music data are referenced when generating the second music data, the similarity between the first chord progression features of the first music data and the second chord progression features of the second music data can be greater than the similarity threshold, thereby ensuring that the chord progression features of the first music data and the second music data are relatively consistent, so that the first music data and the second music data have higher harmony and consistency in terms of musicality. The similarity threshold can be a pre-set threshold. By comparing the chord types at different time points represented by the first chord progression features and the chord types at different time points represented by the second chord progression features, the similarity between the first chord progression features and the second chord progression features is obtained. For example, the first chord progression feature indicates that the chord types at various time points are type 1, type 2, and type 1, respectively, and the second chord progression feature indicates that the chord types at various time points are type 1, type 2, type 4, and type 5, respectively. The similarity is predetermined to be equal to the number of identical chord types between the first chord progression feature and the second chord progression feature (including repeated chord types) divided by the maximum number of chord types of the first chord progression feature and the second chord progression feature (including repeated chord types), then the similarity between the first chord progression feature and the second chord progression feature is determined to be 50%.

[0037] In one embodiment, the audio feature is processed according to the first chord progression feature and the text feature by a music processing model to generate the second music data, including:

[0038] By means of the music processing model, feature compression processing is performed on the audio feature and the noise feature of the random noise data according to the text feature to obtain a first feature; wherein the first feature is used to represent information related to the processing instruction in the first music data and information related to the processing instruction in the random noise data; the random noise data is noise data randomly generated by the music processing model for the first music data;

[0039] Through the music processing model, according to the first chord progression feature and the text feature, the first feature is subjected to feature weight adjustment processing and feature restoration processing to generate second music data. The feature value in the first feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration; the feature weight adjustment processing is used to adjust the feature weight.

[0040] In this embodiment, the music processing model can randomly generate noise data for the first music data, and the randomly generated noise data is random noise data. The audio feature extraction unit in the music processing model can also process the random noise data to extract the noise features of the random noise data. Then, through the music processing model, feature compression processing is performed on the audio features and the noise features according to the text features to obtain the first features corresponding to the first music data and the random noise data. The first features corresponding to the first music data and the random noise data can be data in the form of a matrix, which is a more abstract and more concentrated effective feature extracted from the audio features of the first music data and the noise features of the random noise data. The first features corresponding to the first music data and the random noise data are used to represent information related to the processing instructions input by the user in the first music data and information related to the processing instructions input by the user in the random noise data, and the related information is the effective information of the first music data and the random noise data.

[0041] Next, through the music processing model, according to the above-mentioned first chord progression feature and the above-mentioned text feature, the first feature corresponding to the first music data and the random noise data is subjected to feature weight adjustment processing and feature restoration processing to generate second music data. Among them, the feature value in the first feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration, and the feature weight adjustment processing is used to adjust the feature weight of the feature value in the first feature.

[0042] It can be seen that through this embodiment, random noise data is generated for the first music data through the music processing model, and the first feature corresponding to the first music data and the random noise data is extracted according to the text feature of the processing instruction, and the first feature is subjected to feature weight adjustment processing and feature restoration processing according to the first chord progression feature of the first music data and the text feature of the processing instruction, so as to generate the second music data. Since the second music data is obtained by subjecting the first feature to feature weight adjustment processing and feature restoration processing based on the first chord progression feature and the text feature, the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data can be greater than the similarity threshold, thereby ensuring that the chord progression features of the first music data and the second music data are relatively consistent, and the second music data is matched with the processing content indicated by the user's processing instruction, thereby generating music data that meets the user's needs.

[0043] In one embodiment, the music processing model is used to perform feature compression processing on the audio feature and the noise feature of the random noise data according to the text feature to obtain the first feature, including:

[0044] The audio features are downsampled using a music processing model, and the noise features and the downsampled audio features are downsampled based on the text features to obtain a first feature.

[0045] In this embodiment, the audio features of the first music data are first downsampled through the music processing model, and then, based on the text features, the noise features of the random noise data and the downsampled audio features are downsampled again to obtain the first features.

[0046] It can be seen that in this embodiment, by downsampling the audio features of the first music data multiple times, the music processing model can strengthen the learning of the audio features of the first music data when generating the first feature, so that the information related to the processing instructions in the first music data represented by the first feature is more accurate, and the second music data that better meets the user's needs is generated based on the first feature.

[0047] like Figure 2 As shown, in one embodiment, the music processing model includes an intermediate processing module, the intermediate processing module includes a first downsampling unit and a second downsampling unit; the first downsampling unit includes a plurality of first downsampling layers connected in sequence; the second downsampling unit includes a plurality of second downsampling layers connected in sequence; based on this, the audio feature is downsampled through the music processing model, and the noise feature and the downsampled audio feature are downsampled according to the text feature to obtain the first feature, including:

[0048] Downsampling the audio features through each first downsampling layer; wherein the input of the first downsampling layer includes the audio features; the input of the n+1th first downsampling layer includes the output of the nth first downsampling layer; n is an integer greater than or equal to 1 and less than or equal to T-1; T is the number of first downsampling layers;

[0049] Through each second downsampling layer, the noise features and the audio features after downsampling are downsampled according to the text features; wherein the input of the first second downsampling layer includes the noise features, the output of the first first downsampling layer and the text features; the input of the m+1th second downsampling layer includes the output of the mth first downsampling layer, the output of the n+1th first downsampling layer and the text features of the processing instructions; m is an integer greater than or equal to 1 and less than or equal to S-1; S is the number of second downsampling layers; the feature of the output of the last second downsampling layer is the first feature.

[0050] refer to Figure 2 In this embodiment, the music processing model includes an intermediate processing module, the intermediate processing module includes a first downsampling unit and a second downsampling unit, the first downsampling unit includes a plurality of first downsampling layers connected in sequence, and the figure takes three first downsampling layers as an example for illustration. For the first downsampling layer, the input is the audio feature of the first music data, for each first downsampling layer after the first downsampling layer, the input is the output of the previous first downsampling layer, and each first downsampling layer is used to downsample the input feature.

[0051] refer to Figure 2 In this embodiment, the second downsampling unit includes a plurality of second downsampling layers connected in sequence, and three second downsampling layers are used as an example in the figure. For the first second downsampling layer, the input includes the text features of the processing instructions, the noise features of the random noise data, and the output of the first first downsampling layer. For each second downsampling layer after the first second downsampling layer, the input includes the text features of the processing instructions, the output of the previous second downsampling layer, and the output of the first downsampling layer of the same number of layers. For example, for the second second downsampling layer, the input includes the text features of the processing instructions, the output of the first second downsampling layer, and the output of the second first downsampling layer. Each second downsampling layer is used to downsample the input features.

[0052] from Figure 2 It can be seen that the number of the first downsampling layers is the same as the number of the second downsampling layers. For the last second downsampling layer, the input includes the text features of the processing instructions, the output of the previous second downsampling layer, and the output of the last first downsampling layer. The feature of the output of the last second downsampling layer is the first feature mentioned above.

[0053] It can be seen that in this embodiment, the audio features are first downsampled through the first downsampling layer, and then the noise features of the random noise data and the audio features after downsampling are downsampled again according to the text features through the second downsampling layer, so that the music processing model can strengthen the learning of the audio features of the first music data when generating the first feature, so that the information related to the processing instructions in the first music data represented by the first feature is more accurate.

[0054] The downsampling process mentioned in each embodiment of the present disclosure is a means for achieving feature compression. In actual implementation, the downsampling process can also be replaced by other feature compression means, and other feature compression means are not limited here.

[0055] like Figure 2 As shown, in one embodiment, the intermediate processing module of the music processing model further includes a feature weight adjustment unit and a feature restoration unit. Based on this, the music processing model is used to perform feature weight adjustment processing and feature restoration processing on the first feature according to the first chord progression feature and the text feature to generate second music data, including:

[0056] By means of a feature weight adjustment unit, the feature weights of the feature values ​​in the first feature are adjusted according to the first chord progression feature, so as to obtain the first feature after the feature weights are adjusted;

[0057] The feature restoration unit performs feature restoration on the first feature after feature weight adjustment according to the text feature to generate second music data.

[0058] like Figure 2 As shown, in this embodiment, the intermediate processing module of the music processing model also includes a feature weight adjustment unit and a feature restoration unit. The input of the feature weight adjustment unit includes the first chord progression feature and the first feature of the first music data. Through the feature weight adjustment unit, according to the first chord progression feature, the feature weight of each feature value in the first feature is adjusted to obtain the first feature after the feature weight adjustment. Among them, the feature value in the first feature is used to represent the information related to the processing instruction in the first music data and the information related to the processing instruction in the random noise data. Each feature value has its own feature weight, which can represent the importance of the feature value when performing feature restoration. The feature weight of each feature value is adjusted, that is, the importance of each feature value in the subsequent feature restoration is adjusted, so that the feature restoration is performed based on the first feature after the feature weight adjustment, and the second music data matching the processing instruction and the second chord progression feature meeting the requirements is obtained.

[0059] Furthermore, if Figure 2As shown, the first feature after the feature weight is adjusted is also input into the feature restoration unit, and the feature restoration unit performs feature restoration on the first feature after the feature weight is adjusted according to the text feature to generate the second music data.

[0060] It can be seen that through this embodiment, first, the feature weight adjustment unit is used to adjust the feature weights of each feature value in the first feature according to the first chord progression feature to obtain the first feature after the feature weight adjustment. Then, the feature restoration unit is used to restore the first feature after the feature weight adjustment according to the text feature to generate the second music data. By adjusting the feature weights of the feature values ​​of the first feature, after feature restoration based on the first feature after the feature weight adjustment, the second music data that matches the processing instruction and whose second chord progression feature meets the requirements can be obtained, thereby improving the accuracy of generating the second music data.

[0061] like Figure 2 As shown, the feature weight adjustment unit includes a feature weight adjustment layer based on the attention mechanism. Based on this, the feature weight adjustment unit adjusts the feature weights of each feature value in the first feature according to the first chord progression feature to obtain the first feature after the feature weight adjustment, including:

[0062] Through the feature weight adjustment layer, according to the first chord progression feature, the feature weights of each feature value in the first feature are adjusted based on the attention mechanism to obtain the first feature after the feature weight adjustment.

[0063] In this embodiment, the feature weight adjustment layer can be used to adjust the feature weights of each feature value in the first feature based on the cross attention mechanism according to the first chord progression feature to obtain the first feature after the feature weight adjustment.

[0064] It should be noted that in Figure 2 In the model shown, the first downsampling layer, the second downsampling layer, the feature weight adjustment layer in the feature weight adjustment unit, and the upsampling layer in the feature restoration unit all adopt a processing strategy based on the cross-attention mechanism. The difference is that in the second downsampling layer and the upsampling layer in the feature restoration unit, the cross-attention mechanism can only be used for feature processing based on the text features of the processing instructions, while in the feature weight adjustment layer, the cross-attention mechanism can be used for feature processing based on the first chord progression feature, so that the similarity between the second chord progression feature of the generated second music data and the first chord progression feature of the first music data is greater than the similarity threshold.

[0065] It can be seen that through this embodiment, the feature weight adjustment layer can be used to efficiently and quickly adjust the feature weights of each feature value in the first feature based on the attention mechanism according to the first chord progression feature, thereby improving the efficiency of obtaining the first feature after the feature weight adjustment.

[0066] like Figure 2 As shown, the feature restoration unit includes a plurality of upsampling layers connected in sequence. Based on this, the feature restoration unit performs feature restoration on the first feature after the feature weight is adjusted according to the text feature to generate the second music data, including:

[0067] Through each upsampling layer, the first feature after the feature weight is adjusted is upsampled according to the text feature; wherein the input of the first upsampling layer includes the first feature after the feature weight is adjusted and the text feature, and the input of the p+1th upsampling layer includes the output of the pth upsampling layer and the text feature; p is an integer greater than or equal to 1 and less than or equal to W-1; W is the number of upsampling layers;

[0068] The features output by the last upsampling layer are decoded to obtain the second music data.

[0069] refer to Figure 2 The feature restoration unit includes multiple upsampling layers connected in sequence. The figure takes three upsampling layers as an example. For the first upsampling layer, the input includes the text features and the first feature of the processing instruction. For each upsampling layer after the first upsampling layer, the input includes the output and text features of the previous upsampling layer. Figure 2 As shown, the number of the second downsampling layer and the upsampling layer is the same, and the second downsampling layer and the upsampling layer are also residually connected, and the output of each second downsampling layer is also used as the input of the corresponding upsampling layer. Each upsampling layer is used to upsample the input features.

[0070] In this embodiment, the features output by the last upsampling layer are also decoded to obtain the second music data.

[0071] It can be seen that in this embodiment, the first feature after the feature weight is adjusted is upsampled according to the text feature through each upsampling layer, and the feature output by the last upsampling layer is decoded to obtain the second music data. Since the first feature after the feature weight is adjusted is upsampled according to the text feature of the processing instruction, the generated second music data can match the user's processing instruction.

[0072] like Figure 2 As shown, in one embodiment, the music processing model further includes a post-processing module, and the post-processing module includes a style guiding unit and a decoding unit, based on which the features output by the last upsampling layer are decoded to obtain second music data, including:

[0073] By means of an emotion style guidance module, the music emotion and / or music style indicated by the text features are identified as decoding guidance information;

[0074] Through the emotional style guidance module, according to the decoding guidance information, the decoding unit is guided to decode the features output by the last upsampling layer to obtain the second music data matching the decoding guidance information.

[0075] In this embodiment, the text features of the processing instructions are also input into the emotional style guidance module, which identifies the musical emotions indicated by the text features as decoding guidance information, or identifies the musical styles indicated by the text features as decoding guidance information, or identifies the musical emotions and musical styles indicated by the text features as decoding guidance information.

[0076] Furthermore, the emotional style guidance module guides the decoding unit to decode the features output by the last upsampling layer according to the decoding guidance information, and obtains the second music data matching the decoding guidance information. The emotional style guidance module can calculate the gap between the decoding guidance information and the second music data obtained by decoding, and provide feedback guidance based on the gap to adjust the audio features and the first chord progression features of the first music data, thereby adjusting the first features and generating the adjusted second music data, so as to finally generate the second music data matching the decoding guidance information.

[0077] It can be seen that in this embodiment, since the music emotion and / or music style indicated by the text feature can be identified as decoding guidance information, and according to the decoding guidance information, the features output by the last upsampling layer are decoded to obtain the second music data, the second music data can be matched with the decoding guidance information, so that the second music data has the music style and / or music emotion expected by the user.

[0078] In a specific example, in the pre-processing module of the music processing model, the audio feature extraction unit includes an auto encoder, which is a type of encoder that can use input information as a learning target and perform characterization learning on the input information, or the audio feature extraction unit can be implemented using an audio feature extraction network of a sound stream model. The chord progression feature extraction unit can be implemented based on existing technologies such as the technology described in the article "Joint Learning: Joint Learning for Multi-Instrument Transcrition and Its Applications".

[0079] In a specific example, the intermediate processing module in the music processing model can be implemented based on the unet network of the diffusion model Diffusion.

[0080] In a specific example, in the post-processing module of the music processing model, the decoding unit can be implemented based on an auto decoder, or a decoding network of a sound stream model. The emotional style guidance unit can be implemented based on a Mulan model, a CLAP model, or other models.

[0081] In a specific example, when the intermediate processing module of the music processing model is implemented based on the unet network of the diffusion model Diffusion, the Spatial Transformer network in the unet network can be replaced by the Chunk Transformer network. Figure 2 In the example, the first downsampling layer, the second downsampling layer, the feature weight adjustment layer, and the upsampling layer can all be Chunk Transformer block transformation networks. The Chunk Transformer block transformation network can cut the input features into feature segments of fixed length and perform feature processing based on the feature segments, thereby removing the restriction on the length of the input features, so that the music processing model can process the first music data of any length and ensure that the length of the second music data is the same as the length of the first music data.

[0082] refer to Figure 2 The first downsampling layer can cut the audio features into feature segments of fixed length and perform feature processing based on the feature segments. The second downsampling layer can cut the noise features into feature segments of fixed length and perform feature processing based on the feature segments. The feature weight adjustment layer can cut the first features into feature segments of fixed length and perform feature processing based on the feature segments. The upsampling layer can cut the processed first features into feature segments of fixed length and perform feature processing based on the feature segments.

[0083] The above describes the detailed process of processing the first music data based on the music processing model. The following describes the training process of the music processing model.

[0084] Figure 3 A flowchart of a training music processing model provided by an embodiment of the present disclosure is shown in FIG. Figure 3 As shown, the process includes:

[0085] Step S302, acquiring sample music data, sample processing instructions in text form associated with the sample music data, and target music data corresponding to the sample music data;

[0086] Step S304, extracting sample chord progression features and sample audio features of the sample music data, and sample text features of the sample processing instructions through the pre-built neural network structure;

[0087] Step S306, training a neural network structure based on the sample chord progression features, the sample audio features, the sample text features and the target music data, and the trained neural network structure is a music processing model.

[0088] Figure 3 In the above step S302, sample music data, sample processing instructions in text form associated with the sample music data, and target music data corresponding to the sample music data are obtained. The target music data is the music data expected to be obtained after processing the sample music data. The sample music data, the sample processing instructions in text form associated with the sample music data, and the target music data can all be obtained in an artificially constructed form.

[0089] In the above step S304, the sample chord progression features and sample audio features of the sample music data are extracted through the pre-built neural network structure, and the sample text features of the sample processing instructions are extracted. Figure 4 A schematic diagram of the training principle of a music processing model provided in an embodiment of the present disclosure is shown in FIG. Figure 4 As shown in FIG, since the emotional style guidance unit can be obtained by pre-training, there is no need to train the emotional style guidance unit in the process of training the music processing model. Figure 2 , Figure 4 There is no emotional style guidance unit.

[0090] refer to Figure 4 ,and Figure 2 The process in is similar to that in which the sample audio features of the sample music data can be extracted by the audio feature extraction unit to be trained, the sample chord progression features of the sample music data can be extracted by the chord progression feature extraction unit to be trained, and the sample text features of the sample processing instructions can be extracted by the text feature extraction unit to be trained. The sample chord progression features are used to represent the chord type information of the sample music data at different time points, and the sample audio features are used to represent the pitch information, music duration information, rhythm information, audio frequency information, etc. of the sample music data. The sample text features at least include the semantic information of the sample processing instructions.

[0091] In the above step S306, a neural network structure is trained based on sample chord progression features, sample audio features, sample text features and target music data, and the trained neural network structure is a music processing model.

[0092] It can be seen that through this embodiment, a music processing model can be efficiently and quickly trained based on sample music data, sample processing instructions in text form associated with the sample music data, and target music data corresponding to the sample music data. Since the sample chord progression features of the sample music data are used when training the music processing model, the music processing model can have good performance in maintaining the consistency of the chord progression features, so that the music data before and after processing has a high degree of harmony and consistency in terms of musicality.

[0093] In one embodiment, training a neural network structure based on sample chord progression features, sample audio features, sample text features and target music data includes:

[0094] Through the neural network structure, sample random noise data is superimposed on the target music data to obtain the target noise data;

[0095] Extract target noise features of target noise data through a neural network structure;

[0096] The neural network structure is trained based on sample chord progression features, sample audio features, sample text features and target noise features.

[0097] like Figure 4 As shown, the music processing model in training can randomly generate noise data for the sample music data, and the randomly generated noise data is the sample random noise data. The audio feature extraction unit in the music processing model can also superimpose the target music data and the sample random noise data to obtain the target noise data, and extract the target noise feature of the target noise data. Then, based on the sample chord progression feature, the sample audio feature, the sample text feature and the target noise feature, the neural network structure is trained.

[0098] It can be seen that through this embodiment, the target music data can be superimposed with sample random noise data to obtain target noise data, and the target noise features of the target noise data can be extracted. Based on the sample chord progression features, sample audio features, sample text features and target noise features, the neural network structure can be trained efficiently and quickly.

[0099] In one embodiment, a neural network structure is trained based on sample chord progression features, sample audio features, sample text features, and target noise features, including:

[0100] Through the neural network structure, the sample audio feature and the target noise feature are subjected to feature compression processing according to the sample text feature to obtain a second feature; the second feature is used to represent information related to the sample processing instruction in the sample music data and information related to the sample processing instruction in the target noise data;

[0101] Through the neural network structure, according to the sample chord progression feature and the sample text feature, the second feature is subjected to feature weight adjustment processing and feature restoration processing to generate processed sample music data; wherein the feature value in the second feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration; the feature weight adjustment processing is used to adjust the feature weight;

[0102] Based on the processed sample music data and target music data, the neural network structure is trained.

[0103] In this embodiment, the second feature is obtained by performing feature compression processing on the sample audio feature and the target noise feature according to the sample text feature through the music processing model in training, that is, the neural network structure. Similar to the first feature, the second feature can be data in the form of a matrix, which is a more abstract and more concentrated effective feature extracted from the sample audio feature of the sample music data and the target noise feature of the target noise data. The second feature is used to represent the information related to the sample processing instruction in the sample music data and the information related to the sample processing instruction in the target noise data, and the related information is the effective information of the sample music data and the target noise data.

[0104] Next, through the neural network structure, according to the sample chord progression feature and the sample text feature, the second feature is subjected to feature weight adjustment processing and feature restoration processing to generate processed sample music data, wherein the feature value in the second feature has a feature weight, the feature weight is used to indicate the importance of the feature value when performing feature restoration, and the feature weight adjustment processing is used to adjust the feature weight. Based on the processed sample music data and the target music data, the neural network structure is trained.

[0105] It can be seen that through this embodiment, since the second feature is subjected to feature weight adjustment processing and feature restoration processing based on the sample chord progression feature and the sample text feature, processed sample music data is generated, and the neural network structure is trained based on the processed sample music data and the target music data. Therefore, when the trained model processes the music data, the similarity between the chord progression features of the music data before and after processing is greater than the similarity threshold, thereby generating music data that meets user needs.

[0106] In one embodiment, a neural network structure is used to perform feature compression processing on sample audio features and target noise features according to sample text features to obtain a second feature, including:

[0107] The sample audio features are downsampled through a neural network structure, and the target noise features and the sample audio features after downsampling are downsampled according to the sample text features to obtain a second feature.

[0108] In this embodiment, through a neural network structure, the sample audio features of the sample music data are first downsampled, and then, according to the sample text features, the target noise features of the target noise data and the sample audio features after downsampling are downsampled again to obtain a second feature.

[0109] It can be seen that in this embodiment, by downsampling the sample audio features of the sample music data multiple times, the sample audio features of the sample music data can be strengthened when training the music processing model, so that the information related to the sample processing instructions in the sample music data represented by the second feature is more accurate, so that the trained music processing model can generate music data that meets user needs.

[0110] like Figure 4 As shown, the pre-built neural network structure includes an intermediate processing module, the intermediate processing module includes a first downsampling unit and a second downsampling unit; the first downsampling unit includes a plurality of first downsampling layers connected in sequence; the second downsampling unit includes a plurality of second downsampling layers connected in sequence; based on this, the sample audio feature is downsampled through the neural network structure, and the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature to obtain the second feature, including:

[0111] Downsampling the sample audio features through each first downsampling layer; wherein the input of the first downsampling layer includes the sample audio features; the input of the n+1th first downsampling layer includes the output of the nth first downsampling layer; n is an integer greater than or equal to 1 and less than or equal to T-1; T is the number of first downsampling layers;

[0112] Through each second downsampling layer, the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature; wherein the input of the first second downsampling layer includes the target noise feature, the output of the first first downsampling layer and the sample text feature; the input of the m+1th second downsampling layer includes the output of the mth first downsampling layer, the output of the n+1th first downsampling layer and the sample text feature; m is an integer greater than or equal to 1 and less than or equal to S-1; S is the number of second downsampling layers; the feature output by the last second downsampling layer is the second feature.

[0113] refer to Figure 4In this embodiment, the neural network includes an intermediate processing module, the intermediate processing module includes a first downsampling unit and a second downsampling unit, the first downsampling unit includes a plurality of first downsampling layers connected in sequence, and the figure takes three first downsampling layers as an example for illustration. For the first downsampling layer, the input is the sample audio features of the sample music data, for each first downsampling layer after the first downsampling layer, the input is the output of the previous first downsampling layer, and each first downsampling layer is used to downsample the input features.

[0114] refer to Figure 4 In this embodiment, the second downsampling unit includes a plurality of second downsampling layers connected in sequence, and three second downsampling layers are used as an example in the figure. For the first second downsampling layer, the input includes the sample text features of the sample processing instruction, the target noise features of the target noise data, and the output of the first first downsampling layer. For each second downsampling layer after the first second downsampling layer, the input includes the sample text features of the sample processing instruction, the output of the previous second downsampling layer, and the output of the first downsampling layer of the same number of layers. For example, for the second second downsampling layer, the input includes the sample text features of the sample processing instruction, the output of the first second downsampling layer, and the output of the second first downsampling layer. Each second downsampling layer is used to downsample the input features.

[0115] from Figure 4 It can be seen that the number of the first downsampling layers is the same as the number of the second downsampling layers. For the last second downsampling layer, the input includes the sample text features of the sample processing instructions, the output of the previous second downsampling layer and the output of the last first downsampling layer. The feature of the output of the last second downsampling layer is the above-mentioned second feature.

[0116] It can be seen that in this embodiment, the sample audio features are first downsampled through the first downsampling layer, and then the target noise features and the sample audio features after downsampling are downsampled again according to the sample text features through the second downsampling layer, so that the neural network structure strengthens the learning of the audio features of the first music data, and makes the information related to the sample processing instructions in the sample music data represented by the second features more accurate.

[0117] like Figure 4 As shown, in one embodiment, the intermediate processing module of the neural network structure further includes a feature weight adjustment unit and a feature restoration unit; based on this, the neural network structure is used to perform feature weight adjustment processing and feature restoration processing on the second feature according to the sample chord progression feature and the sample text feature, and the processed sample music data is generated, including:

[0118] By means of a feature weight adjustment unit, the feature weight of each feature value in the second feature is adjusted according to the sample chord progression feature, so as to obtain a second feature after the feature weight is adjusted;

[0119] The feature restoration unit performs feature restoration on the second feature after feature weight adjustment according to the sample text feature, thereby generating processed sample music data.

[0120] like Figure 4 As shown, in this embodiment, the intermediate processing module of the neural network structure also includes a feature weight adjustment unit and a feature restoration unit. The input of the feature weight adjustment unit includes the sample chord progression feature and the second feature of the sample music data. The feature weight adjustment unit adjusts the feature weight of each feature value in the second feature according to the sample chord progression feature to obtain the second feature after the feature weight adjustment. Among them, the feature value in the second feature is used to represent the information related to the sample processing instruction in the sample music data and the information related to the sample processing instruction in the target noise data. Each feature value has its own feature weight, which can represent the importance of the feature value when performing feature restoration. Adjusting the feature weight of each feature value means adjusting the importance of each feature value in the subsequent feature restoration, so as to perform feature restoration based on the second feature after the feature weight adjustment to obtain the processed sample music data.

[0121] Furthermore, if Figure 4 As shown, the second feature after the feature weight is adjusted is also input into the feature restoration unit, and the feature restoration unit performs feature restoration on the second feature after the feature weight is adjusted according to the sample text feature to generate processed sample music data.

[0122] It can be seen that through this embodiment, first, the feature weight adjustment unit is used to adjust the feature weights of each feature value in the second feature according to the sample chord progression feature to obtain the second feature after the feature weight adjustment. Then, the feature restoration unit is used to restore the second feature after the feature weight adjustment according to the sample text feature to generate processed sample music data. By adjusting the feature weights of the feature values ​​of the second feature, after feature restoration based on the second feature after the feature weight adjustment, processed sample music data that matches the sample processing instruction and whose chord progression feature meets the requirements can be obtained, thereby improving the accuracy of generating the processed sample music data and the accuracy of model training.

[0123] like Figure 4As shown, the feature weight adjustment unit includes a feature weight adjustment layer based on the attention mechanism; based on this, the feature weight adjustment unit adjusts the feature weights of each feature value in the second feature according to the sample chord progression feature to obtain the second feature after the feature weight adjustment, including:

[0124] Through the feature weight adjustment layer, according to the sample chord progression features, based on the attention mechanism, the feature weights of each feature value in the second feature are adjusted to obtain the second feature after the feature weight adjustment.

[0125] In this embodiment, the feature weight adjustment layer can be used to adjust the feature weights of each feature value in the second feature based on the cross attention mechanism according to the sample chord progression features to obtain the second feature after the feature weight adjustment.

[0126] It should be noted that in Figure 4 In the network structure shown, the first downsampling layer, the second downsampling layer, the feature weight adjustment layer in the feature weight adjustment unit, and the upsampling layer in the feature restoration unit all adopt a processing strategy based on the cross-attention mechanism. The difference is that in the second downsampling layer and the upsampling layer in the feature restoration unit, the cross-attention mechanism can only be used for feature processing according to the sample text features of the sample processing instructions, and in the feature weight adjustment layer, the cross-attention mechanism can be used for feature processing according to the sample chord progression features, so that the similarity between the generated processed chord progression features of the sample music data and the sample chord progression features of the sample music data is greater than the similarity threshold, thereby improving the accuracy of model training.

[0127] It can be seen that through this embodiment, the feature weight adjustment layer can be used to efficiently and quickly adjust the feature weights of each feature value in the second feature based on the attention mechanism according to the sample chord progression features, thereby improving the efficiency of obtaining the second feature after the feature weight adjustment.

[0128] like Figure 4 As shown, the feature restoration unit includes a plurality of upsampling layers connected in sequence; based on this, the feature restoration unit performs feature restoration on the second feature after the feature weight is adjusted according to the sample text feature to generate processed sample music data, including:

[0129] Through each upsampling layer, the second feature after the feature weight is adjusted is upsampled according to the sample text feature; wherein the input of the first upsampling layer includes the second feature after the feature weight is adjusted and the sample text feature, and the input of the p+1th upsampling layer includes the output of the pth upsampling layer and the sample text feature; p is an integer greater than or equal to 1 and less than or equal to W-1; W is the number of upsampling layers;

[0130] The features output by the last upsampling layer are decoded to obtain the processed sample music data.

[0131] refer to Figure 4 , the feature restoration unit includes multiple upsampling layers connected in sequence, Figure 4 In the figure, three upsampling layers are used as an example. For the first upsampling layer, the input includes the sample text feature and the second feature of the sample processing instruction. For each upsampling layer after the first upsampling layer, the input includes the output of the previous upsampling layer and the sample text feature. Figure 4 As shown, the number of the second downsampling layer and the upsampling layer is the same, and the second downsampling layer and the upsampling layer are also residually connected, and the output of each second downsampling layer is also used as the input of the corresponding upsampling layer. Each upsampling layer is used to upsample the input features.

[0132] In this embodiment, the features output by the last upsampling layer are also decoded to obtain processed sample music data.

[0133] It can be seen that in this embodiment, the second feature after the feature weight is adjusted is upsampled according to the sample text feature through each upsampling layer, and the feature output by the last upsampling layer is decoded to obtain the processed sample music data. Since the second feature after the feature weight is adjusted is upsampled according to the sample text feature of the sample processing instruction, the generated processed sample music data can match the sample processing instruction.

[0134] In this embodiment, after obtaining the processed sample music data, the neural network structure can be trained based on the difference between the processed sample music data and the target music data, or the model can be trained based on the principle of the diffusion model by predicting the sample random noise data added during the model training process. The specific training method is not limited here.

[0135] It should be noted that the application process of the music processing model is quite similar to the training process, so the training process can also be explained with reference to the aforementioned application process.

[0136] In summary, through the above-mentioned music processing model, the music data can be processed and the chord progression characteristics of the music data before and after the processing can be made more consistent, so that the music data before and after the processing has higher harmony and consistency in musicality.

[0137] An embodiment of the present disclosure further provides an audio processing device, which is used to implement the above-mentioned audio processing method. Figure 5 A schematic diagram of the structure of an audio processing device provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, the device comprises:

[0138] An acquisition unit 51, configured to acquire first music data and a processing instruction in text form associated with the first music data;

[0139] An extraction unit 52, configured to extract the first chord progression feature and the audio feature of the first music data and the text feature of the processing instruction through a music processing model;

[0140] A generation unit 53 is used to process the audio feature according to the first chord progression feature and the text feature through the music processing model to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

[0141] Optionally, the generating unit 53 is specifically configured to:

[0142] The music processing model performs feature compression processing on the audio feature and the noise feature of the random noise data according to the text feature to obtain a first feature; wherein the first feature is used to represent information related to the processing instruction in the first music data and information related to the processing instruction in the random noise data; the random noise data is noise data randomly generated by the music processing model for the first music data;

[0143] Through the music processing model, according to the first chord progression feature and the text feature, the first feature is subjected to feature weight adjustment processing and feature restoration processing to generate second music data; wherein the feature value in the first feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration; the feature weight adjustment processing is used to adjust the feature weight.

[0144] Optionally, the generating unit 53 is further specifically configured to:

[0145] The audio feature is downsampled by the music processing model, and the noise feature and the downsampled audio feature are downsampled according to the text feature to obtain the first feature.

[0146] Optionally, the music processing model includes a first downsampling unit and a second downsampling unit; the first downsampling unit includes a plurality of first downsampling layers connected in sequence; the second downsampling unit includes a plurality of second downsampling layers connected in sequence; the generating unit 53 is further specifically used for:

[0147] Downsampling the audio features through each first downsampling layer; wherein the input of the first downsampling layer includes the audio features; the input of the n+1th first downsampling layer includes the output of the nth first downsampling layer; n is an integer greater than or equal to 1 and less than or equal to T-1; T is the number of first downsampling layers;

[0148] Through each second downsampling layer, the noise feature and the audio feature after downsampling are downsampled according to the text feature; wherein the input of the first second downsampling layer includes the noise feature, the output of the first first downsampling layer and the text feature; the input of the m+1th second downsampling layer includes the output of the mth first downsampling layer, the output of the n+1th first downsampling layer and the text feature; m is an integer greater than or equal to 1 and less than or equal to S-1; S is the number of second downsampling layers; and the feature of the output of the last second downsampling layer is the first feature.

[0149] Optionally, the music processing model includes a feature weight adjustment unit and a feature restoration unit; the generation unit 53 is further specifically used for:

[0150] By means of the feature weight adjustment unit, according to the first chord progression feature, the feature weight of each feature value in the first feature is adjusted to obtain the first feature after the feature weight is adjusted;

[0151] The feature restoration unit performs feature restoration on the first feature after feature weight adjustment according to the text feature to generate second music data.

[0152] Optionally, the feature weight adjustment unit includes a feature weight adjustment layer based on an attention mechanism; the generating unit 53 is further specifically used for:

[0153] Through the feature weight adjustment layer, according to the first chord progression feature, the feature weight of each feature value in the first feature is adjusted based on the attention mechanism to obtain the first feature after the feature weight is adjusted.

[0154] Optionally, the feature restoration unit includes a plurality of upsampling layers connected in sequence; the generating unit 53 is further specifically configured to:

[0155] Through each upsampling layer, the first feature after the feature weight is adjusted is upsampled according to the text feature; wherein the input of the first upsampling layer includes the first feature after the feature weight is adjusted and the text feature; the input of the p+1th upsampling layer includes the output of the pth upsampling layer and the text feature; p is an integer greater than or equal to 1 and less than or equal to W-1; W is the number of upsampling layers;

[0156] The features output by the last upsampling layer are decoded to obtain the second music data.

[0157] Optionally, the music processing model includes an emotion style guiding unit and a decoding unit; the generating unit 53 is further specifically used for:

[0158] By means of the emotion style guidance module, the music emotion and / or music style indicated by the text feature is identified as decoding guidance information;

[0159] The emotional style guidance module guides the decoding unit to decode the features output by the last upsampling layer according to the decoding guidance information, so as to obtain the second music data matching the decoding guidance information.

[0160] Optionally, a model training unit is further included, for:

[0161] Acquire sample music data, sample processing instructions in text form associated with the sample music data, and target music data corresponding to the sample music data;

[0162] Extracting sample chord progression features and sample audio features of the sample music data and sample text features of the sample processing instructions through a pre-built neural network structure;

[0163] Based on the sample chord progression features, the sample audio features, the sample text features and the target music data, the neural network structure is trained, and the trained neural network structure is the music processing model.

[0164] Optionally, the model training unit is specifically used for:

[0165] Through the neural network structure, the sample random noise data is superimposed on the target music data to obtain target noise data;

[0166] Extracting target noise features of the target noise data through the neural network structure;

[0167] The neural network structure is trained based on the sample chord progression features, the sample audio features, the sample text features and the target noise features.

[0168] Optionally, the model training unit is further specifically used for:

[0169] By means of the neural network structure, the sample audio feature and the target noise feature are subjected to feature compression processing according to the sample text feature to obtain a second feature; the second feature is used to represent information related to the sample processing instruction in the sample music data and information related to the sample processing instruction in the target noise data;

[0170] Through the neural network structure, according to the sample chord progression feature and the sample text feature, the second feature is subjected to feature weight adjustment processing and feature restoration processing to generate processed sample music data; wherein the feature value in the second feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration; the feature weight adjustment processing is used to adjust the feature weight;

[0171] The neural network structure is trained based on the processed sample music data and the target music data.

[0172] Optionally, the model training unit is further specifically used for:

[0173] The sample audio feature is downsampled by the neural network structure, and the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature to obtain the second feature.

[0174] Optionally, the neural network structure includes a first downsampling unit and a second downsampling unit; the first downsampling unit includes a plurality of first downsampling layers connected in sequence; the second downsampling unit includes a plurality of second downsampling layers connected in sequence; and the model training unit is further specifically used for:

[0175] Downsampling the sample audio features through each first downsampling layer; wherein the input of the first downsampling layer includes the sample audio features; the input of the n+1th first downsampling layer includes the output of the nth first downsampling layer; n is an integer greater than or equal to 1 and less than or equal to T-1; T is the number of first downsampling layers;

[0176] Through each second downsampling layer, the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature; wherein the input of the first second downsampling layer includes the target noise feature, the output of the first first downsampling layer and the sample text feature; the input of the m+1th second downsampling layer includes the output of the mth first downsampling layer, the output of the n+1th first downsampling layer and the sample text feature; m is an integer greater than or equal to 1 and less than or equal to S-1; S is the number of second downsampling layers; the feature of the output of the last second downsampling layer is the second feature.

[0177] Optionally, the neural network structure includes a feature weight adjustment unit and a feature restoration unit; the model training unit is further specifically used for:

[0178] By means of the feature weight adjustment unit, the feature weight of each feature value in the second feature is adjusted according to the sample chord progression feature to obtain the second feature after the feature weight is adjusted;

[0179] The feature restoration unit performs feature restoration on the second feature after feature weight adjustment according to the sample text feature to generate processed sample music data.

[0180] Optionally, the feature weight adjustment unit includes a feature weight adjustment layer based on an attention mechanism; and the model training unit is further specifically used for:

[0181] Through the feature weight adjustment layer, the feature weight of each feature value in the second feature is adjusted based on the attention mechanism according to the sample chord progression feature, so as to obtain the second feature after the feature weight is adjusted.

[0182] Optionally, the feature restoration unit includes a plurality of upsampling layers connected in sequence; and the model training unit is further specifically used for:

[0183] Through each upsampling layer, the second feature after the feature weight is adjusted is upsampled according to the sample text feature; wherein the input of the first upsampling layer includes the second feature after the feature weight is adjusted and the sample text feature; the input of the p+1th upsampling layer includes the output of the pth upsampling layer and the sample text feature; p is an integer greater than or equal to 1 and less than or equal to W-1; W is the number of upsampling layers;

[0184] The features output by the last upsampling layer are decoded to obtain the processed sample music data.

[0185] The audio processing device in the embodiment of the present disclosure can implement the various processes of the above-mentioned audio processing method embodiment and achieve the same effects and functions, which will not be repeated here.

[0186] An embodiment of the present disclosure further provides an electronic device, Figure 6 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure, such as Figure 6As shown, the electronic device may have relatively large differences due to different configurations or performances, and may include one or more processors 601 and memory 602, and one or more applications or data may be stored in the memory 602. Among them, the memory 602 can be a short-term storage or a persistent storage. The application stored in the memory 602 may include one or more modules (not shown in the figure), and each module may include a series of computer executable instructions in the electronic device. Furthermore, the processor 801 can be configured to communicate with the memory 602 and execute a series of computer executable instructions in the memory 602 on the electronic device. The electronic device may also include one or more power supplies 603, one or more wired or wireless network interfaces 604, one or more input or output interfaces 605, one or more keyboards 606, etc.

[0187] In a specific embodiment, the electronic device includes a processor; and a memory configured to store computer executable instructions, wherein when the computer executable instructions are executed, the processor implements the following process:

[0188] acquiring first music data and a processing instruction in text form associated with the first music data;

[0189] Extracting the first chord progression feature and the audio feature of the first music data and the text feature of the processing instruction through the music processing model;

[0190] Through the music processing model, the audio feature is processed according to the first chord progression feature and the text feature to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

[0191] The electronic device in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0192] Another embodiment of the present disclosure further provides a computer-readable storage medium, wherein the computer-readable storage medium is used to store computer-executable instructions, and when the computer-executable instructions are executed by a processor, the following process is implemented:

[0193] acquiring first music data and a processing instruction in text form associated with the first music data;

[0194] Extracting the first chord progression feature and the audio feature of the first music data and the text feature of the processing instruction through the music processing model;

[0195] Through the music processing model, the audio feature is processed according to the first chord progression feature and the text feature to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

[0196] The storage medium in the embodiment of the present disclosure can implement each process of the above-mentioned audio processing method embodiment and achieve the same effect and function, which will not be repeated here.

[0197] In various embodiments of the present disclosure, the computer-readable storage medium includes a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0198] In the 1990s, improvements to a technology could be clearly distinguished as hardware improvements (for example, improvements to the circuit structure of diodes, transistors, switches, etc.) or software improvements (improvements to the method flow). However, with the development of technology, many improvements to the method flow today can be regarded as direct improvements to the hardware circuit structure. Designers almost always obtain the corresponding hardware circuit structure by programming the improved method flow into the hardware circuit. Therefore, it cannot be said that an improvement in a method flow cannot be implemented using a hardware entity module. For example, a programmable logic device (PLD) (such as a field programmable gate array (FPGA)) is such an integrated circuit whose logical function is determined by the user's programming of the device. Designers can "integrate" a digital system on a PLD by programming it themselves, without having to ask a chip manufacturer to design and produce a dedicated integrated circuit chip. Moreover, nowadays, instead of manually making integrated circuit chips, this kind of programming is mostly implemented by "logic compiler" software, which is similar to the software compiler used when developing and writing programs, and the original code before compilation must also be written in a specific programming language, which is called hardware description language (HDL). There is not only one HDL, but many kinds, such as ABEL (Advanced Boolean Expression Language), AHDL (Altera Hardware Description Language), Confluence, CUPL (Cornell University Programming Language), HDCal, JHDL (Java Hardware Description Language), Lava, Lola, MyHDL, PALASM, RHDL (Ruby Hardware Description Language), etc. The most commonly used ones are VHDL (Very-High-Speed ​​Integrated Circuit Hardware Description Language) and Verilog. Those skilled in the art should also know that it is only necessary to program the method flow slightly in the above-mentioned hardware description languages ​​and program it into the integrated circuit, and then it is easy to obtain the hardware circuit that implements the logic method flow.

[0199] The controller can be implemented in any appropriate manner, for example, the controller can take the form of a microprocessor or processor and a computer-readable medium storing a computer-readable program code (such as software or firmware) that can be executed by the (micro)processor, a logic gate, a switch, an application-specific integrated circuit (ASIC), a programmable logic controller, and an embedded microcontroller. Examples of controllers include, but are not limited to, the following microcontrollers: ARC 625D, Atmel AT91SAM, Microchip PIC18F26K20, and Silicone Labs C8051F320. The memory controller can also be implemented as part of the control logic of the memory. Those skilled in the art also know that in addition to implementing the controller in a purely computer-readable program code manner, the controller can be implemented in the form of a logic gate, a switch, an application-specific integrated circuit, a programmable logic controller, and an embedded microcontroller by logically programming the method steps. Therefore, this controller can be considered as a hardware component, and the devices included therein for implementing various functions can also be regarded as structures within the hardware component. Or even, the devices for implementing various functions can be regarded as both software modules for implementing the method and structures within the hardware component.

[0200] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer. Specifically, the computer may be, for example, a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or a combination of any of these devices.

[0201] For the convenience of description, the above devices are described in terms of functions and are divided into various units. Of course, when implementing the embodiments of the present disclosure, the functions of each unit can be implemented in the same or multiple software and / or hardware.

[0202] Those skilled in the art will appreciate that one or more embodiments of the present disclosure may be provided as a method, system or computer program product. Therefore, one or more embodiments of the present disclosure may take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, one or more embodiments of the present disclosure may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0203] The present disclosure is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present disclosure. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0204] These computer program instructions may also be stored in a computer readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture including an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0205] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process in the computer or other programmable device. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0206] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0207] One or more embodiments of the present disclosure may be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. One or more embodiments of the present disclosure may also be practiced in distributed computing environments in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules may be located in local and remote computer storage media, including storage devices.

[0208] Each embodiment in the present disclosure is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the system embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0209] The above description is only an embodiment of the present disclosure and is not intended to limit the present disclosure. For those skilled in the art, the present disclosure may have various modifications and variations. Any modification, equivalent substitution, improvement, etc. made within the spirit and principle of the present disclosure shall be included in the scope of the claims of the present disclosure.

Claims

1. An audio processing method, characterized in that: include: acquiring first music data and a processing instruction in text form associated with the first music data; Extracting the first chord progression feature and the audio feature of the first music data and the text feature of the processing instruction through the music processing model; Through the music processing model, the audio feature is processed according to the first chord progression feature and the text feature to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

2. The method according to claim 1, characterized in that The step of processing the audio feature according to the first chord progression feature and the text feature through the music processing model to generate second music data includes: The music processing model performs feature compression processing on the audio feature and the noise feature of the random noise data according to the text feature to obtain a first feature; wherein the first feature is used to represent information related to the processing instruction in the first music data and information related to the processing instruction in the random noise data; the random noise data is noise data randomly generated by the music processing model for the first music data; Through the music processing model, according to the first chord progression feature and the text feature, the first feature is subjected to feature weight adjustment processing and feature restoration processing to generate second music data; wherein the feature value in the first feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration; the feature weight adjustment processing is used to adjust the feature weight.

3. The method according to claim 2, characterized in that The step of performing feature compression processing on the audio feature and the noise feature of the random noise data according to the text feature through the music processing model to obtain the first feature includes: The audio feature is downsampled by the music processing model, and the noise feature and the downsampled audio feature are downsampled according to the text feature to obtain the first feature.

4. The method according to claim 3, characterized in that The music processing model includes a first downsampling unit and a second downsampling unit; the first downsampling unit includes a plurality of first downsampling layers connected in sequence; the second downsampling unit includes a plurality of second downsampling layers connected in sequence; the audio feature is downsampled by the music processing model, and the noise feature and the downsampled audio feature are downsampled according to the text feature to obtain the first feature, including: Downsampling the audio features through each first downsampling layer; wherein the input of the first downsampling layer includes the audio features; the input of the n+1th first downsampling layer includes the output of the nth first downsampling layer; n is an integer greater than or equal to 1 and less than or equal to T-1; T is the number of first downsampling layers; Through each second downsampling layer, the noise feature and the audio feature after downsampling are downsampled according to the text feature; wherein the input of the first second downsampling layer includes the noise feature, the output of the first first downsampling layer and the text feature; the input of the m+1th second downsampling layer includes the output of the mth first downsampling layer, the output of the n+1th first downsampling layer and the text feature; m is an integer greater than or equal to 1 and less than or equal to S-1; S is the number of second downsampling layers; and the feature of the output of the last second downsampling layer is the first feature.

5. The method according to claim 2, characterized in that: The music processing model includes a feature weight adjustment unit and a feature restoration unit; the music processing model performs feature weight adjustment processing and feature restoration processing on the first feature according to the first chord progression feature and the text feature to generate second music data, including: By means of the feature weight adjustment unit, according to the first chord progression feature, the feature weight of each feature value in the first feature is adjusted to obtain the first feature after the feature weight is adjusted; The feature restoration unit performs feature restoration on the first feature after feature weight adjustment according to the text feature to generate second music data.

6. The method according to claim 5, characterized in that The feature weight adjustment unit includes a feature weight adjustment layer based on an attention mechanism; the feature weight adjustment unit adjusts the feature weight of each feature value in the first feature according to the first chord progression feature to obtain the first feature after the feature weight is adjusted, including: Through the feature weight adjustment layer, according to the first chord progression feature, the feature weight of each feature value in the first feature is adjusted based on the attention mechanism to obtain the first feature after the feature weight is adjusted.

7. The method according to claim 5, characterized in that The feature restoration unit includes a plurality of upsampling layers connected in sequence; the feature restoration unit performs feature restoration on the first feature after feature weight adjustment according to the text feature to generate second music data, including: Through each upsampling layer, the first feature after the feature weight is adjusted is upsampled according to the text feature; wherein the input of the first upsampling layer includes the first feature after the feature weight is adjusted and the text feature; the input of the p+1th upsampling layer includes the output of the pth upsampling layer and the text feature; p is an integer greater than or equal to 1 and less than or equal to W-1; W is the number of upsampling layers; The features output by the last upsampling layer are decoded to obtain the second music data.

8. The method according to claim 7, characterized in that The music processing model includes an emotion style guiding unit and a decoding unit; the decoding of the features output by the last upsampling layer to obtain the second music data includes: By means of the emotion style guidance module, the music emotion and / or music style indicated by the text feature is identified as decoding guidance information; The emotional style guidance module guides the decoding unit to decode the features output by the last upsampling layer according to the decoding guidance information, so as to obtain the second music data matching the decoding guidance information.

9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: Acquire sample music data, sample processing instructions in text form associated with the sample music data, and target music data corresponding to the sample music data; Extracting sample chord progression features and sample audio features of the sample music data and sample text features of the sample processing instructions through a pre-built neural network structure; Based on the sample chord progression features, the sample audio features, the sample text features and the target music data, the neural network structure is trained, and the trained neural network structure is the music processing model.

10. The method according to claim 9, characterized in that The step of training the neural network structure based on the sample chord progression feature, the sample audio feature, the sample text feature and the target music data includes: Through the neural network structure, the sample random noise data is superimposed on the target music data to obtain target noise data; Extracting target noise features of the target noise data through the neural network structure; The neural network structure is trained based on the sample chord progression features, the sample audio features, the sample text features and the target noise features.

11. The method according to claim 10, characterized in that The training of the neural network structure based on the sample chord progression feature, the sample audio feature, the sample text feature and the target noise feature includes: By means of the neural network structure, the sample audio feature and the target noise feature are subjected to feature compression processing according to the sample text feature to obtain a second feature; the second feature is used to represent information related to the sample processing instruction in the sample music data and information related to the sample processing instruction in the target noise data; Through the neural network structure, according to the sample chord progression feature and the sample text feature, the second feature is subjected to feature weight adjustment processing and feature restoration processing to generate processed sample music data; wherein the feature value in the second feature has a feature weight, and the feature weight is used to indicate the importance of the feature value when performing feature restoration; the feature weight adjustment processing is used to adjust the feature weight; The neural network structure is trained based on the processed sample music data and the target music data.

12. The method according to claim 11, characterized in that The step of performing feature compression processing on the sample audio feature and the target noise feature according to the sample text feature through the neural network structure to obtain a second feature includes: The sample audio feature is downsampled by the neural network structure, and the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature to obtain the second feature.

13. The method according to claim 12, characterized in that The neural network structure includes a first downsampling unit and a second downsampling unit; the first downsampling unit includes a plurality of first downsampling layers connected in sequence; the second downsampling unit includes a plurality of second downsampling layers connected in sequence; the neural network structure is used to downsample the sample audio feature, and the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature to obtain the second feature, including: Downsampling the sample audio features through each first downsampling layer; wherein the input of the first downsampling layer includes the sample audio features; the input of the n+1th first downsampling layer includes the output of the nth first downsampling layer; n is an integer greater than or equal to 1 and less than or equal to T-1; T is the number of first downsampling layers; Through each second downsampling layer, the target noise feature and the sample audio feature after downsampling are downsampled according to the sample text feature; wherein the input of the first second downsampling layer includes the target noise feature, the output of the first first downsampling layer and the sample text feature; the input of the m+1th second downsampling layer includes the output of the mth first downsampling layer, the output of the n+1th first downsampling layer and the sample text feature; m is an integer greater than or equal to 1 and less than or equal to S-1; S is the number of second downsampling layers; the feature of the output of the last second downsampling layer is the second feature.

14. The method according to claim 11, characterized in that The neural network structure includes a feature weight adjustment unit and a feature restoration unit; the neural network structure performs feature weight adjustment processing and feature restoration processing on the second feature according to the sample chord progression feature and the sample text feature to generate processed sample music data, including: By means of the feature weight adjustment unit, the feature weight of each feature value in the second feature is adjusted according to the sample chord progression feature to obtain the second feature after the feature weight is adjusted; The feature restoration unit performs feature restoration on the second feature after feature weight adjustment according to the sample text feature to generate processed sample music data.

15. The method according to claim 14, characterized in that The feature weight adjustment unit includes a feature weight adjustment layer based on an attention mechanism; the feature weight adjustment unit adjusts the feature weight of each feature value in the second feature according to the sample chord progression feature to obtain the second feature after the feature weight is adjusted, including: Through the feature weight adjustment layer, the feature weight of each feature value in the second feature is adjusted based on the attention mechanism according to the sample chord progression feature, so as to obtain the second feature after the feature weight is adjusted.

16. The method according to claim 14, characterized in that The feature restoration unit includes a plurality of upsampling layers connected in sequence; the feature restoration unit performs feature restoration on the second feature after feature weight adjustment according to the sample text feature to generate processed sample music data, including: Through each upsampling layer, the second feature after the feature weight is adjusted is upsampled according to the sample text feature; wherein the input of the first upsampling layer includes the second feature after the feature weight is adjusted and the sample text feature; the input of the p+1th upsampling layer includes the output of the pth upsampling layer and the sample text feature; p is an integer greater than or equal to 1 and less than or equal to W-1; W is the number of upsampling layers; The features output by the last upsampling layer are decoded to obtain the processed sample music data.

17. An audio processing device, characterized in that: include: an acquisition unit, configured to acquire first music data and a processing instruction in text form associated with the first music data; an extraction unit, configured to extract, by means of a music processing model, a first chord progression feature and an audio feature of the first music data, and a text feature of the processing instruction; A generating unit is used to process the audio feature according to the first chord progression feature and the text feature through the music processing model to generate second music data; the similarity between the first chord progression feature of the first music data and the second chord progression feature of the second music data is greater than a similarity threshold.

18. An electronic device, characterized in that: include: processor; as well as, A memory configured to store computer executable instructions, which, when executed, cause the processor to implement the steps of the method according to any one of claims 1 to 16.

19. A computer-readable storage medium, characterized in that: The computer-readable storage medium is used to store computer-executable instructions, and the computer-executable instructions, when executed by a processor, implement the steps of the method described in any one of claims 1 to 16.

Images