Model generation method, sound effect description generation method, equipment, medium and product

Abstract

The invention discloses a sound mixing sound effect description model generation method, a sound effect description generation method, equipment, a medium and a product in the technical field of computers. A sound mixing sound effect description model used for generating corresponding sound mixing sound effect description information for an input sound mixing audio can be obtained through training. The sound mixing effect description information generated by the model can guide a user to adjust the sound mixing audio, and a listener can judge whether the sound mixing effect of the sound mixing audio is appropriate or not by means of the sound mixing effect description information instead of experience of the listener, so that the listener can master the sound mixing skill more easily and know the advantages and disadvantages of the audio accurately, and the user experience is improved. And the universality and the convenience of music production are improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a method for generating a mixed sound effect description model, a sound effect description generation method, a device, a medium and a product. Background Art

[0002] At present, it is difficult to evaluate the mixing effect of the mixed audio after it is generated by a deep learning model. Listeners can only rely on their own experience to judge whether the mixing effect of the mixed audio is appropriate. Summary of the invention

[0003] In view of this, the purpose of this application is to provide a method, device, medium and product for generating a mixed sound effect description model, which can enable the model to learn the ability to generate mixed sound effect description information, and the mixed sound effect description information output by the obtained mixed sound effect description model can guide the user to adjust the mixed audio. The specific scheme is as follows:

[0004] To achieve the above objectives, on the one hand, the present application provides a method for generating a mixed sound effect description model, comprising:

[0005] Obtaining sample audio, audio question text and sound effect description text; the sample audio is obtained by mixing the original audio, the audio question text is used to inquire about the difference between the sample audio and the original audio in target sound effect parameters, and the sound effect description text is a response to the audio question text;

[0006] Processing the sample audio and the audio question text by the mixed sound effect description model to be trained to obtain a predicted sound effect description text;

[0007] Based on the predicted sound effect description text and the sound effect description text, the parameters of the mixed sound effect description model to be trained are optimized until the convergence condition is reached, so as to obtain a trained mixed sound effect description model, which is used to generate corresponding mixed sound effect description information for the input mixed audio.

[0008] Optionally, the process of generating the sample audio includes:

[0009] At least one of a reverberation parameter, a compression parameter, a delay parameter, and an equalization parameter in a DSP mixing link is adjusted according to the mixing sound effect parameter group to obtain a plurality of DSP mixing links with different mixing parameters; the mixing sound effect parameter group includes: a plurality of groups of parameter vectors composed of at least one of a reverberation parameter, a compression parameter, a delay parameter, and an equalization parameter;

[0010] Using the multiple DSP mixing links to perform mixing processing on multiple target audios, to obtain mixed audios corresponding to the multiple target audios respectively;

[0011] Each mixed audio is used as the sample audio.

[0012] Optionally, the process of generating the sound effect description text includes:

[0013] The mixed sound effect description information of the sample audio input by the user is used as the sound effect description text.

[0014] In another aspect, the present application also provides a method for generating a sound effect description, comprising:

[0015] Obtaining a target mixed audio and a question text corresponding to the target mixed audio;

[0016] The target mixed audio and the question text are input into a mixed sound effect description model so that the mixed sound effect description model generates a mixed sound effect description text for the question text; wherein the mixed sound effect description model is generated according to any of the methods described above; and the mixed sound effect description information is used to guide mixing optimization of the target mixed audio.

[0017] Optionally, the method further comprises:

[0018] Optimizing the mixed sound effect of the target mixed audio according to the mixed sound effect description information to obtain an optimized target mixed audio;

[0019] Return to the step of obtaining the target mixed audio and the question text of the target mixed audio, until the obtained mixed sound effect description information indicates that the target mixed audio does not need to be optimized any more, and the optimal target mixed audio is obtained.

[0020] Optionally, optimizing the mixed sound effect of the target mixed audio according to the mixed sound effect description information includes:

[0021] The DSP mixing link is used to perform at least one of reverberation processing, compression processing, delay processing and equalization processing on the target mixed audio according to the mixed audio effect description information.

[0022] Optionally, the method further comprises:

[0023] The audio question text in the training process of the mixed sound effect description model is used as the question text corresponding to the target mixed audio.

[0024] On the other hand, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the aforementioned corresponding method.

[0025] On the other hand, the present application further provides a storage medium, in which computer executable instructions are stored. When the computer executable instructions are loaded and executed by a processor, the aforementioned corresponding method is implemented.

[0026] On the other hand, the present application also provides a computer program product, including a computer program / instruction, which implements any of the methods described above when executed by a processor.

[0027] Based on sample audio, audio question text and sound effect description text, this application can train a mixed sound effect description model for generating corresponding mixed sound effect description information for input mixed audio. Among them, the sample audio is obtained by mixing the original audio, the audio question text is used to inquire about the difference between the sample audio and the original audio in the target sound effect parameters, and the sound effect description text is a response to the audio question text. The mixed sound effect description information generated by the mixed sound effect description model obtained by this application can guide users to adjust the mixed audio. Listeners can use the mixed sound effect description information instead of their own experience to judge whether the mixing effect of the mixed audio is appropriate, so that listeners can more easily master mixing skills, accurately understand the advantages and disadvantages of audio, and improve the universality and convenience of music production.

[0028] Correspondingly, the sound effect description generation method, mixing method, equipment, medium and product provided in this application also have the above-mentioned technical effects. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0030] Figure 1 A schematic diagram of a physical architecture applicable to the present application provided for the present application;

[0031] Figure 2 A flow chart of a method for generating a mixed sound effect description model provided in this application;

[0032] Figure 3 A flow chart of a method for generating a sound effect description provided in this application;

[0033] Figure 4 A flow chart of another method for generating a mixed sound effect description model provided in this application;

[0034] Figure 5 A flow chart of a mixing method provided for this application;

[0035] Figure 6 A schematic diagram of a mixing process provided for this application;

[0036] Figure 7 A schematic diagram of a model training process provided for this application;

[0037] Figure 8 A server structure diagram provided for this application;

[0038] Fig. 9 A terminal structure diagram provided for this application. DETAILED DESCRIPTION

[0039] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application. In addition, in the embodiments of the present application, "first", "second", etc. are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0040] After generating mixed audio using a deep learning model, it is difficult to evaluate the mixing effect of the mixed audio, and listeners can only rely on their own experience to judge whether the mixing effect of the mixed audio is appropriate.

[0041] In view of the above-mentioned problems currently existing, the present application proposes a mixed sound effect description model generation scheme, which can be trained to obtain a mixed sound effect description model. The mixed sound effect description model can generate corresponding mixed sound effect description information for the input mixed audio. The mixed sound effect description information can guide the user to adjust the mixed audio. The listener can use the mixed sound effect description information rather than his own experience to judge whether the mixing effect of the mixed audio is appropriate, thereby making it easier for the listener to master mixing skills, accurately understand the advantages and disadvantages of the audio, and improve the universality and convenience of music production.

[0042] For ease of understanding, the physical framework to which this application applies is first introduced.

[0043] It should be understood that the method for generating a mixed sound effect description model provided in the present application can be applied to a system or program having a mixed sound effect description model generation function and a mixing function. Specifically, the system or program having a mixed sound effect description model generation function and a mixing function can be run in a server, a personal computer, or other device.

[0044] like Figure 1 As shown, Figure 1 This is a schematic diagram of the physical architecture applicable to this application. Figure 1In the invention, a system or program having a mixed sound effect description model generation function and a mixing function can be run on a server, and the server obtains sample audio, audio question text and sound effect description text from other terminal devices through a network; the sample audio and audio question text are processed by the mixed sound effect description model to be trained to obtain a predicted sound effect description text; based on the predicted sound effect description text and the sound effect description text, the parameters of the mixed sound effect description model to be trained are optimized until the convergence condition is reached, and a trained mixed sound effect description model is obtained, and the mixed sound effect description model is used to generate corresponding mixed sound effect description information for the input mixed audio.

[0045] Depend on Figure 1 It can be seen that the server can establish communication connections with multiple devices, and the server obtains sample audio, audio question text and sound effect description text from these devices. The server trains and obtains a mixed sound effect description model based on this. The server pre-stores a pre-trained language model, and the server converts the pre-trained language model into a mixed sound effect description model based on the sample audio, audio question text and sound effect description text.

[0046] Figure 1 A variety of terminal devices are shown in FIG. In actual scenarios, more or fewer types of terminal devices may participate in the process of generating the mixed sound effect description model. The specific number and type depend on the actual scenario and are not limited here. In addition, Figure 1 One server is shown in the figure, but in an actual scenario, multiple servers may be involved, and the specific number of servers depends on the actual scenario.

[0047] It should be noted that the mixed sound effect description model generation method provided in this embodiment can be performed offline, that is, the server locally stores sample audio, audio question text and sound effect description text, and it can directly use the solution provided in this application to train the mixed sound effect description model.

[0048] It is understandable that the above-mentioned system and program with the function of generating a mixing sound effect description model and mixing function can be used as a kind of cloud service program. The specific operation mode depends on the actual scenario and is not limited here.

[0049] Specifically, after generating the mixed sound effect description model, the obtained mixed sound effect description model can be used to describe the mixed sound effects of any mixed song, so that the user can optimize the sound effect parameters of the mixed song based on the description.

[0050] Combined with the above commonalities, please see Figure 2 , Figure 2 A flow chart of a method for generating a mixed sound effect description model provided in an embodiment of the present application. Figure 2 As shown, the method for generating a mixed sound effect description model may include the following steps:

[0051] S201, obtaining sample audio, audio question text and sound effect description text; the sample audio is obtained by mixing the original audio, the audio question text is used to inquire about the difference between the sample audio and the original audio in target sound effect parameters, and the sound effect description text is a response to the audio question text.

[0052] The audio question text and the corresponding sound effect description text are as follows:

[0053] Audio question text 1: What are the characteristics of the low frequency of this audio compared to the standard audio?

[0054] The sound effect description text corresponding to audio problem text 1: The low frequency is too strong and should be adjusted much weaker.

[0055] Audio question text 2: What are the characteristics of the loudness of this audio compared to standard audio?

[0056] The sound effect description text corresponding to audio problem text 2: The loudness of this audio is too low and should be increased.

[0057] In one example, the sample audio generation process includes: adjusting at least one of the reverberation parameter, compression parameter, delay parameter and equalization parameter in the DSP mixing link according to the mixing sound effect parameter group, and obtaining multiple DSP mixing links with different mixing parameters; the mixing sound effect parameter group includes: multiple groups of parameter vectors composed of at least one of the reverberation parameter, compression parameter, delay parameter and equalization parameter; using multiple DSP mixing links to mix multiple target audios, and obtaining mixed audios corresponding to the multiple target audios; using each mixed audio as the sample audio. The target audio can be any song or any audio, such as: the original audio in the master or a released song.

[0058] In one example, the process of generating the sound effect description text includes: using the mixed sound effect description information of the sample audio input by the user as the sound effect description text, and the sound effect description text can also be generated through a natural language model.

[0059] S202: Process the sample audio and the audio question text through the mixed sound effect description model to be trained to obtain a predicted sound effect description text.

[0060] S203. Based on the predicted sound effect description text and the sound effect description text, the parameters of the mixed sound effect description model to be trained are optimized until convergence conditions are reached to obtain a trained mixed sound effect description model, which is used to generate corresponding mixed sound effect description information for the input mixed audio.

[0061] In this embodiment, based on the predicted sound effect description text and the sound effect description text, the parameters of the mixed sound effect description model to be trained are optimized, including: inputting the sample audio and its sound effect description text into the mixed sound effect description model to be trained, so that the mixed sound effect description model to be trained outputs the mixed sound effect description text corresponding to the sample audio; calculating the loss value between the mixed sound effect description text and the sound effect description text of the sample audio; if the loss value meets the model convergence condition, the current mixed sound effect description model to be trained is used as the mixed sound effect description model; otherwise, after updating the model parameters of the current mixed sound effect description model according to the loss value, the current mixed sound effect description model is iteratively trained; when the loss value meets the model convergence condition, the current model is used as the mixed sound effect description model.

[0062] It can be seen that this embodiment can train a mixed sound effect description model for generating corresponding mixed sound effect description information for input mixed audio based on sample audio, audio question text and sound effect description text. Among them, the sample audio is obtained by mixing the original audio, the audio question text is used to inquire about the difference between the sample audio and the original audio in the target sound effect parameters, and the sound effect description text is a response to the audio question text. The mixed sound effect description information generated by the mixed sound effect description model obtained by this application can guide users to adjust the mixed audio. Listeners can use the mixed sound effect description information instead of their own experience to judge whether the mixing effect of the mixed audio is appropriate, so that listeners can more easily master mixing skills, accurately understand the advantages and disadvantages of audio, and improve the universality and convenience of music production.

[0063] See also Figure 3 , Figure 3 A flow chart of a method for generating a sound effect description provided in an embodiment of the present application. Figure 3 As shown, the method may include the following steps:

[0064] S301: Obtain target mixed audio and question text corresponding to the target mixed audio.

[0065] S302: Input the target mixed audio and question text into a mixed sound effect description model, so that the mixed sound effect description model generates a mixed sound effect description text for the question text.

[0066] The mixed sound effect description model is generated according to the method described in any of the aforementioned embodiments; and the mixed sound effect description information is used to guide the mixing optimization of the target mixed audio.

[0067] In one example, the present embodiment further optimizes the mixed sound effects of the target mixed audio according to the mixed sound effect description information to obtain the optimized target mixed audio; returns to the step of obtaining the target mixed audio and the problem text of the target mixed audio, until the obtained mixed sound effect description information indicates that the target mixed audio does not need to be optimized any more, and the optimal target mixed audio is obtained. It can be seen that the present embodiment can input the optimized target mixed audio into the mixed sound effect description model, so that the mixed sound effect description model generates the corresponding mixed sound effect description information again, and then enables the DSP mixing link to further optimize the audio mixing effect according to the mixed sound effect description information, thereby achieving further optimization of the audio mixing effect.

[0068] In one example, optimizing the mixed sound effect of the target mixed audio according to the mixed sound effect description information includes: using a DSP mixing link to perform at least one of reverberation processing, compression processing, delay processing and equalization processing on the target mixed audio according to the mixed sound effect description information.

[0069] In one example, this embodiment uses the audio question text in the mixed sound effect description model training process as the question text corresponding to the target mixed audio.

[0070] It can be seen that this embodiment enables the mixed sound effect description model to generate a mixed sound effect description text for the question text based on the target mixed audio and the question text corresponding to the target mixed audio. Based on this mixed sound effect description text, it can be clear whether the mixing effect of the mixed audio is appropriate, thereby making it easier for listeners to master mixing skills, accurately understand the advantages and disadvantages of the audio, and improve the universality and convenience of music production.

[0071] Figure 4 Another flow chart of a method for generating a mixed sound effect description model provided in an embodiment of the present application. Figure 4 As shown, the method for generating a mixed sound effect description model may include the following steps:

[0072] S401: Acquire multiple target audios.

[0073] In this embodiment, the target audio may be any song or any audio, such as the original audio in the master tape or a released song.

[0074] S402: Perform mixing processing on multiple target audios according to the mixing audio effect parameter group to obtain multiple mixed audios.

[0075] It should be noted that the mixing sound effect parameter group in this embodiment includes: multiple groups of parameter vectors composed of reverberation parameters, compression parameters, delay parameters and / or equalization parameters. A group of parameter vectors includes reverberation parameters, compression parameters, delay parameters and / or equalization parameters.

[0076] In one embodiment, multiple target audios are mixed according to a mixing sound effect parameter group to obtain multiple mixed audios, including: adjusting reverberation parameters, compression parameters, delay parameters and / or equalization parameters in a DSP mixing link according to the mixing sound effect parameter group to obtain multiple DSP mixing links with different mixing parameters; the mixing sound effect parameter group includes: multiple groups of parameter vectors composed of reverberation parameters, compression parameters, delay parameters and / or equalization parameters; using multiple DSP mixing links to mix each target audio to obtain multiple mixed audios corresponding to each target audio.

[0077] It should be noted that the DSP mixing link is specifically a program script or hardware module that implements a mixing function. The DSP mixing link adjusts the parameters such as reverberation, compression, delay and / or equalization according to the mixing sound effect parameter group. For example: in the mixing sound effect parameter group, the reverberation parameter value is A, the compression parameter value is B, the delay parameter value is C, and the equalization parameter value is D. Then the DSP mixing link sets the reverberation value of the corresponding audio to A, the compression value to B, the delay value to C, and the equalization value to D.

[0078] S403: Label each mixed audio with mixed sound effect description information to obtain multiple training audios.

[0079] Among them, the annotated mixed sound effect description information can be input by the user, and can also be generated by a natural language model. For example: each mixed audio and annotated question is input into the natural language model, so that the natural language model describes the characteristics of the mixed parameters such as the loudness and frequency of the input mixed audio. Among them, the annotated question can be: What are the characteristics of the low frequency and loudness of the mixed audio relative to the original audio. Correspondingly, the mixed sound effect description information can be: the mixed audio has too low a low frequency and too low a loudness relative to the original audio. The original audio is: the unmixed audio corresponding to the input mixed audio, that is, the target audio obtained in step S401. In one embodiment, annotating the mixed sound effect description information for each mixed audio includes: annotating the mixed sound effect description information of each mixed audio input by the user on each mixed audio. The mixed sound effect description information can be text, or it can be audio or picture.

[0080] Among them, the mixing sound effect description information marked by the user covers human subjective feelings, and can describe the difference between the mixing effect of the same mixed audio heard by humans and the mixing effect determined by mixing technology; for example: using the DSP mixing link to enhance the low frequency and increase the reverberation of a certain audio, but when people hear the mixed audio, they do not feel that the low frequency is significantly improved, and this difference can be reflected through the mixing sound effect description information marked by the user.

[0081] S404: Use multiple training audios to train a pre-trained language model, so that the pre-trained language model learns the ability to generate mixed sound effect description information, and obtains a mixed sound effect description model.

[0082] The mixed sound effect description model is used to generate corresponding mixed sound effect description information for the input mixed audio.

[0083] In one embodiment, a pre-trained language model (i.e., a mixed sound effect description model to be trained) is trained using multiple training audios so that the pre-trained language model learns the ability to generate mixed sound effect description information to obtain a mixed sound effect description model, including: obtaining training data from multiple training audios; inputting the training data into the pre-trained language model so that the pre-trained language model outputs a mixed sound effect description text of the training data; calculating a loss value between the mixed sound effect description text and the mixed sound effect description information annotated in the training data; if the loss value meets the model convergence condition, using the current pre-trained language model as the mixed sound effect description model; otherwise, after updating the model parameters of the current pre-trained language model according to the loss value, additional training data is obtained from multiple training audios to iteratively train the current pre-trained language model until the loss value meets the model convergence condition.

[0084] The pre-trained language model may be any language model that can identify functions such as song categories and song styles.

[0085] It can be seen that after mixing multiple target audios, this embodiment adds mixed sound effect description information to them, thereby obtaining model training audio. Based on this training audio, the pre-trained language model can learn the ability to generate mixed sound effect description information, thereby converting the pre-trained language model into a mixed sound effect description model. The mixed sound effect description model can then generate corresponding mixed sound effect description information for the input mixed audio, and the mixed sound effect description information can guide the user to adjust the mixed audio. The listener can use the mixed sound effect description information rather than his own experience to judge whether the mixing effect of the mixed audio is appropriate, thereby making it easier for the listener to master mixing skills, accurately understand the pros and cons of the audio, and improve the universality and convenience of music production.

[0086] See also Figure 5 , Figure 5 A flow chart of a mixing method provided in an embodiment of the present application. Figure 5 As shown, the mixing method may include the following steps:

[0087] S501, obtaining a target song.

[0088] In this embodiment, the target song may be any song, such as an original song in a master tape or a released song.

[0089] S502: Input the target song into the mixed sound effect description model so that the mixed sound effect description model generates corresponding mixed sound effect description information.

[0090] The mixed sound effect description model is generated according to the method described in the above embodiment.

[0091] S503: Optimize the mixed sound effects of the target song according to the mixed sound effect description information.

[0092] In one embodiment, the mixed sound effects of the target song are optimized according to the mixed sound effect description information, including: utilizing a DSP mixing link to perform reverberation processing, compression processing, delay processing and / or equalization processing on the target song according to the mixed sound effect description information; the DSP mixing link can be specifically a program script that implements a mixing function, and the mixing technology used here can refer to the mixing technology used in step S402, and the two can learn from each other.

[0093] In order to further improve the mixed sound effect of the target song, the optimized target song can be input into the mixed sound effect description model so that the mixed sound effect description model can generate the corresponding mixed sound effect description information again, and then the optimized target song and the corresponding mixed sound effect description information can be constructed as the training audio of the mixed sound effect description model; the mixed sound effect description model can be trained again using the training audio, thereby achieving retraining and re-optimization of the model. Alternatively, the optimized target song can be reverberated, compressed, delayed and / or equalized again using the DSP mixing link according to the mixed sound effect description information regenerated by the mixed sound effect description model, thereby achieving re-optimization of the song mixing effect.

[0094] It can be seen that this embodiment uses the trained mixing sound effect description model to describe the parameter characteristics of the mixing sound effects of any song. The description information can guide the user to adjust the mixed audio. The listener can use the mixing sound effect description information rather than his own experience to judge whether the mixing effect of the mixed audio is appropriate, thereby making it easier for the listener to master mixing skills, accurately understand the advantages and disadvantages of the audio, and improve the universality and convenience of music production.

[0095] In one example, the training data can be generated by the following process: a large amount of high-definition audio is passed through a basic audio effect chain (such as a DSP mixing processing chain), thereby batch-generating a large amount of audio processed with audio effects, obtaining multiple mixed versions of the same audio, and retaining the audio effect parameter vector corresponding to the audio effect chain. Afterwards, these audio processed with audio effects are annotated with text at the audio effect level in the form of questions and answers.

[0096] See also Figure 6, select standard audio (i.e., music produced by standard commercial mixing), and then use the DSP mixing processing link to transform the standard audio into n mixed versions, with different reverberation parameters, compression parameters, delay parameters and / or equalization parameters. Since the DSP mixing processing link is based on the sound effect parameter vector (i.e., mixing sound effect parameters), this step can batch produce audio data processed by the sound effect link.

[0097] Next, we conduct Q&A on the music information related to the sound effects for the n mixed versions mentioned above. Through this type of Q&A, we can obtain a large number of sound effect text annotations for the mixed version audio, which can be used for the subsequent training of the large language model. If the sound effect text annotation (i.e., mixed sound effect description information) is determined by the model, then the model can be determined based on the relative distance between the sound effect parameter vectors of the standard audio and the mixed audio.

[0098] Let’s take the following text question and answer example for further explanation:

[0099] Q1: What are the characteristics of the low frequency of this audio compared to standard audio?

[0100] A1: The low frequency is too strong and should be adjusted down a lot.

[0101] Q2: What are the characteristics of the loudness of this audio compared to standard audio?

[0102] A2: The audio volume is too low and should be increased.

[0103] Next, we will train the large language model. This solution uses a pre-trained music large language model (i.e., the mixed sound effect description model to be trained). The model already has the ability to recognize basic audio features (such as music genres and types of music), so this solution only requires fine-tuning of the model. The input in the model training process is the audio processed by the sound effect link (i.e., the mixed version), the sound effect Q&A questions, and the annotated sound effect text mentioned above. The output is the relevant answer. The training process strives to output answers that are as close as possible to the annotated sound effect text. Figure 7 As shown, the loss value between the answer output by the model and the annotated sound effect text is calculated to update the model parameters of the large language model until the loss value meets the model convergence condition or reaches the preset maximum number of iterations.

[0104] After the model training is completed, the input of the model application process is the audio processed by the sound effect link (i.e. the mixed version) and the corresponding sound effect question and answer questions, and the output is the relevant answer, i.e. the text description of the sound effect of the music (i.e. the mixed sound effect description information).

[0105] Afterwards, the sound effect text description output by the sound effect large language model (i.e., the mixed sound effect description model) can be further used to optimize and adjust the corresponding sound effect parameter vector to obtain an optimized mixed version. The sound effect parameter vector can be output by the sound effect large language model.

[0106] This embodiment solves the problem of difficulty in obtaining training data by means of question-answer annotation. Moreover, training on a pre-trained large language model can significantly shorten the model training cycle and improve training efficiency. In addition, this solution only requires the use of finished songs as model input data, which greatly reduces the difficulty of data acquisition. At the same time, the annotation information of the training data of this solution can be only text, which reduces the difficulty of annotation. With the help of the large music language model, this solution can use the data-driven characteristics of deep learning to obtain more effective audio and sound effect information. In addition, the audio and sound effect features output by the model can be used for subsequent sound effect guidance, so that subsequent sound effects can be flexibly adjusted according to the scene and user preferences.

[0107] This solution can lower the threshold for mixing, making it easier for amateur music lovers to master mixing skills and improve their music production level; by analyzing and optimizing the sound effect features through a large language model, users can more accurately understand the pros and cons of music, which will help realize data-driven music production and improve the market adaptability of music works. This solution can obtain audio sound effect features, including: text description and vectors composed of sound effect parameters. The vector composed of sound effect parameters can then be further adjusted based on the text description, which increases the flexibility of the solution.

[0108] In one example, the annotated text description can be automatically generated by the model in addition to being input by the user. Or the pre-trained large language model can be used to process the unlabeled mixed audio with a self-supervised or unsupervised strategy to obtain a sound effect large language model. Of course, multiple mixed songs can also be input into the sound effect large language model at one time, so that the sound effect large language model can learn functions such as multi-track mixing and sound effect style transfer.

[0109] An electronic device provided in an embodiment of the present application is introduced below. The electronic device described below and related implementation steps of the above embodiment can be referenced to each other.

[0110] Furthermore, the present application also provides an electronic device. The electronic device can be Figure 8 The server 50 shown may also be Fig. 9 Terminal 60 is shown. Figure 8 and Fig. 9 All of them are structural diagrams of electronic devices according to an exemplary embodiment, and the contents in the diagrams cannot be regarded as any limitation on the scope of use of the present application.

[0111] Figure 8 A schematic diagram of the structure of a server provided in an embodiment of the present application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 is used to store a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the method disclosed in any of the aforementioned embodiments.

[0112] In this embodiment, the power supply 53 is used to provide working voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows is any communication protocol that can be applied to the technical solution of the present application, and is not specifically limited here; the input and output interface 55 is used to obtain external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs and is not specifically limited here.

[0113] In addition, the memory 52, as a carrier for storing resources, can be a read-only memory, a random access memory, a disk or an optical disk, etc. The resources stored thereon include an operating system 521, a computer program 522 and data 523, etc. The storage method can be temporary storage or permanent storage.

[0114] The operating system 521 is used to manage and control the hardware devices and computer programs 522 on the server 50, so as to realize the operation and processing of the data 523 in the memory 52 by the processor 51, and it can be Windows Server, Netware, Unix, Linux, etc. In addition to including computer programs that can be used to complete the methods disclosed in any of the above embodiments, the computer program 522 can also further include computer programs that can be used to complete other specific tasks. In addition to including data such as application update information, the data 523 can also include data such as application developer information.

[0115] Fig. 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal 60 may specifically include but is not limited to a smart phone, a tablet computer, a laptop computer or a desktop computer.

[0116] Generally, the terminal 60 in this embodiment includes: a processor 61 and a memory 62 .

[0117] Among them, the processor 61 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. The processor 61 can be implemented in at least one hardware form of DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the awake state, also known as a CPU (Central Processing Unit); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 61 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0118] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include a high-speed random access memory, and a non-volatile memory, such as one or more disk storage devices, flash memory storage devices. In this embodiment, the memory 62 is at least used to store the following computer program 621, wherein, after the computer program is loaded and executed by the processor 61, it can implement the relevant steps in the method performed by the terminal side disclosed in any of the aforementioned embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, etc., and the storage method may be temporary storage or permanent storage. Among them, the operating system 622 may include Windows, Unix, Linux, etc. Data 623 may include, but is not limited to, update information of the application.

[0119] In some embodiments, the terminal 60 may further include a display screen 63 , an input / output interface 64 , a communication interface 65 , a sensor 66 , a power supply 67 , and a communication bus 68 .

[0120] Those skilled in the art will understand that Fig. 9 The structure shown in the figure does not constitute a limitation on the terminal 60, and the terminal 60 may include more or less components than those shown in the figure.

[0121] A storage medium provided in an embodiment of the present application is introduced below. The storage medium described below and the related implementation steps of the above embodiment can be referenced to each other.

[0122] Furthermore, the embodiment of the present application also discloses a storage medium, in which computer executable instructions are stored, and when the computer executable instructions are loaded and executed by the processor, the method disclosed in any of the above embodiments is implemented. For the specific steps of the method, reference can be made to the corresponding contents disclosed in the above embodiments, and no further description will be given here.

[0123] A computer program product provided in an embodiment of the present application is introduced below. The computer program product described below and the related implementation steps of the above embodiment can be referenced to each other.

[0124] Furthermore, the embodiments of the present application also disclose a computer program product, including a computer program / instruction, which implements the method described in any of the above embodiments when executed by a processor. The specific steps of the method can refer to the corresponding contents disclosed in the above embodiments, and will not be repeated here.

[0125] It should be pointed out that the above is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

[0126] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0127] Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for those skilled in the art, according to the idea of ​​the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A method for generating a mixed sound effect description model, characterized in that: include: Get sample audio, audio question text, and sound effect description text; The sample audio is obtained by mixing the original audio, the audio question text is used to inquire about the difference between the sample audio and the original audio in the target sound effect parameters, and the sound effect description text is a response to the audio question text; Processing the sample audio and the audio question text by the mixed sound effect description model to be trained to obtain a predicted sound effect description text; Based on the predicted sound effect description text and the sound effect description text, the parameters of the mixed sound effect description model to be trained are optimized until the convergence condition is reached, so as to obtain a trained mixed sound effect description model, which is used to generate corresponding mixed sound effect description information for the input mixed audio.

2. The method according to claim 1, characterized in that The generation process of the sample audio includes: At least one of a reverberation parameter, a compression parameter, a delay parameter, and an equalization parameter in a DSP mixing link is adjusted according to the mixing sound effect parameter group to obtain a plurality of DSP mixing links with different mixing parameters; the mixing sound effect parameter group includes: a plurality of groups of parameter vectors composed of at least one of a reverberation parameter, a compression parameter, a delay parameter, and an equalization parameter; Using the multiple DSP mixing links to perform mixing processing on multiple target audios, to obtain mixed audios corresponding to the multiple target audios respectively; Each mixed audio is used as the sample audio.

3. The method according to claim 1, characterized in that The generation process of the sound effect description text includes: The mixed sound effect description information of the sample audio input by the user is used as the sound effect description text.

4. A method for generating a sound effect description, characterized in that: include: Obtaining a target mixed audio and a question text corresponding to the target mixed audio; The target mixed audio and the question text are input into a mixed sound effect description model so that the mixed sound effect description model generates a mixed sound effect description text for the question text; wherein the mixed sound effect description model is generated according to the method described in any one of claims 1 to 3; and the mixed sound effect description information is used to guide mixing optimization of the target mixed audio.

5. The method according to claim 4, characterized in that The method further comprises: Optimizing the mixed sound effect of the target mixed audio according to the mixed sound effect description information to obtain an optimized target mixed audio; Return to the step of obtaining the target mixed audio and the question text of the target mixed audio, until the obtained mixed sound effect description information indicates that the target mixed audio does not need to be optimized any more, and the optimal target mixed audio is obtained.

6. The method according to claim 5, characterized in that Optimizing the mixed sound effect of the target mixed audio according to the mixed sound effect description information includes: The DSP mixing link is used to perform at least one of reverberation processing, compression processing, delay processing and equalization processing on the target mixed audio according to the mixed audio effect description information.

7. The method according to claim 4, characterized in that The method further comprises: The audio question text in the training process of the mixed sound effect description model is used as the question text corresponding to the target mixed audio.

8. An electronic device, characterized in that: The electronic device comprises a processor and a memory; wherein the memory is used to store a computer program, and the computer program is loaded and executed by the processor to implement the method according to any one of claims 1 to 7.

9. A storage medium, characterized in that: The storage medium stores computer executable instructions, and when the computer executable instructions are loaded and executed by the processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images