Model generation method, sound effect description generation method, device, medium and product

Abstract

This application discloses a method for generating a mixed audio effect description model, a method for generating audio effect descriptions, an apparatus, a medium, and a product in the field of computer technology. This solution, based on sample audio, audio problem text, and audio effect description text, can be trained to obtain a mixed audio effect description model for generating corresponding mixed audio effect description information for input mixed audio. The mixed audio effect description information generated by this model can guide users in adjusting mixed audio. Listeners can use the mixed audio effect description information, rather than their own experience, to judge whether the mixing effect of the mixed audio is appropriate, thus making it easier for listeners to master mixing techniques, accurately understand the strengths and weaknesses of audio, and improve the universality and convenience of music production.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a sound effect description model generation method for mixing, a sound effect description generation method, equipment, a medium and a product. BACKGROUND

[0002] Currently, after a deep learning model is used to generate mixed audio, it is difficult to evaluate the mixing effect of the mixed audio, and a listener can only rely on his own experience to determine whether the mixing effect of the mixed audio is appropriate. SUMMARY

[0003] Therefore, the present application aims to provide a sound effect description model generation method for mixing, equipment, a medium and a product, which can enable the model to learn the generation ability of sound effect description information, and the sound effect description information output by the sound effect description model for mixing can guide the user to adjust the mixed audio. The specific scheme is as follows:

[0004] To achieve the above-mentioned purpose, on the one hand, the present application provides a sound effect description model generation method for mixing, comprising:

[0005] obtaining a sample audio, an audio question text and a sound effect description text; the sample audio is obtained by mixing an original audio, the audio question text is used to inquire about the difference between the sample audio and the original audio in a target sound effect parameter, and the sound effect description text is a response to the audio question text;

[0006] processing the sample audio and the audio question text by a to-be-trained sound effect description model for mixing to obtain a predicted sound effect description text;

[0007] based on the predicted sound effect description text and the sound effect description text, performing parameter optimization on the to-be-trained sound effect description model for mixing until a convergence condition is reached, to obtain a trained sound effect description model for mixing, which is used to generate corresponding sound effect description information for an input mixed audio.

[0008] Optionally, the generation process of the sample audio comprises:

[0009] adjusting at least one of a reverb parameter, a compression parameter, a delay parameter and an equalization parameter in a DSP mixing link according to a mixing sound effect parameter group to obtain a plurality of DSP mixing links with different mixing parameters; the mixing sound effect parameter group comprises a plurality of parameter vectors composed of at least one of the reverb parameter, the compression parameter, the delay parameter and the equalization parameter;

[0010] mixing a plurality of target audios by using the plurality of DSP mixing links to obtain mixed audios corresponding to the plurality of target audios, respectively;

[0011] mixing audio of each sample audio as the sample audio.

[0012] Optionally, the generating process of the sound effect description text comprises:

[0013] mixing sound effect description information of the sample audio input by the user as the sound effect description text.

[0014] In another aspect, the present application also provides a sound effect description generation method, comprising:

[0015] obtaining target mixing audio and problem text corresponding to the target mixing audio;

[0016] inputting the target mixing audio and the problem text into a mixing sound effect description model, so that the mixing sound effect description model generates mixing sound effect description text for the problem text; wherein the mixing sound effect description model is generated according to any one of the preceding methods; and the mixing sound effect description information is used to guide mixing optimization of the target mixing audio.

[0017] Optionally, the method further comprises:

[0018] optimizing mixing sound effect of the target mixing audio according to the mixing sound effect description information to obtain optimized target mixing audio;

[0019] returning to the step of obtaining target mixing audio and problem text corresponding to the target mixing audio until the obtained mixing sound effect description information indicates that the target mixing audio does not need to be optimized any more, and the optimal target mixing audio is obtained.

[0020] Optionally, optimizing mixing sound effect of the target mixing audio according to the mixing sound effect description information comprises:

[0021] using a DSP mixing link to perform at least one of reverb processing, compression processing, delay processing and equalization processing on the target mixing audio according to the mixing sound effect description information.

[0022] Optionally, the method further comprises:

[0023] using audio problem text in the training process of the mixing sound effect description model as the problem text corresponding to the target mixing audio.

[0024] In another aspect, the present application also provides an electronic device, comprising a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to realize the corresponding method.

[0025] In yet another aspect, the present application also provides a storage medium, wherein the storage medium stores computer executable instructions, and the computer executable instructions are loaded and executed by a processor to implement the foregoing corresponding method.

[0026] In yet another aspect, the present application also provides a computer program product, comprising computer program / instructions, and the computer program / instructions are executed by a processor to implement the method according to any one of the foregoing aspects.

[0027] Based on the sample audio, the audio question text and the sound effect description text, the present application can train a sound effect description model for mixing sound, which is used to generate corresponding sound effect description information of the mixed audio. The sample audio is obtained by mixing the original audio, the audio question text is used to inquire the difference between the sample audio and the original audio in the target sound effect parameter, and the sound effect description text is a response to the audio question text. The sound effect description information generated by the sound effect description model obtained by the present application can guide the user to adjust the mixed audio. The listener can judge whether the mixing effect of the mixed audio is appropriate by means of the sound effect description information instead of his own experience, so that the listener can more easily master the mixing skill, accurately understand the pros and cons of the audio, and improve the universality and convenience of music production.

[0028] Correspondingly, the sound effect description generation method, the mixing method, the device, the medium and the product provided by the present application also have the above technical effects. BRIEF DESCRIPTION OF DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the drawings needed to be used in the embodiments or the prior art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the present application, and those skilled in the art can obtain other drawings according to the provided drawings without creative labor.

[0030] Figure 1 A physical architecture schematic diagram to which the present application is applicable is provided for the present application;

[0031] Figure 2 A sound effect description model generation method flowchart is provided for the present application;

[0032] Figure 3 A sound effect description generation method flowchart is provided for the present application;

[0033] Figure 4 Another sound effect description model generation method flowchart is provided for the present application;

[0034] Figure 5 A mixing method flowchart is provided for the present application;

[0035] Figure 6 A schematic diagram of a mixing process provided in this application;

[0036] Figure 7 A schematic diagram of a model training process provided in this application;

[0037] Figure 8 A server architecture diagram provided for this application;

[0038] Figure 9 A terminal structure diagram provided for this application. Detailed Implementation

[0039] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application. Furthermore, in the embodiments of this application, "first," "second," etc., are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0040] After generating mixed audio using a deep learning model, it is difficult to evaluate the mixing effect of the mixed audio. Listeners can only rely on their own experience to judge whether the mixing effect of the mixed audio is appropriate.

[0041] In view of the aforementioned problems, this application proposes a scheme for generating a mixing sound effect description model, which can be trained to generate a mixing sound effect description model. This model can generate corresponding mixing sound effect description information for the input mixed audio. This mixing sound effect description information can guide users to adjust the mixed audio. Listeners can use the mixing sound effect description information, rather than their own experience, to judge whether the mixing effect of the mixed audio is appropriate, thereby making it easier for listeners to master mixing techniques, accurately understand the strengths and weaknesses of the audio, and improve the universality and convenience of music production.

[0042] To facilitate understanding, the physical framework to which this application applies will be introduced first.

[0043] It should be understood that the method for generating a mixing sound effect description model provided in this application can be applied to systems or programs that have the functions of generating a mixing sound effect description model and mixing. Specifically, systems or programs that have the functions of generating a mixing sound effect description model and mixing can run on devices such as servers and personal computers.

[0044] like Figure 1 As shown, Figure 1 This is a schematic diagram of the physical architecture to which this application applies. Figure 1In this context, a system or program with the ability to generate a mixing sound effect description model and a mixing function can run on a server. This server acquires sample audio, audio question text, and sound effect description text from other terminal devices via a network. The sample audio and audio question text are processed by the mixing sound effect description model to be trained to obtain predicted sound effect description text. Based on the predicted sound effect description text and the sound effect description text, the parameters of the mixing sound effect description model to be trained are optimized until the convergence condition is met, resulting in a trained mixing sound effect description model. The mixing sound effect description model is used to generate corresponding mixing sound effect description information for the input mixed audio.

[0045] Depend on Figure 1 It is known that the server can establish communication connections with multiple devices, obtaining sample audio, audio question text, and sound effect description text from these devices. The server then trains a mixed audio effect description model based on this data. The server pre-stores a pre-trained language model, and converts this pre-trained language model into a mixed audio effect description model based on the sample audio, audio question text, and sound effect description text.

[0046] Figure 1 The diagram illustrates various terminal devices. In real-world scenarios, more or fewer types of terminal devices may participate in the process of generating the audio mixing effect description model. The specific number and types depend on the actual scenario and are not limited here. Additionally... Figure 1 The image shows one server, but in real-world scenarios, multiple servers can be involved, with the specific number depending on the actual situation.

[0047] It should be noted that the method for generating the audio mixing effect description model provided in this embodiment can be performed offline. That is, the server locally stores sample audio, audio problem text and sound effect description text, and can directly use the scheme provided in this application to train the audio mixing effect description model.

[0048] It is understandable that the aforementioned systems and programs with functions for generating sound effect description models and mixing functions can be considered as a type of cloud service program. The specific operating mode depends on the actual scenario and is not limited here.

[0049] Specifically, after generating the mixing sound effect description model, the obtained mixing sound effect description model can be used to describe the mixing sound effects of any mixed song, so that users can optimize the sound effect parameters of the mixed song based on this description.

[0050] Based on the above commonalities, please refer to Figure 2 , Figure 2 This is a flowchart illustrating a method for generating a mixed audio effect description model, as provided in an embodiment of this application. Figure 2 As shown, the method for generating the audio mixing effect description model may include the following steps:

[0051] S201, obtain sample audio, audio question text and sound effect description text; the sample audio is obtained by mixing the original audio, the audio question text is used to inquire the difference between the sample audio and the original audio in the target sound effect parameter, and the sound effect description text is a response to the audio question text.

[0052] The audio question text and the corresponding sound effect description text are as follows:

[0053] Audio question text 1: What are the characteristics of the low frequency of the audio relative to the standard audio?

[0054] The sound effect description text corresponding to the audio question text 1 is: the low frequency is too strong and should be weakened.

[0055] Audio question text 2: What are the characteristics of the loudness of the audio relative to the standard audio?

[0056] The sound effect description text corresponding to the audio question text 2 is: the loudness of the audio is too low and should be enhanced.

[0057] In an example, the generation process of the sample audio includes: adjusting at least one of the reverb parameter, the compression parameter, the delay parameter and the equalization parameter in the DSP mixing link according to the mixing sound effect parameter group to obtain a plurality of DSP mixing links with different mixing parameters; the mixing sound effect parameter group includes a plurality of parameter vectors composed of at least one of the reverb parameter, the compression parameter, the delay parameter and the equalization parameter; mixing a plurality of target audios by using a plurality of DSP mixing links to obtain a plurality of mixing audios corresponding to the target audios respectively; and taking each mixing audio as a sample audio. The target audio can be any song, any audio, such as the original audio in the master tape or the published song, etc.

[0058] In an example, the generation process of the sound effect description text includes: taking the mixing sound effect description information of the sample audio input by the user as the sound effect description text, or generating the sound effect description text by using a natural language model.

[0059] S202, process the sample audio and the audio question text by using the mixing sound effect description model to be trained to obtain a predicted sound effect description text.

[0060] S203, based on the predicted sound effect description text and the sound effect description text, optimize the parameters of the mixing sound effect description model to be trained until a convergence condition is reached, obtain a trained mixing sound effect description model, and the mixing sound effect description model is used to generate corresponding mixing sound effect description information for the input mixing audio.

[0061] In the embodiment, the parameter of the to-be-trained sound effect description model of mixed audio is optimized based on the predicted sound effect description text and the sound effect description text, including: inputting the sample audio and the sound effect description text thereof into the to-be-trained sound effect description model of mixed audio, so that the to-be-trained sound effect description model of mixed audio outputs the sound effect description text of mixed audio corresponding to the sample audio; calculating the loss value between the sound effect description text of mixed audio and the sound effect description text of the sample audio; if the loss value meets the model convergence condition, the current to-be-trained sound effect description model of mixed audio is taken as the sound effect description model of mixed audio; otherwise, the model parameter of the current sound effect description model of mixed audio is updated according to the loss value, and the current sound effect description model of mixed audio is iteratively trained; when the loss value meets the model convergence condition, the current model is taken as the sound effect description model of mixed audio.

[0062] It can be seen that, based on the sample audio, the audio question text and the sound effect description text, the sound effect description model of mixed audio for generating the corresponding sound effect description information of mixed audio input can be trained. The sample audio is obtained by mixing the original audio, the audio question text is used to inquire the difference between the sample audio and the original audio in the target sound effect parameter, and the sound effect description text is a response to the audio question text. The sound effect description information generated by the sound effect description model obtained by the application can guide the user to adjust the mixed audio. The listener can judge whether the mixed effect of the mixed audio is appropriate by means of the mixed sound effect description information, rather than his own experience, so that the listener can more easily master the mixing skill, accurately understand the pros and cons of the audio, and improve the universality and convenience of music production.

[0063] Please refer to Figure 3 , Figure 3 A sound effect description generation method flowchart is provided for the embodiments of the application. As shown in Figure 3 , the method can include the following steps:

[0064] S301, obtaining target mixed audio and question text corresponding to the target mixed audio.

[0065] S302, inputting the target mixed audio and the question text into the sound effect description model of mixed audio, so that the sound effect description model of mixed audio generates the sound effect description text for the question text.

[0066] The sound effect description model of mixed audio is generated according to the method described in any of the preceding embodiments; the sound effect description information is used to guide the mixed audio optimization of the target mixed audio.

[0067] In one example, this embodiment further optimizes the mixing effect of the target mixed audio according to the mixing effect description information to obtain the optimized target mixed audio; then it returns to the step of obtaining the target mixed audio and the problem text of the target mixed audio, until the obtained mixing effect description information indicates that no further optimization of the target mixed audio is needed, thus obtaining the optimal target mixed audio. It can be seen that this embodiment can input the optimized target mixed audio into the mixing effect description model, so that the mixing effect description model can generate the corresponding mixing effect description information again, and then the DSP mixing link can further optimize the audio mixing effect according to this mixing effect description information, thereby achieving further optimization of the audio mixing effect.

[0068] In one example, optimizing the mixing effect of a target mixed audio according to the mixing effect description information includes: using a DSP mixing link to perform at least one of reverb processing, compression processing, delay processing, and equalization processing on the target mixed audio according to the mixing effect description information.

[0069] In one example, this embodiment uses the audio problem text generated during the training process of the audio mixing description model as the problem text corresponding to the target audio mixing.

[0070] As can be seen, this embodiment generates a mixing effect description text for the problem text based on the target mixed audio and the corresponding problem text. The mixing effect description text can clearly indicate whether the mixing effect of the mixed audio is appropriate, thereby making it easier for listeners to master mixing techniques, accurately understand the strengths and weaknesses of the audio, and improve the universality and convenience of music production.

[0071] Figure 4 A flowchart illustrating another method for generating a mixed audio effect description model, provided in an embodiment of this application. Figure 4 As shown, the method for generating the audio mixing effect description model may include the following steps:

[0072] S401, Acquire multiple target audio files.

[0073] In this embodiment, the target audio can be any song or any audio file, such as the original audio from the master tape or a released song.

[0074] S402. Mix multiple target audios according to the mixing effect parameter group to obtain multiple mixed audios.

[0075] It should be noted that the mixing audio effect parameter set in this embodiment includes multiple parameter vectors consisting of reverb parameters, compression parameters, delay parameters, and / or equalization parameters. Each parameter vector includes reverb parameters, compression parameters, delay parameters, and / or equalization parameters.

[0076] In one implementation, multiple target audios are mixed according to a set of mixing effect parameters to obtain multiple mixed audios. This includes: adjusting the reverb, compression, delay, and / or equalization parameters in the DSP mixing links according to the set of mixing effect parameters to obtain multiple DSP mixing links with different mixing parameters; the set of mixing effect parameters includes multiple parameter vectors composed of reverb, compression, delay, and / or equalization parameters; and using multiple DSP mixing links to mix each target audio to obtain multiple mixed audios corresponding to each target audio.

[0077] It should be noted that a DSP mixing link is specifically a program script or hardware module that implements mixing functionality. This DSP mixing link adjusts the values ​​of parameters such as reverb, compression, delay, and / or equalization according to the mixing effect parameter set. For example, if the reverb parameter is set to value A, the compression parameter to value B, the delay parameter to value C, and the equalization parameter to value D in the mixing effect parameter set, then the DSP mixing link will set the reverb value to A, the compression value to B, the delay value to C, and the equalization value to D for the corresponding audio.

[0078] S403. Label each mixed audio with mixing effect description information to obtain multiple training audios.

[0079] The labeled audio mixing description information can be input by the user or generated by a natural language model. For example, each audio mix and the labeled question can be input into a natural language model so that the model can describe the characteristics of the mixing parameters, such as loudness and frequency, of the input audio mix. The labeled question could be: "What are the characteristics of the low frequencies and loudness of this audio mix compared to the original audio?" Correspondingly, the audio mixing description information could be: "Compared to the original audio, this audio mix has excessively low low frequencies and excessively low loudness." The original audio is the unmixed audio corresponding to the input audio mix, which is the target audio obtained in step S401. In one embodiment, labeling each audio mix with audio mixing description information includes: labeling each audio mix with audio mixing description information input by the user. The audio mixing description information can be text, audio, or an image.

[0080] Among them, the user-annotated mixing effect description information covers human subjective perception, which can describe the difference between the mixing effect heard by a person and the mixing effect determined by mixing technology. For example, the low frequency and reverberation of a certain audio are enhanced by using a DSP mixing link, but when a person hears the mixed audio, they do not feel that the low frequency is significantly enhanced. This difference can be reflected by the user-annotated mixing effect description information.

[0081] S404. Use multiple training audios to train a pre-trained language model, enabling the pre-trained language model to learn the ability to generate mixed sound effect description information, thus obtaining a mixed sound effect description model.

[0082] Among them, the mixing sound effect description model is used to generate corresponding mixing sound effect description information for the input mixed audio.

[0083] In one implementation, a pre-trained language model (i.e., a mixing sound effect description model to be trained) is trained using multiple training audios. This enables the pre-trained language model to learn the ability to generate mixing sound effect description information, resulting in a mixing sound effect description model. The process includes: acquiring training data from multiple training audios; inputting the training data into the pre-trained language model so that the pre-trained language model outputs mixing sound effect description text from the training data; calculating the loss value between the mixing sound effect description text and the mixing sound effect description information labeled in the training data; if the loss value meets the model convergence condition, then the current pre-trained language model is used as the mixing sound effect description model; otherwise, after updating the model parameters of the current pre-trained language model according to the loss value, additional training data is taken from multiple training audios to iteratively train the current pre-trained language model until the loss value meets the model convergence condition.

[0084] Among them, the pre-trained language model can be any language model that can identify song categories, song styles, etc.

[0085] As can be seen, in this embodiment, after mixing multiple target audios, mixing sound effect description information is added to them to obtain model training audio. Based on this training audio, the pre-trained language model can learn the ability to generate mixing sound effect description information, thereby converting the pre-trained language model into a mixing sound effect description model. Then, the mixing sound effect description model can generate corresponding mixing sound effect description information for the input mixed audio. This mixing sound effect description information can guide users to adjust the mixed audio. Listeners can use the mixing sound effect description information, rather than their own experience, to judge whether the mixing effect of the mixed audio is appropriate, thus making it easier for listeners to master mixing skills, accurately understand the strengths and weaknesses of the audio, and improve the universality and convenience of music production.

[0086] Please see Figure 5 , Figure 5 This is a flowchart illustrating a mixing method provided in an embodiment of this application. Figure 5 As shown, the mixing method may include the following steps:

[0087] S501, Obtain the target song.

[0088] In this embodiment, the target song can be any song, such as the original song in the master tape or a released song.

[0089] S502. Input the target song into the mixing sound effect description model so that the mixing sound effect description model can generate the corresponding mixing sound effect description information.

[0090] The audio mixing effect description model is generated according to the method described in the aforementioned embodiments.

[0091] S503. Optimize the mixing effects of the target song according to the mixing effect description information.

[0092] In one implementation, optimizing the mixing effect of a target song according to the mixing effect description information includes: using a DSP mixing link to perform reverb processing, compression processing, delay processing and / or equalization processing on the target song according to the mixing effect description information; the DSP mixing link can be specifically a program script that implements mixing function, and the mixing technology used here can refer to the mixing technology used in step S402, and the two can be mutually referenced.

[0093] To further improve the mixing effect of the target song, the optimized target song can be input into the mixing effect description model, allowing the model to generate corresponding mixing effect description information again. Then, the optimized target song and the corresponding mixing effect description information are used to construct the training audio for the mixing effect description model. The training audio is then used to retrain the mixing effect description model, thus achieving retraining and re-optimization. Alternatively, the DSP mixing link can be used to apply reverb, compression, delay, and / or equalization processing to the optimized target song based on the mixing effect description information regenerated by the mixing effect description model, thereby achieving further optimization of the song's mixing effect.

[0094] As can be seen, this embodiment uses a trained mixing sound effect description model to describe the parameter characteristics of the mixing sound effect of any song. This description information can guide users to adjust the mixed audio. Listeners can use the mixing sound effect description information, rather than their own experience, to judge whether the mixing effect of the mixed audio is appropriate, thereby making it easier for listeners to master mixing skills, accurately understand the strengths and weaknesses of the audio, and improve the universality and convenience of music production.

[0095] In one example, the training data can be generated as follows: A large amount of high-definition audio is passed through a basic audio effects processing chain (such as a DSP mixing processing chain), thereby generating a large number of audio files with audio effects processing, resulting in multiple mixed versions of the same audio file, while retaining the audio effect parameter vectors corresponding to the audio effects processing chain. Then, for these audio files with audio effects processing, they are annotated with text at the audio effects level through a question-and-answer format.

[0096] Please see Figure 6The process involves selecting standard audio (i.e., music produced using standard commercial mixes), and then using a DSP mixing processing link to transform the standard audio into n mix versions, each with different reverb, compression, delay, and / or equalization parameters. Since the DSP mixing processing link is based on audio effect parameter vectors (i.e., mix audio effect parameters), this step can batch-produce audio data processed by the audio effect link.

[0097] The following section presents a question-and-answer session regarding the sound effects and related music information for the aforementioned n remix versions. This question-and-answer format allows for the acquisition of a large amount of sound effect text annotations for the remix audio, which can be used for training a large language model. If the sound effect text annotations (i.e., remix sound effect descriptions) are determined using a model, the model can be based on the relative distance between the sound effect parameter vectors of the standard audio and the remix audio.

[0098] The following textual question-and-answer example will be used to further illustrate this:

[0099] Q1: What are the characteristics of the low frequencies in this audio compared to standard audio?

[0100] A1: Its low frequencies are too strong and should be reduced significantly.

[0101] Q2: What are the characteristics of the loudness of this audio compared to the standard audio?

[0102] A2: The audio volume is too low; it should be increased.

[0103] The following section describes the training of the large language model. This approach uses a pre-trained music large language model (i.e., the mixed sound effect description model to be trained). The model already possesses the ability to recognize basic audio features (such as music genres and types), therefore, this approach only requires fine-tuning. The input to the model training process consists of audio processed through the sound effect link (i.e., the mixed version), sound effect question-and-answer questions, and the annotated sound effect text mentioned above. The output is the relevant answer, and the training process aims to ensure that the output answer is as close as possible to the annotated sound effect text. For example... Figure 7 As shown, the loss value between the model's output answer and the labeled audio text is used to update the model parameters of the large language model until the loss value meets the model convergence condition or reaches the preset maximum number of iterations.

[0104] After the model is trained, the input to the model during the application process is the audio (i.e., the mixed version) processed by the audio effect link and the corresponding audio effect Q&A questions. The output is the relevant answer, i.e. the audio effect text description of the music (i.e., the mixed audio effect description information).

[0105] Then, the audio effect text descriptions output by the audio effect big language model (i.e., the audio mixing audio effect description model) can be used to further optimize and adjust the corresponding audio effect parameter vectors to obtain an optimized mixing version. The audio effect parameter vectors can be output by the audio effect big language model.

[0106] This embodiment addresses the challenge of acquiring training data by employing a question-and-answer annotation approach. Furthermore, training on a pre-trained large language model significantly shortens the training cycle and improves training efficiency. Additionally, this solution only requires finished songs as input data, greatly reducing the difficulty of data acquisition. Moreover, the annotation information for the training data can be simply text, further simplifying the annotation process. Leveraging the large music language model, this solution utilizes the data-driven nature of deep learning to acquire more effective audio and sound effect information. Furthermore, the audio and sound effect features output by the model can be used for subsequent sound effect guidance, allowing for flexible adjustments based on the scenario and user preferences.

[0107] This solution lowers the barrier to entry for mixing, making it easier for amateur music enthusiasts to master mixing techniques and improve their music production skills. By analyzing and optimizing sound effect features through a large language model, users can more accurately understand the strengths and weaknesses of music, which will help achieve data-driven music production and improve the market adaptability of musical works. This solution can obtain audio sound effect features, including a vector composed of text descriptions and sound effect parameters. The vector composed of sound effect parameters can then be further adjusted based on the text descriptions, increasing the flexibility of the solution.

[0108] In one example, the labeled text descriptions can be generated automatically by the model in addition to user input. Alternatively, a pre-trained large language model can be used to process unlabeled mixed audio using self-supervised or unsupervised strategies to obtain a large language model for sound effects. Of course, multiple mixed songs can also be input into the large language model for sound effects at once, allowing the model to learn multi-track mixing, sound effect style transfer, and other functions.

[0109] The following describes an electronic device provided by an embodiment of this application. The implementation steps of the electronic device described below can be referred to in conjunction with those of the above embodiments.

[0110] Furthermore, embodiments of this application also provide an electronic device. The aforementioned electronic device can be, for example,... Figure 8 The server 50 shown can also be as follows: Figure 9 Terminal 60 is shown. Figure 8 and Figure 9 These are all diagrams illustrating the structure of an electronic device according to an exemplary embodiment. The content in the diagrams should not be considered as any limitation on the scope of this application.

[0111] Figure 8 This is a schematic diagram of a server structure provided in an embodiment of this application. The server 50 may specifically include: at least one processor 51, at least one memory 52, a power supply 53, a communication interface 54, an input / output interface 55, and a communication bus 56. The memory 52 stores a computer program, which is loaded and executed by the processor 51 to implement the relevant steps in the methods disclosed in any of the foregoing embodiments.

[0112] In this embodiment, the power supply 53 is used to provide operating voltage for each hardware device on the server 50; the communication interface 54 can create a data transmission channel between the server 50 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 55 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0113] In addition, the memory 52, as a carrier for resource storage, can be a read-only memory, random access memory, disk, or optical disk, etc. The resources stored on it include operating system 521, computer program 522, and data 523, etc., and the storage method can be temporary storage or permanent storage.

[0114] The operating system 521 manages and controls the various hardware devices on the server 50 and the computer program 522 to enable the processor 51 to perform operations and processing on the data 523 in the memory 52. ​​The operating system 521 can be Windows Server, Netware, Unix, Linux, etc. The computer program 522, in addition to including computer programs capable of performing the methods disclosed in any of the foregoing embodiments, may further include computer programs capable of performing other specific tasks. The data 523 may include application update information and other data, as well as application developer information.

[0115] Figure 9 This is a schematic diagram of the structure of a terminal provided in an embodiment of this application. The terminal 60 may specifically include, but is not limited to, a smartphone, tablet computer, laptop computer, or desktop computer.

[0116] Typically, the terminal 60 in this embodiment includes a processor 61 and a memory 62.

[0117] The processor 61 may include one or more processing cores, such as a quad-core processor or an octa-core processor. The processor 61 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). The processor 61 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, the processor 61 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, the processor 61 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0118] The memory 62 may include one or more computer-readable storage media, which may be non-transitory. The memory 62 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In this embodiment, the memory 62 is used to store at least the following computer program 621, which, after being loaded and executed by the processor 61, is capable of implementing the relevant steps in the methods executed by the terminal side as disclosed in any of the foregoing embodiments. In addition, the resources stored in the memory 62 may also include an operating system 622 and data 623, and the storage method may be temporary storage or permanent storage. The operating system 622 may include Windows, Unix, Linux, etc. The data 623 may include, but is not limited to, application update information.

[0119] In some embodiments, the terminal 60 may further include a display screen 63, an input / output interface 64, a communication interface 65, a sensor 66, a power supply 67, and a communication bus 68.

[0120] Those skilled in the art will understand that Figure 9 The structure shown does not constitute a limitation on terminal 60 and may include more or fewer components than shown.

[0121] The following describes a storage medium provided in an embodiment of this application. The implementation steps of the storage medium described below can be referred to in conjunction with those of the above embodiments.

[0122] Furthermore, this application also discloses a storage medium storing computer-executable instructions. When these computer-executable instructions are loaded and executed by a processor, they implement the method disclosed in any of the foregoing embodiments. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0123] The following describes a computer program product provided by an embodiment of this application. The implementation steps of the computer program product described below can be referred to in conjunction with those of the above embodiments.

[0124] Furthermore, embodiments of this application also disclose a computer program product, including a computer program / instructions, which, when executed by a processor, implements the method described in any of the foregoing embodiments. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0125] It should be noted that the above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

[0126] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0127] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for generating a mixed audio effect description model, characterized in that, include: Obtain sample audio, audio problem text, and sound effect description text; The sample audio is obtained by mixing the original audio. The audio question text is used to inquire about the differences between the sample audio and the original audio in terms of target sound effect parameters. The sound effect description text is a response to the audio question text. The sound effect description text describes the difference between the mixing effect heard by a person and the mixing effect determined by mixing technology. The sample audio and the audio question text are processed by the mixed audio effect description model to be trained to obtain the predicted audio effect description text; Based on the predicted sound effect description text and the sound effect description text, the parameters of the mixed sound effect description model to be trained are optimized until the convergence condition is reached, and a trained mixed sound effect description model is obtained. The mixed sound effect description model is used to generate corresponding mixed sound effect description information for the input mixed audio.

2. The method according to claim 1, characterized in that, The process of generating the sample audio includes: According to the mixing effect parameter group, at least one of the reverb parameter, compression parameter, delay parameter and equalization parameter in the DSP mixing link is adjusted to obtain multiple DSP mixing links with different mixing parameters; the mixing effect parameter group includes: multiple parameter vectors composed of at least one of the reverb parameter, compression parameter, delay parameter and equalization parameter; The multiple DSP mixing links are used to mix multiple target audios to obtain mixed audios corresponding to the multiple target audios respectively; Each mixed audio is used as the sample audio.

3. The method according to claim 1, characterized in that, The process of generating the sound effect description text includes: The sound effect description information of the sample audio input by the user is used as the sound effect description text.

4. A method for generating sound effect descriptions, characterized in that, include: Obtain the target mixed audio and the corresponding question text; The target mixed audio and the problem text are input into the mixing sound effect description model, so that the mixing sound effect description model generates a mixing sound effect description text for the problem text; wherein, the mixing sound effect description model is generated according to any one of claims 1 to 3; the mixing sound effect description information is used to guide the mixing optimization of the target mixed audio.

5. The method according to claim 4, characterized in that, The method further includes: The mixing effect of the target mixed audio is optimized according to the mixing effect description information to obtain the optimized target mixed audio; Return to the step of obtaining the target mixed audio and the problem text of the target mixed audio, until the obtained mixed sound effect description information indicates that there is no need to further optimize the target mixed audio, and the optimal target mixed audio is obtained.

6. The method according to claim 5, characterized in that, Optimize the mixing effects of the target mixed audio according to the mixing effect description information, including: Using a DSP mixing link, at least one of the following processes is performed on the target mixed audio according to the mixed audio effect description information: reverb processing, compression processing, delay processing, and equalization processing.

7. The method according to claim 4, characterized in that, The method further includes: The audio problem text generated during the training process of the audio mixing effect description model is used as the problem text corresponding to the target audio mixing.

8. An electronic device, characterized in that, The electronic device includes a processor and a memory; wherein the memory is used to store a computer program, which is loaded and executed by the processor to implement the method as claimed in any one of claims 1 to 7.

9. A storage medium, characterized in that, The storage medium stores computer-executable instructions, which, when loaded and executed by a processor, implement the method as described in any one of claims 1 to 7.

10. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the method described in any one of claims 1 to 7.

Images