Sound effect adjusting method and training method and device of sound effect adjusting multi-mode large model

Abstract

The invention relates to a sound effect adjustment method, a training method of a sound effect adjustment multi-modal large model, computer equipment and a storage medium. The method comprises the following steps: in response to a sound effect adjustment request for target music sent by a terminal, obtaining an original audio of the target music; inputting the original audio of the target music into a target encoder module of the trained sound effect adjustment multi-mode large model to obtain audio features of the target music, inputting the audio features into a projection module, and converting the audio features into music description semantic features of the target music; the music description semantic feature is a semantic feature of a description text of the target music; and inputting the music description semantic features of the target music into the large language model module to obtain sound effect adjustment parameters of the target music. By adopting the method, a user does not need to manually select a sound effect adjustment mode, and the adjusted parameters can be determined according to the target music, so that the sound effect adjustment effect can be improved.

Description

Technical Field

[0001] The present application relates to the field of music adjustment technology, and in particular to a sound effect adjustment method, a training method for a large multimodal model of sound effect adjustment, a computer device, a storage medium, and a computer program product. Background Art

[0002] With the development of music adjustment technology, a technology for adjusting music sound effects has emerged. This technology can pre-create sound effect materials with different perceptual effects in music applications. When adding sound effect materials to music, these sound effect materials can be displayed in the music application. Users can then select the sound effect materials and adjust the music using the selected sound effect materials.

[0003] However, the above-mentioned sound effect adjustment method relies on the user to manually select the sound effect adjustment method. Therefore, it may be difficult for the user to select the appropriate sound effect to adjust. Or, after selecting the appropriate sound effect for the currently playing music and adjusting it, if the music being played changes, the currently selected sound effect will not be applicable to the changed music. Therefore, the sound effect adjustment effect of the above-mentioned sound effect adding method is relatively poor. Summary of the Invention

[0004] Based on this, it is necessary to provide a sound effect adjustment method that can improve the sound effect adjustment effect, a training method, device, computer equipment, computer-readable storage medium and computer program product for a large multimodal model of sound effect adjustment in response to the above technical problems.

[0005] In a first aspect, the present application provides a method for adjusting sound effects, comprising:

[0006] In response to a sound effect adjustment request for target music sent by a terminal, obtaining original audio of the target music;

[0007] Inputting the original audio of the target music into the target encoder module of the trained sound effect adjustment multimodal large model to obtain audio features of the target music, inputting the audio features into the projection module of the sound effect adjustment multimodal large model, converting the audio features into music description semantic features of the target music; the music description semantic features are semantic features of the description text of the target music; and inputting the music description semantic features of the target music into the large language model module of the sound effect adjustment multimodal large model to obtain sound effect adjustment parameters of the target music;

[0008] The sound effect adjustment parameters of the target music are used to adjust the sound effects of the original audio of the target music to obtain the target audio of the target music, and the target audio data of the target music is used to be played on the terminal.

[0009] In a second aspect, the present application also provides a method for training a large multimodal model for sound effect adjustment, the method comprising:

[0010] Inputting the first sample music into a target encoder module of a to-be-trained multimodal model for sound effect adjustment to obtain audio features of the first sample music, and inputting the audio features of the first sample music into a projection module of the to-be-trained multimodal model for sound effect adjustment to obtain descriptive semantic features of the first sample music;

[0011] Inputting the descriptive semantic features of the first sample music into the large language model module of the to-be-trained multimodal large model for sound effect adjustment to obtain a predicted descriptive text of the first sample music, and training the projection module of the to-be-trained multimodal large model for sound effect adjustment based on the difference between the predicted descriptive text of the first sample music and the actual descriptive text of the first sample music to obtain an initially trained multimodal large model for sound effect adjustment;

[0012] Inputting the second sample music into the target encoder module of the initially trained sound effect adjustment multimodal large model to obtain audio features of the second sample music, and inputting the audio features of the second sample music into the projection module of the initially trained sound effect adjustment multimodal large model to obtain descriptive semantic features of the second sample music;

[0013] The descriptive semantic features of the second sample music are input into the large language model module of the sound effect adjustment multimodal large model after the initial training to obtain the predicted sound effect adjustment parameters of the second sample music, and based on the difference between the predicted sound effect adjustment parameters of the second sample music and the actual sound effect adjustment parameters of the second sample music, the projection module of the sound effect adjustment multimodal large model after the initial training is trained to obtain the trained sound effect adjustment multimodal large model.

[0014] In a third aspect, the present application further provides a sound effect adjustment device, comprising:

[0015] a target music acquisition module, configured to obtain the original audio of the target music in response to a sound effect adjustment request for the target music sent by the terminal;

[0016] an adjustment parameter acquisition module for inputting the original audio of the target music into a target encoder module of a trained sound effect adjustment multimodal large model to obtain audio features of the target music, inputting the audio features into a projection module of the sound effect adjustment multimodal large model, converting the audio features into music description semantic features of the target music; the music description semantic features are semantic features of a description text of the target music; and inputting the music description semantic features of the target music into a large language model module of the sound effect adjustment multimodal large model to obtain sound effect adjustment parameters of the target music;

[0017] The sound effect adjustment parameters of the target music are used to adjust the sound effects of the original audio of the target music to obtain the target audio of the target music, and the target audio data of the target music is used to be played on the terminal.

[0018] In a fourth aspect, the present application further provides a training device for a large multimodal model of sound effect adjustment, comprising:

[0019] a first semantic acquisition module, configured to input a first sample music into a target encoder module of a large multimodal model for sound effect adjustment to be trained, to obtain audio features of the first sample music, and to input the audio features of the first sample music into a projection module of the large multimodal model for sound effect adjustment to be trained, to obtain descriptive semantic features of the first sample music;

[0020] a first projection training module, configured to input the descriptive semantic features of the first sample music into the large language model module of the to-be-trained multimodal sound effect adjustment large model to obtain a predicted descriptive text of the first sample music, and train the projection module of the to-be-trained multimodal sound effect adjustment large model based on the difference between the predicted descriptive text of the first sample music and the actual descriptive text of the first sample music, so as to obtain an initially trained multimodal sound effect adjustment large model;

[0021] a second semantic acquisition module, configured to input the second sample music into the target encoder module of the initially trained sound effect adjustment multimodal large model to obtain audio features of the second sample music, and input the audio features of the second sample music into the projection module of the initially trained sound effect adjustment multimodal large model to obtain descriptive semantic features of the second sample music;

[0022] The second projection training module is used to input the descriptive semantic features of the second sample music into the large language model module of the sound effect adjustment multimodal large model after the initial training to obtain the predicted sound effect adjustment parameters of the second sample music, and train the projection module of the sound effect adjustment multimodal large model after the initial training based on the difference between the predicted sound effect adjustment parameters of the second sample music and the actual sound effect adjustment parameters of the second sample music to obtain the trained sound effect adjustment multimodal large model.

[0023] In a fifth aspect, the present application further provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of the method described in any one of the embodiments of the first aspect or the second aspect are implemented.

[0024] In a sixth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the method described in any one of the embodiments of the first aspect or the second aspect.

[0025] In a seventh aspect, the present application also provides a computer program product, comprising a computer program, which, when executed by a processor, implements the steps of the method described in any embodiment of the first aspect or the second aspect.

[0026] The above-mentioned sound effect adjustment method, training method, device, computer equipment, storage medium and computer program product of the sound effect adjustment multimodal large model, when a user initiates a sound effect adjustment request, can input the original audio of the target music that needs to adjust the sound effect into the sound effect adjustment multimodal large model, and the target encoder module of the sound effect adjustment multimodal large model obtains the audio features, and then uses the projection module to convert the audio features of the target music into music description semantic features, and then inputs the music description semantic features into the large language model module to obtain sound effect adjustment parameters to adjust the sound effect of the target music. In this way, the user does not need to manually select the sound effect adjustment method, and the adjustment parameters can be determined according to the target music, so the effect of the sound effect adjustment can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the following briefly introduces the drawings required for use in the embodiments or related technical descriptions. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0028] Figure 1 A diagram illustrating an application environment of a sound effect adjustment method according to an embodiment;

[0029] Figure 2 1 is a flow chart of a sound effect adjustment method according to an embodiment;

[0030] Figure 3 1 is a flow chart of obtaining target music after sound effect adjustment in one embodiment;

[0031] Figure 4 1 is a flow chart of a method for training a large multimodal model for sound effect adjustment in one embodiment;

[0032] Figure 5 A schematic diagram of an interactive interface for adjusting sound effects in one embodiment;

[0033] Figure 6 A schematic diagram of an interactive interface for adjusting sound effects in another embodiment;

[0034] Figure 7 Schematic diagram of interaction of a method for adjusting music sound effects in one embodiment;

[0035] Figure 8 is an interactive schematic diagram of a music sound effect adjustment method in another embodiment;

[0036] Figure 9 is an interactive schematic diagram of a music sound effect adjustment method in yet another embodiment;

[0037] Figure 10 Schematic diagram of the structure of a multimodal large model in one embodiment;

[0038] Figure 11 1 is a flow chart of obtaining tuned audio in one embodiment;

[0039] Figure 12 is a structural block diagram of a sound effect adjustment device in one embodiment;

[0040] Figure 13 A structural block diagram of a training device for a large multimodal model for sound effect adjustment according to one embodiment;

[0041] Figure 14 FIG. 1 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of this application more clear, the following further describes this application in detail with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain this application and are not intended to limit this application.

[0043] The sound effect adjustment method provided in the embodiment of the present application can be applied to Figure 1In the application environment shown, terminal 101 communicates with server 102 via a network. Specifically, when a user initiates a sound effect adjustment request through terminal 101, server 102 obtains the original audio of the target music to be adjusted. The original audio can then be input into the target encoder module of the trained sound effect adjustment multimodal large model. The target encoder module extracts the audio features of the target music and inputs the audio features into the projection module of the sound effect adjustment multimodal large model to convert them into music description semantic features of the target music. The music description semantic features are then input into the large language model module of the sound effect adjustment multimodal large model to obtain the sound effect adjustment parameters of the target music. The sound effect adjustment parameters are then used to adjust the sound effect of the target music to obtain the target audio after sound effect adjustment. Terminal 101 can be, but is not limited to, various personal computers, laptops, smartphones, tablets, IoT devices, and portable wearable devices. IoT devices can be smart speakers, smart TVs, smart air conditioners, smart car devices, etc. Server 102 can be implemented as a standalone server or a server cluster consisting of multiple servers.

[0044] In one embodiment, Figure 2 As shown, a sound effect adjustment method is provided, which is applied to Figure 1 Taking the server 102 in the example as an example, the following steps are included:

[0045] Step S201 , in response to a sound effect adjustment request for target music sent by terminal 101 , obtaining the original audio of the target music.

[0046] Among them, the target music refers to the music that needs to be adjusted for sound effects. The music can be a complete song or a music clip in the complete song. The target music can be sent by the terminal 101 to the server 102 when the user initiates a sound effect adjustment request for adjusting the sound effects. The sound effect adjustment request can be triggered by the user through the sound effect adjustment control displayed on the terminal. The sound effect adjustment request can be triggered by clicking the control. The original audio refers to the audio signal before the sound effect of the target music is adjusted.

[0047] Specifically, when a user needs to adjust the sound effects of a target music, the user can initiate a sound effect adjustment request for the target music through terminal 101, and server 102 can obtain the original audio of the target music based on the sound effect adjustment request. The original audio can be pre-stored in the server, or it can be sent to server 102 together with the sound effect adjustment request initiated by terminal 101.

[0048] Step S202: Inputting the original audio of the target music into the target encoder module of the trained sound effect adjustment multimodal large model to obtain audio features of the target music; inputting the audio features into the projection module of the sound effect adjustment multimodal large model to convert the audio features into music description semantic features of the target music; the music description semantic features are semantic features of the description text of the target music; and inputting the music description semantic features of the target music into the large language model module of the sound effect adjustment multimodal large model to obtain sound effect adjustment parameters of the target music;

[0049] The sound effect adjustment parameters of the target music are used to adjust the sound effect of the original audio of the target music to obtain the target audio of the target music, and the target audio data of the target music is used to be played on the terminal 101.

[0050] The trained sound effect adjustment multimodal large model refers to a pre-trained large language model for implementing the sound effect adjustment function for music. The large language model mainly includes the following three modules: a target encoder module, a projection module, and a large language model module. The encoder module is mainly used to extract the audio features of the target music. The audio features refer to the corresponding audio information features of the target music. The projection module is used to convert the audio features into music description semantic features. The music description semantic features refer to the textual semantic features of the description text of the target music. For example, the description text may include at least one of the description text related to the rhythm, the description text related to the emotion, and the description text related to the style of the target music. The textual semantic features are the semantic features of the at least one description text. The large language model module is a module for using the music description semantic features to output the sound effect adjustment parameters of the target music. The sound effect adjustment parameters can be used to indicate the sound effect adjustment method of the original audio of the target music, and obtain the target audio after sound effect adjustment according to the above sound effect adjustment method. The target audio can be played on the terminal 101, thereby achieving the sound effect adjustment of the target music.

[0051] Specifically, after the server 102 obtains the original audio of the target music, the original audio can be input into the trained sound effect adjustment multimodal large model, and the target encoder module of the sound effect adjustment multimodal large model extracts the audio features of the target music. Then, the music and text can be aligned through the projection module of the sound effect adjustment multimodal large model, that is, the audio features of the target music are converted into music description semantic features, and finally the music description semantic features are input into the large language model module of the sound effect adjustment multimodal large model to output sound effect adjustment parameters for adjusting the sound effects of the target music. The sound effect adjustment parameters can be used to adjust the sound effects of the original audio of the target music to obtain the target audio of the target music, that is, the sound effect adjustment multimodal large model can directly adjust the sound effects of the original audio of the target music according to the sound effect adjustment parameters to obtain the target audio of the target music, or the terminal 101 can adjust the original audio according to the sound effect adjustment parameters according to the sound effect adjustment multimodal large model to obtain the target audio of the target music.

[0052] In the above-mentioned embodiment of the sound effect adjustment method, when a user initiates a sound effect adjustment request, the original audio of the target music whose sound effect needs to be adjusted can be input into the sound effect adjustment multimodal large model, and the target encoder module of the sound effect adjustment multimodal large model can obtain audio features. The projection module can then be used to convert the audio features of the target music into music description semantic features, and then the music description semantic features can be input into the large language model module to obtain sound effect adjustment parameters to adjust the sound effect of the target music. In this way, the user does not need to manually select the sound effect adjustment method, and the adjustment parameters can be determined based on the target music, thereby improving the effect of the sound effect adjustment.

[0053] In one embodiment, Figure 3 As shown, step S202 may further include:

[0054] Step S301: Acquire the sound effect adjustment keywords of the target music according to the sound effect adjustment auxiliary text of the target music.

[0055] The sound effect adjustment auxiliary text is used to improve the sound effect adjustment parameters for the target music. It is text information input into the sound effect adjustment multimodal large model in addition to the original audio of the target music. In this embodiment, in addition to inputting the original audio of the target music into the sound effect adjustment multimodal large model, the server 102 can also additionally input text information for assisting in adjusting the original sound effects of the target music in order to further improve the accuracy of the sound effect adjustment, namely, the sound effect adjustment auxiliary text. For example, it can be the sound effect adjustment requirement text additionally input by the user, and the sound effect adjustment keyword refers to the keyword extracted from the sound effect adjustment auxiliary text that is related to the sound effect adjustment of the original audio of the target music. The keyword can be extracted by the keyword recognition module based on the sound effect adjustment auxiliary text by adding a keyword recognition module to the sound effect adjustment multimodal large model.

[0056] Specifically, in this embodiment, the sound effect adjustment parameter output of the target music will not only be affected by the target music itself, but also by the sound effect adjustment auxiliary text for the target music. Therefore, if the server receives the sound effect adjustment auxiliary text for the target music, it can further implement sound effect adjustment keyword extraction based on the sound effect adjustment auxiliary text, thereby obtaining the sound effect adjustment keywords for the target music.

[0057] Step S302: Inputting the music description semantic features and the sound effect adjustment keywords of the target music into the large language model module of the sound effect adjustment multimodal large model, so that the large language model module obtains the sound effect adjustment text features corresponding to the sound effect adjustment keywords, and then concatenates the music description semantic features with the sound effect adjustment text features to obtain comprehensive text features.

[0058] Step S303: forward propagating the comprehensive text features through the large language model module to obtain the sound effect adjustment parameters of the target music.

[0059] The sound effect adjustment text feature refers to the text feature vector corresponding to the sound effect adjustment keyword. In this embodiment, the input of the large language model module includes the following two parts: the music description semantic feature output by the projection module, and the sound effect adjustment keyword extracted based on the sound effect adjustment auxiliary text. The comprehensive text feature is a text feature that includes both the music description semantic feature and the sound effect adjustment text feature.

[0060] Specifically, after the server 102 obtains the sound effect adjustment keyword, it can input the sound effect adjustment keyword and the music description semantic features of the target music output by the projection module into the large language model module. The large language model module can first extract the text features of the sound effect adjustment keyword, and after obtaining the sound effect adjustment text features, it can then feature splice the music description semantic features with the extracted sound effect adjustment text features to obtain comprehensive text features. Finally, the comprehensive text features are forward propagated through the large language model module to output the sound effect adjustment parameters of the target music.

[0061] In this embodiment, the target music after sound effect adjustment can be obtained by obtaining the sound effect adjustment auxiliary text for the target music, and the sound effect adjustment keywords of the target music extracted from the sound effect adjustment auxiliary text are input into the large language model module in the sound effect adjustment multimodal large model together with the sound effect adjustment text features of the target music, thereby obtaining the target music after sound effect adjustment. In this way, the accuracy of the sound effect adjustment can be further improved.

[0062] Furthermore, before step S301, the method may also include: obtaining the sound effect adjustment auxiliary text of the target music sent by the terminal 101, where the sound effect adjustment auxiliary text is a descriptive text input by the user to the terminal 101 to reflect the user's demand for sound effect adjustment of the target music.

[0063] In this embodiment, the sound effect adjustment auxiliary text can be a sound effect adjustment demand text input by the user to the terminal 101. The text can reflect the user's sound effect adjustment demand for the target music. Specifically, when the user initiates a sound effect adjustment request for the target music through the terminal 101, the user can also enter a description text for representing the sound effect adjustment demand. The description text can be used as the sound effect adjustment auxiliary text and sent by the terminal 101 to the server 102. The server 102 can then receive the sound effect adjustment auxiliary text representing the user's sound effect adjustment demand for the target music. In this way, the server 102 can adjust the sound effect of the target music based on the target music and the sound effect adjustment demand input by the user for the target music, thereby obtaining the target music after sound effect adjustment.

[0064] For example, the sound effect adjustment auxiliary text input by the user may be: "Help me adjust the music to have more atmosphere", and the recognized sound effect adjustment keyword may be: "Enhance the sense of atmosphere", thereby using the sound effect adjustment keyword to assist in sound effect adjustment.

[0065] In this embodiment, when the user inputs sound effect adjustment auxiliary text for reflecting the user's sound effect adjustment needs for the target music, the server 102 can receive the above-mentioned sound effect adjustment auxiliary text sent by the terminal 101. In this way, the sound effect adjustment can be achieved according to the user's sound effect adjustment needs and the original audio of the target music. Therefore, the target music after sound effect adjustment can be more in line with user needs, thereby further improving the intelligence of the sound effect adjustment.

[0066] In addition, step S301 may also include: obtaining music playback scene information of the target music played on the terminal, the music playback scene information can reflect the playback environment characteristics of the target music when played on the terminal 101; converting the music playback scene information into sound effect adjustment auxiliary text of the target music.

[0067] Among them, the music playback scene information may refer to information related to the characteristics of the external playback environment when the target music is played on the terminal 101, such as the temperature information, weather information, and geographic location information corresponding to the external playback scene, etc. The music playback external scene information may be obtained by user input or obtained autonomously by the server 102. For example, the temperature information may be obtained by the server 102 by receiving the temperature information collected by the temperature sensor, and the weather information may be extracted by the server 102 from a database storing weather information, etc.

[0068] Afterwards, the server 102 can convert the above-mentioned music playback scene information into sound effect adjustment auxiliary text for assisting the output of sound effect adjustment parameters. For example, the music playback scene information may be: "The temperature is minus 5 degrees Celsius", and the sound effect adjustment auxiliary text obtained after conversion can be: "The temperature is low", and the sound effect adjustment keyword obtained based on the sound effect adjustment auxiliary text can be: "Enhance the warmth of the human voice", thereby using the sound effect adjustment keyword converted from the music playback scene information to assist in sound effect adjustment.

[0069] In this embodiment, the sound effect adjustment auxiliary text can also be obtained by converting the music playback scene information, that is, the sound effect adjustment parameters of the target music can be related to the playback environment when the target music is played on the terminal 101. Specifically, the server 102 can collect the music playback scene information of the target music, and then convert the music playback scene information into the sound effect adjustment auxiliary text, and input it into the large language model module together with the music description semantic features of the target music, so as to realize the sound effect adjustment of the target music.

[0070] In one embodiment, the sound effect adjustment method may also include: obtaining a sound effect adjustment feedback text generated by the sound effect adjustment multimodal large model for the target audio of the target music, wherein the sound effect adjustment feedback text is used to describe the sound effect adjustment effect of the target audio of the target music; and sending the sound effect adjustment feedback text to the terminal 101 so that the terminal 101 displays the sound effect adjustment feedback text.

[0071] The sound effect adjustment feedback text is a text used to describe the sound effect adjustment effect of the target audio after the target music sound effect is adjusted. The feedback text can also be generated by a sound effect adjustment multimodal large model. For example, the sound effect adjustment feedback text can include the sensory effect after the target music is sound-adjusted.

[0072] Specifically, in addition to realizing the sound effect adjustment function of the target music, the sound effect adjustment multimodal large model can also generate sound effect adjustment feedback text related to the target music after the sound effect adjustment, and then send the sound effect adjustment feedback text to the terminal 101 for display as a user feedback text for user feedback.

[0073] For example, when the user inputs the target music A and completes the sound effect adjustment, the output sound effect adjustment feedback text can be: "Under the large model sound effect processing, the outline of the instrument is clearer and the timbre is brighter, as if every note is shining, allowing you to better immerse yourself in the delicacy and breadth of the music", to characterize the effect of the sound effect adjustment. If the user also inputs the sound effect adjustment requirement text for the target music B: "Help me adjust the music to have more atmosphere", the output sound effect adjustment feedback text can also be: "Okay, it has been intelligently adjusted for you. The atmosphere of the song has been appropriately enhanced through the external surround effect, which will bring you a more immersive listening experience."

[0074] In this embodiment, in addition to being used to implement sound effect adjustment processing for target music, the multimodal large model of sound effect adjustment can also be used to generate sound effect adjustment feedback text for the sound effect adjustment effect of the target music after sound effect adjustment and send it to terminal 101 for display. In this way, user feedback function can be provided for sound effect adjustment, further enriching the diversity of sound effect adjustment functions.

[0075] In one embodiment, after step S202, the method may also include: sending the sound effect adjustment parameters of the target music to the terminal 101, so that the terminal 101 uses the sound effect adjustment parameters to adjust the sound effect of the original audio of the target music to obtain the target audio of the target music; or after the large language model module uses the sound effect adjustment parameters to adjust the sound effect of the original audio of the target music to obtain the target audio of the target music, the target audio of the target music is sent to the terminal 101.

[0076] In this embodiment, after obtaining the sound effect adjustment parameters, the sound effect adjustment parameters are used to adjust the original audio of the target music to generate the target audio. There are two ways to do this: the first way is that the server 102 outputs the sound effect adjustment parameters for the target music through the sound effect adjustment multimodal large model, and the client 101 then uses the sound effect adjustment parameters to perform sound effect adjustment processing, thereby obtaining the target audio after sound effect adjustment. The second way is that the server 102 directly uses the sound effect adjustment parameters through the sound effect adjustment multimodal large model to complete the sound effect adjustment for the target music, thereby generating the target audio after sound effect adjustment.

[0077] Specifically, if sound effect adjustment processing is implemented using method 1, server 102 may obtain the sound effect adjustment parameters. Then, it may return the sound effect adjustment parameters to terminal 101. Terminal 101 may then perform sound effect adjustment processing on the original audio of the target music through a standard digital signal processing process, namely, a digital signal processing link (DSP link), thereby obtaining the target music after sound effect adjustment. This processing method can effectively reduce the computational workload of the large language model module, thereby improving the efficiency of sound effect adjustment.

[0078] If the sound effect adjustment process is implemented by the second method, the server 102 can directly use the sound effect adjustment parameters to implement the sound effect adjustment process on the original audio of the target music through the large language model module to obtain the target audio. This processing method can better optimize the processing effect of the sound effect adjustment.

[0079] In this embodiment, there may be multiple ways to obtain the target audio after sound effect adjustment. The terminal 101 may utilize a digital signal processing link to perform sound effect adjustment processing on the target music according to the sound effect adjustment parameters to obtain the target music after sound effect adjustment. Alternatively, the target audio after sound effect adjustment may be directly generated through a large language model module. This embodiment can enrich the diversity of sound effect adjustment methods.

[0080] In one embodiment, the original audio of the target music includes: the original audio of at least one verse segment of the target music and the original audio of at least one chorus segment of the target music; the sound effect adjustment parameters of the target music include: the sound effect adjustment parameters of each verse segment in at least one verse segment and the sound effect adjustment parameters of each chorus segment in at least one chorus segment; wherein the sound effect adjustment parameters of each verse segment are respectively used to perform sound effect adjustment on the original audio of each verse segment to obtain the target audio of each verse segment, and the sound effect adjustment parameters of each chorus segment are respectively used to perform sound effect adjustment on the original audio of each chorus segment to obtain the target audio of each chorus segment.

[0081] The verse and chorus segments of the target music can be extracted by identifying them through chorus point detection. In this embodiment, since the verse and chorus of the same song may give different feelings, the sound effect adjustment effect can be further improved by applying different sound effect adjustment parameters to different segments of the complete music song.

[0082] Specifically, the server 102 can first use the chorus point detection technology to identify the verse segment and the chorus segment of the target music, and then input the original audio of each verse segment and the chorus segment as the original audio of the target music into the sound effect adjustment multimodal large model. Therefore, the sound effect adjustment parameters output by the sound effect adjustment multimodal large model can also be the sound effect adjustment parameters corresponding to each verse segment, and the sound effect adjustment parameters corresponding to each chorus segment, and the sound effect adjustment parameters of each verse segment can be used to adjust the original audio of each verse segment to obtain the target audio of the corresponding verse segment, and the sound effect adjustment parameters of each chorus segment can be used to adjust the original audio of each chorus segment to obtain the target audio of the corresponding chorus segment. In this way, the sound effect adjustment of each verse segment and the chorus segment is achieved, thereby further improving the effect of the sound effect adjustment.

[0083] In this embodiment, the server 102 can also use the original audio of each main song segment and chorus segment as the original audio of the target music, so as to adjust the sound effects of each main song segment and chorus segment separately, thereby further improving the effect of the sound effect adjustment.

[0084] In one embodiment, Figure 4 As shown in the figure, a training method for a large multimodal model of sound effect adjustment is provided, and the method is applied to Figure 1 Taking the server 102 in the example as an example, the following steps are included:

[0085] Step S401: Input the first sample music into the target encoder module of the sound effect adjustment multimodal large model to be trained to obtain the audio features of the first sample music, and input the audio features of the first sample music into the projection module of the sound effect adjustment multimodal large model to be trained to obtain the descriptive semantic features of the first sample music.

[0086] The first sample music is sample music that carries actual description text as a label. The actual description text refers to the real music description text corresponding to the first sample music. The music description text can be used to describe the first sample music.

[0087] The target encoder module refers to the encoder module used in the multimodal large model of sound effect adjustment to be trained. The encoder module can be pre-trained and can be used to extract audio features from sample music. The projection module is the module in the multimodal large model of sound effect adjustment used to convert audio features into semantic features describing music, that is, the module used to align the semantics of music and text.

[0088] Specifically, the server 102 can input the first sample music into the target encoder module in the sound effect adjustment multimodal model to be trained, so that the target encoder module can obtain the audio features corresponding to the first sample music, and then the audio features can be further input into the projection module in the sound effect adjustment multimodal model to be trained to obtain the corresponding descriptive semantic features.

[0089] Step S402: input the descriptive semantic features of the first sample music into the large language model module of the sound effect adjustment multimodal large model to be trained to obtain the predicted description text of the first sample music, and train the projection module of the sound effect adjustment multimodal large model to be trained based on the difference between the predicted description text of the first sample music and the actual description text of the first sample music to obtain the sound effect adjustment multimodal large model after initial training.

[0090] The predicted description text is the predicted music description text for the first sample music output by the large language model module. Specifically, server 102 can input the descriptive semantic features of the first sample music into the large language model module in the to-be-trained multimodal sound effect adjustment large model. The large language model module then outputs the predicted description text corresponding to the first sample music. The difference between the predicted description text and the actual description text is then used to train the projection module in the to-be-trained multimodal sound effect adjustment large model, thereby obtaining an initially trained multimodal sound effect adjustment large model.

[0091] Step S403: Input the second sample music into the target encoder module of the sound effect adjustment multimodal large model after initial training to obtain the audio features of the second sample music, and input the audio features of the second sample music into the projection module of the sound effect adjustment multimodal large model after initial training to obtain the descriptive semantic features of the second sample music.

[0092] The second sample music refers to the sample music carrying actual sound effect adjustment parameters, and the actual sound effect adjustment parameters refer to the real sound effect adjustment parameters used to adjust the sound effects of the second sample music.

[0093] After completing the first stage of training for the sound effect adjustment multimodal large model and obtaining the initially trained sound effect adjustment multimodal large model, it is necessary to use the second sample music to perform the second stage of training on the sound effect adjustment multimodal large model, that is, by inputting the second sample music into the target encoder module of the initially trained sound effect adjustment multimodal large model to obtain the audio features of the second sample music, and further inputting the audio features of the second sample music into the projection module of the initially trained sound effect adjustment multimodal large model to obtain the descriptive semantic features of the second sample music.

[0094] Step S404: input the descriptive semantic features of the second sample music into the large language model module of the sound effect adjustment multimodal large model after initial training to obtain the predicted sound effect adjustment parameters of the second sample music; and train the projection module of the sound effect adjustment multimodal large model after initial training based on the difference between the predicted sound effect adjustment parameters of the second sample music and the actual sound effect adjustment parameters of the second sample music to obtain the trained sound effect adjustment multimodal large model.

[0095] The predicted sound effect adjustment parameters refer to the sound effect adjustment parameters for the second sample music output by the large language model module, and the trained sound effect adjustment multimodal large model refers to the sound effect adjustment multimodal large model for implementing sound effect adjustment that is finally obtained after completing the second stage of training. Specifically, the server 102 can input the descriptive semantic features of the second sample music into the large language model module in the sound effect adjustment multimodal large model after initial training, and the large language model module outputs the predicted sound effect adjustment parameters corresponding to the second sample music, thereby using the difference between the predicted sound effect adjustment parameters and the actual sound effect adjustment parameters to further train the projection module in the sound effect adjustment multimodal large model after initial training to obtain the trained sound effect adjustment multimodal large model.

[0096] In the above-mentioned training method of the multimodal large model for sound effect adjustment, the training of the projection module in the multimodal large model for sound effect adjustment is completed through the first sample music and the second sample music, thereby ensuring that the sound effect adjustment of music is performed using the multimodal large model for sound effect adjustment. Through the introduction of this model, the user does not need to manually select the sound effect adjustment method when adjusting the sound effect, but instead determines the sound effect adjustment method according to the target music, thereby improving the effect of the sound effect adjustment.

[0097] In addition, before step S402, it can also include: inputting the third sample music into the initial encoder module of the sound effect adjustment multimodal large model to be trained, obtaining the audio features of the third sample music, and generating predicted music based on the audio features of the third sample music; training the initial encoder module of the sound effect adjustment multimodal large model to be trained based on the difference between the predicted music and the third sample music to obtain the target encoder module of the sound effect adjustment multimodal large model to be trained.

[0098] In this embodiment, the target encoder module in the sound effect adjustment multimodal large model can also be obtained in advance through sample music training, and the third sample music is the sample music used to train the target encoder module, and the third sample music does not carry the corresponding label.

[0099] Specifically, server 102 may collect a third sample music and input the third sample music into an encoder module to be trained, i.e., an initial encoder module. The initial encoder module may then output audio features corresponding to the third sample music, i.e., third audio features. The third audio features may then be restored, for example, by a decoder to obtain predicted music. The difference between the predicted music and the third sample music is then used to train the initial encoder module, thereby obtaining a target encoder module.

[0100] In this embodiment, the target encoder module used in the sound effect adjustment multimodal large model can be obtained by training the initial encoder module using the third sample music. In this way, the accuracy of the target encoder module in extracting audio features can be improved.

[0101] In one embodiment, an interactive sound effect adjustment method based on a multimodal large model is also provided. The sound effect adjustment function currently provided is usually to pre-produce some sound effects (such as increasing the low frequency of 125Hz by 3dB), and then Figure 5 As shown in the figure, "perceptual effects" (such as wide surround, electronic sound effects, surface ACG, galactic symphony, ..., wide field surround, etc.) can be presented to users through the sound effect interface for selection. Or a more professional interface can be opened, such as adding the selection of sound effect characteristics (such as blues, classical, jazz, etc.), and adding settings such as fidelity, reverberation intensity, surround intensity, etc. Figure 6 shown.

[0102] However, the above adjustment method has the following disadvantages: Figure 5 The adjustment method provided does not allow users to directly perceive what the music will sound like after different sound effects are adjusted. Figure 6 The users may not know how to adjust the sound effects in the provided adjustment methods, and a single preset sound effect cannot be suitable for all songs. There will inevitably be a problem that after turning on the sound effect, the effect is good for some songs, but not for other songs.

[0103] Therefore, this embodiment provides an interactive sound effect adjustment method based on a multimodal large model. This method analyzes the song a user is currently listening to based on the multimodal large model and generates an appropriate sound effect solution to improve the listening experience. This method also supports conversational tuning solutions, with inputs including but not limited to user needs and external scenarios. It also employs segment-level effect adjustment to address the issue of poor sound effect solutions caused by emotional changes from the verse to the chorus, effectively improving the overall listening experience.

[0104] This embodiment can have the following three interaction implementation modes:

[0105] Method 1: If Figure 7 As shown, the input is music, and the output is the tuned music and user feedback. An example of user feedback is as follows: "With the large model sound effect processing, the instrument's outline is clearer and the timbre is brighter, as if every note shines, allowing you to better immerse yourself in the delicacy and breadth of the music."

[0106] Method 2: If Figure 8 As shown, the system inputs music and user requirements, and outputs the tuned music and user feedback. An example user requirement is: "Please adjust the music to have a better atmosphere." An example user feedback is: "Okay, I've intelligently adjusted it for you. The ambient sound of the song has been appropriately enhanced through the surround sound effects, which will give you a more immersive listening experience."

[0107] Method 3: If Figure 9 As shown, the input is music and external scene information, and the output is music with sound effects and user feedback. An example external scene is as follows: Location - Beijing, Temperature -5°C, Heavy Snow. Example user feedback is as follows: "I know it's snowing heavily outside today. The large model sound effect appropriately adjusts the atmosphere of the music, making the overall tone warmer. I hope to bring you a better and more comfortable listening experience."

[0108] The specific implementation process of this embodiment can be as follows:

[0109] 1. Training data preparation:

[0110] In order to achieve good performance of MM-LLM, a large amount of music and labels are required for training. The details are as follows:

[0111] (1) Massive unlabeled music data: The encoder is subsequently trained through self-supervision in order to effectively extract music features.

[0112] (2) 10,000+ labeled data: The goal is to align music content with higher-level semantics such as rhythm, emotion, and style. At the same time, at the model level, it can narrow the "distance" between similar music and better prepare for subsequent semantic association with tuning data. This data is relatively easy to obtain, as mainstream music platforms have a certain amount of data assets accumulated, and "pseudo-labels" can also be generated through some excellent open source models.

[0113] (3) 1000+ sound effect tuning data, used to let MM-LLM learn how to generate sound effect parameters. There are two options:

[0114] a. Manually mark how to adjust parameters such as frequency response and sound field for some typical songs;

[0115] b. Based on historical data on sound effects, identify the user’s most preferred sound effects for each song, and then use the corresponding parameters of these sound effects.

[0116] Note 1: To support user input and provide feedback, it is best to have some perception data corresponding to the sound effect tuning. This can be mined based on knowledge or user comments. The following is an example:

[0117] EffectFeelingMap={

[0118] "Low frequency":{

[0119] "enhancement":{

[0120] "Thickness": ["thick", "full", "roaring", "solid", "impact", "energy"],

[0121] "Fullness": ["sense of space", "envelopment", "solid", "rich"],

[0122] "strength": ["power", "tough", "hardcore", "hard"]

[0123] },

[0124] Note 2: If you need to provide separate tuning solutions for different devices (headphones, speakers), you can add some tuning parameters for the corresponding devices as additional training data. This way, MM-LLM can better combine the characteristics of the music and the current device to output sound effect parameters.

[0125] 2. Structure and training of multimodal large models:

[0126] The typical structure of a multimodal large model is as follows Figure 10As shown in the figure, the model can be composed of a modal encoder, an input projector, and the backbone of a large language model. The audio data can be first input into the modal encoder, and after using the modal encoder to extract features of the music, the music and text (such as rhythm, emotion, style, etc.) are aligned through the projection module, i.e., the input projector. Finally, the music and text information such as user input or external scenes are input into the backbone of the large language model to generate sound effect parameters and user feedback.

[0127] At the training level, large language models are generally pre-trained. The encoder can be trained using (1) of the above data, or using a pre-trained model, while the projection module is trained using the above data (2) and (3).

[0128] 3. Model reasoning and application:

[0129] In the inference stage, music information is Figure 10 The structure can be used for input. Given the high degree of freedom in user input, keywords can be extracted first, such as "enhanced atmosphere," and then passed to the LLM along with the music information after the Projector. For external scenes, the LLM can be used to perform some interpretation, such as "outside temperature -5 degrees Celsius" -> "enhanced vocal warmth." The music information after the Projector can be cached in the database because it is independent of user / external input.

[0130] After obtaining the audio parameters, such as Figure 11 As shown, audio can be processed through a standard DSP process to obtain tuned audio. This step can be performed on the client side. The client only needs to download the tuning parameters and user feedback output by the MM-LLM, thereby reducing network transmission requirements.

[0131] The above reasoning scheme describes using a single sound effect parameter for the entire song. However, considering that the verse and chorus of the same song may give people different feelings, further optimizing the sound effect parameters to use segments as units will achieve better results. The specific changes are as follows:

[0132] 1. The input of MM-LLM has also been changed from the whole song to fragments, and the acquisition of fragments is achieved through technologies such as chorus detection.

[0133] 2. When processing audio, it is necessary to consider the auditory continuity of adjacent segments and add appropriate smoothing.

[0134] 3. The reliance on training data is reduced, and annotation can be performed in units of fragments instead of the entire song.

[0135] In addition, the output of the multimodal large model is not limited to text form. The (sound effect parameters + DSP processing link) can be replaced with directly generated processed music. Although the amount of calculation will be larger and may not be real-time, the upper limit of the effect will be higher because it is no longer restricted to a limited number of mixing parameters. Correspondingly, the "sound effect tuning data" in the training data is also replaced with "optimized music content" to allow the model to automatically learn the difference between input and output audio. In addition to the above-mentioned sound effect parameters, parameters related to volume and sound quality can also be included. For example, in a car scene, when the car speed is relatively fast, the following interaction can be performed: "It is detected that you are driving relatively fast and the noise in the car is a little loud. The volume is intelligently increased for you, and the voice is clearer, so that you can better enjoy the beauty of music."

[0136] Through this embodiment, the sound effect capability can be made "thousands of songs and thousands of faces", which improves the user's happiness index when listening to music, and supports the interaction between users and external scenes, thereby improving the playability and sense of technology of the sound effect function.

[0137] It should be understood that, although the various steps in the flowcharts involved in the various embodiments described above are displayed in sequence according to the instructions of the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and these steps can be executed in other orders. Moreover, at least a portion of the steps in the flowcharts involved in the various embodiments described above can include multiple steps or multiple stages, and these steps or stages are not necessarily executed and completed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a portion of steps or stages in other steps.

[0138] Based on the same inventive concept, the embodiments of the present application also provide a sound effect adjustment device for implementing the above-mentioned sound effect adjustment method, and a sound effect adjustment multimodal large model training device for the sound effect adjustment multimodal large model training method. The implementation solution provided by the device is similar to the implementation solution recorded in the above-mentioned method, so the specific limitations in the embodiments of one or more sound effect adjustment devices and the sound effect adjustment multimodal large model training device provided below can be found in the above-mentioned limitations on the sound effect adjustment method and the sound effect adjustment multimodal large model training method, and will not be repeated here.

[0139] In one embodiment, Figure 12 As shown, a sound effect adjustment device is provided, including: a target music acquisition module 1201 and an adjustment parameter acquisition module 1202, wherein:

[0140] The target music acquisition module 1201 is configured to obtain the original audio of the target music in response to a sound effect adjustment request for the target music sent by the terminal;

[0141] The adjustment parameter acquisition module 1202 is configured to input the original audio of the target music into the target encoder module of the trained sound effect adjustment multimodal large model to obtain audio features of the target music, input the audio features into the projection module of the sound effect adjustment multimodal large model, convert the audio features into music description semantic features of the target music; the music description semantic features are semantic features of the description text of the target music; and input the music description semantic features of the target music into the large language model module of the sound effect adjustment multimodal large model to obtain sound effect adjustment parameters of the target music.

[0142] Among them, the sound effect adjustment parameters of the target music are used to adjust the sound effects of the original audio of the target music to obtain the target audio of the target music, and the target audio data of the target music is used to be played on the terminal.

[0143] In one embodiment, Figure 13 As shown, a training device for a large multimodal model of sound effect adjustment is provided, comprising: a first semantic acquisition module 1301, a first projection training module 1302, a second semantic acquisition module 1303 and a second projection training module 1304, wherein:

[0144] A first semantic acquisition module 1301 is configured to input the first sample music into a target encoder module of a large multimodal model for sound effect adjustment to be trained, thereby obtaining audio features of the first sample music, and to input the audio features of the first sample music into a projection module of the large multimodal model for sound effect adjustment to be trained, thereby obtaining descriptive semantic features of the first sample music;

[0145] A first projection training module 1302 is configured to input the descriptive semantic features of the first music sample into the large language model module of the multimodal large model of sound effect adjustment to be trained, obtain a predicted descriptive text of the first music sample, and train the projection module of the multimodal large model of sound effect adjustment to be trained based on the difference between the predicted descriptive text of the first music sample and the actual descriptive text of the first music sample, so as to obtain an initially trained multimodal large model of sound effect adjustment;

[0146] A second semantic acquisition module 1303 is configured to input the second sample music into the target encoder module of the initially trained sound effect adjustment multimodal large model to obtain audio features of the second sample music, and input the audio features of the second sample music into the projection module of the initially trained sound effect adjustment multimodal large model to obtain descriptive semantic features of the second sample music;

[0147] The second projection training module 1304 is used to input the descriptive semantic features of the second sample music into the large language model module of the sound effect adjustment multimodal large model after initial training to obtain the predicted sound effect adjustment parameters of the second sample music, and train the projection module of the sound effect adjustment multimodal large model after initial training based on the difference between the predicted sound effect adjustment parameters of the second sample music and the actual sound effect adjustment parameters of the second sample music to obtain the trained sound effect adjustment multimodal large model.

[0148] Each module in the aforementioned sound effect adjustment device and the training device for a large multimodal model of sound effect adjustment can be implemented in whole or in part through software, hardware, or a combination thereof. Each module can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.

[0149] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 14 As shown. The computer device includes a processor, a memory, an input / output interface (Input / Output, abbreviated as I / O) and a communication interface. The processor, memory and input / output interface are connected through a system bus, and the communication interface is connected to the system bus through the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The database of the computer device is used to store target music data. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, it implements a sound effect adjustment method or a training method for a large multimodal model of sound effect adjustment.

[0150] Those skilled in the art will understand that Figure 14 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0151] In one embodiment, a computer device is further provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.

[0152] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.

[0153] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.

[0154] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant regulations.

[0155] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the above-mentioned embodiments. In particular, any reference to memory, database, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The databases involved in the various embodiments provided herein may include at least one of a relational database and a non-relational database. Non-relational databases may include, but are not limited to, distributed databases based on blockchains. The processors involved in the various embodiments provided herein may be, but are not limited to, general-purpose processors, central processing units (CPUs), graphics processing units (GPUs), digital signal processors (DSPs), programmable logic devices (PLDs), data processing logic devices based on quantum computing, and the like.

[0156] The technical features of the above embodiments can be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0157] The above-described embodiments merely represent several implementation methods of the present application. While the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the present application. It should be noted that a person of ordinary skill in the art may make various modifications and improvements without departing from the spirit of the present application, and these modifications and improvements fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be determined by the appended claims.

Claims

1. A sound effect adjustment method, characterized in that: The method comprises: In response to a sound effect adjustment request for target music sent by a terminal, obtaining original audio of the target music; Inputting the original audio of the target music into the target encoder module of the trained sound effect adjustment multimodal large model to obtain audio features of the target music, inputting the audio features into the projection module of the sound effect adjustment multimodal large model, converting the audio features into music description semantic features of the target music; the music description semantic features are semantic features of the description text of the target music; and inputting the music description semantic features of the target music into the large language model module of the sound effect adjustment multimodal large model to obtain sound effect adjustment parameters of the target music; The sound effect adjustment parameters of the target music are used to adjust the sound effects of the original audio of the target music to obtain the target audio of the target music, and the target audio data of the target music is used to be played on the terminal.

2. The method according to claim 1, characterized in that The step of inputting the music description semantic features of the target music into the large language model module of the sound effect adjustment multimodal large model to obtain the sound effect adjustment parameters of the target music includes: Acquiring a sound effect adjustment keyword for the target music according to the sound effect adjustment auxiliary text for the target music; Inputting the music description semantic features of the target music and the sound effect adjustment keywords of the target music into the large language model module of the sound effect adjustment multimodal large model, so that the large language model module obtains the sound effect adjustment text features corresponding to the sound effect adjustment keywords, and splicing the music description semantic features with the sound effect adjustment text features to obtain comprehensive text features; The comprehensive text features are forward propagated through the large language model module to obtain the sound effect adjustment parameters of the target music.

3. The method according to claim 2, characterized in that Also includes: Acquire the sound effect adjustment auxiliary text of the target music sent by the terminal, where the sound effect adjustment auxiliary text is a descriptive text input by the user to the terminal to reflect the user's sound effect adjustment requirements for the target music.

4. The method according to claim 2, characterized in that Also includes: Acquire music playing scene information of the target music played on the terminal, where the music playing scene information can reflect the playing environment characteristics of the target music when played on the terminal; The music playing scene information is converted into sound effect adjustment auxiliary text of the target music.

5. The method according to claim 1, characterized in that The method further comprises: Obtaining a sound effect adjustment feedback text generated by the sound effect adjustment multimodal large model for the target audio of the target music, wherein the sound effect adjustment feedback text is used to describe the sound effect adjustment effect of the target audio of the target music; The sound effect adjustment feedback text is sent to the terminal, so that the terminal displays the sound effect adjustment feedback text.

6. The method according to claim 1, characterized in that After obtaining the sound effect adjustment parameters of the target music, the method further includes: Sending the sound effect adjustment parameters of the target music to the terminal, so that the terminal uses the sound effect adjustment parameters to adjust the sound effect of the original audio of the target music to obtain the target audio of the target music; or After the large language model module uses the sound effect adjustment parameters to perform sound effect adjustment on the original audio of the target music to obtain the target audio of the target music, the target audio of the target music is sent to the terminal.

7. The method according to claim 1, characterized in that The original audio of the target music includes: the original audio of at least one verse segment of the target music and the original audio of at least one chorus segment of the target music; The sound effect adjustment parameters of the target music include: sound effect adjustment parameters of each verse segment in the at least one verse segment, and sound effect adjustment parameters of each chorus segment in the at least one chorus segment; wherein the sound effect adjustment parameters of each verse segment are respectively used to perform sound effect adjustment on the original audio of each verse segment to obtain the target audio of each verse segment, and the sound effect adjustment parameters of each chorus segment are respectively used to perform sound effect adjustment on the original audio of each chorus segment to obtain the target audio of each chorus segment.

8. A method for training a large multimodal model for sound effect adjustment, characterized in that: The method comprises: Inputting the first sample music into a target encoder module of a to-be-trained multimodal model for sound effect adjustment to obtain audio features of the first sample music, and inputting the audio features of the first sample music into a projection module of the to-be-trained multimodal model for sound effect adjustment to obtain descriptive semantic features of the first sample music; Inputting the descriptive semantic features of the first sample music into the large language model module of the to-be-trained multimodal large model for sound effect adjustment to obtain a predicted descriptive text of the first sample music, and training the projection module of the to-be-trained multimodal large model for sound effect adjustment based on the difference between the predicted descriptive text of the first sample music and the actual descriptive text of the first sample music to obtain an initially trained multimodal large model for sound effect adjustment; Inputting the second sample music into the target encoder module of the initially trained sound effect adjustment multimodal large model to obtain audio features of the second sample music, and inputting the audio features of the second sample music into the projection module of the initially trained sound effect adjustment multimodal large model to obtain descriptive semantic features of the second sample music; The descriptive semantic features of the second sample music are input into the large language model module of the sound effect adjustment multimodal large model after the initial training to obtain the predicted sound effect adjustment parameters of the second sample music, and based on the difference between the predicted sound effect adjustment parameters of the second sample music and the actual sound effect adjustment parameters of the second sample music, the projection module of the sound effect adjustment multimodal large model after the initial training is trained to obtain the trained sound effect adjustment multimodal large model.

9. The method according to claim 8, characterized in that Before inputting the first sample music into the target encoder module of the sound effect adjustment multimodal large model to be trained, the method further includes: Inputting the third sample music into the initial encoder module of the multimodal large model of sound effect adjustment to be trained, obtaining audio features of the third sample music, and generating predicted music based on the audio features of the third sample music; The initial encoder module of the sound effect adjustment multimodal large model to be trained is trained according to the difference between the predicted music and the third sample music to obtain the target encoder module of the sound effect adjustment multimodal large model to be trained.

10. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 9 are implemented.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

12. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 9 are implemented.

Images