Audio and sound effect processing method and device, computer equipment and storage medium

Abstract

The invention discloses an audio and sound effect processing method and device, computer equipment and a storage medium. The audio and sound effect processing method comprises the steps of receiving a target audio; processing the target audio to obtain M * N audio paragraphs of the target audio, M * N paragraph information of the target audio and global semantic features of the target audio; performing feature extraction on the M * N audio paragraphs of the target audio to obtain target dimension features of the M * N audio paragraphs; integrating the global semantic feature of the target audio, the target dimension features of the M * N audio paragraphs and the M * N paragraph information of the target audio to obtain a target feature of the target audio; and determining a target sound effect template based on the target feature of the target audio and the initial sound effect template, so as to adjust the target sound effect through the target sound effect template. The content of each audio paragraph is deeply analyzed, so that the processed audio accurately conveys deep information such as emotion and theme expressed by the processed audio, and the audio sound effect processing effect is further improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and more specifically, to an audio sound effect processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] As user needs become more personalized, audio and sound processing technology is also constantly innovating. It is mainly used in fields such as music production, movies, games, and communications.

[0003] Currently, audio effect processing can process audio through system preset sound effect parameters to meet user needs, or use simple audio feature analysis, such as volume, frequency distribution, etc., to automatically adjust the sound effects.

[0004] However, the above methods lack in-depth analysis of the audio content, resulting in the processed audio being unable to accurately convey the deep information such as the emotions and themes it expresses, resulting in poor audio sound effect processing. Summary of the Invention

[0005] The main purpose of this application is to provide an audio sound effect processing method, device, computer equipment and storage medium, which can deeply analyze the audio content so that the processed audio can accurately convey the deep information such as the emotions and themes it expresses, thereby improving the audio sound effect processing effect.

[0006] To achieve the above objectives, in a first aspect, the present application provides an audio effect processing method, which is applied to a server, comprising:

[0007] Receive target audio;

[0008] Processing the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and global semantic features of the target audio, where M and N are integers greater than 1, and the M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio;

[0009] Perform feature extraction on the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments;

[0010] Integrate the global semantic features of the target audio, the target dimension features of the M*N audio segments, and the M*N segment information of the target audio to obtain the target features of the target audio;

[0011] Based on the target features of the target audio and the initial sound effect template, a target sound effect template is determined, so as to adjust the target sound effect through the target sound effect template.

[0012] In one embodiment, the target audio is processed to obtain an audio segment of the target audio, segment information of the target audio, and a global semantic feature of the target audio, including:

[0013] Performing structural analysis on the target audio to obtain the audio segments and segment information of the target audio;

[0014] Feature extraction is performed on the target audio to obtain the global semantic features of the target audio.

[0015] In one embodiment, a structural analysis is performed on the target audio to obtain M*N audio segments of the target audio and M*N segment information of the target audio, including:

[0016] Perform track splitting on the target audio to obtain M audio tracks corresponding to the target audio;

[0017] For each audio track in the M audio tracks corresponding to the target audio, divide each audio track into sections to obtain N audio sections corresponding to each audio track;

[0018] Summarize the N audio segments corresponding to each audio track to obtain M*N audio segments;

[0019] For each audio segment in the M*N audio segments, the segment information of each audio segment is obtained, and the segment information of each audio segment is summarized to obtain M*N segment information.

[0020] In one embodiment, feature extraction is performed on the target audio to obtain the global semantic features of the target audio, including:

[0021] Get the preset multimodal large model;

[0022] The target audio is subjected to feature extraction through a preset multimodal large model to obtain the global semantic features of the target audio.

[0023] In one embodiment, feature extraction is performed on the target audio using a preset multimodal large model to obtain global semantic features of the target audio, including:

[0024] Constructing a first prompt word, wherein the first prompt word is used to guide a preset multimodal large model to output a global semantic feature of the target audio;

[0025] Integrate the target audio and audio text to obtain input data;

[0026] The input data and the first prompt word are input into a preset multimodal large model to obtain the global semantic features of the target audio.

[0027] In one embodiment, the target dimension features include at least music features and speech features;

[0028] Perform feature extraction on the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments, including:

[0029] Acquire a target dimension, wherein the target dimension includes at least music and speech;

[0030] Feature extraction is performed on the M*N audio segments according to music and speech respectively, to obtain music features and speech features of the M*N audio segments.

[0031] In one embodiment, feature extraction is performed on M*N audio segments according to music and speech, respectively, to obtain music features and speech features of the M*N audio segments, including:

[0032] For each audio segment in the M*N audio segments, extract music features and speech features from each audio segment to obtain music features and speech features of each audio segment;

[0033] The music features and speech features of each audio segment are summarized to obtain the music features and speech features of M*N audio segments.

[0034] In one embodiment, based on the target features of the target audio and the initial sound effect template, a target sound effect template is determined to adjust the target sound effect using the target sound effect template, including:

[0035] Obtaining an initial sound effect template, wherein the initial sound effect template matches the global semantic features of the target audio;

[0036] Based on the target features of the target audio and the initial sound effect template, a second prompt word is constructed to guide the preset multimodal agent to output the target sound effect template;

[0037] The second prompt word is input into a preset multimodal intelligent agent, and a target sound effect template is output to adjust the target sound effect through the target sound effect template, wherein the target sound effect template includes at least a sound effect template for each audio segment in the M*N audio segments.

[0038] In a second aspect, an embodiment of the present application provides an audio effect processing method, which is applied to a client, comprising:

[0039] Receive a target sound effect template;

[0040] In response to the user inputting target audio and target text into a preset multimodal intelligent agent, a sound effect adjustment template is obtained, wherein the target text is used to describe the user's needs;

[0041] The target sound effect template and the sound effect adjustment template are superimposed to obtain an adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template.

[0042] In a third aspect, an embodiment of the present application provides an audio and sound effect processing device, comprising:

[0043] A first receiving module, configured to receive target audio;

[0044] an audio processing module, configured to process the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and a global semantic feature of the target audio, wherein M and N are integers greater than 1, and wherein the M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio;

[0045] A feature extraction module is used to extract features from the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments;

[0046] A feature integration module is used to integrate the global semantic features of the target audio, the target dimensional features of the M*N audio segments, and the M*N segment information of the target audio to obtain the target features of the target audio;

[0047] The first adjustment module is used to determine a target sound effect template based on the target feature of the target audio and the initial sound effect template, so as to adjust the target sound effect through the target sound effect template.

[0048] In a fourth aspect, an embodiment of the present application provides an audio and sound effect processing device, comprising:

[0049] A second receiving module is used to receive target audio and target sound effect template;

[0050] A template output module is configured to obtain a sound effect adjustment template in response to a user inputting a target audio and a target text into a preset multimodal agent, wherein the target text is used to describe the user's needs;

[0051] The second adjustment module is used to superimpose the target sound effect template and the sound effect adjustment template to obtain an adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template.

[0052] In a fifth aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of any of the above methods when executing the computer program.

[0053] In a sixth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the steps of any of the above methods are implemented.

[0054] In a seventh aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements the steps of any of the above methods when executed by a processor.

[0055] An embodiment of the present application provides an audio sound effect processing method, apparatus, computer equipment and storage medium, including: first receiving target audio, then processing the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio and global semantic features of the target audio, wherein M and N are integers greater than 1, wherein the M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio, then performing feature extraction on the M*N audio segments of the target audio to obtain target dimensional features of the M*N audio segments, and then integrating the global semantic features of the target audio, the target dimensional features of the M*N audio segments and the M*N segment information of the target audio to obtain target features of the target audio, thereby determining a target sound effect template based on the target features of the target audio and an initial sound effect template, so as to adjust the target sound effect through the target sound effect template. This application processes the target audio to obtain global semantic features to deeply analyze the audio content, and integrates the global semantic features with the target dimensional features and paragraph information of the audio paragraphs to deeply analyze the content of each audio paragraph, so that the processed audio can accurately convey the deep information such as the emotions and themes it expresses, thereby improving the audio sound effect processing effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0056] The drawings that constitute part of this application are used to provide a further understanding of this application and make other features, objects and advantages of this application more apparent. The illustrative embodiment drawings of this application and their descriptions are used to explain this application and do not constitute an improper limitation of this application. In the drawings:

[0057] Figure 1 This is a structural diagram of an audio and sound effect processing system provided in an embodiment of the present application;

[0058] Figure 2 This is an application scenario diagram of an audio sound effect processing method provided by an embodiment of the present application;

[0059] Figure 3 This is a flowchart of an audio effect processing method provided by an embodiment of the present application;

[0060] Figure 4 Schematic diagram of paragraph information of an audio paragraph provided in an embodiment of the present application;

[0061] Figure 5 This is a schematic diagram of the effector combination of the basic sound effect template provided in an embodiment of the present application;

[0062] Figure 6 This is a flowchart of another audio effect processing method provided by an embodiment of the present application;

[0063] Figure 7 This is a structural diagram of an audio and sound effect processing device provided in an embodiment of the present application;

[0064] Figure 8 This is a structural diagram of another audio and sound effect processing device provided in an embodiment of the present application;

[0065] Figure 9 It is a schematic diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0067] The terms "first," "second," "third," "fourth," and so on (if any) in the specification and claims of this application and the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or sequential sequence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the application described herein can be implemented in orders other than those illustrated or described herein.

[0068] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program that has a predetermined function and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0069] It should be understood that in the various embodiments of the present application, the size of the serial number of each process does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0070] It should be understood that in this application, "include" and "have" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or apparatus that includes a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products or apparatuses.

[0071] It should be understood that in this application, "multiple" means two or more. "And / or" is merely a description of the association relationship of associated objects, indicating that three relationships can exist. For example, and / or B can represent three situations: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the previous and next associated objects are in an "or" relationship. "Contains A, B and C", "Contains A, B, C" means that A, B, and C are all included, "Contains A, B or C" means that one of A, B, and C is included, and "Contains A, B and / or C" means that any one, any two, or any three of A, B, and C are included.

[0072] It should be understood that, in this application, "B corresponding to A," "B corresponding to A," "A corresponds to B," or "B corresponds to A" means that B is associated with A and B can be determined based on A. Determining B based on A does not mean determining B based solely on A; B can also be determined based on A and / or other information. A and B match when the similarity between A and B is greater than or equal to a preset threshold.

[0073] Depending on the context, "if" as used herein may be interpreted as "when" or "when" or "in response to determining" or "in response to detecting."

[0074] The data involved in this application may be data authorized by the tester or fully authorized by all parties. The collection, dissemination, and use of the data shall comply with the relevant laws, regulations, and standards of the relevant countries and regions. The implementation methods / examples of this application may be combined with each other.

[0075] The following specific embodiments are used to describe the technical solution of the present application in detail. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments.

[0076] First, to facilitate understanding of the solution of this application, the terms in this solution are explained as follows:

[0077] LLM (Large Language Model): used for natural language interaction and semantic analysis, such as GPT-4 and DeepSeek.

[0078] MM-LM (Multi-Modal Large Language Model): supports multimodal inputs such as audio, text, and images, such as QwenAudiod.

[0079] Graphic equalizer: An audio effects unit used to adjust the frequency response of an audio signal.

[0080] Compressor: An audio effect, dynamic range control, used to balance the dynamic range of an audio signal.

[0081] Reverb: An audio effect that provides rich spatial processing.

[0082] Delay: An audio effect that provides an echo effect.

[0083] Next, the present application solution will be described through specific embodiments with reference to the accompanying drawings.

[0084] As user needs become more personalized, audio and sound processing technology is also constantly innovating. It is mainly used in fields such as music production, movies, games, and communications.

[0085] Currently, audio effects processing can be tailored to user needs using system-preset sound effect parameters. For example, some music playback software offers preset sound effect parameter configurations based on musical genres. For example, pop music might enhance the bass to make the rhythm stand out, while classical music might emphasize the mid- and high-frequency parts, allowing users to hear the instrument's timbre more clearly.

[0086] Alternatively, simple audio feature analysis, such as volume and frequency distribution, can be used to automatically adjust the sound effects, such as automatically reducing background noise in quieter music passages.

[0087] These methods lack the depth and breadth of understanding music. Relying solely on genre or simple audio features for sound processing, they fail to accurately grasp the deeper semantic information conveyed by the music, such as emotions and themes. For example, a melancholic pop song processed using common sound effects presets may fail to highlight its emotional expression, resulting in the sound effects appearing flat and inconsistent with the actual emotion of the music.

[0088] Furthermore, the analysis of musical structure isn't detailed enough, and the varying sound effects requirements of different sections aren't fully considered, making it impossible to dynamically adjust the sound effects. For example, for a song with multiple sections, including an intro, verse, chorus, and outro, the aforementioned sound effects processing methods struggle to achieve smooth transitions and personalized adaptation of sound effect parameters between sections, resulting in a less coherent and unnatural sound throughout the song.

[0089] Furthermore, there are limitations in user interaction. Most of the aforementioned sound effect adjustments are automatically performed by the system, leaving users with limited preset options to choose from. Users cannot intuitively express their preferences for sound effects through natural language or other means, lacking the flexibility and convenience of personalized customization.

[0090] In summary, the above methods are relatively simple, and most of them can only adjust the sound effects based on the type of music or simple audio features. They lack in-depth analysis of the audio content, resulting in the processed audio being unable to accurately convey the deep information such as the emotions and themes it expresses, resulting in poor audio sound effect processing.

[0091] To solve the above problems, this application proposes an audio sound effect processing method.

[0092] See also Figure 1 , Figure 1 A schematic diagram of the structure of an audio and sound effect processing system provided in an embodiment of the present application.

[0093] The audio sound effect processing system of the present application includes a server and a client, wherein the server is used to comprehensively analyze the music and generate a personalized sound effect configuration. This sound effect configuration generally refers to an adjusted sound effect template (i.e., a target sound effect template), and the audio can be adjusted by the sound effect parameters in the target sound effect template. The client is used to receive the audio file (i.e., target audio) and the target sound effect template sent from the server when the user listens to audio, such as listening to songs, so that the user can generate an adjusted sound effect template by conducting a natural language dialogue with the sound effect interaction Agent module, and then superimpose the adjusted sound effect template with the target sound effect template to achieve the adjustment of the sound effect parameters, so as to provide users with a personalized sound effect experience.

[0094] The server includes a basic template preset module, a music structure analysis module, a global semantic feature extraction module, a paragraph signal feature extraction module, and a sound effect agent module, while the client includes a sound effect interaction module.

[0095] Basic Template Presets: This module provides a complete audio processing toolchain, encompassing multiple effects from graphic equalizers to compressors. Its modular design supports independent configuration and flexible adjustment, and provides foundational sound effect templates for different musical genres. The basic sound effect targets are configured with sound effect parameters.

[0096] Music structure analysis module: Uses deep learning technology to perform structural analysis on audio, accurately identifying and dividing five basic audio segments, including prelude, verse, and chorus.

[0097] Global semantic feature extraction module: With the help of a large multimodal language model, audio and text information are integrated to extract the global semantic features of the song, including key information such as style, emotion, theme, and instrument.

[0098] Paragraph signal feature extraction module: Performs multi-dimensional feature analysis on each audio paragraph, extracts features from three aspects: audio, speech, and music structure, and comprehensively analyzes the characteristics of the audio paragraph.

[0099] Sound Effect Agent Module: Based on large model technology, it is responsible for generating personalized sound effect configurations based on music characteristics and user needs.

[0100] Sound effect interaction module: Utilizes a large language model to enable natural language interaction between users and clients. Users can describe their desired sound effects through natural language, and the sound effect parameters are analyzed and adjusted in real time to meet their personalized needs.

[0101] The system utilizes a layered and progressive approach to sound effects processing, starting with pre-setting basic sound effect templates, global optimization, and then segment-level refinement, ultimately enabling personalized customization through natural language interaction. This architectural design enables deep audio understanding, precise sound effect adaptation, and convenient user interaction.

[0102] See also Figure 2 , Figure 2 A schematic diagram of an application scenario of an audio sound effect processing method provided in an embodiment of the present application.

[0103] The server 202 communicates with the client 204. The server 202 receives the target audio and then processes the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and global semantic features of the target audio, where M and N are integers greater than 1, and the M*N audio segments of the target audio correspond one-to-one with the M*N segment information of the target audio. Feature extraction is then performed on the M*N audio segments of the target audio to obtain target dimensional features of the M*N audio segments. The global semantic features of the target audio, the target dimensional features of the M*N audio segments, and the M*N segment information of the target audio are then integrated to obtain target features of the target audio. A target sound effect template is then determined based on the target features of the target audio and the initial sound effect template, so as to adjust the target sound effect using the target sound effect template.

[0104] The client 204 receives the target audio and target sound effect template sent by the server 202, and then inputs the target audio and target text into the preset multimodal intelligent agent, and outputs the sound effect adjustment template, wherein the target text is used to describe user needs, thereby superimposing the target sound effect template and the sound effect adjustment template to obtain the adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template.

[0105] The client 204 and the server 202 can communicate with each other through any communication method, including but not limited to network communication, and the above-mentioned network can include but not limited to: wired network, wireless network, wherein the wired network includes: local area network, metropolitan area network and wide area network, and the wireless network includes: Bluetooth, WIFI, and other networks that realize wireless communication. The client 204 includes but is not limited to at least one of the following: mobile phone (such as Android phone, iOS phone, etc.), laptop computer, tablet computer, PDA, mobile Internet device (Mobile Internet Device, MID), PAD, desktop computer, smart TV, etc. The server 202 can be an on-site server or a remote server, wherein both the on-site server and the remote server can be implemented as independent servers or a service cluster composed of multiple servers. The above is only an example, and no limitation is made to this in this embodiment.

[0106] See also Figure 3 , Figure 3 This is a flow chart of an audio effect processing method provided in an embodiment of the present application. Figure 3 As shown, this method is applied to Figure 2 The server 202 shown includes the following steps:

[0107] Step S301: receiving target audio;

[0108] Step S302: Process the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and global semantic features of the target audio.

[0109] The target audio is any audio, such as music audio, speech audio, synthesized audio, etc.

[0110] The M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio, where M and N are integers greater than 1.

[0111] In one embodiment, the target audio is processed to obtain the audio segments of the target audio, the segment information of the target audio, and the global semantic features of the target audio, including: first performing a structural analysis on the target audio to obtain the audio segments of the target audio and the segment information of the target audio, and then performing feature extraction on the target audio to obtain the global semantic features of the target audio.

[0112] The method includes performing structural analysis on the target audio to obtain M*N audio segments and M*N segment information of the target audio, including: performing track splitting processing on the target audio to obtain M audio tracks corresponding to the target audio; dividing each audio track into segments for each of the M audio tracks corresponding to the target audio to obtain N audio segments corresponding to each audio track; aggregating the N audio segments corresponding to each audio track to obtain M*N audio segments; obtaining segment information of each audio segment in the M*N audio segments, and aggregating the segment information of each audio segment to obtain M*N segment information.

[0113] For example, assuming that the target audio is music audio, M=4, N=4, the music audio is firstly divided into four tracks, which can be accurately divided into drums, bass, vocals and other instruments.

[0114] Since Demucs is an audio source separation model based on deep learning, it can learn the characteristic patterns of different instruments and sounds in audio, thereby achieving accurate separation. Therefore, the track separation processing of this application can use a pre-trained Demucs model. The Demucs model uses its powerful audio source separation capabilities to decompose the input music audio file into four independent tracks: drums, bass, vocals, and other instruments. In actual operation, the music audio file is input into the Demucs model. After learning from a large amount of training data, the model can automatically identify and separate these four main tracks and output them as independent music track files.

[0115] Then, each of the above music tracks is divided into sections. For example, the bass part is divided into four audio sections: prelude, verse, chorus, interlude and outing. Other music tracks are also divided according to the above section division method, which will not be repeated here.

[0116] The open-source All-In-One tool can be used to segment each music track. This tool provides efficient music structure analysis and can accurately identify sections such as the prelude, verse, chorus, interlude, and outro in the audio. It is based on a deep learning model and has been trained with a large amount of music data to understand the characteristics and patterns of different music sections. Input the split master audio or other comprehensive audio tracks into the All-In-One tool, and the tool will output the following: Figure 4 The segment information of the music segment shown includes the start time (start_time), end time (end_time) and segment label (label) of each segment.

[0117] In one embodiment, feature extraction is performed on the target audio to obtain global semantic features of the target audio, including: first obtaining a preset multimodal large model, and then performing feature extraction on the target audio through the preset multimodal large model to obtain global semantic features of the target audio.

[0118] Among them, the target audio is feature extracted through a preset multimodal large model to obtain the global semantic features of the target audio, including: constructing a first prompt word, wherein the first prompt word is used to guide the preset multimodal large model to output the global semantic features of the target audio; integrating the target audio and the audio text to obtain input data; inputting the input data and the first prompt word into the preset multimodal large model to obtain the global semantic features of the target audio.

[0119] Among them, the preset multimodal large models include but are not limited to Qwen-Audio model, Step-Audio model, etc.

[0120] For example, because the Qwen-Audio model can process both audio and text input data simultaneously, and has learned the complex relationships between audio and text data through large-scale pre-training, it has excellent semantic understanding capabilities and can provide in-depth and accurate semantic feature extraction services for audio analysis. Therefore, this application selects the Qwen-Audio model, an advanced multimodal large language model, to achieve the required semantic feature extraction.

[0121] First, it is necessary to set a clear role and task for the Qwen-Audio model, namely "professional music analysis assistant", and construct a special first prompt to guide the Qwen-Audio model to output global semantic features of music audio that meet the requirements.

[0122] Specifically,

[0123] Role (Description): "You are a professional music analysis assistant, skilled at extracting key semantic features from music audio and lyrics. Your task is to analyze key information such as the genre, emotion, rhythm, and main instrumentation of the music based on the input music information, and present it in a clear and concise manner."

[0124] Output format requirements: "Please output your analysis results in the following format:

[0125] Music style: [Style type, such as pop, rock, classical, etc.]

[0126] Emotional expression: [emotional description, such as joy, sadness, excitement, etc.]

[0127] Rhythm content: [rhythm overview, such as fast, slow, etc.]

[0128] Primary Instrument: [list of instruments, e.g. guitar, piano, drums, etc.]"

[0129] Input data: The audio to be analyzed, such as music audio (denoted as audio) and music audio text, such as lyrics text (denoted as lyrics) are used as input data.

[0130] The first clue is: "Analyze the song {audio}, its lyrics are {lyrics}."

[0131] Then, the audio and lyrics information are integrated and input into the QwenAudio model to start the music semantic feature analysis. The QwenAudio model generates global semantic features about the music audio based on the constructed first prompt word and input data. It is presented in a preset format and directly reflects the core semantic information of the music audio.

[0132] The global semantic feature extraction of the embodiment of the present application focuses on in-depth analysis of music audio and its lyrics information, and accurately extracts the global semantic features of music. These features cover key elements such as music style (such as pop, rock, classical, etc.), emotional expression (such as cheerful, sad, passionate, etc.), theme content (such as love, nature, inspirational, etc.) and main instruments (such as guitar, piano, drums, etc.). Through the extraction of these semantic features, detailed semantic guidance can be provided for subsequent sound effect processing, so that sound effect optimization can be closely aligned with the content and emotional expression of the music, thereby significantly enhancing the appeal and expressiveness of the music and bringing a more immersive listening experience to users.

[0133] Step S303: performing feature extraction on the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments.

[0134] The target dimension features include at least music features and speech features.

[0135] In one embodiment, feature extraction is performed on M*N audio segments of target audio to obtain target dimension features of the M*N audio segments. The target dimension must first be obtained, where the target dimension includes at least music and speech. Then, feature extraction is performed on the M*N audio segments according to music and speech respectively to obtain music features and speech features of the M*N audio segments.

[0136] The method further comprises the following steps: performing feature extraction on the M*N audio segments according to music and speech respectively to obtain music features and speech features of the M*N audio segments, including: performing music feature extraction and speech feature extraction on each audio segment in the M*N audio segments to obtain music features and speech features of each audio segment; and aggregating the music features and speech features of each audio segment to obtain music features and speech features of the M*N audio segments.

[0137] For example,

[0138] To extract the music features of each audio segment, we need to extract the corresponding spectral features, dynamic features, and spatial features. Specifically:

[0139] Spectral characteristics: The spectrum of the audio signal for each audio segment is first calculated using a Fast Fourier Transform (FFT) to determine the energy distribution across different frequency bands. The spectrum is then analyzed for features such as the spectral centroid (indicating the center of the spectrum and reflecting the brightness of the timbre), spectral spread (describing the distribution of spectral energy and measuring the richness of the timbre), and harmonic proportion (evaluating the proportion of harmonic components in the audio, which is related to the purity and pleasantness of the sound).

[0140] Dynamic features: Calculate the short-term energy of the audio signal for each audio segment, analyzing the frequency band dynamic range (the difference between the highest and lowest energy levels, reflecting the amplitude of changes in the intensity of the music), peak level (the point of maximum energy in the music, which determines the peak volume of the audio), and average level (reflecting the overall loudness of the audio). These features can reflect the dynamic changes and energy distribution of the music.

[0141] Spatial features: Utilize the left and right channel information of each audio to calculate stereo width (reflects the spatial distribution range of the audio in the left and right channels. A narrow width will make the sound appear thin, while a wide width may cause the sound to be too separated), phase correlation (measures the similarity between the left and right channel signals, affecting the fusion and positioning of the sound), and sound image position (determines the position of the sound in the stereo field, such as left, right, or center), etc., to provide a basis for spatial sound processing.

[0142] To extract the speech features of each audio segment, we need to extract the corresponding spectral features, dynamic features, and spatial features. Specifically:

[0143] Spectral characteristics: Using spectrum analysis technology, we calculate features such as the vocal ratio (the proportion of vocal signal energy in the overall audio, reflecting the prominence of the vocal in the music), the harmonic-to-noise ratio (a measure of the ratio of harmonic components to noise components in the vocals, related to the clarity and purity of the sound), the spectral centroid (reflecting the brightness of the vocal tone), the spectral roll-off (describing the tendency of spectral energy to decay with increasing frequency, reflecting the richness of the high-frequency components of the sound), and the spectral flatness (evaluating the flatness of the spectral energy, related to the timbre characteristics of the sound).

[0144] Dynamic characteristics: Analyze the dynamic range of the vocal part (reflecting the amplitude of changes in the strength of the vocals, affecting the emotional expression of the sound), the envelope average (reflecting the average change trend of the vocal energy), the envelope standard deviation (measuring the discrete degree of vocal energy changes, reflecting the stability of the sound), the envelope maximum (the maximum energy point in the vocals, determining the peak intensity of the sound), the envelope minimum (the minimum energy point in the vocals, affecting the delicacy of the sound), etc., to capture the strength and emotional changes of the vocals.

[0145] Pitch characteristics: Using a pitch detection algorithm, we determine features such as pitch range (the lowest to highest pitch of the human voice, reflecting the width of the sound range), pitch stability (a measure of the smoothness of pitch changes, related to the accuracy and stability of the sound), vibrato depth (reflecting the amplitude of the vibrato, reflecting the emotional color of the sound), and vibrato frequency (a measure of the frequency changes of the vibrato, related to the expressiveness of the sound) to accurately describe the pitch characteristics of the human voice.

[0146] The segment feature extraction implemented in this application enables detailed, in-depth, multi-dimensional feature analysis of each audio segment. By extracting a series of key features from both the musical and speech dimensions, the system aims to comprehensively analyze the characteristics of the audio segment, providing accurate data support for subsequent sound effect optimization. This allows the sound effect processing to be more closely aligned with the actual content and emotional expression of the music, enhancing the appeal of the audio effects and the listening experience.

[0147] Step S304: Integrate the global semantic features of the target audio, the target dimensional features of the M*N audio segments, and the M*N segment information of the target audio to obtain the target features of the target audio.

[0148] The global semantic features of the target audio, the target dimensional features of the M*N audio segments and the M*N segment information of the target audio are integrated. The integration methods include but are not limited to splicing and combination. For example, the global semantic features of the target audio can be divided according to the audio segments to obtain the semantic features of each audio segment, and then combined according to the audio segments. Combination one is the target dimensional features of the first audio segment, the first segment information and the semantic features of the first audio segment; combination two is the target dimensional features of the second audio segment, the second segment information and the semantic features of the first segment, and so on.

[0149] Step S305: Determine a target sound effect template based on the target features of the target audio and the initial sound effect template, so as to adjust the target sound effect through the target sound effect template.

[0150] In one embodiment, a target sound effect template is determined based on the target features of the target audio and the initial sound effect template, so as to adjust the target sound effect through the target sound effect template. The initial sound effect template must be obtained first, wherein the initial sound effect template matches the global semantic features of the target audio, and then based on the target features of the target audio and the initial sound effect template, a second prompt word is constructed to guide the preset multimodal intelligent agent to output the target sound effect template, and then the second prompt word is input into the preset multimodal intelligent agent to output the target sound effect template, so as to adjust the target sound effect through the target sound effect template, wherein the target sound effect template includes at least the sound effect template of each audio segment in the M*N audio segments.

[0151] Among them, Figure 5 As shown, the initial sound effect template is a combination of effects, including the parameters of each effect. A possible combination of effects includes a graphic equalizer, reverb, delay, stereo enhancer, and compressor. In addition to the required parameters, each effect also has a parameter to control whether it is enabled. Therefore, the effects enabled in the initial sound effect template may vary and depend on the professional tuning selection.

[0152] Taking music audio as an example, the initial sound effect templates include a series of sound effect templates meticulously designed by professional tuners for different mainstream music genres, such as pop, rock, classical, electronic, and jazz. These templates fully consider the characteristics of each genre and common listening needs. For example, sound effect templates for rock music typically emphasize strong bass, impactful mid- and high-frequency sounds, and appropriate reverberation to highlight the dynamics of guitar and drum beats; while sound effect templates for classical music focus more on the purity of tone, balance between instruments, and a natural reverberation space to recreate the atmosphere of a live performance.

[0153] Therefore, the corresponding initial sound effect template can be matched based on the global semantic features of the target audio.

[0154] Among them, the preset multimodal agents are mainly multimodal agents that support real-time processing of text, voice, and video, such as Google Gemini 2.05, which can generate dynamic sound effects based on the understanding of audio features using preset effectors and defined initial sound effect templates.

[0155] First, we need to define a multimodal agent. Specifically:

[0156] 1. Role: Agent is an intelligent sound design assistant that automatically generates or adjusts appropriate sound configurations, known as target sound templates, based on input target characteristics and user needs. It emulates the thinking of professional audio engineers, combining music theory and audio processing knowledge to provide users with a more personalized and high-quality sound experience.

[0157] 2. Supported effects. This section defines the sound processing process and available effect types. Each effect has a clear function, parameters, and value range. For example:

[0158] 1) Graphic Equalizer

[0159] Parameter Eqs: Controls the volume gain of each frequency band, with a typical range of -12dB to +12dB.

[0160] 2) Reverb

[0161] parameter:

[0162] Reverberation time (time): Reverberation decay time, ranging from 0.1 seconds to 60 seconds, default 2 seconds.

[0163] Reverb Size: The size of the simulated space, ranging from 0 to 100, with a default of 50.

[0164] Reverb intensity (wl): Reverb signal intensity, ranging from 0% to 100%, with the default being 50%.

[0165] Mix: The ratio of dry signal to wet signal, ranging from 0% to 100%, with the default value being 30%.

[0166] 3. Sound effect input and output formats. The input and output sound effect formats are all in json format, including a set of used effects such as:

[0167] {

[0168] "Graph EQ":{"eqs":[0,3,0,-0.8,1.5,4.8,5.8,3.2,-0.5,2]},

[0169] "Reverb":{"time":2,"Size":40,"wl":30%,"mix":10%}

[0170] }

[0171] 4. Effector combination rules

[0172] 1) The sequence must be: Graphic Equalizer → Dynamic Range Compressor → Reverb → Delay → Stereo Enhancer → Limiter. Follow the professional audio processing workflow to ensure optimal sound quality.

[0173] 2) Linkage rule: When reverberation is enabled, lower the compressor threshold appropriately to enhance the reverberation effect.

[0174] 3) Detect high-energy audio and automatically optimize the effector parameters.

[0175] After the above content is defined, the second prompt word is input into the multimodal agent, and the target sound effect template is output. The second prompt word is as follows:

[0176] "This is a song with {global features}, consisting of N sections, each with features {Section 1: Features}, {Section 2: Features}, and so on, {Section N: Features}. The initial sound effect template is set to: {Sound Effect Template}. Taking these features into consideration, please fine-tune the sound effects for each section based on the basic sound effect template to create a sound effect configuration that precisely matches the style and emotion of the song."

[0177] When the multimodal agent receives the second prompt word, it uses its internal sound effect generation logic and rules to adjust and optimize the initial sound effect template to obtain the target sound effect template. For example, it first makes adjustments to the initial sound effect template based on global semantic features such as musical style and emotion, and then makes adjustments to each paragraph based on the target dimensional features of the audio paragraph to ensure the consistency of the overall style. At the same time, it considers the transition between paragraphs and generates adjustment values ​​for the sound effect when adjusting based on the user's natural language description. The adjustment value of the sound effect is the sound effect parameter value in the target sound effect template, allowing the user to adjust and optimize the target audio through the sound effect parameter values ​​in the target sound effect template.

[0178] An embodiment of the present application provides an audio sound effect processing method including: first receiving a target audio, then processing the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and global semantic features of the target audio, wherein M and N are integers greater than 1, wherein the M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio, then performing feature extraction on the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments, and then integrating the global semantic features of the target audio, the target dimension features of the M*N audio segments, and the M*N segment information of the target audio to obtain target features of the target audio, thereby determining a target sound effect template based on the target features of the target audio and an initial sound effect template, so as to adjust the target sound effect through the target sound effect template. This application processes the target audio to obtain global semantic features to deeply analyze the audio content, and integrates the global semantic features with the target dimensional features and paragraph information of the audio paragraphs to deeply analyze the content of each audio paragraph, so that the processed audio can accurately convey the deep information such as the emotions and themes it expresses, thereby improving the audio sound effect processing effect.

[0179] See also Figure 6 , Figure 6 This is a flow chart of an audio effect processing method provided in an embodiment of the present application. Figure 6 As shown, this method is applied to Figure 2The client 204 shown includes the following steps:

[0180] Step S601: Receive a target sound effect template.

[0181] After the server generates the target sound effect template, it will send it to the client. When the client plays the target audio, it can apply the target sound effect template to configure the sound effect of the target audio to ensure linear connection when switching sound effects.

[0182] Step S602: In response to the user inputting the target audio and target text into the preset multimodal agent, a sound effect adjustment template is obtained.

[0183] Among them, the target text is used to describe user needs.

[0184] The client is configured with a preset multimodal agent, among which the preset multimodal agent is mainly a multimodal agent that supports real-time processing of text, voice, and video, such as Google Gemini 2.05, which can generate dynamic sound effects based on the understanding of audio features using preset effectors and defined initial sound effect templates.

[0185] When a user requests personalized sound effects adjustment, they input target audio and target text, where the target text is described in natural language, such as "stronger vocals," "stronger accompaniment," or "stronger drums." These descriptions and target audio are provided as input to a pre-defined multimodal agent, such as Google Gemini 2.05. Google Gemini 2.05 generates a corresponding sound effect adjustment template based on the user's instructions.

[0186] Step S603: superimposing the target sound effect template and the sound effect adjustment template to obtain an adjusted sound effect template, so as to adjust the target sound effect by using the adjusted sound effect template.

[0187] After generating the sound effect adjustment template, the target sound effect template is superimposed on the sound effect adjustment template to obtain the adjusted sound effect template. During the superposition process, the sound effect parameters are ensured to be within a reasonable range to avoid distortion or unnatural sound effects.

[0188] The sound effect parameters in the adjusted sound effect template will be applied in real time to the target audio played on the client, allowing users to obtain a personalized sound effect experience.

[0189] The embodiment of the present application provides an audio sound effect processing method, comprising: first receiving a target sound effect template, then responding to the user inputting the target audio and target text into a preset multimodal intelligent agent, obtaining a sound effect adjustment template, wherein the target text is used to describe the user's needs, and then superimposing the target sound effect template and the sound effect adjustment template to obtain an adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template. The present application uses natural language processing technology such as a preset multimodal intelligent agent to allow the user to input instructions such as "voice enhancement", and the system parses and fine-tunes the sound effect parameters in real time to accurately meet personalized sound effect needs, realize personalized sound effect configuration, and improve user experience.

[0190] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0191] The following are device embodiments of the present application. For details not fully described therein, please refer to the corresponding method embodiments described above.

[0192] Figure 7 A schematic diagram of the structure of an audio and sound effect processing device provided in an embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown. The audio and sound effect processing device includes a first receiving module 701, an audio processing module 702, a feature extraction module 703, a feature integration module 704, and a first adjustment module 705. The details are as follows:

[0193] A first receiving module 701 is configured to receive target audio;

[0194] An audio processing module 702 is configured to process the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and global semantic features of the target audio, where M and N are integers greater than 1, and the M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio;

[0195] A feature extraction module 703 is configured to extract features from the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments;

[0196] A feature integration module 704 is configured to integrate the global semantic features of the target audio, the target dimensional features of the M*N audio segments, and the M*N segment information of the target audio to obtain target features of the target audio;

[0197] The first adjustment module 705 is configured to determine a target sound effect template based on the target features of the target audio and the initial sound effect template, so as to adjust the target sound effect using the target sound effect template.

[0198] In one embodiment, the audio processing module 702 is further configured to perform structural analysis on the target audio to obtain the audio segments of the target audio and segment information of the target audio;

[0199] Feature extraction is performed on the target audio to obtain the global semantic features of the target audio.

[0200] In one embodiment, the audio processing module 702 is further configured to perform track splitting processing on the target audio to obtain M audio tracks corresponding to the target audio;

[0201] For each audio track in the M audio tracks corresponding to the target audio, divide each audio track into sections to obtain N audio sections corresponding to each audio track;

[0202] Summarize the N audio segments corresponding to each audio track to obtain M*N audio segments;

[0203] For each audio segment in the M*N audio segments, the segment information of each audio segment is obtained, and the segment information of each audio segment is summarized to obtain M*N segment information.

[0204] In one embodiment, the audio processing module 702 is further configured to obtain a preset multimodal macro model;

[0205] The target audio is subjected to feature extraction through a preset multimodal large model to obtain the global semantic features of the target audio.

[0206] In one embodiment, the audio processing module 702 is further configured to construct a first prompt word, wherein the first prompt word is used to guide the preset multimodal large model to output the global semantic features of the target audio;

[0207] Integrate the target audio and audio text to obtain input data;

[0208] The input data and the first prompt word are input into a preset multimodal large model to obtain the global semantic features of the target audio.

[0209] In one embodiment, the target dimension features include at least music features and speech features;

[0210] The feature extraction module 703 is further used to obtain target dimensions, wherein the target dimensions include at least music and speech;

[0211] Feature extraction is performed on the M*N audio segments according to music and speech respectively, to obtain music features and speech features of the M*N audio segments.

[0212] In one embodiment, the feature extraction module 703 is further configured to extract music features and speech features from each of the M*N audio segments to obtain music features and speech features of each audio segment.

[0213] The music features and speech features of each audio segment are summarized to obtain the music features and speech features of M*N audio segments.

[0214] In one embodiment, the first adjustment module 705 is further configured to obtain an initial sound effect template, wherein the initial sound effect template matches the global semantic features of the target audio;

[0215] Based on the target features of the target audio and the initial sound effect template, a second prompt word is constructed to guide the preset multimodal agent to output the target sound effect template;

[0216] The second prompt word is input into a preset multimodal intelligent agent, and a target sound effect template is output to adjust the target sound effect through the target sound effect template, wherein the target sound effect template includes at least a sound effect template for each audio segment in the M*N audio segments.

[0217] An embodiment of the present application provides an audio sound effect processing device: used to first receive target audio, then process the target audio, obtain M*N audio paragraphs of the target audio, M*N paragraph information of the target audio, and global semantic features of the target audio, wherein M and N are integers greater than 1, wherein the M*N audio paragraphs of the target audio correspond one-to-one to the M*N paragraph information of the target audio, then perform feature extraction on the M*N audio paragraphs of the target audio to obtain target dimensional features of the M*N audio paragraphs, and then integrate the global semantic features of the target audio, the target dimensional features of the M*N audio paragraphs, and the M*N paragraph information of the target audio to obtain target features of the target audio, thereby determining a target sound effect template based on the target features of the target audio and an initial sound effect template, so as to adjust the target sound effect through the target sound effect template. This application processes the target audio to obtain global semantic features to deeply analyze the audio content, and integrates the global semantic features with the target dimensional features and paragraph information of the audio paragraphs to deeply analyze the content of each audio paragraph, so that the processed audio can accurately convey the deep information such as the emotions and themes it expresses, thereby improving the audio sound effect processing effect.

[0218] Figure 8 A schematic diagram of the structure of an audio and sound effect processing device provided by another embodiment of the present application is shown. For ease of explanation, only the parts related to the embodiment of the present application are shown. The audio and sound effect processing device includes a second receiving module 801, a template output module 802, and a second adjustment module 803. The details are as follows:

[0219] The second receiving module 801 is used to receive target audio and target sound effect template;

[0220] The template output module 802 is configured to obtain a sound effect adjustment template in response to a user inputting a target audio and a target text into a preset multimodal agent, wherein the target text is used to describe the user's needs;

[0221] The second adjustment module 803 is configured to superimpose the target sound effect template and the sound effect adjustment template to obtain an adjusted sound effect template, so as to adjust the target sound effect using the adjusted sound effect template.

[0222] The embodiment of the present application provides an audio sound effect processing device: it is used to first receive a target sound effect template, and then, in response to the user inputting the target audio and target text into a preset multimodal intelligent agent, obtain a sound effect adjustment template, wherein the target text is used to describe the user's needs, and then the target sound effect template and the sound effect adjustment template are superimposed to obtain an adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template. The present application uses natural language processing technology such as a preset multimodal intelligent agent to allow the user to input instructions such as "voice enhancement", and the system parses and fine-tunes the sound effect parameters in real time to accurately meet personalized sound effect needs, realize personalized sound effect configuration, and improve user experience.

[0223] This application Figure 9 A schematic diagram of a computer device is provided. Figure 9 As shown, the computer device 9 of this embodiment includes: a processor 901, a memory 902, and steps of the audio effect processing method embodiment stored in the memory 902 and executable on the processor 901, such as Figure 3 Steps 301 to 305 shown, or Figure 6 Alternatively, when the processor 901 executes the computer program 903, the functions of the modules / units in the above-mentioned embodiments of the audio and sound effect processing device are realized, for example Figure 7 The functions of the modules / units 701 to 705 shown, or Figure 8 Functions of modules / units 801 to 803 are shown.

[0224] The present application also provides a readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, it is used to implement the audio and sound effect processing methods provided in the various embodiments described above.

[0225] Among them, the readable storage medium can be a computer storage medium or a communication medium. Communication media include any medium that facilitates the transmission of computer programs from one place to another. Computer storage media can be any available medium that can be accessed by a general-purpose or special-purpose computer. For example, a readable storage medium is coupled to a processor so that the processor can read information from the readable storage medium and write information to the readable storage medium. Of course, the readable storage medium can also be an integral part of the processor. The processor and the readable storage medium can be located in an application-specific integrated circuit (ASIC). In addition, the ASIC can be located in a user device. Of course, the processor and the readable storage medium can also exist in a communication device as discrete components. The readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0226] The present application also provides a program product, the program product including execution instructions stored in a readable storage medium. At least one processor of a device can read the execution instructions from the readable storage medium, and at least one processor executes the execution instructions so that the device implements the audio and sound effect processing methods provided in the various embodiments described above.

[0227] In the embodiments of the above-mentioned devices, it should be understood that the processor may be a central processing unit (CPU), or may be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), etc. The general-purpose processor may be a microprocessor or any conventional processor. The steps of the method disclosed in this application may be directly implemented as being executed by a hardware processor, or may be executed by a combination of hardware and software modules in the processor.

[0228] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An audio effect processing method, characterized in that: Applied to the server, including: Receive target audio; Processing the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and global semantic features of the target audio, where M and N are integers greater than 1, and the M*N audio segments of the target audio and the M*N segment information of the target audio are in one-to-one correspondence; Performing feature extraction on the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments; Integrating the global semantic features of the target audio, the target dimensional features of the M*N audio segments, and the M*N segment information of the target audio to obtain the target features of the target audio; Based on the target features of the target audio and the initial sound effect template, a target sound effect template is determined, so as to adjust the target sound effect through the target sound effect template.

2. The audio effect processing method according to claim 1, wherein: The processing of the target audio to obtain an audio segment of the target audio, segment information of the target audio, and a global semantic feature of the target audio includes: Performing structural analysis on the target audio to obtain audio segments of the target audio and segment information of the target audio; Feature extraction is performed on the target audio to obtain global semantic features of the target audio.

3. The audio effect processing method according to claim 2, wherein: The performing structural analysis on the target audio to obtain M*N audio segments of the target audio and M*N segment information of the target audio includes: Performing track splitting processing on the target audio to obtain M audio tracks corresponding to the target audio; For each audio track of the M audio tracks corresponding to the target audio, divide the audio track into sections to obtain N audio sections corresponding to the audio track; Summarize the N audio segments corresponding to each audio track to obtain M*N audio segments; For each audio segment in the M*N audio segments, the segment information of each audio segment is obtained, and the segment information of each audio segment is summarized to obtain M*N segment information.

4. The audio effect processing method according to claim 2, wherein: The extracting features of the target audio to obtain global semantic features of the target audio includes: Get the preset multimodal large model; The target audio is subjected to feature extraction using the preset multimodal large model to obtain global semantic features of the target audio.

5. An audio effect processing method, characterized in that: Applied to the client, including: Receive a target sound effect template; In response to the user inputting the target audio and target text into a preset multimodal agent, a sound effect adjustment template is obtained, wherein the target text is used to describe the user's needs; The target sound effect template and the sound effect adjustment template are superimposed to obtain an adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template.

6. An audio effect processing device, characterized in that: include: A first receiving module, configured to receive target audio; an audio processing module, configured to process the target audio to obtain M*N audio segments of the target audio, M*N segment information of the target audio, and a global semantic feature of the target audio, wherein M and N are integers greater than 1, and wherein the M*N audio segments of the target audio correspond one-to-one to the M*N segment information of the target audio; A feature extraction module, configured to extract features from the M*N audio segments of the target audio to obtain target dimension features of the M*N audio segments; A feature integration module is used to integrate the global semantic features of the target audio, the target dimensional features of the M*N audio segments, and the M*N segment information of the target audio to obtain the target features of the target audio; The first adjustment module is used to determine a target sound effect template based on the target feature of the target audio and the initial sound effect template, so as to adjust the target sound effect through the target sound effect template.

7. An audio effect processing device, characterized in that: include: A second receiving module is used to receive a target sound effect template; A template output module is configured to obtain a sound effect adjustment template in response to a user inputting the target audio and target text into a preset multimodal agent, wherein the target text is used to describe the user's needs; The second adjustment module is used to superimpose the target sound effect template and the sound effect adjustment template to obtain an adjusted sound effect template, so as to adjust the target sound effect through the adjusted sound effect template.

8. A computer device, characterized in that: comprising a memory, and one or more processors communicatively connected to the memory; The memory stores instructions that can be executed by the one or more processors, and the instructions are executed by the one or more processors to enable the one or more processors to implement the audio sound effect processing method as described in any one of claims 1 to 4 or 5.

9. A computer-readable storage medium, characterized in that The method comprises a program or an instruction, which, when executed on a computer, implements the audio and sound effect processing method as claimed in any one of claims 1 to 4 or 5.

10. A computer program product, characterized in that It includes a computer program, which, when executed by a processor, implements the audio sound effect processing method according to any one of claims 1 to 4 or 5.

Images