A multi-modal multi-audio generation method and device, equipment and medium

Abstract

The application discloses a multi-modal-oriented multi-audio generation method and device, equipment and medium, comprising: task decomposition is performed on the multi-modal input to obtain a first audio script set; the first audio script set is optimized according to the completeness evaluation result of the first audio script set to obtain a second audio script set; an audio generation model is assigned to each audio script according to the type of each audio script in the second audio script set to obtain a first audio generation model set; the first audio generation model set is optimized according to the adaptability evaluation result of the first audio generation model set to obtain a second audio generation model set; audio is generated by the second audio generation model set to obtain an audio set; and the audio set is optimized according to the accuracy evaluation result of the audio set to generate a multi-audio output corresponding to the multi-modal input. Therefore, by implementing the application, high-quality multi-modal-oriented multi-audio generation can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a multi-audio generation method and device for multi-modal, equipment and medium. BACKGROUND

[0002] With the development of multimedia and artificial intelligence technology, the demand for generating corresponding audio content from multi-modal inputs such as text, images or videos is growing. For example, in game development, film production and VR / AR experience, realistic environmental sound effects, dialogue voices and background music are automatically generated according to the scene. In recent years, the technology of single-modal to audio generation has made great progress, including text-to-audio generation (such as text-to-speech, text-to-sound effects or text-to-music) and video-to-audio generation (such as generating corresponding voice commentary or sound effects for videos). However, these methods are often limited to a single type of audio or limited input modalities, lacking the ability to handle the synchronous generation of multiple types of audio in complex multi-modal scenarios.

[0003] For multi-modal to multi-audio generation, some explorations have recently emerged. For example, AudioX proposes a unified model based on diffusion Transformer, which can generate audio from text, video and image inputs, but this model mainly focuses on sound effects and music, does not cover types such as speech and songs, and the model training process is very computationally intensive and relies on large-scale paired data that has not been publicly disclosed. In addition, some works attempt to concatenate multiple pre-trained audio models for calling, for example, AudioGPT and WavCraft use proxy models to connect multiple pre-trained audio models to perform speech and music generation tasks, and LVAS-Agent designs a multi-agent framework inspired by professional dubbing processes, but these solutions are either still limited to a single audio task (such as dubbing) or lack a unified decision center and self-correction mechanism, making it difficult to effectively handle complex scenarios of multi-audio events, and the generated results often have incomplete or inconsistent problems. SUMMARY

[0004] The present application provides a multi-audio generation method, device, equipment and medium for multi-modal, which can realize the generation of multiple input modalities to multiple categories of audio, and the final audio output has high accuracy.

[0005] The present application provides a multi-audio generation method for multi-modal, comprising:

[0006] Task decomposition and script planning are performed on the multi-modal input to obtain a first audio script set;

[0007] Based on the integrity evaluation result of the first audio script set, the first audio script set is optimized to obtain a second audio script set; wherein, the integrity evaluation result of the first audio script set is obtained by evaluating the integrity of the first audio script set based on the audio events contained in the multimodal input;

[0008] Based on the type of each audio script in the second audio script set, an audio generation model is assigned to each audio script to obtain the first audio generation model set.

[0009] Based on the adaptability evaluation results of the first audio generation model set, the first audio generation model set is optimized to obtain a second audio generation model set; wherein, the adaptability evaluation results of the first audio generation model set are obtained by evaluating the adaptability of each audio generation model according to the type of each audio script.

[0010] Based on the second set of audio scripts, audio is generated using the second set of audio generation models to obtain an audio set;

[0011] Based on the accuracy evaluation results of the audio set, the audio set is optimized to generate multiple audio outputs corresponding to the multimodal input; wherein, the accuracy evaluation results of the audio set are obtained by evaluating the accuracy of each generated audio based on the content of each audio script.

[0012] This invention, through scripted task decomposition of multimodal input and integrity evaluation and optimization of audio scripts, ensures that all audio events are identified. By dynamically assigning audio generation models to each audio script and optimizing the adaptability of these models, it ensures that each type of audio is generated by the most suitable model. Furthermore, by generating audio using each audio generation model and optimizing its accuracy, it ensures that the final audio output has high accuracy. Compared to existing technologies that only target a single type of audio or a limited input modality, this application can generate multiple audio types in multimodal scenarios, and the final audio output has high accuracy.

[0013] Furthermore, the process of decomposing and planning the multimodal input to obtain a first audio script set includes:

[0014] Feature analysis is performed on the multimodal input to obtain the overall audio requirements;

[0015] Based on semantic and temporal information, the overall audio requirement is decomposed into multiple audio events;

[0016] Multiple audio scripts are generated based on the type of each audio event; one audio script corresponds to one audio event.

[0017] The generated audio scripts are integrated to obtain the first audio script set.

[0018] The embodiments of the present invention decompose multimodal inputs into scripted tasks, which facilitates subsequent specialized processing for each event.

[0019] Further, the step of optimizing the first audio script set based on the integrity evaluation result of the first audio script set to obtain the second audio script set includes:

[0020] The first audio script set is evaluated according to a preset first requirement; wherein, the first requirement specifically means that all audio events included in the multimodal input are described by audio scripts;

[0021] Complete the missing audio scripts in the first audio script set until the first audio script set meets the first requirement;

[0022] The original audio scripts are integrated with the supplementary audio scripts to obtain a second set of audio scripts.

[0023] The embodiments of the present invention can ensure that all audio events are identified by optimizing the audio script through integrity assessment.

[0024] Further, the step of assigning an audio generation model to each audio script based on its type in the second audio script set to obtain the first audio generation model set includes:

[0025] Based on the type of each audio script in the second audio script set, an audio generation model is selected from a preset model library; wherein, the model library consists of multiple pre-trained audio generation models;

[0026] The selected audio generation models are integrated to obtain the first audio generation model set.

[0027] The embodiments of the present invention perform audio generation tasks through a pre-trained audio generation model, which can reduce the cost of realizing multimodal to multi-audio generation.

[0028] Further, the step of optimizing the first audio generation model set based on the adaptability evaluation results of the first audio generation model set to obtain the second audio generation model set includes:

[0029] According to the preset second requirement, each audio generation model in the first audio generation model set is evaluated; wherein, the second requirement is specifically: the type of the audio generation model is compatible with the type of the audio script;

[0030] Replace the audio generation models in the first audio generation model set that do not meet the second requirement until all audio generation models meet the second requirement;

[0031] The audio generation models that meet the second requirement are integrated to obtain the second audio generation model set.

[0032] The embodiments of the present invention can ensure that each type of audio is generated by the most suitable model by performing adaptability evaluation and optimization on the audio generation model.

[0033] Further, the step of generating an audio set from the second audio generation model set based on the second audio script set includes:

[0034] Each audio generation model in the second audio generation model set generates audio based on its corresponding audio script, resulting in multiple audio files;

[0035] The generated audio files are then integrated to obtain an audio set.

[0036] The embodiments of the present invention generate audio through various audio generation models, and can execute various audio scripts generated from multimodal inputs.

[0037] Further, the step of optimizing the audio set based on the accuracy evaluation results of the audio set to generate multi-audio outputs corresponding to the multimodal input includes:

[0038] According to a preset third requirement, each audio in the audio set is evaluated; wherein, the third requirement is specifically: the generated audio can accurately reproduce the content of the corresponding audio script;

[0039] The audio files that do not meet the third requirement in the audio set are regenerated until all audio files meet the third requirement.

[0040] The audio samples that meet the third requirement are synthesized to generate the multi-audio output corresponding to the multimodal input.

[0041] The embodiments of the present invention can ensure that the final audio output has high accuracy by optimizing the accuracy of the audio through accuracy evaluation.

[0042] Another embodiment of the present invention provides a multi-modal multi-audio generation device, including: an audio script generation module, an audio script evaluation module, an audio model allocation module, an audio model evaluation module, an audio generation module, and an audio evaluation module;

[0043] The audio script generation module is used to perform task decomposition and script planning on multimodal input to obtain a first audio script set;

[0044] The audio script evaluation module is used to optimize the first audio script set based on the integrity evaluation result of the first audio script set to obtain a second audio script set; wherein, the integrity evaluation result of the first audio script set is obtained by evaluating the integrity of the first audio script set based on the audio events contained in the multimodal input;

[0045] The audio model allocation module is used to allocate an audio generation model to each audio script according to the type of each audio script in the second audio script set, thereby obtaining the first audio generation model set;

[0046] The audio model evaluation module is used to optimize the first audio generation model set based on the adaptability evaluation result of the first audio generation model set to obtain a second audio generation model set; wherein, the adaptability evaluation result of the first audio generation model set is obtained by evaluating the adaptability of each audio generation model based on the type of each audio script.

[0047] The audio generation module is used to generate audio from the second audio generation model set based on the second audio script set to obtain an audio set.

[0048] The audio evaluation module is used to optimize the audio set based on the accuracy evaluation result of the audio set, and generate multiple audio outputs corresponding to the multimodal input; wherein, the accuracy evaluation result of the audio set is obtained by evaluating the accuracy of each generated audio based on the content of each audio script.

[0049] Another embodiment of the present invention provides a terminal device, including: a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements the steps of a multi-modal multi-audio generation method as described in the present invention.

[0050] Another embodiment of the present invention also provides a computer-readable storage medium item, including: a stored computer program, which, when the computer program is running, controls the device where the computer-readable storage medium is located to perform steps of a multi-modal multi-audio generation method as described in the present invention. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the multi-modal multi-audio generation method provided in an embodiment of the present invention.

[0052] Figure 2 This is a flowchart illustrating a multi-modal multi-audio generation system provided in an embodiment of the present invention.

[0053] Figure 3 This is an experimental comparison diagram between the multi-modal multi-audio generation system provided in this embodiment of the invention and other systems;

[0054] Figure 4 This is a schematic diagram of the structure of a multi-modal multi-audio generation device provided in an embodiment of the present invention. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings of the embodiments. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the application; the terms “comprising” and “having”, and any variations thereof, in the specification, claims, and foregoing description of the drawings are intended to cover non-exclusive inclusion.

[0057] In the description of the embodiments of this application, technical terms such as "first" and "second" are used only to distinguish different objects and should not be construed as indicating or implying relative importance or implicitly specifying the number, specific order, or primary and secondary relationship of the indicated technical features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly defined.

[0058] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0059] In the description of the embodiments in this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship.

[0060] In the description of the embodiments of this application, the term "multiple" refers to two or more (including two), similarly, "multiple sets" refers to two or more (including two sets), and "multiple pieces" refers to two or more (including two pieces).

[0061] See Figure 1 To address the problem that existing technologies only target a single type of audio or a limited range of input modalities, and generate inaccurate audio, an embodiment of the present invention provides a multi-modal multi-audio generation method, comprising steps S101 to S106:

[0062] Step S101: Perform task decomposition and script planning on the multimodal input to obtain the first audio script set.

[0063] Further, step S101 includes:

[0064] Feature analysis is performed on the multimodal input to obtain the overall audio requirements;

[0065] Based on semantic and temporal information, the overall audio requirement is decomposed into multiple audio events;

[0066] Multiple audio scripts are generated based on the type of each audio event; one audio script corresponds to one audio event.

[0067] The generated audio scripts are integrated to obtain the first audio script set.

[0068] Preferably, a generative expert agent is used to decompose and plan the multimodal input to generate a structured audio script; wherein the audio script includes attributes such as the type of the corresponding audio event (e.g., sound effects, speech, and music), content description, start / end time, and volume.

[0069] The embodiments of the present invention decompose multimodal inputs into scripted tasks, which facilitates subsequent specialized processing for each event.

[0070] Step S102: Based on the integrity evaluation result of the first audio script set, optimize the first audio script set to obtain a second audio script set; wherein, the integrity evaluation result of the first audio script set is obtained by evaluating the integrity of the first audio script set based on the audio events contained in the multimodal input.

[0071] Further, step S102 includes:

[0072] The first audio script set is evaluated according to a preset first requirement; wherein, the first requirement specifically means that all audio events included in the multimodal input are described by audio scripts;

[0073] Complete the missing audio scripts in the first audio script set until the first audio script set meets the first requirement;

[0074] The original audio scripts are integrated with the supplementary audio scripts to obtain a second set of audio scripts.

[0075] Preferably, a supervisory expert agent is used to evaluate the first audio script set and to provide feedback on any missing audio events to the generator expert agent, which then regenerates the audio scripts based on the feedback.

[0076] The embodiments of the present invention can ensure that all audio events are identified by optimizing the audio script through integrity assessment.

[0077] Step S103: Based on the type of each audio script in the second audio script set, assign an audio generation model to each audio script to obtain the first audio generation model set.

[0078] Further, step S103 includes:

[0079] Based on the type of each audio script in the second audio script set, an audio generation model is selected from a preset model library; wherein, the model library consists of multiple pre-trained audio generation models;

[0080] The selected audio generation models are integrated to obtain the first audio generation model set.

[0081] Preferably, the model library includes audio generation models in different fields such as sound effects, speech dialogue, background music and song performance. For example, text-to-speech models of different styles / languages, music generation models of various genres, and synthesis models of environmental noise and object sounds.

[0082] Preferably, a generative expert agent is used for model allocation; wherein, before generation, the model in each domain will self-check whether the audio script is complete (e.g., the model in the speech domain checks whether the dialogue is complete and natural, and the model in the music domain checks whether the melody matches the context), and fine-tune the audio script accordingly; the models in different domains will also share information and review the consistency of the audio scripts with each other (e.g., whether the rhythm of the background music and the speech dialogue conflict, and which side needs to be reduced in volume, etc.).

[0083] The embodiments of the present invention perform audio generation tasks through a pre-trained audio generation model, which can reduce the cost of realizing multimodal to multi-audio generation.

[0084] Step S104: Based on the adaptability evaluation results of the first audio generation model set, optimize the first audio generation model set to obtain a second audio generation model set; wherein, the adaptability evaluation results of the first audio generation model set are obtained by evaluating the adaptability of each audio generation model according to the type of each audio script.

[0085] Further, step S104 includes:

[0086] According to the preset second requirement, each audio generation model in the first audio generation model set is evaluated; wherein, the second requirement is specifically: the type of the audio generation model is compatible with the type of the audio script;

[0087] Replace the audio generation models in the first audio generation model set that do not meet the second requirement until all audio generation models meet the second requirement;

[0088] The audio generation models that meet the second requirement are integrated to obtain the second audio generation model set.

[0089] Preferably, a supervisory expert agent is used to evaluate whether the selected audio generation model and its generation parameters are suitable for each audio event, ensuring that each type of audio is generated by the most suitable audio generation model.

[0090] The embodiments of the present invention can ensure that each type of audio is generated by the most suitable model by performing adaptability evaluation and optimization on the audio generation model.

[0091] Step S105: Based on the second audio script set, audio is generated using the second audio generation model set to obtain an audio set.

[0092] Further, step S105 includes:

[0093] Each audio generation model in the second audio generation model set generates audio based on its corresponding audio script, resulting in multiple audio files;

[0094] The generated audio files are then integrated to obtain an audio set.

[0095] The embodiments of the present invention generate audio through various audio generation models, and can execute various audio scripts generated from multimodal inputs.

[0096] Step S106: Based on the accuracy evaluation result of the audio set, optimize the audio set to generate multiple audio outputs corresponding to the multimodal input; wherein, the accuracy evaluation result of the audio set is obtained by evaluating the accuracy of each generated audio based on the content of each audio script.

[0097] Further, step S106 includes:

[0098] According to a preset third requirement, each audio in the audio set is evaluated; wherein, the third requirement is specifically: the generated audio can accurately reproduce the content of the corresponding audio script;

[0099] The audio files that do not meet the third requirement in the audio set are regenerated until all audio files meet the third requirement.

[0100] The audio samples that meet the third requirement are synthesized to generate the multi-audio output corresponding to the multimodal input.

[0101] Preferably, a supervised expert agent is used to evaluate and improve each audio track in the audio set through a tree-like exploration approach. Specifically, for each audio track, a "mind tree" is constructed: the root node is the initially generated audio, and the child nodes represent improved audio tracks generated based on different feedback. The supervised expert agent listens to and evaluates each generated audio track and provides feedback. If an audio track is not ideal (e.g., the music's mood doesn't match the visuals), the generative expert agent is instructed to regenerate it based on the feedback, forming a new branch node. This process can be viewed as the generative expert agent continuously backtracking and exploring along the mind tree, trying different generation paths: such as changing to a backup model, adjusting generation parameters, or introducing additional hints. Each time a new audio track is generated, the supervised expert agent re-evaluates it; if it still doesn't meet the requirements, it continues iterating until satisfactory results are achieved. Finally, the optimally evaluated audio track is selected for synthesis.

[0102] The embodiments of the present invention can ensure that the final audio output has high accuracy by optimizing the accuracy of the audio through accuracy evaluation.

[0103] This invention, through scripted task decomposition of multimodal input and integrity evaluation and optimization of audio scripts, ensures that all audio events are identified. By dynamically assigning audio generation models to each audio script and optimizing the adaptability of these models, it ensures that each type of audio is generated by the most suitable model. Furthermore, by generating audio using each audio generation model and optimizing its accuracy, it ensures that the final audio output has high accuracy. Compared to existing technologies that only target a single type of audio or a limited input modality, this application can generate multiple audio types in multimodal scenarios, and the final audio output has high accuracy.

[0104] like Figure 2 As shown, based on the above method embodiments, a corresponding system embodiment is provided; wherein, the system includes a generating expert agent and a supervising expert agent.

[0105] The system's process includes three stages:

[0106] In Phase 1, the generative expert agent decomposes the multimodal input into several manageable sub-events and generates a preliminary audio script; the supervisory expert agent, as an audio planning expert, evaluates the fit between the generated audio script (such as sub-event categories, timing, and volume) and the input content, provides modification suggestions, and optimizes the script.

[0107] Phase 2: The generative expert agent assigns an appropriate audio generation model to each sub-event; the supervisory expert agent, acting as a domain review expert, evaluates whether the selected audio generation model and its generation parameters are suitable for each audio event.

[0108] In Phase 3, each audio generation model performs audio generation; the supervisory expert agent acts as a professional audio evaluator, evaluating each generated audio file and promptly reporting any problems to the generation expert agent for correction.

[0109] The embodiments of the present invention achieve a deep understanding of multimodal inputs and a division of labor and collaboration in multi-audio outputs through close cooperation between the generating expert intelligent agent and the supervising expert intelligent agent.

[0110] To demonstrate the process and beneficial effects of a multi-modal audio generation system, this invention provides a specific implementation. The input is a video clip of a street performance in a city, along with a corresponding text description: "On a bustling commercial street, a street performer is singing and playing song A, with lyrics B. Pedestrians stop to watch and applaud, while fireworks bloom in the sky." The system needs to generate a comprehensive audio for this scene, including ambient sound effects, audience voices, and the song's vocals.

[0111] First, the system reads the video and text, extracts key elements (such as crowds and fireworks) and timeline information from the video, and generates a preliminary audio script by combining the text description. The system divides the entire scene into four sub-events: (1) pedestrian footsteps and cheers, type "sound effect", which occur for most of the video to create a street crowd atmosphere; (2) fireworks explosions, type "sound effect", which occur at a specific time (when the fireworks appear); (3) street environmental noise, type "sound effect", which exists continuously as background noise; (4) songs sung by artists, type "song", which runs throughout the video. For each sub-event, the audio script records its start / end time, volume, and brief description. For example, event (1) is marked as "Type: sound effect, object: crowd applause and cheers, start: 2nd second, end: 7th second, description: footsteps of pedestrians and applause of onlookers"; event (4) is marked as "Type: song, content: singing song A, style: original style, emotion: nostalgia", etc. Subsequently, the supervisory expert agent, acting as the audio planning expert, reviewed the audio script, confirming that all sound elements described in the text were included. Particular attention was paid to the fact that the audio script covered key elements such as pedestrian cheers, fireworks, and a song, and that the timing was reasonable (the fireworks sound did not conflict with the song's climax). After slightly adjusting the time stamps for the fireworks event (to precisely match the timing of the fireworks in the video), the audio script was finalized.

[0112] Secondly, the system calls the corresponding audio generation model for each of the four sub-events. For event (4), the generative expert agent, acting as a song expert, first analyzes the song title and lyrics, calls a pre-trained Chinese vocal synthesis model, and selects a vocal configuration that matches the timbre of the street performer (e.g., a young male voice with live reverberation). The generative expert agent also autonomously checks the matching of lyrics and melody to ensure that the synthesized performance conforms to the style of the original song, and adds appropriate emotional rendering when necessary (e.g., a slightly weathered voice to match the mood of the lyrics). For events (1), (2), and (3), since they all belong to the environmental / effects sound domain, the generative expert agent, acting as a sound effects expert, calls multiple pre-trained sound effect generation models: one of which is good at synthesizing everyday environmental sounds such as footsteps and noise, and another is good at special effects such as loud explosions. Based on the event description, the generative expert agent decides to use the environmental sound model to generate continuous street background sounds and crowd trampling / clapping sounds (events (1) and (3)), and to use the explosion sound model to generate the booming sound of fireworks (event (2)). On the other hand, the song expert and the sound effects expert collaborated: they discussed the ratio of background sound to song volume to ensure that the song (event (4)) was the main track, the human voice was clearly identifiable, and the volume of the cheers and fireworks sound effects was appropriately reduced to enhance the atmosphere of the scene. The supervisory expert agent participated in the above process as a domain review expert, providing suggestions on model selection and parameter settings. For example, it suggested that the song expert add microphone effects appropriately during synthesis to simulate the realism of street performance; it suggested that the sound effects expert adjust the delay of the fireworks sound to synchronize it with the fireworks flash in the video. After multi-party collaboration, the final generation scheme for all four events was determined: the song was sung by a specific vocal synthesis model, the background ambient sound and human voice footsteps were generated by an ambient sound model, and the fireworks were generated by an explosion sound model.

[0113] Finally, each audio generation model began generating audio according to a predetermined scheme. First, the entire song audio was generated—a 10-second Chinese vocal track containing a chorus with given lyrics. After generation, a supervisory expert agent, acting as an audio evaluation expert, listened to the song segment and found the lyrics correct and the pitch and rhythm appropriate, thus approving the result. Next, background ambient sound tracks, crowd sound tracks, and fireworks sound tracks were generated. The supervisory expert agent carefully checked the synchronization between these two audio tracks and the video, noting that the fireworks sound was about 0.5 seconds ahead of the actual fireworks display, and pointed out this timing discrepancy. The generation expert agent then backtracked to the generation node in the tree-like thought structure, generated sibling nodes, adjusted the trigger time parameters of the fireworks sound, and regenerated the audio to precisely match the fireworks display frames in the video. For the background ambient sound track, the supervisory expert agent focused on the appropriate volume. The results showed that some street noise and footsteps could be heard between vocal segments, adding a sense of presence without interfering with the main song, thus approving the track. Finally, the system blends and overlays the four tracks—song, crowd sounds, ambient sounds, and fireworks sounds—to generate the final audio output, which is then synthesized and displayed with the original video. Throughout the process, with repeated feedback from the supervising expert agent and multiple trials and refinements by the generative expert agent, each sound achieved the desired effect.

[0114] like Figure 3 As shown, based on the above specific embodiments, corresponding comparative experimental embodiments are provided; Figure 3 In the diagram, (I)-(IV) represent the audio generated by the four sub-events parsed by the multimodal multi-audio generation system ((a) Ours): (I) the sound of footsteps of pedestrians and cheers of the audience, (II) the sound of fireworks exploding, (III) the ambient noise of surrounding shops, and (IV) the song sung by a street performer. Other models ((b) MMAudio, (c) FoleyCrafter, (d) AudioX) can only capture a single sub-event, resulting in incomplete audio output.

[0115] like Figure 4 As shown, based on the above method embodiments, corresponding apparatus embodiments are provided;

[0116] An embodiment of the present invention provides a multi-audio generation device for multimodal use, including: an audio script generation module 401, an audio script evaluation module 402, an audio model allocation module 403, an audio model evaluation module 404, an audio generation module 405, and an audio evaluation module 406.

[0117] The audio script generation module 401 is used to perform task decomposition and script planning on the multimodal input to obtain a first audio script set;

[0118] The audio script evaluation module 402 is used to optimize the first audio script set according to the integrity evaluation result of the first audio script set to obtain a second audio script set; wherein, the integrity evaluation result of the first audio script set is obtained by evaluating the integrity of the first audio script set based on the audio events contained in the multimodal input.

[0119] The audio model allocation module 403 is used to allocate an audio generation model to each audio script according to the type of each audio script in the second audio script set, so as to obtain the first audio generation model set;

[0120] The audio model evaluation module 404 is used to optimize the first audio generation model set based on the adaptability evaluation result of the first audio generation model set to obtain a second audio generation model set; wherein, the adaptability evaluation result of the first audio generation model set is obtained by evaluating the adaptability of each audio generation model based on the type of each audio script.

[0121] The audio generation module 405 is used to generate audio from the second audio generation model set based on the second audio script set to obtain an audio set.

[0122] The audio evaluation module 406 is used to optimize the audio set based on the accuracy evaluation result of the audio set, and generate multiple audio outputs corresponding to the multimodal input; wherein, the accuracy evaluation result of the audio set is obtained by evaluating the accuracy of each generated audio based on the content of each audio script.

[0123] In this embodiment of the invention, the audio script generation module 401 includes: a feature analysis submodule, a task decomposition submodule, a script planning submodule, and a script integration submodule;

[0124] The feature analysis submodule is used to perform feature analysis on multimodal input to obtain the overall audio requirements;

[0125] The task decomposition submodule is used to decompose the overall audio requirements into multiple audio events based on semantic and temporal information.

[0126] The script planning submodule is used to generate multiple audio scripts based on the type of each audio event; wherein, one audio event corresponds to one audio script.

[0127] The script integration submodule is used to integrate the generated audio scripts to obtain the first audio script set.

[0128] The embodiments of the present invention decompose multimodal inputs into scripted tasks, which facilitates subsequent specialized processing for each event.

[0129] In this embodiment of the invention, the audio script evaluation module 402 includes: a first evaluation submodule, a first optimization submodule, and a first integration submodule;

[0130] The first evaluation submodule is used to evaluate the first audio script set according to a preset first requirement; wherein, the first requirement specifically means that all audio events included in the multimodal input are described by audio scripts;

[0131] The first optimization submodule is used to complete the missing audio scripts in the first audio script set until the first audio script set meets the first requirement;

[0132] The first integration submodule is used to integrate the original audio script with the supplementary audio script to obtain a second audio script set.

[0133] The embodiments of the present invention can ensure that all audio events are identified by optimizing the audio script through integrity assessment.

[0134] In this embodiment of the invention, the audio model allocation module 403 includes: a model allocation submodule and a model integration submodule;

[0135] The model allocation submodule is used to select an audio generation model from a preset model library according to the type of each audio script in the second audio script set; wherein, the model library consists of multiple pre-trained audio generation models;

[0136] The model integration submodule is used to integrate the selected audio generation models to obtain the first audio generation model set.

[0137] The embodiments of the present invention perform audio generation tasks through a pre-trained audio generation model, which can reduce the cost of realizing multimodal to multi-audio generation.

[0138] In this embodiment of the invention, the audio model evaluation module 404 includes: a second evaluation submodule, a second optimization submodule, and a second integration submodule;

[0139] The second evaluation submodule is used to evaluate each audio generation model in the first audio generation model set according to a preset second requirement; wherein, the second requirement is specifically: the type of the audio generation model is compatible with the type of the audio script;

[0140] The second optimization submodule is used to replace audio generation models in the first audio generation model set that do not meet the second requirement until all audio generation models meet the second requirement.

[0141] The second integration submodule is used to integrate the audio generation models that meet the second requirement to obtain a second audio generation model set.

[0142] The embodiments of the present invention can ensure that each type of audio is generated by the most suitable model by performing adaptability evaluation and optimization on the audio generation model.

[0143] In this embodiment of the invention, the audio generation module 405 includes: an audio generation submodule and an audio integration submodule;

[0144] The audio generation submodule is used by each audio generation model in the second audio generation model set to generate audio according to its corresponding audio script, thereby obtaining multiple audio files.

[0145] The audio integration submodule is used to integrate the generated audio files to obtain an audio set.

[0146] The embodiments of the present invention generate audio through various audio generation models, and can execute various audio scripts generated from multimodal inputs.

[0147] In this embodiment of the invention, the audio evaluation module 406 includes: a third evaluation submodule, a third optimization submodule, and an audio output submodule;

[0148] The third evaluation submodule is used to evaluate each audio in the audio set according to a preset third requirement; wherein, the third requirement is specifically: the generated audio can accurately reproduce the content of the corresponding audio script;

[0149] The third optimization submodule is used to regenerate the audio in the audio set that does not meet the third requirement until all audio meets the third requirement.

[0150] The audio output submodule is used to synthesize the audio signals that meet the third requirement to generate the multi-audio output corresponding to the multimodal input.

[0151] The embodiments of the present invention can ensure that the final audio output has high accuracy by optimizing the accuracy of the audio through accuracy evaluation.

[0152] It is understood that the above-described device embodiments correspond to the method embodiments of the present invention, and can implement a multi-modal multi-audio generation method provided by any of the above-described method embodiments of the present invention.

[0153] This invention employs an audio script generation module to decompose multimodal inputs into scripted tasks, and an audio script evaluation module to optimize the integrity of the audio scripts, ensuring that all audio events are recognized. An audio model allocation module dynamically assigns audio generation models to each audio script, and an audio model evaluation module optimizes the suitability of these models, ensuring that each type of audio is generated by the most suitable model. Furthermore, the audio generation module generates audio, and the audio evaluation module optimizes its accuracy, ensuring high accuracy in the final audio output. Compared to existing technologies that only target a single type of audio or a limited number of input modalities, this application can generate multiple audio formats in multimodal scenarios, and the final audio output has high accuracy.

[0154] It should be noted that the device embodiments described above are merely illustrative, and some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Furthermore, in the accompanying drawings of the device embodiments provided by this invention, the connection relationships between modules indicate that they have communication connections, which can specifically be implemented as one or more communication buses or signal lines. Those skilled in the art can understand and implement this without any creative effort.

[0155] Based on the above embodiment of a multimodal multi-audio generation method, another embodiment of the present invention provides a terminal device, which includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor. When the processor executes the computer program, it implements a multimodal multi-audio generation method according to any embodiment of the present invention.

[0156] For example, in this embodiment, the computer program can be divided into one or more modules, which are stored in the memory and executed by the processor to complete the present invention. The one or more modules may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in the terminal device.

[0157] The terminal device may be a desktop computer, laptop, handheld computer, or cloud server, etc. The terminal device may include, but is not limited to, a processor and a memory.

[0158] The processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the terminal device, connecting all parts of the terminal device via various interfaces and lines.

[0159] Based on the above-described method embodiments, another embodiment of the present invention provides a computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to execute a multi-modal multi-audio generation method as described in any of the above-described method embodiments of the present invention.

[0160] The modules / units integrated in the device / terminal equipment, if implemented as software functional units and sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above embodiments of the present invention can also be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, a recording medium, a USB flash drive, a portable hard drive, a magnetic disk, an optical disk, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium, etc.

[0161] The above description represents the preferred embodiments of the present invention. It should be noted that those skilled in the art can make various improvements and modifications without departing from the principles of the present invention, and these improvements and modifications are also considered to be within the scope of protection of the present invention.

Claims

1. A multi-modal multi-audio generation method, characterized in that, include: Task decomposition and script planning are performed on the multimodal input to obtain the first audio script set; Based on the integrity evaluation result of the first audio script set, the first audio script set is optimized to obtain a second audio script set; wherein, the integrity evaluation result of the first audio script set is obtained by evaluating the integrity of the first audio script set based on the audio events contained in the multimodal input; Based on the type of each audio script in the second audio script set, an audio generation model is assigned to each audio script to obtain the first audio generation model set. Based on the adaptability evaluation results of the first audio generation model set, the first audio generation model set is optimized to obtain a second audio generation model set; wherein, the adaptability evaluation results of the first audio generation model set are obtained by evaluating the adaptability of each audio generation model according to the type of each audio script. Based on the second set of audio scripts, audio is generated using the second set of audio generation models to obtain an audio set; Based on the accuracy evaluation results of the audio set, the audio set is optimized to generate multiple audio outputs corresponding to the multimodal input; wherein, the accuracy evaluation results of the audio set are obtained by evaluating the accuracy of each generated audio based on the content of each audio script.

2. The multi-audio generation method for multimodal audio as described in claim 1, characterized in that, The process of decomposing and planning the multimodal input to obtain a first audio script set includes: Feature analysis is performed on the multimodal input to obtain the overall audio requirements; Based on semantic and temporal information, the overall audio requirement is decomposed into multiple audio events; Multiple audio scripts are generated based on the type of each audio event; one audio script corresponds to one audio event. The generated audio scripts are integrated to obtain the first audio script set.

3. The multi-audio generation method for multimodal audio as described in claim 1, characterized in that, The step of optimizing the first audio script set based on the integrity evaluation result of the first audio script set to obtain the second audio script set includes: The first audio script set is evaluated according to a preset first requirement; wherein, the first requirement specifically means that all audio events included in the multimodal input are described by audio scripts; Complete the missing audio scripts in the first audio script set until the first audio script set meets the first requirement; The original audio scripts are integrated with the supplementary audio scripts to obtain a second set of audio scripts.

4. The multi-audio generation method for multimodal audio as described in claim 1, characterized in that, The step of assigning an audio generation model to each audio script based on its type in the second audio script set, thereby obtaining a first audio generation model set, includes: Based on the type of each audio script in the second audio script set, an audio generation model is selected from a preset model library; wherein, the model library consists of multiple pre-trained audio generation models; The selected audio generation models are integrated to obtain the first audio generation model set.

5. The multi-audio generation method for multimodal audio as described in claim 1, characterized in that, The step of optimizing the first audio generation model set based on the adaptability evaluation results of the first audio generation model set to obtain a second audio generation model set includes: According to the preset second requirement, each audio generation model in the first audio generation model set is evaluated; wherein, the second requirement is specifically: the type of the audio generation model is compatible with the type of the audio script; Replace the audio generation models in the first audio generation model set that do not meet the second requirement until all audio generation models meet the second requirement; The audio generation models that meet the second requirement are integrated to obtain the second audio generation model set.

6. The multi-audio generation method for multimodal audio as described in claim 1, characterized in that, The step of generating an audio set from the second audio script set and the second audio generation model set includes: Each audio generation model in the second audio generation model set generates audio based on its corresponding audio script, resulting in multiple audio files; The generated audio files are then integrated to obtain an audio set.

7. The multi-audio generation method for multimodal audio as described in claim 1, characterized in that, The step of optimizing the audio set based on the accuracy evaluation results of the audio set to generate multi-audio outputs corresponding to the multimodal input includes: According to a preset third requirement, each audio in the audio set is evaluated; wherein, the third requirement is specifically: the generated audio can accurately reproduce the content of the corresponding audio script; The audio files that do not meet the third requirement in the audio set are regenerated until all audio files meet the third requirement. The audio samples that meet the third requirement are synthesized to generate the multi-audio output corresponding to the multimodal input.

8. A multi-audio frequency generation device for multimodal audio, characterized in that, include: The system includes an audio script generation module, an audio script evaluation module, an audio model allocation module, an audio model evaluation module, an audio generation module, and an audio evaluation module. The audio script generation module is used to perform task decomposition and script planning on multimodal input to obtain a first audio script set; The audio script evaluation module is used to optimize the first audio script set based on the integrity evaluation result of the first audio script set to obtain a second audio script set; wherein, the integrity evaluation result of the first audio script set is obtained by evaluating the integrity of the first audio script set based on the audio events contained in the multimodal input; The audio model allocation module is used to allocate an audio generation model to each audio script according to the type of each audio script in the second audio script set, thereby obtaining the first audio generation model set; The audio model evaluation module is used to optimize the first audio generation model set based on the adaptability evaluation result of the first audio generation model set to obtain a second audio generation model set; wherein, the adaptability evaluation result of the first audio generation model set is obtained by evaluating the adaptability of each audio generation model based on the type of each audio script. The audio generation module is used to generate audio from the second audio generation model set based on the second audio script set to obtain an audio set. The audio evaluation module is used to optimize the audio set based on the accuracy evaluation result of the audio set, and generate multiple audio outputs corresponding to the multimodal input; wherein, the accuracy evaluation result of the audio set is obtained by evaluating the accuracy of each generated audio based on the content of each audio script.

9. A terminal device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein when the processor executes the computer program, it implements a multi-modal multi-audio generation method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, include: A stored computer program, wherein, when the computer program is executed, it controls the device containing the computer-readable storage medium to perform a multi-modal multi-audio generation method as described in any one of claims 1-7.

Images