Information generation method and device, electronic equipment and storage medium

Abstract

Embodiments of the present disclosure provide an information generation method and device, electronic equipment and storage medium. The method comprises: obtaining multi-modal input data, the multi-modal input data comprising at least two media data, the multi-modal input data being used to represent a target intention based on the at least two media data; processing the multi-modal input data by a music processing agent, and calling at least one music processing plug-in according to a processing result to obtain media resources corresponding to each music processing plug-in; and combining the media resources corresponding to each music processing plug-in to generate visual music information matching the target intention. The multi-modal input data composed of at least two media data is obtained by the music processing agent to better understand the target intention of the user, and then the music processing plug-in is called to obtain the corresponding media resources, so that the visual music information matching the target intention is generated, the visual music information generated finally is more accurate, and the interaction efficiency is improved.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and in particular to an information generation method, apparatus, electronic device, and storage medium. Background Technology

[0002] Currently, with the rise and development of large models, content generation capabilities based on large models are being used more and more widely, such as for article continuation, content summarization, and image generation based on the semantic understanding and reasoning capabilities of large models.

[0003] In existing technologies, music processing functions based on large models are usually implemented by processing user-input descriptive text through large models. However, in existing solutions, large models have difficulty accurately obtaining user intent, resulting in poor performance and low interaction efficiency in music processing functions based on large models. Summary of the Invention

[0004] This disclosure provides an information generation method, apparatus, electronic device, and storage medium to overcome the problems of poor performance and low interaction efficiency in music processing functions.

[0005] In a first aspect, embodiments of this disclosure provide an information generation method, including:

[0006] Acquire multimodal input data, which includes at least two types of media data, and use the multimodal input data to characterize the target intent based on the at least two types of media data; process the multimodal input data through a music processing agent, and call at least one music processing plugin according to the processing result to obtain the media resources corresponding to each music processing plugin; combine the media resources corresponding to each music processing plugin to generate visual music information that matches the target intent.

[0007] Secondly, embodiments of this disclosure provide an information generation apparatus, comprising:

[0008] An acquisition module is used to acquire multimodal input data, the multimodal input data including at least two types of media data, the multimodal input data being used to characterize the target intent based on the at least two types of media data;

[0009] The processing module is used to process the multimodal input data through a music processing agent, and to call at least one music processing plugin according to the processing result to obtain the media resources corresponding to each music processing plugin.

[0010] The generation module is used to combine the media resources corresponding to each of the music processing plugins to generate visual music information that matches the target intent.

[0011] Thirdly, embodiments of this disclosure provide an electronic device, including: a processor and a memory;

[0012] The memory stores computer-executed instructions;

[0013] The processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the information generation method as described in the first aspect and various possible designs of the first aspect.

[0014] Fourthly, embodiments of this disclosure provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the information generation method described in the first aspect and various possible designs of the first aspect.

[0015] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the information generation method described in the first aspect and various possible designs of the first aspect.

[0016] The information generation method, apparatus, electronic device, and storage medium provided in this embodiment acquire multimodal input data, which includes at least two types of media data. This multimodal input data is used to characterize a target intent based on the at least two types of media data. A music processing agent processes the multimodal input data and, based on the processing result, calls at least one music processing plugin to obtain media resources corresponding to each plugin. The media resources corresponding to each music processing plugin are then combined to generate visual music information matching the target intent. By acquiring multimodal input data composed of at least two types of media data and performing inference through the music processing agent, the target intent of the user is better understood. Then, music processing plugins are called to obtain corresponding media resources. After combining these media resources, visual music information matching the target intent is generated. This makes the large model's understanding of the user's intent more accurate, allowing for the use of more suitable music processing plugins to complete the music processing function. The resulting visual music information is more accurate and better meets user needs. Furthermore, because the accuracy of the user's intent is improved, there is no need for repeated supplementation and modification of the large model input, thus improving the interactive efficiency during the music processing function implementation process. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this disclosure. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0018] Figure 1 This is an application scenario diagram of the information generation method provided in the embodiments of this disclosure;

[0019] Figure 2 Flowchart of the information generation method provided in the embodiments of this disclosure Figure 1 ;

[0020] Figure 3 This is a schematic diagram illustrating a process for generating visualized music information, provided as an embodiment of the present disclosure.

[0021] Figure 4 for Figure 2 A flowchart illustrating the specific implementation of step S103 in the illustrated embodiment;

[0022] Figure 5 for Figure 4 A flowchart illustrating the specific implementation of step S1031 in the illustrated embodiment;

[0023] Figure 6 Flowchart of the information generation method provided in the embodiments of this disclosure Figure 2 ;

[0024] Figure 7 for Figure 6 A flowchart illustrating the specific implementation of step S203 in the illustrated embodiment;

[0025] Figure 8 for Figure 6 A flowchart illustrating the specific implementation of step S204A in the illustrated embodiment;

[0026] Figure 9 This is a schematic diagram illustrating the invocation process of a music processing plugin, as provided in an embodiment of this disclosure.

[0027] Figure 10 This is a schematic diagram illustrating a process for generating visualized music information, provided as an embodiment of the present disclosure.

[0028] Figure 11 This is a structural block diagram of the information generation apparatus provided in the embodiments of this disclosure;

[0029] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0030] Figure 13 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0031] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0032] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals are provided for users to choose to authorize or refuse.

[0033] The application scenarios of the embodiments of this disclosure are explained below:

[0034] The information generation method provided in this disclosure can be applied to applications (APPs) with music search and playback functions, such as music applications, intelligent AI assistant programs, and browser programs. More specifically, it can be applied to application scenarios such as music search and intelligent background music based on AI assistants. The execution subject of this embodiment can be a terminal device running the aforementioned application with music search and playback functions, a server deploying the server corresponding to the aforementioned application, or other electronic devices that perform similar functions. When the execution subject is a terminal device, the terminal device executes the method provided in this embodiment by running the aforementioned application; when the execution subject is a server, the server of the aforementioned application with music search and playback functions can run partially or entirely on the server, and the method provided in this embodiment is executed on the server side, while the terminal device runs the client of the application. Communication between the server and the terminal device is based on server-client communication, thereby enabling the terminal device to obtain the execution result of the method provided in this embodiment and display it as needed.

[0035] In some embodiments, the terminal device or server can implement the information generation method provided in this disclosure by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be program-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be local applications, i.e., programs that need to be installed in the operating system to run, or mini-programs embedded in any app, i.e., programs that run in a browser environment. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin; the specific implementation can be configured as needed. Furthermore, in implementing the information generation method provided in this disclosure, the terminal device can execute the method by running computer-executable instructions or computer programs set locally, or by calling computer-executable instructions or computer programs set in an external server. In some embodiments, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be interactive processing services that can be invoked by terminal devices.

[0036] Figure 1 This is an application scenario diagram of the information generation method provided in the embodiments of this disclosure, with reference to... Figure 1 As shown in the diagram, taking a terminal device as an example, the terminal device runs a target application with music search functionality. The user enters their request text through the target application's interface, such as "Find songs released by artist X from 2000 to 2010." The target application then performs semantic understanding using a large model and conducts an online search, retrieving matching query results from the internet or the target application's data server. The application then displays the song title, playback link, and other information within the target application, thus responding to the user's request (song search). For example, as shown in the diagram, the interface displays the text "The following results were found for you," along with the corresponding song playback link.

[0037] In the prior art, for example Figure 1In the illustrated use cases of music processing functions, when user needs are relatively clear and simple, using a large model to semantically understand the requirement text can accurately execute the corresponding functional processing steps, thereby obtaining a processing result that meets the user's needs. However, for some more complex application scenarios and needs, such as intelligent background music for speech or images, music search based on emotional features, and music recognition, it is difficult to accurately and clearly describe the user's intent using only the requirement text. Since large models typically use text as input for recognition and reasoning, they often struggle to accurately identify user intent, leading to biased or even erroneous processing instructions. This results in a reduction in the effectiveness of the music processing function and lowers its performance. Furthermore, describing user needs solely through requirement text increases the design complexity of the requirement text, causing users to repeatedly modify and debug the description text, ultimately leading to low interaction efficiency.

[0038] This disclosure provides an information generation method to solve the above-mentioned problems.

[0039] refer to Figure 2 , Figure 2 Flowchart of the information generation method provided in the embodiments of this disclosure Figure 1 The method of this embodiment can be applied to a terminal device or a server. In one possible implementation, for a terminal device executing the method provided in this embodiment, the terminal device can execute locally deployed program code to implement the information generation method provided in this embodiment. In another possible implementation, a server can be used to deploy functional services implemented based on the information generation method provided in this embodiment, and the terminal device can access the server and call the corresponding functional services to implement the information generation method provided in this embodiment. For example, the information generation method provided in this embodiment includes:

[0040] Step S101: Acquire multimodal input data, which includes at least two types of media data. The multimodal input data is used to characterize the target intent based on at least two types of media data.

[0041] Step S102: The music processing agent processes the multimodal input data and calls at least one music processing plugin based on the processing result to obtain the media resources corresponding to each music processing plugin.

[0042] Step S103: Combine the media resources corresponding to each music processing plugin to generate visual music information that matches the target intent.

[0043] refer to Figure 1The illustrated application scenario diagram illustrates the information generation method provided, using a terminal device as the execution subject. For example, the terminal device runs a target application and provides an interactive interface to the user. The user selects data from the interface (local or cloud), uses voice or keyboard input, etc., to load multimodal input data. This multimodal input data consists of at least two types of media data, used to represent the target intent based on these media data. Specifically, the media data includes text, images, audio, and video. In one example, the multimodal input data consists of text and images, which can be represented as RData_1 = [text_1, pic_1, pic_2]. Here, RData_1 is the multimodal input data, text_1 is a piece of text, and pic_1 and pic_2 are two different images. In another example, the multimodal input data consists of images, audio, and video, which can be represented as RData_2 = [pic_1, audio_1, video_1]. In this dataset, RData_2 represents multimodal input data, pic_1 is an image, audio_1 is an audio clip, and video_1 is a video clip. At least two types of media data are used to represent the target intent. For example, the multimodal input data consists of text and video data. The text data contains the phrase "provide background music for the following video." Thus, by using this text and video data, the user's target intent is accurately described, solving the problem that it is difficult to clearly express user needs using only text data (the requirement text).

[0044] Furthermore, the aforementioned multimodal input data can be obtained by the terminal device in response to the user's corresponding operation. For example, text data can be obtained by responding to the text entered by the user in the text box in the interactive interface, or video data can be obtained by loading a local video in response to the user's selection command in the interactive interface, and so on. The specific implementation method for the terminal device to obtain multimodal input data can be set as needed, and will not be elaborated here.

[0045] Furthermore, after acquiring multimodal input data, the terminal device invokes a pre-built music processing agent to process it. This music processing agent is a program with music processing capabilities used to perform music-related tasks. It is implemented based on a large model, leveraging the language understanding, reasoning, and generation capabilities provided by the large model to assist the music processing agent in making final decisions and executing actions. Specifically, the music processing agent performs specific skill actions by invoking corresponding music processing plugins. These plugins are programs used to implement specific skills and functions, and are connected to the music processing agent in a pluggable manner. The music processing agent can load music processing plugins to use their functions and unload them after use, reducing interface and resource consumption. For example, when the music processing agent needs to execute skill action A (e.g., intelligent background music), it invokes a music processing plugin ex_A capable of executing skill action A. Correspondingly, after the music processing plugin completes the corresponding action, it will output a corresponding processing result, namely the media resource corresponding to the music processing plugin. The media resource can be one or more of text, images, audio, and video.

[0046] In this process, after processing multimodal input data, the music processing agent may invoke one or more music processing plugins to perform corresponding actions based on the user's target intent, thereby obtaining one or more corresponding media resources. Then, these media resources need to be combined to obtain a processing result that matches the user's target intent for display, i.e., visualized music information.

[0047] Figure 3 This is a schematic diagram illustrating a process for generating visualized music information according to an embodiment of the present disclosure. The following is in conjunction with... Figure 3 To further explain the above process, such as... Figure 3 As shown, for example, the multimodal input data consists of text (Text_1) and an image (Pic_1), and its target intent can be represented as "to add background music to an image." After the steps described above, where the music processing agent calls the corresponding music processing plugin E1 for processing, three media resources are obtained: a piece of text (Text_2), an image (Pic_2), and an audio (Audio_1). These three media resources are combined in a certain layout to form visualized music information, which is then displayed on the terminal device. More specifically, the text (Text_2) may be, for example, an introduction to the image content and / or the audio characteristics and name; the image (Pic_2) may be, for example, a music album cover similar to the image (Pic_1); and the audio (Audio_1) is the background music result for the image (Pic_1).

[0048] Furthermore, in one possible implementation, such as Figure 4 As shown, the specific implementation of step S103 includes:

[0049] Step S1031: Obtain a visualization template that matches the target intent. The visualization template is used to determine the layout style of the visualized music information.

[0050] Step S1032: Based on the visualization template, combine the media resources generated by each music processing plugin to generate visualized music information that matches the target intent.

[0051] For example, refer to Figure 3 As shown, in the process of generating and displaying visualized music information within the interactive interface of a target application, one possible implementation is based on a fixed style template. For example, text resources are fixed in the first display position within the interactive interface, video resources are fixed in the second display position, etc., to generate visualized music information and present the processing result. In another possible implementation, as in the steps of this embodiment, a visualization template matching the target intent is obtained, i.e., the visualization template is dynamically obtained. Then, the media resources are placed according to the layout style indicated by the visualization template. Therefore, different visualization templates are obtained when the target intent is different, resulting in different layout styles for the visualized music information. For example, when the first type of intent (e.g., intelligent background music) is present, the generated audio resources are placed in the first display position within the interactive interface; when the second type of intent (e.g., music recognition) is present, the generated audio resources are placed in the second display position within the interactive interface. Through the steps of this embodiment, the visualized music information can present a layout style matching the user's intent, improving information display efficiency and effect.

[0052] Furthermore, in one possible implementation, the visualization template includes a first visualization template, which is used to characterize the display position of the target media resource corresponding to the target intent, such as... Figure 5 As shown, the specific implementation of step S1031 includes:

[0053] Step S1031A-1: The music processing agent performs intent recognition on the multimodal input data to obtain intent features that describe the target intent.

[0054] Step S1031A-2: Based on the generative model, process the intent features and generate the first visualization template.

[0055] For example, in the steps of this embodiment, firstly, the music processing agent performs intent recognition on the multimodal input data to obtain intent features. Specifically, the intent features can be feature vectors or feature matrices composed of feature values. The target intent is described by the feature vectors or feature matrices. The more feature values ​​there are, the more complex and accurate the target intent can be expressed, thereby achieving a precise expression of the target intent. Next, the intent features are input into a pre-trained generative model. The generative model's capabilities are used to generate a visualization template matching the intent features, i.e., a first visualization template. This first visualization template is used to represent the display position of at least one media resource. The first visualization template constrains the first display position corresponding to the text resource, more specifically, for example, a prominent display position in the center of the interactive interface, thereby satisfying a specific target intent. For the positions of media resources not constrained by the first visualization template, they can be determined based on a basic template, or the first visualization template can constrain the positions of all target media resources corresponding to the target intent.

[0056] In this embodiment, intent features describing the target intent are obtained by performing intent recognition on the multimodal input data. A first visualization template is then generated based on these intent features. This first visualization template is a layout design specifically for the target media resources corresponding to the target intent. Therefore, the number of media resource slots in the first visualization template is determined based on the target intent, rather than being a fixed number. That is, there is no need to reserve media resource slots not involved in the target intent, thus allowing the layout style represented by the first visualization template to better match the intent features, improving layout rationality and the efficiency of visualizing music information.

[0057] In another possible implementation, the visualization template includes a second visualization template, which is used to characterize the user interface layout based on the target number of media resource bits. Accordingly, the specific implementation of step S1031 includes:

[0058] Step S1031B-1: The music processing agent performs intent recognition on the multimodal input data to obtain the target intent category and the number of media resource bits corresponding to the target intent;

[0059] Step S1031B-2: Based on the target intent category and the number of media resource bits, obtain the corresponding second visualization template. The second visualization template is used to represent the user interface layout composed of media resource bits based on the target number.

[0060] For example, in this embodiment, a music processing agent performs intent recognition on the multimodal input data to obtain the target intent category corresponding to the target intent. The target intent category is one of the preset intent categories and can be represented by a category identifier. More specifically, the target intent category includes at least one of the following: intelligent background music, music search, song recognition, and music introduction. Then, based on the target intent category, a corresponding visualization template is retrieved from a pre-generated visualization template library to obtain a second visualization template. This second visualization template is used to represent the user interface layout based on the target number of media resource bits.

[0061] Furthermore, in one possible implementation, the number of media resource bits is a fixed value corresponding to the target intent category. For example, when the target intent category is intelligent background music, the corresponding number of bits is 3; when the target intent category is music search, the corresponding number of bits is 2, and so on. The above mapping relationship can be set as needed and is not limited. In another possible implementation, the number of media resource bits is a value independent of the target intent category, obtained after the music processing agent performs intent recognition on the multimodal input data. Specifically, it is implemented by the capabilities provided by the pre-trained music processing agent; the principle is not elaborated here.

[0062] In this embodiment, the music processing agent performs intent recognition on the multimodal input data to obtain the target intent category and the number of media resource bits. Then, based on the target intent category and the number of media resource bits, a second visualization template representing the layout of the user interface is determined, which improves the layout rationality of the second visualization template. At the same time, the second visualization template is pre-generated, which has a fast matching speed and low computational resource overhead, thereby improving the generation speed of visualized music information.

[0063] The information generation method provided in this embodiment acquires multimodal input data, which includes at least two types of media data. This multimodal input data is used to characterize the target intent based on these two types of media data. A music processing agent processes the multimodal input data and, based on the processing results, calls at least one music processing plugin to obtain the media resources corresponding to each plugin. The media resources corresponding to each music processing plugin are then combined to generate visual music information that matches the target intent. By acquiring multimodal input data composed of at least two types of media data and performing inference through the music processing agent, the user's target intent is better understood. Then, music processing plugins are called to obtain the corresponding media resources. After combining these media resources, visual music information matching the target intent is generated. This makes the large model's understanding of the user's intent more accurate, allowing for the use of more suitable music processing plugins to complete the music processing function. The resulting visual music information is more accurate and better meets user needs. Furthermore, because the accuracy of the user's intent is improved, there is no need to repeatedly supplement or modify the large model input, thus improving the interactive efficiency during the music processing function implementation process.

[0064] refer to Figure 6 , Figure 6 Flowchart of the information generation method provided in the embodiments of this disclosure Figure 2 This embodiment is in Figure 2 Based on the illustrated embodiment, step S102 is further refined, and the information generation method includes:

[0065] Step S201: Obtain multimodal input data, which includes at least two types of media data. The multimodal input data is used to characterize the target intent based on at least two types of media data.

[0066] Step S202: The music processing agent performs intent recognition on the multimodal input data to obtain the target intent category corresponding to the target intent.

[0067] Step S203: Determine the corresponding target music processing plugin based on the target intent category.

[0068] For example, combined Figure 2The illustrated embodiment describes a process where, after obtaining multimodal input data, a music processing agent is invoked to perform intent recognition on the multimodal input data, thereby obtaining the target intent category corresponding to the multimodal input data. Specifically, for example, the music processing agent first extracts features from the multimodal input data to obtain an intent feature matrix or intent feature vector representing the target intent. Then, the intent feature matrix or intent feature vector is classified into a preset category, thus determining the target intent category. For example, preset categories include: intelligent background music, music search, song recognition, and music introduction. These target intent categories correspond to function categories or skill categories. Then, based on the target intent category, a corresponding music processing plugin capable of implementing the corresponding function or skill is selected, i.e., the target music processing plugin. There is a pre-configured mapping relationship between the intent category and the music processing plugin, or the music processing agent has the ability to load and use music processing plugins corresponding to the preset intent category, thus realizing the step of determining the corresponding target music processing plugin according to the target intent category.

[0069] For example, in another possible implementation, the music processing plugin is implemented based on workflow invocation. Specifically, based on the aforementioned target intent category, a corresponding workflow capable of implementing that target intent category is selected, i.e., the target workflow. Then, during the execution of the target workflow, when a more specific function or skill needs to be implemented, the corresponding music processing plugin, i.e., the target music processing plugin, is invoked. Here, there is a pre-configured mapping relationship between the intent category and the workflow, and the specific implementation of the workflow is prior knowledge. Alternatively, the music processing agent has the ability to load and use workflows corresponding to preset intent categories, thereby realizing the steps of determining the corresponding workflow based on the target intent category, and then determining and invoking the target music processing plugin.

[0070] One possible implementation is, such as Figure 7 As shown, the specific implementation of step S203 includes:

[0071] Step S2031: Based on the target intent category, obtain the corresponding visualization template. The visualization template is used to represent at least one target media resource that constitutes the visualized music information.

[0072] Step S2032: Based on the visualization template, obtain the target music processing plugin used to generate each target media resource.

[0073] In another possible implementation, after determining the target intent category, a corresponding visualization template can be determined first. Different intent categories correspond to different visualization templates, allowing the visualized music information to be presented with different target media resources and different layout styles among them. Then, based on the target media resources (e.g., text or video resources) corresponding to the visualization template, a corresponding workflow is triggered. This then calls the music processing plugin within the workflow, essentially using the result (media resource) to deduce the plugin that generated the result (target music processing plugin). Finally, based on this, the control parameters input to the target music processing plugin are used to characterize and control the target intent, ensuring that the media resources output by the target music processing plugin satisfy the target intent.

[0074] In this embodiment, the steps involve first determining a visualization template, and then using the target media resources in the visualization template to reverse-determine the target music processing plugin. Since the visualization template is directly determined based on the target intent category, the accuracy of the generated visualization music information and its matching degree with the target intent can be improved.

[0075] Step S204: Call the target music processing plugin to obtain the corresponding media resources.

[0076] For example, after identifying the target music processing plugin, the music processing agent invokes the target music processing plugin to use its capabilities, thereby generating media resources that match the target intent.

[0077] In another possible implementation, after step S202, the following is also included:

[0078] Step S203A: Based on the target intent category, the music processing agent extracts parameters from the multimodal input data to obtain the feature parameters corresponding to the target intent category.

[0079] Step S203B: Convert the feature parameters into control parameters that match the target music processing plugin.

[0080] For example, in another implementation, after determining the target intent category, a music processing agent extracts parameters from the multimodal input data to obtain feature parameters corresponding to the target intent category. The target intent category corresponds to a function category or skill category. Therefore, the feature parameters corresponding to the target intent category are the input parameters used to execute the function or skill equivalent to the target intent category. For example, if the target intent category is intelligent background music, the feature parameters can be parameters representing image style, content, and other information obtained by the music processing agent after extracting parameters from the image data in the multimodal input data. Then, based on the previously obtained interface rules of the target music processing plugin, the feature parameters are converted into control parameters matching the target music processing plugin for input into the target music processing plugin in subsequent steps.

[0081] Accordingly, the specific implementation of step S204 includes:

[0082] Step S204A: Based on the control parameters, call the target music processing plugin to obtain the corresponding media resources.

[0083] In this embodiment, by extracting parameters from the multimodal input data, feature parameters corresponding to the target intent category are obtained. These feature parameters are then converted into control parameters and input into the target music processing plugin. The processing of the target music processing plugin is constrained by the control parameters, thereby making the media resources output by the target music processing plugin more accurate.

[0084] Furthermore, in one possible implementation, such as Figure 8 As shown, the specific implementation of step S204A includes:

[0085] Step S2041: Call the target music processing plugin to obtain the corresponding music processing workflow.

[0086] Step S2042: Based on the music processing workflow, process the control parameters of the target music processing plugin in sequence to generate plugin output resources.

[0087] Step S2043: Convert the format of the plugin output resources to generate media resources with the target format, wherein the target format corresponds to the target intent.

[0088] For example, the above steps illustrate the specific working principle of the target music processing plugin. First, by calling the target music processing plugin, the corresponding program is executed, thereby obtaining the processing flow recorded in the program, i.e., the music processing workflow. This music processing workflow can consist of multiple work nodes. Then, based on each work node of the music processing workflow, the control parameters are processed sequentially. For example, the control parameters can be input at the first work node of the music processing workflow or at a non-first work node. Afterward, the music processing workflow processes sequentially in an iterative manner (i.e., the output of the previous work node is used as the input of the next work node) until the last work node is completed, at which point the plugin output resources are output. Finally, based on a preset display format, or based on the text format or image format corresponding to the target intent type, or based on the format requirements of the visualization template, the plugin output resources are format-converted to generate media resources with the target format. The purpose of this format conversion step is to unify the format of the plugin output resources from different music processing workflows, thereby facilitating the combination of media resources in subsequent steps. The specific format conversion method can be configured as needed and is not limited here.

[0089] Figure 9 This is a schematic diagram illustrating the invocation process of a music processing plugin according to an embodiment of the present disclosure. The following is in conjunction with... Figure 9 To further explain the above process, such as... Figure 9 As shown, firstly, the user inputs multimodal input data through the interactive interface. Then, the historical dialogue record and the above multimodal input data are sent to the music processing agent for processing. The music processing agent judges the intent based on the multimodal input data and obtains the corresponding target intent category as "intelligent background music". Then, the corresponding workflow Workflow_1 is determined. Workflow_1 contains a worker node P1 that calls the intelligent background music plugin Plugin_1. The intelligent background music plugin Plugin_1 is deployed on the server. After the worker node P1 corresponding to the intelligent background music plugin Plugin_1 is triggered, it performs recommended music analysis based on the control parameters (image or video frame extraction) input to the intelligent background music plugin to obtain music information that matches the above control parameters (such as music name and album cover image). The information is then returned to the music processing agent, which continues to execute the next work node P2 in Workflow_1. This work node is used to convert the format of the music information and combine the music name, album cover image, and other content in the music information based on the visualization template to form visualized music information, which is displayed in the interactive interface in the form of "text + card". The text is the music name, and the card is used to carry the album cover image.

[0090] Step S205: Combine the media resources corresponding to each music processing plugin to generate visual music information that matches the target intent.

[0091] After obtaining the media resources corresponding to each music processing plugin based on the above steps, these resources can be further combined with visualization templates to form visualized music information with a layout style that matches the target intent. For specific implementation methods and technical effects, please refer to [link / reference needed]. Figure 2 The relevant descriptions in the illustrated embodiments will not be repeated here.

[0092] Furthermore, Figure 10 This is a schematic diagram illustrating a process for generating visualized music information according to an embodiment of the present disclosure. The following is in conjunction with... Figure 10 The steps of the above embodiments will be described in more detail, such as... Figure 10 As shown, firstly, the user inputs multimodal input data, including text, images, videos, and audio, through the interactive interface. Then, the historical dialogue record and the multimodal input data are sent to the music processing agent. After obtaining the historical dialogue record and multimodal input data, the music processing agent first performs intent recognition and parameter extraction based on system prompts, obtaining the workflow corresponding to the target intent category and the corresponding control parameters. Then, based on the control parameters, it executes the corresponding workflow to implement the corresponding music processing tasks, such as the intelligent background music task, music search task, song recognition task, and music encyclopedia task shown in the figure. During execution, it calls the corresponding target music processing plugins, thereby generating the media resources output by each plugin. Finally, based on a visualization template, it combines and lays out the content of the media resources to generate visual music information matching the target intent, and displays this visual music information on the interactive interface, thus completing the music processing process.

[0093] Optionally, after step S205, the method further includes:

[0094] Step S206: For the visualized music information, load and display at least one interactive control, wherein the interactive control corresponds one-to-one with the media resources constituting the visualized music information, and the interactive control is used to control the display state of the corresponding media resources.

[0095] For example, visualized music information can be displayed in the interactive interface of the target application in the form of interactive cards. After generating the visualized music information, corresponding interactive controls can be further configured in the interactive cards corresponding to the visualized music information. For example, playback controls for music and video resources in the visualized music information, and zoom and page-turning controls for image resources in the visualized music information. Then, based on the above-loaded interactive controls, the display status of media resources can be controlled, such as playback and pause controls for music resources, and zoom and page-turning controls for image resources. The loading of the above-mentioned interactive controls can be implemented through interactive function units in the target application. The interactive function units can obtain different interactive controls from local or external sources and load them into the interactive interface of the target application.

[0096] In this embodiment, the implementation methods of steps S201 and S205 are the same as those of this disclosure. Figure 2 The implementation methods of steps S101 and S103 in the illustrated embodiment are the same, and will not be described in detail here.

[0097] Corresponding to the information generation method in the above embodiments, Figure 11 This is a structural block diagram of the information generation apparatus provided in the embodiments of this disclosure. The method described in the above embodiments can be executed by this information generation apparatus, which can be implemented by software and / or hardware, and can be integrated into an electronic device with certain data processing capabilities. The electronic device may include, but is not limited to, mobile terminals with big data processing capabilities, as well as fixed terminals with big data processing capabilities such as desktop computers and supercomputers.

[0098] For ease of explanation, only the parts relevant to embodiments of this disclosure are shown. (Refer to...) Figure 11 The information generation device 3 includes:

[0099] The acquisition module 31 is used to acquire multimodal input data, which includes at least two types of media data. The multimodal input data is used to characterize the target intent based on at least two types of media data.

[0100] The processing module 32 is used to process multimodal input data through a music processing agent, and call at least one music processing plugin according to the processing result to obtain the media resources corresponding to each music processing plugin.

[0101] The generation module 33 is used to combine the media resources corresponding to each music processing plugin to generate visual music information that matches the target intent.

[0102] According to one or more embodiments of this disclosure, the processing module 32 is specifically configured to: perform intent recognition on multimodal input data through a music processing agent to obtain the target intent category corresponding to the target intent; determine the corresponding target music processing plugin based on the target intent category; call the target music processing plugin to obtain the corresponding media resources.

[0103] According to one or more embodiments of this disclosure, the processing module 32 is further configured to: extract parameters from the multimodal input data based on the target intent category using a music processing agent to obtain feature parameters corresponding to the target intent category; convert the feature parameters into control parameters that match the target music processing plugin; when the processing module 32 calls the target music processing plugin to obtain the corresponding media resources, it is specifically configured to: call the target music processing plugin based on the control parameters to obtain the corresponding media resources.

[0104] According to one or more embodiments of this disclosure, when the processing module 32 calls the target music processing plugin to obtain the corresponding media resources, it is specifically used to: call the target music processing plugin to obtain the corresponding music processing workflow; based on the music processing workflow, process the control parameters of the target music processing plugin in sequence to generate plugin output resources; and perform format conversion on the plugin output resources to generate media resources with the target format, wherein the target format corresponds to the target intent.

[0105] According to one or more embodiments of this disclosure, when the processing module 32 determines the corresponding target music processing plugin based on the target intent category, it is specifically used to: obtain the corresponding visualization template based on the target intent category, wherein the visualization template is used to characterize at least one target media resource constituting the visualization music information; and obtain the target music processing plugin for generating each target media resource based on the visualization template.

[0106] According to one or more embodiments of this disclosure, the target intent category includes at least one of the following: intelligent background music, music search, song recognition, and music introduction.

[0107] According to one or more embodiments of this disclosure, the generation module 33 is specifically used for: obtaining a visualization template that matches the target intent, the visualization template being used to determine the layout style of the visualization music information; and combining the media resources generated by each music processing plugin based on the visualization template to generate visualization music information that matches the target intent.

[0108] According to one or more embodiments of this disclosure, the visualization template includes a first visualization template. When the generation module 33 obtains a visualization template that matches the target intent, it is specifically used to: perform intent recognition on the multimodal input data through a music processing agent to obtain intent features used to describe the target intent; process the intent features based on the generation model to generate the first visualization template, which is used to characterize the display position of the target media resource corresponding to the target intent.

[0109] According to one or more embodiments of this disclosure, the visualization template includes a second visualization template. When the generation module 33 obtains a visualization template that matches the target intent, it is specifically used to: perform intent recognition on the multimodal input data through a music processing agent to obtain the target intent category and the number of media resource bits corresponding to the target intent; and obtain the corresponding second visualization template according to the target intent category and the number of media resource bits. The second visualization template is used to characterize the user interface layout composed of media resource bits based on the target number.

[0110] According to one or more embodiments of this disclosure, the generation module 33 is further configured to: load and display at least one interactive control for the visualized music information, wherein the interactive control corresponds one-to-one with the media resources constituting the visualized music information, and the interactive control is used to control the display state of the corresponding media resources.

[0111] The acquisition module 31, processing module 32, and generation module 33 are connected sequentially. The information generation device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, so it will not be described again here.

[0112] Figure 12 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure, such as... Figure 12 As shown, the electronic device 4 includes:

[0113] Processor 41, and memory 42 communicatively connected to processor 41;

[0114] Memory 42 stores instructions executed by the computer;

[0115] The processor 41 executes computer execution instructions stored in the memory 42 to achieve, for example, Figures 2-10 The information generation method in the illustrated embodiment.

[0116] Optionally, the processor 41 and the memory 42 are connected via a bus 43.

[0117] For relevant instructions, please refer to the corresponding text. Figures 2-10 The relevant descriptions and effects of the steps in the corresponding embodiments are understood, and will not be elaborated on here.

[0118] This disclosure provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement this disclosure. Figures 2-10 The information generation method provided in any of the corresponding embodiments.

[0119] This disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements this disclosure. Figures 2-10 The information generation method provided in any of the corresponding embodiments.

[0120] To implement the above embodiments, this disclosure also provides an electronic device.

[0121] refer to Figure 13 The diagram illustrates a structural schematic of an electronic device 900 suitable for implementing embodiments of the present disclosure. The electronic device 900 can be a terminal device or a server. The terminal device can include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers, portable media players (PMPs), and in-vehicle terminals (e.g., in-vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 13 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0122] like Figure 13 As shown, the electronic device 900 may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 into a random access memory (RAM) 903. The RAM 903 also stores various programs and data required for the operation of the electronic device 900. The processing unit 901, ROM 902, and RAM 903 are interconnected via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0123] Typically, the following devices can be connected to I / O interface 905: input devices 906 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 907 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 908 including, for example, magnetic tapes, hard disks, etc.; and communication devices 909. Communication device 909 allows electronic device 900 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 13 An electronic device 900 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0124] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by a processing device 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0125] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0126] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0127] The aforementioned computer-readable medium carries one or more programs, which, when executed by the electronic device, cause the electronic device to perform the methods shown in the above embodiments.

[0128] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, and C++, and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0129] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0130] The units or modules described in the embodiments of this disclosure can be implemented in software or hardware. The names of the units or modules do not necessarily limit the specific unit itself.

[0131] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0132] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0133] In a first aspect, according to one or more embodiments of this disclosure, an information generation method is provided, comprising:

[0134] Acquire multimodal input data, which includes at least two types of media data, and use the multimodal input data to characterize the target intent based on the at least two types of media data; process the multimodal input data through a music processing agent, and call at least one music processing plugin according to the processing result to obtain the media resources corresponding to each music processing plugin; combine the media resources corresponding to each music processing plugin to generate visual music information that matches the target intent.

[0135] According to one or more embodiments of this disclosure, the step of processing the multimodal input data through a music processing agent and calling at least one music processing plugin based on the processing result to obtain media resources corresponding to each music processing plugin includes: performing intent recognition on the multimodal input data through a music processing agent to obtain a target intent category corresponding to the target intent; determining the corresponding target music processing plugin based on the target intent category; calling the target music processing plugin to obtain the corresponding media resources.

[0136] According to one or more embodiments of this disclosure, the method further includes: extracting parameters from the multimodal input data using a music processing agent based on the target intent category to obtain feature parameters corresponding to the target intent category; converting the feature parameters into control parameters that match the target music processing plugin; and calling the target music processing plugin to obtain corresponding media resources, including: calling the target music processing plugin based on the control parameters to obtain corresponding media resources.

[0137] According to one or more embodiments of this disclosure, the step of invoking the target music processing plugin to obtain the corresponding media resources includes: invoking the target music processing plugin to obtain the corresponding music processing workflow; processing the control parameters of the target music processing plugin sequentially based on the music processing workflow to generate plugin output resources; and converting the format of the plugin output resources to generate media resources with a target format, wherein the target format corresponds to the target intent.

[0138] According to one or more embodiments of this disclosure, determining the corresponding target music processing plugin based on the target intent category includes: obtaining a corresponding visualization template based on the target intent category, wherein the visualization template is used to characterize at least one target media resource constituting the visualized music information; and obtaining a target music processing plugin for generating each of the target media resources based on the visualization template.

[0139] According to one or more embodiments of this disclosure, the target intent category includes at least one of the following: intelligent background music, music search, song recognition, and music introduction.

[0140] According to one or more embodiments of this disclosure, the step of combining media resources corresponding to each of the music processing plugins to generate visual music information matching the target intent includes: obtaining a visualization template matching the target intent, the visualization template being used to determine the layout style of the visual music information; and combining media resources generated by each of the music processing plugins based on the visualization template to generate visual music information matching the target intent.

[0141] According to one or more embodiments of this disclosure, the visualization template includes a first visualization template, and the step of obtaining a visualization template matching the target intent includes: performing intent recognition on the multimodal input data through a music processing agent to obtain intent features for describing the target intent; processing the intent features based on a generative model to generate the first visualization template, wherein the first visualization template is used to characterize the display position of the target media resource corresponding to the target intent.

[0142] According to one or more embodiments of this disclosure, the visualization template includes a second visualization template, and the step of obtaining a visualization template matching the target intent includes: performing intent recognition on the multimodal input data through a music processing agent to obtain the target intent category and the number of media resource bits corresponding to the target intent; obtaining a corresponding second visualization template based on the target intent category and the number of media resource bits, wherein the second visualization template is used to characterize the user interface layout composed of media resource bits based on the target number.

[0143] According to one or more embodiments of this disclosure, the method further includes: loading and displaying at least one interactive control for the visualized music information, wherein the interactive control corresponds one-to-one with the media resources constituting the visualized music information, and the interactive control is used to control the display state of the corresponding media resources.

[0144] Secondly, according to one or more embodiments of this disclosure, an information generation apparatus is provided, comprising:

[0145] An acquisition module is used to acquire multimodal input data, the multimodal input data including at least two types of media data, the multimodal input data being used to characterize the target intent based on the at least two types of media data;

[0146] The processing module is used to process the multimodal input data through a music processing agent, and to call at least one music processing plugin according to the processing result to obtain the media resources corresponding to each music processing plugin.

[0147] The generation module is used to combine the media resources corresponding to each of the music processing plugins to generate visual music information that matches the target intent.

[0148] According to one or more embodiments of this disclosure, the processing module is specifically configured to: perform intent recognition on the multimodal input data through a music processing agent to obtain a target intent category corresponding to the target intent; determine a corresponding target music processing plugin based on the target intent category; and call the target music processing plugin to obtain the corresponding media resources.

[0149] According to one or more embodiments of this disclosure, the processing module is further configured to: extract parameters from the multimodal input data based on the target intent category using a music processing agent to obtain feature parameters corresponding to the target intent category; convert the feature parameters into control parameters that match the target music processing plugin; when the processing module 32 calls the target music processing plugin to obtain the corresponding media resources, it is specifically configured to: call the target music processing plugin based on the control parameters to obtain the corresponding media resources.

[0150] According to one or more embodiments of this disclosure, when the processing module calls the target music processing plugin to obtain the corresponding media resources, it is specifically configured to: call the target music processing plugin to obtain the corresponding music processing workflow; based on the music processing workflow, process the control parameters of the target music processing plugin sequentially to generate plugin output resources; and perform format conversion on the plugin output resources to generate media resources with a target format, wherein the target format corresponds to the target intent.

[0151] According to one or more embodiments of this disclosure, when the processing module determines the corresponding target music processing plugin based on the target intent category, it is specifically configured to: obtain a corresponding visualization template based on the target intent category, wherein the visualization template is used to characterize at least one target media resource constituting the visualized music information; and obtain a target music processing plugin for generating each of the target media resources based on the visualization template.

[0152] According to one or more embodiments of this disclosure, the target intent category includes at least one of the following: intelligent background music, music search, song recognition, and music introduction.

[0153] According to one or more embodiments of this disclosure, the generation module is specifically configured to: obtain a visualization template matching the target intent, the visualization template being used to determine the layout style of the visualized music information; and, based on the visualization template, combine the media resources generated by each of the music processing plugins to generate visualized music information matching the target intent.

[0154] According to one or more embodiments of this disclosure, the visualization template includes a first visualization template. When the generation module obtains a visualization template that matches the target intent, it is specifically used to: perform intent recognition on the multimodal input data through a music processing agent to obtain intent features for describing the target intent; process the intent features based on the generation model to generate the first visualization template, wherein the first visualization template is used to characterize the display position of the target media resource corresponding to the target intent.

[0155] According to one or more embodiments of this disclosure, the visualization template includes a second visualization template. When the generation module obtains a visualization template that matches the target intent, it is specifically used to: perform intent recognition on the multimodal input data through a music processing agent to obtain the target intent category and the number of media resource bits corresponding to the target intent; and obtain a corresponding second visualization template based on the target intent category and the number of media resource bits. The second visualization template is used to characterize the user interface layout composed of media resource bits based on the target number.

[0156] According to one or more embodiments of this disclosure, the generation module is further configured to: load and display at least one interactive control for the visualized music information, wherein the interactive control corresponds one-to-one with the media resources constituting the visualized music information, and the interactive control is used to control the display state of the corresponding media resources.

[0157] Thirdly, according to one or more embodiments of the present disclosure, an electronic device is provided, comprising: at least one processor and a memory;

[0158] The memory stores computer-executed instructions;

[0159] The at least one processor executes computer execution instructions stored in the memory, causing the at least one processor to perform the information generation method as described in the first aspect and various possible designs of the first aspect.

[0160] Fourthly, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein computer-executable instructions are stored therein, and when a processor executes the computer-executable instructions, the information generation method described in the first aspect and various possible designs of the first aspect is implemented.

[0161] Fifthly, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the information generation method as described in the first aspect and various possible designs of the first aspect.

[0162] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0163] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0164] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. An information generation method, characterized in that, include: Acquire multimodal input data, which includes at least two types of media data, and use the multimodal input data to characterize the target intent based on the at least two types of media data; The multimodal input data is processed by a music processing agent, and at least one music processing plugin is invoked based on the processing result to obtain the media resources corresponding to each music processing plugin. The media resources corresponding to each of the aforementioned music processing plugins are combined to generate visual music information that matches the target intent.

2. The method according to claim 1, characterized in that, The process of processing the multimodal input data through a music processing agent and invoking at least one music processing plugin based on the processing result to obtain the media resources corresponding to each music processing plugin includes: The music processing agent performs intent recognition on the multimodal input data to obtain the target intent category corresponding to the target intent; Based on the stated target intent category, determine the corresponding target music processing plugin. The target music processing plugin is invoked to obtain the corresponding media resources.

3. The method according to claim 2, characterized in that, The method further includes: Based on the target intent category, the music processing agent extracts parameters from the multimodal input data to obtain the feature parameters corresponding to the target intent category; The feature parameters are converted into control parameters that match the target music processing plugin; The target music processing plugin is invoked to obtain the corresponding media resources, including: Based on the control parameters, the target music processing plugin is invoked to obtain the corresponding media resources.

4. The method according to claim 2, characterized in that, The process of calling the target music processing plugin to obtain the corresponding media resources includes: Invoke the target music processing plugin to obtain the corresponding music processing workflow; Based on the music processing workflow, the control parameters of the target music processing plugin are processed sequentially to generate plugin output resources; The plugin output resources are format-converted to generate media resources with a target format, wherein the target format corresponds to the target intent.

5. The method according to claim 2, characterized in that, The step of determining the corresponding target music processing plugin based on the target intent category includes: Based on the target intent category, a corresponding visualization template is obtained, wherein the visualization template is used to represent at least one target media resource constituting the visualized music information; Based on the visualization template, obtain the target music processing plugin used to generate each of the target media resources.

6. The method according to claim 2, characterized in that, The target intent category includes at least one of the following: Intelligent music selection, music search, song recognition, and music description.

7. The method according to claim 1, characterized in that, The step of combining the media resources corresponding to each of the music processing plugins to generate visual music information that matches the target intent includes: Obtain a visualization template that matches the target intent, the visualization template being used to determine the layout style of the visualized music information; Based on the visualization template, the media resources generated by each of the music processing plugins are combined to generate visual music information that matches the target intent.

8. The method according to claim 7, characterized in that, The visualization template includes a first visualization template, and the step of obtaining a visualization template that matches the target intent includes: The music processing agent performs intent recognition on the multimodal input data to obtain intent features that describe the target intent. Based on the generative model, the intent features are processed to generate the first visualization template, which is used to represent the display position of the target media resource corresponding to the target intent.

9. The method according to claim 7, characterized in that, The visualization template includes a second visualization template, and obtaining the visualization template that matches the target intent includes: The music processing agent performs intent recognition on the multimodal input data to obtain the target intent category and the number of media resource bits corresponding to the target intent; Based on the target intent category and the number of media resource slots, a corresponding second visualization template is obtained. The second visualization template is used to represent the user interface layout composed of media resource slots based on the target number.

10. The method according to claim 1, characterized in that, The method further includes: For the visualized music information, at least one interactive control is loaded and displayed, wherein the interactive control corresponds one-to-one with the media resources constituting the visualized music information, and the interactive control is used to control the display state of the corresponding media resources.

11. An information generation device, characterized in that, include: An acquisition module is used to acquire multimodal input data, the multimodal input data including at least two types of media data, the multimodal input data being used to characterize the target intent based on the at least two types of media data; The processing module is used to process the multimodal input data through a music processing agent, and to call at least one music processing plugin according to the processing result to obtain the media resources corresponding to each music processing plugin. The generation module is used to combine the media resources corresponding to each of the music processing plugins to generate visual music information that matches the target intent.

12. An electronic device, characterized in that, include: Processor and memory; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the information generation method as described in any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, implement the information generation method as described in any one of claims 1 to 10.

14. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the information generation method as described in any one of claims 1 to 10.

Images