Method, device, equipment and product for generating multimedia materials

Abstract

The invention relates to a method, a device, equipment and a product for generating multimedia materials. The method includes receiving, at a first interface, a selection of a first workflow of a plurality of workflows by a user, each workflow of the plurality of workflows being configured to perform a task associated with generating a multimedia material. The method includes receiving a descriptive input of a user at a first interface. In addition, the method also includes displaying a multimedia material on the first interface, the multimedia material being generated based on the descriptive input by invoking a first workflow.

Description

Technical Field

[0001] This disclosure relates to the field of computers, and more specifically to methods, apparatus, devices and products for generating multimedia materials. Background Technology

[0002] Multimodal generation has become a standard capability for content creation platforms. These platforms are no longer limited to generating single text or images, but now support multimodal output including text, images, audio, and video. Simultaneously, users' demands for the quality of generated content are increasing. Generalized, mass-produced content can no longer meet users' diverse needs; they increasingly expect personalized, customized multimedia materials generated through the platform. For example, users may want generated multimedia materials to match their own style preferences, or they may want them to be adaptable to specific scenarios. Summary of the Invention

[0003] In a first aspect of the embodiments of this disclosure, a method for generating multimedia material is provided. The method includes receiving, on a first interface, a user's selection of a first workflow from a plurality of workflows, each workflow being configured to perform a task associated with generating multimedia material. The method also includes receiving descriptive input from the user on the first interface. Furthermore, the method includes displaying multimedia material on the first interface, the multimedia material being generated based on the descriptive input by invoking the first workflow.

[0004] In a second aspect of the embodiments of this disclosure, an apparatus for generating multimedia materials is provided. The apparatus includes a workflow selection module configured to receive, on a first interface, a user's selection of a first workflow from a plurality of workflows, each workflow being configured to perform a task associated with generating multimedia materials. The apparatus also includes a user input module configured to receive descriptive input from the user on the first interface. Furthermore, the apparatus includes a multimedia material output module configured to display multimedia materials on the first interface, the multimedia materials being generated based on the descriptive input by invoking the first workflow.

[0005] In a third aspect of the embodiments of this disclosure, an electronic device is provided. The electronic device includes a processor and a memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to the first aspect.

[0006] In a fourth aspect of the embodiments of this disclosure, a computer program product is provided having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method of the first aspect.

[0007] The summary section is provided to present the chosen concepts in a simplified form, which will be further described in the detailed description below. The summary section is not intended to identify key or principal features of the claimed subject matter, nor is it intended to limit the scope of the claimed subject matter. Attached Figure Description

[0008] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein:

[0009] Figure 1 A schematic diagram of an example environment in which several embodiments of the present disclosure may be implemented is shown;

[0010] Figure 2 A flowchart of a method for generating multimedia materials according to some embodiments of the present disclosure is shown;

[0011] Figures 3A to 3E A schematic diagram illustrating example steps for generating multimedia materials according to some embodiments of the present disclosure is shown;

[0012] Figure 4 A schematic diagram of an example interface for editing multimedia materials according to some embodiments of the present disclosure is shown;

[0013] Figure 5 A schematic diagram of an example interface for a publishing workflow according to some embodiments of the present disclosure is shown;

[0014] Figure 6 A schematic diagram illustrating the process of generating multimedia materials according to some embodiments of the present disclosure is shown;

[0015] Figure 7 A block diagram of an apparatus for generating multimedia materials according to some embodiments of the present disclosure is shown; and

[0016] Figure 8 A block diagram of a device capable of implementing several embodiments of the present disclosure is shown. Detailed Implementation

[0017] It is understood that all user-related data involved in this technical solution should be obtained and used only after authorization from the user. This means that if it is necessary to use a user's personal information in this technical solution, the user's explicit consent and authorization are required before obtaining this data; otherwise, no related data collection and use will be carried out. It should also be understood that when implementing this technical solution, relevant laws and regulations should be strictly followed in the process of data collection, use, and storage, and necessary technical measures should be taken to protect user data security and ensure the secure use of data.

[0018] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0019] For example, upon receiving a user's proactive request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0020] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0021] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0022] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0023] In the description of embodiments of this disclosure, the term "comprising" and similar terms should be understood as open-ended inclusion, i.e., "including but not limited to". The term "based on" should be understood as "at least partially based on". The term "one embodiment" or "the embodiment" should be understood as "at least one embodiment". The terms "first", "second", etc., may refer to different or the same objects unless explicitly stated. Other explicit and implicit definitions may also be included below.

[0024] As mentioned above, the development of automatic content generation technology has lowered the barrier to personalized creation and driven users to shift from content "consumers" to content "producers." Users hope to create content that meets their personal expectations through content generation platforms. In related technologies, taking text-to-video generation as an example, users need to first open the text-to-image generation page, input prompts to generate an image, and then switch to the image-to-video generation page to input the text-to-image result to continue generating the video. In short, users need to switch between multiple operations to complete a generation task, resulting in interruptions in the creation process and an unsmooth experience. Furthermore, taking text-to-image generation as another example, users can generate images by inputting prompts. If the generated style does not meet the user's expectations, the user can only adjust the generated image by repeatedly adjusting the prompts. This method is cumbersome, time-consuming, and fails to meet users' personalized creation needs.

[0025] To address this, this disclosure proposes a method for generating multimedia materials. In this method, based on descriptive input from a user on a first interface, multiple capabilities integrated within a workflow can be invoked to generate multimedia materials. This approach ensures a seamless and continuous multimedia material generation process, avoiding interruptions in the creative process. Furthermore, the entire process is implemented within the first interface, enabling users to complete the entire creation process from conception to final product in a unified interactive environment, effectively improving creative efficiency and the continuity of the user experience.

[0026] Figure 1 A schematic diagram of an example environment 100 in which various embodiments of the present disclosure may be implemented is shown. Reference Figure 1 Environment 100 includes a dialog box 102, which can run on a processing device. The processing device can be any device with processing capabilities. Optionally, the processing device can be a smartphone, tablet, laptop, desktop computer, workstation, or smart wearable device, etc. Within the dialog box 102, users can generate multimedia materials such as text, images, video, audio, or live photos. For example, the dialog box 102 can be deployed on a content publishing platform, video editing platform, or social media platform, etc.

[0027] In environment 100, a user can initiate a request to generate multimedia materials. For example, a component for generating multimedia materials can be provided on the user interface. Clicking this component triggers the multimedia material generation request. In some embodiments, one or more components can be provided on the user interface, which may correspond to multiple workflows 108 associated with multimedia material generation. Optionally, the workflow is configured to perform one or more tasks associated with generating multimedia materials. Workflows may include, for example, text-to-image, text-to-video, video super-resolution, image-to-video, image-to-text, etc. When a user clicks on multiple workflows 108, a list of multiple workflows can be expanded and displayed, as shown in the reference. Figure 1As shown. Users can select a workflow (e.g., the selected workflow can be referred to as the first workflow) by interacting with (e.g., clicking) the corresponding controls. (Reference) Figure 1 Users can click the "Wensheng Video" control to enter the Wensheng Video workflow.

[0028] In environment 100, dialogue 102 can also provide a dialogue interface (also known as a first interface) and input boxes on the user interface. (See reference) Figure 1 The dialog box 102 may also include an input box 104, through which descriptive input 110 from the user can be received in some embodiments. The input box 104 may also include an image adding control 106, through which images or videos can be added by clicking the image adding control 106 or by dragging an image or video onto the image adding control 106. In some embodiments, the descriptive input 110 describes the multimedia material the user expects to generate. Optionally, the multimedia material the user expects to generate is associated with the goal of the workflow selected by the user. (See reference...) Figure 1 For example, the workflow goal might include generating a video based on text. Descriptive input 110 could include: "Generate a video of a flower blooming, with petals slowly unfolding. A flower made of paper folds stands in the center of a white background. The petals should be made of multi-layered transparent frosted glass, with light reflecting off its surface like a mirror. Combine hyper-realistic rendering technology with ultra-fine details and architectural lighting structures. The image should be shadow-free." When the user finishes inputting, descriptive input 110 can be displayed in the dialog box.

[0029] In environment 100, dialogue 102 can display the generated multimedia material 112 on the dialogue interface. This multimedia material 112 is generated based on descriptive input 110 by invoking a workflow selected by the user. This workflow may include multiple capabilities for generating the multimedia material 112, such as semantic understanding capabilities, the ability to generate images from text, the ability to improve image resolution, the ability to generate video from images, and so on, but is not limited to these. In some embodiments, dialogue 102 can generate the user-desired multimedia material 112 in one go by sequentially invoking these capabilities, avoiding creative breakpoints.

[0030] This approach ensures a seamless workflow in generating multimedia materials, preventing interruptions in the creative process. Furthermore, the entire process is implemented within a single interface, allowing users to complete the entire creative process from conception to finished product in a unified interactive environment, effectively improving creative efficiency and the continuity of the experience.

[0031] The following will combine Figures 2 to 6The method according to embodiments of this disclosure is described in detail. For ease of understanding, the specific data mentioned in the following description are exemplary and not intended to limit the scope of this disclosure. It is understood that the embodiments described below may also include additional actions not shown and / or actions shown may be omitted, and the scope of this disclosure is not limited in this respect.

[0032] Figure 2 A flowchart of a method 200 for generating multimedia material according to some embodiments of the present disclosure is shown. Method 200 can be performed by an apparatus for generating multimedia material, which may be, for example, a standalone device or system. The apparatus can be implemented in software and / or hardware. References Figure 2 Method 200 includes boxes 202, 204 and 206.

[0033] In box 202, the first interface receives a user's selection of a first workflow among multiple workflows, each workflow being configured to perform a task associated with generating multimedia content. In some embodiments, a workflow may be associated with multimedia content generation. For example, a workflow may include one or more capabilities for generating multimedia content. (See reference) Figure 1 Multiple workflows 108 can be displayed on dialog 102 (also known as the first interface) for the user to select. In some embodiments, a workflow can be an execution process that organizes a set of related tasks according to a preset logical order, rules, and dependencies to achieve the workflow's objectives. (See reference) Figure 1 The goal of a text-based video workflow can include generating video from text. To achieve this goal, it can include a set of tasks such as semantic understanding, generating images from text, improving image resolution, and generating video from images. It should be understood that this is merely an example, and the text-based video workflow can also be implemented using other sets of tasks, and is not limited to this.

[0034] In box 204, descriptive input from the user is received on the first interface. In some embodiments, the user may describe the multimedia material to be generated through a dialog interface (also referred to as the first interface). Descriptive input may include any of the following: text or images, and of course, may also include the user's voice input. (See reference...) Figure 1 Descriptive input can also be associated with the workflow selected by the user (also known as the first workflow). For example, if the user selects a workflow for text-to-video conversion, then the user input can be text. If the user selects an image-to-text conversion workflow, then the user input can be an image, or a combination of images and text.

[0035] In box 206, multimedia content is displayed on the first interface. This multimedia content is generated based on descriptive input by invoking a first workflow. In some embodiments, this first workflow can be invoked to generate multimedia content based on the user's descriptive input. (See reference) Figure 1 Based on user input 110, a video can be generated, which is the final multimedia material 112.

[0036] This approach ensures a seamless workflow in generating multimedia materials, preventing interruptions in the creative process. Furthermore, the entire process is implemented within a single interface, allowing users to complete the entire creative process from conception to finished product in a unified interactive environment, effectively improving creative efficiency and the continuity of the experience.

[0037] Figures 3A to 3E A schematic diagram illustrating example steps for generating multimedia materials according to some embodiments of the present disclosure is shown. The following is in conjunction with... Figures 3A to 3E The process of generating multimedia materials will be described in detail below. It should be noted that the process described here can complement the aforementioned method 200. Figure 3A The example interface 300A of the homepage is shown. (Reference) Figure 3A The example interface 300A includes a sidebar area 301, a creation area 305, a function entry point 308, and a case preview area 309. The sidebar area 301 displays multiple controls sequentially, such as the creation entry point 302 (including the homepage and smart canvas), asset management 303, and agile entry point 304 (including creating a new project, portrait photography, and a minimalist background). In some embodiments, a project may include a dialog (see reference). Figure 1 Dialogue 102 shown.

[0038] In some embodiments, the sidebar area 301 can serve as navigation information and remain permanently on the side of the display interface. For example, when a user clicks the "Home" control, they can directly enter the dialog interface (also known as the first interface), where they can select one or more workflows to generate the desired multimedia materials. When a user clicks the "Smart Canvas" control (also known as the second control), they can enter the Smart Canvas interface, where they can create workflows for generating multimedia materials based on the canvas. In some embodiments, existing workflows can be managed as assets. When a user clicks the "Asset Management" control 303, they can enter the asset management interface to browse and find existing workflows. Users can quickly enter the corresponding dialog by clicking the Agile Entry control 304. For example, when a user clicks the "New Project" control, they can enter the new project interface. "Portrait Photography" and "Minimalist Background" correspond to recently created projects; when a user clicks the corresponding control, they can directly enter the corresponding project. It should be understood that this is merely an example, and this disclosure does not limit the layout and number of controls on the display interface.

[0039] refer to Figure 3A In the creation area 305, users can add images, text, and other descriptions. Furthermore, users can determine the model, workflow, and aspect ratio used in the current dialogue through the area below 306. Afterwards, the user clicks the upload control 307, triggering entry into the dialogue interface to begin generating multimedia materials. Function entry points 308 can include commonly used or core controls, such as model training, smart canvas, multiple workflows, and agile batch processing, serving as quick access points. In some embodiments, the "multiple workflows" control (also called the first control) can be used to select a workflow. Optionally, when a user clicks the "multiple workflows" control, multiple workflows associated with multimedia material generation can be expanded and displayed around it. For example, multiple workflows can be displayed through drop-down menus, pop-ups, etc. In some embodiments, the "smart canvas" control (also called the second control) can be used to generate a workflow. Optionally, when a user clicks the "smart canvas" control, the smart canvas interface can be accessed. In some embodiments, the "agile batch processing" control (also called the fifth control) can be used for batch processing. Optionally, when a user clicks the "Agile Batch" control, the dialog interface can receive a set of user input (also known as a descriptive input set). The selected workflow then generates a set of multimedia materials based on this set and displays the multimedia materials in the dialog interface. For example, for the "Image to Video" workflow, a user can input multiple images at once, and the dialog interface can display the corresponding videos for each image simultaneously. The case preview area 309 can include workflow cases for the project. For example, a thumbnail of the generated content corresponding to the workflow case can be displayed on the control of the workflow case. When a user clicks on the workflow case, they can initiate a one-click copy or call its corresponding project. Optionally, the example interface 300A can also include other function entry points, such as experiences for popular models and management controls for creative teams.

[0040] In some embodiments, users can create content in two ways. One is through a dialog interface, where they can generate multimedia materials by selecting a workflow. The other is through a smart canvas, where they can create a workflow for generating multimedia materials by building a workflow. Optionally, users can select a workflow (also called the first workflow) from multiple workflows displayed by the "Multiple Workflows" control, enter a content description, and trigger the dialog interface by clicking control 307.

[0041] refer to Figures 3B to 3D This demonstrates example steps for generating multimedia content within a dialog interface. Figure 3B Example interface 300B of the dialog interface is shown. Figure 3C Example interface 300C of the dialog interface is shown. Figure 3D A sample interface 300D of the dialog interface is shown. (Reference) Figure 3B The example interface 300B includes a sidebar area 312, a dialog area 314, a reference area 316, and a control 318 (also known as a first control). The sidebar area 312 is pulled up. Optionally, when the user clicks or drags the sidebar area 312, the sidebar area 312 can be expanded, and the user can quickly access the corresponding function by operating the controls within it.

[0042] refer to Figure 3B Taking a skincare poster project as an example, a portion of the dialogue content is shown. Users can enter descriptions of multimedia materials in the input boxes of the dialogue interface; these descriptions can be text, images, or a combination of both. In some embodiments, users can first select a workflow (e.g., text-to-image) using control 318 and then enter text in the input boxes, referencing... Figure 3B The text includes: "This poster's overall style is fresh and dreamy, aiming to showcase the subject's skin condition...". Based on the input text, the "Text-to-Image" workflow generates three images that meet the "skin-beautifying" goal. Upon completion, dialog area 314 displays "Image generation complete, preview on the right." At this point, the user can click on any image and preview it on the right side of dialog area 314. (Continue to refer to...) Figure 3B After the image is generated, users can preview it in the right-hand area and also edit, regenerate, or delete the generated content by manipulating the corresponding controls (such as the controls below the three generated images).

[0043] In some embodiments, users can switch workflows within the current dialog interface. (See reference) Figure 3B Users can click control 318 to enter the example interface 300C. (Reference) Figure 3C The area surrounding control 318 displays multiple controls for various workflows. The user clicks "Image to Video" to access the example interface 300D. (Reference) Figure 3DIn the "Image to Image" workflow, the image selected by the user is used as part of the user's descriptive input. The text entered by the user in the input box, "Convert this poster into a video, leave everything else unchanged, with slight movement of the characters," is used as another part of that descriptive input. The "Image to Video" workflow then generates the video based on the user's input. When generation is complete, dialog area 314 displays "Image to video complete, preview on the right." This way, users don't need to switch interfaces or generate multimedia materials across functions and modules; instead, they can generate the desired multimedia materials within a unified dialog interface. It should be understood that only a portion of the dialog is shown here as an example, and this disclosure does not limit the project to only these two workflows.

[0044] Continue to refer to Figures 3B to 3D Example interfaces 300B, 300C, and 300D can also include a reference area 316. Reference area 316 can, for example, include two modules: an inspiration square and historical materials. The inspiration square can include reference materials associated with the currently selected workflow, and the historical materials can include historical materials associated with the currently selected workflow. Users can switch between the two modules for display. Figure 3B When a user selects the "Text-to-Image" workflow, the Inspiration Plaza can display 15 reference materials associated with the workflow for the user's reference. It should be understood that users can switch to the Historical Materials module, which displays one or more historical materials associated with the "Text-to-Image" workflow for reference. These multimedia materials can include videos, images, audio, text, etc. They can assist users in creation and also support one-click replication.

[0045] In some embodiments, a user can select a preferred resource (e.g., a preferred image style) from a plurality of displayed reference resources. When a user selects one or more reference resources (e.g., by clicking, dragging, etc.), an input template corresponding to the selected reference resource is displayed in the dialog interface. For example, in a "text-to-image" workflow, the input template for multimedia resources may include text. Optionally, the input template can be displayed in the input box of the dialog interface. In some embodiments, if the user determines the currently displayed input template, the reference resource corresponding to that input template can be determined as part of the user's descriptive input, and the descriptive input added by the user through the input box can be considered as another part of that descriptive input. This other part may, for example, be a supplement to the content of the input template, making it more closely aligned with the user's expectations.

[0046] In some embodiments, a user can enter the sample interface of the smart canvas by clicking the "Smart Canvas" control on the sample interface 300A. Figure 3EA sample interface 300E of the Smart Canvas is shown. Users can build workflows within the Smart Canvas interface. For example, one or more of the aforementioned workflows can be generated within the Smart Canvas interface. (Reference) Figure 3E Take a video super-resolution workflow as an example. Video super-resolution can be, for example, lossless processing of a video, detail restoration, etc., to generate a better and larger video. The multimedia materials corresponding to this workflow can include, for example, a better and larger video. In some embodiments, the example interface 300E can display multiple components associated with the generation of the multimedia materials. These components can correspond to multiple capabilities. Capabilities can include, for example, text generation capabilities, image generation capabilities, video generation capabilities, multimodal capabilities, image matting and segmentation capabilities, image editing capabilities, color grading filter capabilities, etc. In some embodiments, the multiple components include one or more of the following: a machine learning model having one or more of the multiple capabilities, a functional module (e.g., a code module, an agent) having one or more of the multiple capabilities, or preset prompts (e.g., a knowledge base).

[0047] In some embodiments, users can select one or more components from these components as task nodes in a workflow to build the workflow. Based on the built workflow, it can be deployed as a corresponding workflow. Optionally, user actions on components may include clicking or dragging components; when a user clicks or drags a component onto the smart canvas interface, that component can be designated as a task node. Optionally, users can also determine the order of components by dragging them, thus obtaining multiple task nodes in the workflow and their order. (Reference) Figure 3E The workflow comprises three task nodes: task node 322 uploads the video, task node 323 performs super-resolution processing on the uploaded video, and task node 324 stores the processed video. Task node 323 can be configured as a large model, code, agent, etc., with corresponding capabilities. It should be understood that this is merely an example; in practice, a workflow may include more task nodes. In some embodiments, a video super-resolution workflow can be deployed based on these task nodes. (Reference) Figure 3E It can be encapsulated into a video super-resolution workflow based on the first task node (e.g., task node 322) and the last task node (e.g., task node 324).

[0048] Continue to refer to Figure 3E Users can click control 326 to deploy it to the content generation platform with a single click. The example interface 300E can also display control 328 (also known as a third control) for publishing the workflow. Users can click control 328 to add the workflow to the aforementioned workflows. For more information on publishing workflows, please refer to the following text. Figure 5The relevant content will not be elaborated here.

[0049] In this way, users can select task nodes according to their expectations or needs to generate corresponding workflows. Based on the generated workflow, multimedia materials that match their own style can be produced, meeting users' personalized needs. Furthermore, by publishing this workflow, it can be added to the aforementioned workflows, and other users can also select this workflow to generate their own multimedia materials.

[0050] Figure 4 A schematic diagram of an example interface 400 for editing multimedia materials according to some embodiments of the present disclosure is shown. In some embodiments, if a user is dissatisfied with the multimedia materials generated by a certain workflow during the process of generating multimedia materials using one or more workflows in the dialog interface, they can enter the smart canvas interface to edit the workflow. For example, after generating a video in example interface 300D, the user can click on the generated video to view details, refer to... Figure 4 As shown. In the example interface 400, in addition to the sidebar area 402, it may also include a multimedia material display area 404, a details area 406, a tool area 408, and a video frame display area 410. Users can view the beauty poster video in the multimedia material display area 404. The details area 406 displays detailed information about the video, such as the model used, aspect ratio, sampler, iteration deployment, etc. The video frame display area 410 can display multiple image frames contained in the video. The tool area 408 displays corresponding editing controls, such as ultra-high definition, lip-sync, sound effects, positioning dialogue, re-edit, regenerate, etc. Users can edit the video by operating these controls. It should be noted that these edits are limited to editing the video itself, such as adding some emoticons or stickers to the video generated by the workflow, and will not change the capabilities of the workflow.

[0051] Continue to refer to Figure 4Users can click the "Smart Canvas" control to enter the Smart Canvas interface, where they can edit the workflow corresponding to the multimedia material. In some embodiments, in response to user actions on the multimedia material, multiple task nodes associated with the multimedia material can be displayed. User actions on the multimedia material may include, for example, clicking on the multimedia material to view its details, and then clicking the "Smart Canvas" control on the details page to enter the Smart Canvas interface. The Smart Canvas interface can display multiple task nodes associated with the multimedia material, such as multiple task nodes corresponding to the workflow. Users can perform operations on the task nodes (e.g., delete, add, or replace task nodes) to generate another workflow (also called a second workflow) based on the initial workflow (also called the first workflow). It should be noted that the above editing does not modify the initial workflow; it can be viewed as copying the initial workflow, and editing or secondary creation on the copied version to generate another workflow.

[0052] Figure 5 A schematic diagram of an example interface 500 for a publishing workflow according to some embodiments of the present disclosure is shown. References Figure 5 In example interface 500, the user generates a new workflow where two avatars can be combined to hug each other, displaying happy expressions while maintaining their individual appearances. If a user action on control 502 (also known as the third control) is detected, such as clicking control 502, it can be determined that the user intends to publish the current workflow (also known as the first workflow). Control 504 (also known as the fourth control) can be displayed on example interface 500. Control 504 is used to determine the configuration information of the current workflow. In some embodiments, the user can drag task nodes of the workflow onto control 504 to define task nodes on control 504. Figure 5 The dashed line with an arrow illustrates the process of the user selecting task node number 19 on control 504. (Reference) Figure 5 When a task node reaches control 504, it can be parsed to determine its corresponding parameters. These parameters are then processed in the settings. By parsing each task node, the workflow's configuration information can be determined based on these parameters. This configuration information may include, for example, the workflow's input and output parameters. Input parameters may indicate user input, and can include images, text, video, and audio. Output parameters may indicate multimedia materials, and can include images, text, video, and audio.

[0053] In this way, users can encapsulate workflows and publish them to multiple workflows, allowing users or other users to quickly use the workflow within the dialog interface. Furthermore, other users don't need to understand complex workflows; they only need to define the inputs and outputs, further enhancing the user experience.

[0054] Figure 6 A schematic diagram of a process 600 for generating multimedia materials according to some embodiments of the present disclosure is shown. (See reference) Figure 6 In the dialog interface 602, multiple workflows 604 can be displayed. In box 606, workflow 1 can be invoked based on the user's selection of one of the multiple workflows 604 (denoted as workflow 1). In box 608, workflow 1 generates content (denoted as content 1) based on the user's descriptive input. In box 610, workflow 2 can be invoked based on the user's selection of one of the multiple workflows 604 (denoted as workflow 2). In box 612, workflow 2 generates content (denoted as content 2) based on the user's descriptive input (if any) and content 1. This process continues, invoking the user-selected workflow 3, generating content 3, and so on, until box 614, where the final content (denoted as multimedia material) is generated. It should be understood that content 1, content 2, ... here are also referred to as multimedia material 1, multimedia material 2, ...

[0055] On canvas interface 616 (also known as the smart canvas interface), a set of nodes 618 for generating workflow tasks can be displayed. In box 620, the user selects a task node (denoted as node 1) from node set 618. In box 622, the user selects a task node (denoted as node 2) from node set 618. And so on, the user selects node 3, node 4, ... until a workflow is generated. In box 624, the user publishes the workflow. This workflow is added to multiple workflows 604 for the user to choose from.

[0056] This approach allows multiple workflows to be displayed within the dialog interface. Switching between workflows ensures a seamless transition in the multimedia content generation process, preventing interruptions in the creative process. It also guarantees that users complete the entire creative process from conception to final product within a unified interactive environment, effectively improving creative efficiency and user experience. Furthermore, users can build and generate workflows on the canvas interface according to their own expectations or needs. User-built workflows can generate multimedia content that matches their style, satisfying their personalized requirements. By publishing the workflow, it can be added to the dialog interface, allowing other users to select it to generate their own multimedia content.

[0057] Figure 7A block diagram of an apparatus for generating multimedia materials according to some embodiments of the present disclosure is shown. (Reference) Figure 7 The device 700 includes a workflow selection module 710, a user input module 720, and a multimedia content output module 730. The workflow selection module 710 is configured to receive a user's selection of a first workflow from a plurality of workflows on a first interface. Each workflow is configured to perform a task associated with generating multimedia content. The user input module 720 is configured to receive descriptive input from the user on the first interface. The multimedia content output module 730 is configured to display multimedia content on the first interface, which is generated based on the descriptive input by invoking the first workflow.

[0058] Figure 8 A block diagram of a device 800 capable of implementing various embodiments of the present disclosure is shown. (See reference...) Figure 8 As shown, device 800 includes a central processing unit (CPU) and / or a graphics processing unit (GPU) 801, which can perform various appropriate actions and processes according to computer program instructions stored in read-only memory (ROM) 802 or loaded from storage unit 808 into random access memory (RAM) 803. The RAM 803 can also store various programs and data required for the operation of device 800. The CPU / GPU 801, ROM 802, and RAM 803 are interconnected via bus 804. Input / output (I / O) interface 805 is also connected to bus 804. Although not shown in... Figure 8 As shown, device 800 may also include a coprocessor.

[0059] Multiple components in device 800 are connected to I / O interface 805, including: input unit 806, such as keyboard, mouse, etc.; output unit 807, such as various types of monitors, speakers, etc.; storage unit 808, such as disk, optical disk, etc.; and communication unit 809, such as network card, modem, wireless transceiver, etc. Communication unit 809 allows device 800 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0060] The various methods or processes described above can be executed by CPU / GPU 801. For example, in some embodiments, the methods can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed on device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by CPU / GPU 801, one or more steps or actions in the methods or processes described above can be performed.

[0061] In some embodiments, the methods and processes described above can be implemented as a computer program product. The computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for performing various aspects of this disclosure.

[0062] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0063] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, a local area network (LAN), a wide area network (WAN), and / or a wireless network, to an external computer or external storage device. The network may include copper cables, fiber optic cables, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0064] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​and conventional procedural programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to execute the computer-readable program instructions, thereby implementing various aspects of this disclosure.

[0065] These computer-readable program instructions can be provided to a processing unit of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processing unit of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner. Thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0066] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0067] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of devices, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those marked in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0068] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles of the embodiments, practical workflows, or technical improvements to technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

[0069] The following are some example implementations of this disclosure.

[0070] Example 1. A method for generating multimedia materials, comprising:

[0071] The first interface receives the user's selection of a first workflow from a plurality of workflows, each of which is configured to perform a task associated with generating multimedia material;

[0072] Receive descriptive input from the user on the first interface; and

[0073] Multimedia materials are displayed on the first interface, which are generated by invoking the first workflow based on the descriptive input.

[0074] Example 2. The method described in Example 1 further includes:

[0075] Display a first control for selecting the first workflow; and

[0076] In response to detecting an operation on the first control, the plurality of workflows associated with the generation of multimedia materials are displayed.

[0077] Example 3. According to the method described in Example 1 or 2, receiving the user's selection of the first workflow among the plurality of workflows on the first interface includes:

[0078] In response to detecting a user's selection of the first workflow, one or more reference materials associated with the first workflow are displayed; and

[0079] In response to the user's operation on the one or more reference materials, the input template corresponding to the one or more reference materials is displayed on the first interface.

[0080] Example 4. The method according to any one of Examples 1-3, wherein receiving descriptive input from the user on the first interface includes:

[0081] In response to the user's operation on the input template, the corresponding reference material is determined as part of the descriptive input.

[0082] Example 5. The method according to any one of Examples 1-4, wherein receiving a user's selection of the first workflow from the plurality of workflows on the first interface further includes:

[0083] In response to detecting a user's selection of the first workflow, one or more historical materials associated with the first workflow are displayed.

[0084] Example 6. The method according to any one of Examples 1-5 further includes:

[0085] Display a second control for generating the first workflow; and

[0086] In response to detecting a user's operation on the second control, multiple components associated with the generation of multimedia materials are displayed, the multiple components corresponding to multiple capabilities;

[0087] In response to user actions on the multiple components, multiple task nodes are obtained; and

[0088] The first workflow is generated based on the multiple task nodes.

[0089] Example 7. The method according to any one of Examples 1-6 further includes:

[0090] Display a third control for publishing the first workflow; and

[0091] In response to detecting an operation on the third control, the first workflow is added to the plurality of workflows.

[0092] Example 8. The method according to any one of Examples 1-7 further includes:

[0093] A fourth control is displayed to determine the configuration information for the first workflow;

[0094] Receive the task node specified by the user on the fourth control; and

[0095] By parsing the task nodes, the configuration information of the first workflow is determined, including input parameters and output parameters.

[0096] Example 9. The method according to any one of Examples 1-8, wherein the plurality of components includes one or more of the following:

[0097] A machine learning model that has one or more of the aforementioned capabilities;

[0098] A functional module having one or more of the aforementioned capabilities; or

[0099] Preset prompts.

[0100] Example 10. The method according to any one of Examples 1-9 further includes:

[0101] In response to a user's operation on the multimedia material, multiple task nodes associated with the multimedia material are displayed; and

[0102] In response to user operations on the multiple task nodes, a second workflow based on the first workflow is generated.

[0103] Example 11. The method according to any one of Examples 1-10 further includes:

[0104] Displays the fifth control for batch processing;

[0105] In response to detecting an operation on the fifth control, a descriptive input set is received from the first interface; and

[0106] The first interface displays a collection of multimedia materials for the descriptive input set.

[0107] Example 12. According to any one of Examples 1-11, the descriptive input includes one or more of the following: text or images, and the multimedia material includes one or more of the following: text, images, video, or audio.

[0108] Example 13. An apparatus for generating multimedia materials, comprising:

[0109] The workflow selection module is configured to receive a user's selection of a first workflow from a plurality of workflows on a first interface, wherein each of the plurality of workflows is configured to perform a task associated with generating multimedia materials;

[0110] A descriptive input module is configured to receive descriptive input from the user on the first interface; and

[0111] The multimedia material output module is configured to display multimedia materials on the first interface, the multimedia materials being generated by calling the first workflow based on the descriptive input.

[0112] Example 14. The apparatus according to Example 13 further includes:

[0113] A first display module is configured to display a first control for selecting the first workflow; and

[0114] The second display module is configured to display the plurality of workflows associated with the generation of multimedia materials in response to detecting an operation on the first control.

[0115] Example 15. The apparatus according to Example 13 or 14, wherein the workflow selection module includes:

[0116] A third display module is configured to display one or more reference materials associated with the first workflow in response to detecting a user's selection of the first workflow; and

[0117] The fourth display module is configured to display the input template corresponding to the one or more reference materials on the first interface in response to the user's operation on the one or more reference materials.

[0118] Example 16. The apparatus according to any one of Examples 13-15, wherein the user input module comprises:

[0119] The first input module is configured to, in response to a user's operation on the input template, determine the corresponding reference material as part of the descriptive input.

[0120] Example 17. The apparatus according to any one of Examples 13-16, wherein the workflow selection module further comprises:

[0121] The fifth display module is configured to display one or more historical materials associated with the first workflow in response to detecting a user's selection of the first workflow.

[0122] Example 18. The apparatus according to any one of Examples 13-17 further includes:

[0123] The sixth display module is configured to display a second control used to generate the first workflow; and

[0124] The seventh display module is configured to display multiple components associated with the generation of multimedia materials in response to detecting user operation on the second control, the multiple components corresponding to multiple capabilities;

[0125] The acquisition module is configured to acquire multiple task nodes in response to user operations on the multiple components; and

[0126] The first generation module is configured to generate the first workflow based on the multiple task nodes.

[0127] Example 19. The apparatus according to any one of Examples 13-18 further includes:

[0128] The eighth display module is configured to display a third control used to publish the first workflow; and

[0129] The publishing module is configured to add the first workflow to the plurality of workflows in response to detecting an operation on the third control.

[0130] Example 20. The apparatus according to any one of Examples 13-19 further includes:

[0131] The ninth display module is configured to display a fourth control used to determine the configuration information for the first workflow;

[0132] The first receiving module is configured to receive a task node determined by the user on the fourth control; and

[0133] The determination module is configured to determine the configuration information of the first workflow by parsing the task nodes, the configuration information including input parameters and output parameters.

[0134] Example 21. The apparatus according to any one of Examples 13-20, wherein the plurality of components includes one or more of the following:

[0135] A machine learning model that has one or more of the aforementioned capabilities;

[0136] A functional module having one or more of the aforementioned capabilities; or

[0137] Preset prompts.

[0138] Example 22. The apparatus according to any one of Examples 13-21 further includes:

[0139] The tenth display module is configured to display multiple task nodes associated with the multimedia material in response to a user's operation on the multimedia material; and

[0140] The second generation module is configured to generate a second workflow based on the first workflow in response to user operations on the multiple task nodes.

[0141] Example 23. The apparatus according to any one of Examples 13-22 further includes:

[0142] The eleventh display module is configured to display the fifth control for batch processing;

[0143] The second receiving module is configured to receive a descriptive input set on the first interface in response to detecting an operation on the fifth control; and

[0144] The twelfth display module is configured to display a collection of multimedia materials for the descriptive input set on the first interface.

[0145] Example 24. The apparatus according to any one of Examples 13-23, wherein the descriptive input includes one or more of the following: text or an image, and the multimedia material includes one or more of the following: text, an image, video, or audio.

[0146] Example 25. An electronic device comprising:

[0147] Processor; and

[0148] A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform actions, the actions including:

[0149] The first interface receives the user's selection of a first workflow from a plurality of workflows, each of which is configured to perform a task associated with generating multimedia material;

[0150] Receive descriptive input from the user on the first interface; and

[0151] Multimedia materials are displayed on the first interface, which are generated by invoking the first workflow based on the descriptive input.

[0152] Example 26. The electronic device according to Example 25 further includes:

[0153] Display a first control for selecting the first workflow; and

[0154] In response to detecting an operation on the first control, the plurality of workflows associated with the generation of multimedia materials are displayed.

[0155] Example 27. An electronic device according to Example 25 or 26, wherein receiving a user's selection of the first workflow among the plurality of workflows on a first interface includes:

[0156] In response to detecting a user's selection of the first workflow, one or more reference materials associated with the first workflow are displayed; and

[0157] In response to the user's operation on the one or more reference materials, the input template corresponding to the one or more reference materials is displayed on the first interface.

[0158] Example 28. An electronic device according to any one of Examples 25-27, wherein receiving descriptive input on the first interface includes:

[0159] In response to the user's operation on the input template, the corresponding reference material is determined as part of the descriptive input.

[0160] Example 29. An electronic device according to any one of Examples 25-28, wherein receiving a user's selection of the first workflow among the plurality of workflows on the first interface further includes:

[0161] In response to detecting a user's selection of the first workflow, one or more historical materials associated with the first workflow are displayed.

[0162] Example 30. The electronic device according to any one of Examples 25-29 further includes:

[0163] Display a second control for generating the first workflow; and

[0164] In response to detecting a user's operation on the second control, multiple components associated with the generation of multimedia materials are displayed, the multiple components corresponding to multiple capabilities;

[0165] In response to user actions on the multiple components, multiple task nodes are obtained; and

[0166] The first workflow is generated based on the multiple task nodes.

[0167] Example 31. The electronic device according to any one of Examples 25-30 further includes:

[0168] Display a third control for publishing the first workflow; and

[0169] In response to detecting an operation on the third control, the first workflow is added to the plurality of workflows.

[0170] Example 32. The electronic device according to any one of Examples 25-31 further includes:

[0171] A fourth control is displayed to determine the configuration information for the first workflow;

[0172] Receive the task node specified by the user on the fourth control; and

[0173] By parsing the task nodes, the configuration information of the first workflow is determined, including input parameters and output parameters.

[0174] Example 33. An electronic device according to any one of Examples 25-32, wherein the plurality of components includes one or more of the following:

[0175] A machine learning model that has one or more of the aforementioned capabilities;

[0176] A functional module having one or more of the aforementioned capabilities; or

[0177] Preset prompts.

[0178] Example 34. The electronic device according to any one of Examples 25-33 further includes:

[0179] In response to a user's operation on the multimedia material, multiple task nodes associated with the multimedia material are displayed; and

[0180] In response to user operations on the multiple task nodes, a second workflow based on the first workflow is generated.

[0181] Example 35. The electronic device according to any one of Examples 25-34 further includes:

[0182] Displays the fifth control for batch processing;

[0183] In response to detecting an operation on the fifth control, a descriptive input set is received from the first interface; and

[0184] The first interface displays a collection of multimedia materials for the descriptive input set.

[0185] Example 36. An electronic device according to any one of Examples 25-35, wherein the descriptive input includes one or more of the following: text or an image, and the multimedia material includes one or more of the following: text, an image, video, or audio.

[0186] Example 37. A computer-readable storage medium having stored thereon computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of Examples 1 to 12.

[0187] Example 38. A computer program product tangibly stored on a computer-readable medium and comprising computer-executable instructions that, when executed by a device, cause the device to perform the method according to any one of Examples 1 to 12.

[0188] Although this disclosure has been described using language specific to structural features and / or device logic, it should be understood that the subject matter defined in the accompanying examples is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative forms of implementing the examples.

Claims

1. A method for generating multimedia materials, comprising: The first interface receives the user's selection of a first workflow from a plurality of workflows, each of which is configured to perform a task associated with generating multimedia material; The first interface receives descriptive input from the user; as well as The multimedia material is displayed on the first interface. The multimedia material is generated by calling the first workflow based on the descriptive input.

2. The method according to claim 1, further comprising: Display a first control for selecting the first workflow; as well as In response to detecting an operation on the first control, the plurality of workflows associated with the generation of multimedia materials are displayed.

3. The method according to claim 2, wherein receiving the user's selection of the first workflow among the plurality of workflows on the first interface includes: In response to detecting a user's selection of the first workflow, one or more reference materials associated with the first workflow are displayed; as well as In response to the user's operation on the one or more reference materials, the input template corresponding to the one or more reference materials is displayed on the first interface.

4. The method according to claim 3, wherein receiving descriptive input from the user on the first interface includes: In response to the user's operation on the input template, the corresponding reference material is determined as part of the descriptive input.

5. The method according to claim 4, wherein receiving the user's selection of the first workflow among the plurality of workflows on the first interface further comprises: In response to detecting a user's selection of the first workflow, one or more historical materials associated with the first workflow are displayed.

6. The method according to claim 1, further comprising: Display a second control used to generate the first workflow; as well as In response to detecting a user's operation on the second control, multiple components associated with the generation of multimedia materials are displayed, the multiple components corresponding to multiple capabilities; In response to user operations on the multiple components, multiple task nodes are obtained; as well as The first workflow is generated based on the multiple task nodes.

7. The method of claim 6, further comprising: Display a third control for publishing the first workflow; as well as In response to detecting an operation on the third control, the first workflow is added to the plurality of workflows.

8. The method according to claim 7, further comprising: A fourth control is displayed to determine the configuration information for the first workflow; Receive the task node specified by the user on the fourth control; as well as By parsing the task nodes, the configuration information of the first workflow is determined, including input parameters and output parameters.

9. The method of claim 8, wherein the plurality of components comprises one or more of the following: A machine learning model that has one or more of the aforementioned capabilities; A functional module having one or more of the aforementioned capabilities; or Preset prompts.

10. The method of claim 9, further comprising: In response to user actions on the multimedia material, multiple task nodes associated with the multimedia material are displayed; as well as In response to user operations on the multiple task nodes, a second workflow based on the first workflow is generated.

11. The method according to claim 1, further comprising: Displays the fifth control for batch processing; In response to detecting an operation on the fifth control, a descriptive input set is received from the first interface; as well as The first interface displays a collection of multimedia materials for the descriptive input set.

12. The method according to claim 1, wherein the descriptive input includes one or more of the following: text or an image, and the multimedia material includes one or more of the following: text, an image, a video, or audio.

13. An apparatus for generating multimedia materials, comprising: The workflow selection module is configured to receive a user's selection of a first workflow from a plurality of workflows on a first interface, wherein each of the plurality of workflows is configured to perform a task associated with generating multimedia materials; The user input module is configured to receive descriptive input from the user on the first interface; as well as The multimedia material output module is configured to display the multimedia material on the first interface, the multimedia material being generated by calling the first workflow based on the descriptive input.

14. An electronic device comprising: processor; as well as A memory coupled to the processor, the memory having instructions stored therein, which, when executed by the processor, cause the electronic device to perform the method according to any one of claims 1 to 12.

15. A computer program product comprising computer-executable instructions, wherein the computer-executable instructions are executed by a processor to implement the method according to any one of claims 1 to 12.

Images