Method for generating multimedia content, electronic device, storage medium and product

Abstract

The present disclosure relates to a multimedia content generation method, an electronic device, a storage medium and a product, and relates to the technical field of multimedia. The multimedia content generation method comprises: displaying a digital avatar of a user in a conversation interface, the digital avatar being generated based on an image from the user; displaying a message sent by the user in the conversation interface; generating multimedia content of the digital avatar based on the digital avatar and the message; and displaying the multimedia content sent to the user and information of an object associated with the multimedia content in the conversation interface. The present disclosure can associate the information that the user wants with the digital avatar of the user, and the recommended object is also associated with the digital avatar. Therefore, the user can intuitively browse the information related to the content that the user is interested in or the content that the user may be interested in, and the efficiency of the user obtaining information is improved.

Description

Technical Field

[0001] This disclosure relates to the field of multimedia technology, and in particular to a method for generating multimedia content, an electronic device, a storage medium, and a product. Background Technology

[0002] In related technologies, a digital avatar can be generated based on an image provided by the user. For example, the user can choose a preset template. This template includes some prompts. Based on these prompts, a digital avatar matching the template can be generated. For example, if the template is a travel photo of a certain scenic spot, then the digital avatar could be a photo of the user traveling to that scenic spot. Summary of the Invention

[0003] This summary section is provided to briefly introduce the concepts, which will be described in detail in the detailed description section below. This summary section is not intended to identify key or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0004] According to some embodiments of this disclosure, a method for generating multimedia content is provided, comprising: displaying a user's digital clone in a dialog interface, the digital clone being generated based on an image from the user; displaying a message sent by the user in the dialog interface; generating multimedia content of the digital clone based on the digital clone and the message; and displaying the multimedia content sent to the user and information about objects associated with the multimedia content in the dialog interface.

[0005] According to some embodiments of the present disclosure, an electronic device is provided, including: a memory; and a processor coupled to the memory, the processor being configured to execute a method for generating multimedia content according to any embodiment of the present disclosure based on instructions stored in the memory.

[0006] According to some embodiments of the present disclosure, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, performs a method for generating multimedia content according to any embodiment of the present disclosure.

[0007] According to some embodiments of the present disclosure, a computer program product is provided that, when run on a computer, enables the computer to implement the multimedia content generation method of any embodiment of the present disclosure.

[0008] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0009] Preferred embodiments of the present disclosure are described below with reference to the accompanying drawings. The accompanying drawings, which are included to provide a further understanding of the present disclosure, and which, together with the following detailed description, are incorporated in and form a part of this specification and are used to explain the present disclosure. It should be understood that the drawings described below only relate to some embodiments of the present disclosure and are not intended to limit the present disclosure. In the drawings:

[0010] Figure 1 A flowchart illustrating a method for generating multimedia content according to some embodiments of the present disclosure is shown.

[0011] Figure 2 A schematic diagram of a dialog interface according to some embodiments of the present disclosure is shown.

[0012] Figure 3 A flowchart illustrating a method for determining associated objects according to some embodiments of the present disclosure is shown.

[0013] Figure 4 A flowchart illustrating a method for determining associated objects according to other embodiments of this disclosure is shown.

[0014] Figure 5 A schematic diagram of a dialog interface according to other embodiments of the present disclosure is shown.

[0015] Figure 6 A flowchart illustrating a method for generating multimedia content for a digital clone according to some embodiments of the present disclosure is shown.

[0016] Figure 7 A schematic diagram of the structure of a multimedia content generation apparatus according to some embodiments of the present disclosure is shown.

[0017] Figure 8 A schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure is shown.

[0018] Figure 9 A schematic diagram of the structure of a computer system according to some embodiments of the present disclosure is shown.

[0019] It should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not necessarily drawn to actual scale. The same or similar reference numerals are used in the various drawings to denote the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. Detailed Implementation

[0020] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. However, it is obvious that the described embodiments are only some embodiments of this disclosure, and not all embodiments. The following description of the embodiments is merely illustrative and is in no way intended to limit this disclosure or its application or use. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0021] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of components and steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0022] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Furthermore, as used in this disclosure, the term "including" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". Therefore, "comprising" and "including" are synonymous. The term "based on" means "at least partially based on".

[0023] Throughout this specification, the terms "one embodiment," "some embodiments," or "embodiment" mean that a specific feature, structure, or characteristic described in connection with an embodiment is included in at least one embodiment of the invention. For example, the term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; and the term "some embodiments" means "at least some embodiments." Furthermore, the appearance of the phrases "in one embodiment," "in some embodiments," or "in an embodiment" in various places throughout the specification does not necessarily refer to the same embodiment, but may refer to the same embodiment.

[0024] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0025] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0026] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0027] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0028] It should be understood that this disclosure does not limit how the image to be applied / processed is obtained. In one embodiment of this disclosure, it can be obtained from a storage device, such as internal or external memory; in another embodiment, it can be captured by a camera component. It should be noted that the acquired image can be a single captured image or a frame from a captured video, and is not particularly limited to these methods.

[0029] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that the type of image is not specifically limited in the context of this specification. Furthermore, an image can be any suitable image, such as a raw image obtained by a camera device, or an image from which specific processing has been performed, such as preliminary filtering, dealiasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that preprocessing operations may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0030] In related technologies, the role of digital avatars is often limited to providing users with various avatars, primarily satisfying users' aesthetic needs. Further applications of digital avatars are rare.

[0031] This application provides a method for generating multimedia content. By generating multimedia content of a digital clone based on messages sent by the user during a dialogue between the user and the intelligent agent, and then sending information about the object associated with the multimedia content to the user, the method can respond to messages sent by the user using the multimedia content of the digital clone.

[0032] Figure 1 A flowchart illustrating a method for generating multimedia content according to some embodiments of the present disclosure is shown. Figure 1 As shown, the generation method of this embodiment includes steps S102 to S108.

[0033] In step S102, the user's digital clone is displayed in the dialog interface. The digital clone is generated based on an image from the user.

[0034] A dialogue interface refers to the interface through which a user interacts with an intelligent agent, such as a chat interface. In this interface, the user can send messages to the intelligent agent in the form of text, voice, multimedia content (such as images, videos, etc.), and files. The intelligent agent will respond to the user based on the content of the messages sent, for example, by replying with one or more messages. The messages replied by the intelligent agent can also be of various types. In some embodiments, the dialogue interface may involve more than two participants, such as a group chat interface.

[0035] Intelligent agents can be implemented in software, hardware, or a combination of both. They can be implemented using machine learning models, such as Large Language Models (LLMs) or Foundation Models. These machine learning models can be generative models, which output target content based on input information. Input information may include prompts, such as dialogue between the user and the agent, content extracted from the dialogue, or other information.

[0036] Generative models include, for example, models that generate data from text or images. The output of a generative model can include text, images, or a combination of both. Of course, the input or output of a generative model can also be data from other modalities, such as audio, video, or a combination of multiple data types. Generative models can be single-modal, such as models that generate text from text ("text-to-text model") or images from images ("image-to-image model"); alternatively, generative models can be cross-modal, meaning the input and output belong to different modalities, such as models that generate images from text ("text-to-image model"); furthermore, the input of a generative model can include multiple modalities, and the output can also include multiple modalities.

[0037] A user's digital avatar refers to a virtual object created for the user with their authorization. This virtual object can be a two-dimensional or three-dimensional image, generated based on an image from the user. This image is user-authorized, including images uploaded or selected by the user. For example, if the image includes the user, the generated digital avatar's shape and appearance can closely resemble the user's.

[0038] When generating a digital clone, in addition to referencing images from the user, the descriptive information sent by the user can also be considered. Therefore, the generated digital clone's appearance can closely resemble the user's image and match the descriptive information. For example, if a user provides their own photo and adds the description "bigger eyes," a digital clone will be generated that resembles the user but has larger eyes.

[0039] Users can create digital avatars through the avatar creation interface or through a dialogue interface. Taking the latter as an example, users can send messages to the agent that include images, as well as messages that include instructions for creating a digital avatar (e.g., "Generate a digital avatar for me based on this photo"). Additionally, users can send messages to the agent that include descriptive information, such as "Add natural makeup and whitening effects."

[0040] Figure 2 A schematic diagram of a dialog interface according to some embodiments of the present disclosure is shown. Figure 2 As shown, dialogue interface 2 displays the conversation between the user and agent A. The content of message 21 sent by the user is "Help me generate a digital clone photo: princess, long dress, morning light, natural makeup, whitening effect". Agent A can generate a digital clone based on this message and the image from the user. After generating the digital clone, it can be displayed via message 22.

[0041] The digital avatar displayed in the dialog interface can be a three-dimensional virtual object. Alternatively, it can be one or more images, such as screenshots of one or more angles of a three-dimensional virtual object. Of course, it can also be displayed in other forms, such as videos, which will not be elaborated on here.

[0042] In step S104, the message sent by the user is displayed in the dialog interface.

[0043] The messages sent by users can be of various types. Semantic understanding of the message can determine whether it includes instructions to generate multimedia content, or whether it contains information related to the multimedia content to be generated, such as the topic of the multimedia content. This semantic understanding can be based on a natural language processing model.

[0044] In some embodiments, after displaying the digital clone, the user sends the message to the agent to send an instruction to further generate multimedia content based on the digital clone.

[0045] In step S106, multimedia content for the digital clone is generated based on the digital clone and the message sent by the user. The generated multimedia content may include images, videos, etc., of the digital clone, and this multimedia content is associated with the message. At least one of the digital clone's actions, clothing, scene, etc., in the generated multimedia content may differ from the originally displayed digital clone. That is, the generated multimedia content may be the result of the digital clone being presented in a different style, and this style is determined based on the message sent by the user.

[0046] For example, if a user sends a message that includes or is related to a specified topic, the generated multimedia content will also be related to that topic.

[0047] In step S108, the multimedia content sent to the user and information about the objects associated with the multimedia content are displayed on the dialog interface. That is, the agent can send one or more messages to the user, which include the generated multimedia content and information about the objects associated with the multimedia content.

[0048] Objects associated with multimedia content can appear directly within the multimedia content, such as items held or worn by the digital avatar. Alternatively, elements within the multimedia content can be related to these objects. For instance, if the multimedia content is an image of the digital avatar at a tourist attraction, then the objects associated with the multimedia content could be airline tickets to that attraction, or hotels near the attraction, etc.

[0049] The information about this object can be descriptive, such as descriptive text, images, or videos. Alternatively, it can be a navigation control; triggering this control displays the object's interface or transaction screen. This allows for the use of a dialog interface to recommend items to the user.

[0050] Through the above embodiments, after creating a user's digital avatar, multimedia content for the digital avatar can be generated based on messages sent by the user, and information about objects associated with that multimedia content can be pushed to the user simultaneously. Thus, during dialogue between the agent and the user, the agent can send the user's digital avatar's multimedia content and related objects based on the content of the dialogue; that is, responding to the user with the digital avatar's multimedia content and associated object information. This allows the agent to associate the information the user wants with the user's digital avatar, while simultaneously recommending objects that are also associated with the digital avatar. Therefore, users can intuitively browse information related to content they are interested in or that they might be interested in, improving the efficiency of information acquisition.

[0051] Since generating multimedia content for a digital avatar is only one scenario in the dialogue between the user and the agent, many conversations between the user and the agent may not involve the aforementioned scenarios. Therefore, in order to respond appropriately to the user, intent recognition can be performed on the message, and multimedia content for the digital avatar can be generated in response to the user's message having the intent to create multimedia content based on the digital avatar. Intent recognition of the message can be performed through keyword recognition, processing through an intent analysis model, or determination through a natural language model (such as a large language model). If the message sent by the user does not have the intent to create multimedia content based on the digital avatar, then multimedia content for the digital avatar is not generated.

[0052] The generated multimedia content can have a specific theme. For example, the theme can be determined based on a user-sent message, and multimedia content can be generated according to the specified theme. Alternatively, multimedia content can be generated directly based on the user-sent message, and then the theme can be extracted. Then, based on this theme, the objects associated with the multimedia content can be determined. The following describes these two methods for determining associated objects as examples.

[0053] Figure 3 A flowchart illustrating a method for determining associated objects according to some embodiments of this disclosure is shown. Figure 3 As shown, the determination method in this embodiment includes steps S302 to S306.

[0054] In step S302, the theme of the multimedia content to be generated is determined based on the message sent by the user.

[0055] The topic can be extracted directly from the message or determined indirectly based on the message. In some embodiments, reference information can be generated based on the message, which may be generated by expanding the message or responding to it, and then the topic can be extracted from the reference information.

[0056] Topic extraction can be achieved using natural language processing models such as language models and topic analysis models. Generating reference information can be achieved using generative models.

[0057] For example, if a user sends a message saying "Generate a birthday greeting video," the theme "birthday greeting" can be directly extracted from that message. As another example, if a user sends a message saying "What color clothes do you think suit me?", after semantic understanding and analysis of the digital clone's appearance and attributes, it can be determined that blue clothes suit the user, so the theme can be set as "blue clothes."

[0058] In step S304, image materials are searched based on the topic.

[0059] For example, you can directly use the topic as a search term to search for image materials, or you can search for image materials based on the topic and its synonyms, near-synonyms, etc. Image materials can be in image format, such as images of certain objects or locations; or, image materials can also be descriptive information or processing information of images, such as colors, filters, special effects, etc.

[0060] In some embodiments, objects associated with the topic are searched, and then image assets associated with those objects are determined. Some topics determined based on user messages may be general and not necessarily correspond to a specific product. For example, for the topic "blue clothes," existing products featuring blue clothes, such as a particular style from a certain brand, can be searched as objects associated with that topic. Then, image assets associated with that product, such as product images or product elements, are used as image assets. Thus, the generated multimedia content of the digital avatar can reflect the relationship between the digital avatar and the actual product, for example, it can reflect the effect of the user wearing the product. Therefore, the generated multimedia content is more valuable to the user and has a closer relationship with the recommended objects, allowing the user to efficiently obtain valuable information.

[0061] Image assets associated with the object include at least one of the following: image assets of the object itself, image assets of the item provided by the object, and image assets showing the effect of using the object. Taking a lipstick as an example, the image assets associated with the object refer to the image assets of the lipstick itself, and the image assets showing the effect of using the object refer to the image assets showing the color of the lips after using the lipstick. If the lipstick is an item provided by an object, then that object could be a shop selling lipsticks.

[0062] In step S306, multimedia content for the digital clone is generated based on the digital clone and the image material. For example, the digital clone and the image material can be merged or otherwise processed.

[0063] In some embodiments, the overlay method between the image material and the digital clone can be determined based on the user's message and image material, such as angle, position, etc., and then image fusion processing can be performed. In some embodiments, the digital clone and image material can also be fused based on an image generation model, such as an "image-generated image model".

[0064] Because the image materials are based on topic search, the generated multimedia content can also reflect that topic. Furthermore, since the topic is determined based on the messages sent by the user, the generated multimedia content can effectively respond to the user's messages.

[0065] The above embodiments determine the topic based on the message sent by the user, then search for image materials based on the topic and generate multimedia content for the digital avatar. Thus, the generated multimedia content can correspond to a specific object, allowing the user to intuitively see the association between the user and the object when sending information about that object to the user for recommendation. If the user is satisfied with the display, they can further learn about the recommended objects based on the sent information about that object.

[0066] Figure 4 A flowchart illustrating a method for determining associated objects according to other embodiments of this disclosure is shown. For example... Figure 4 As shown, the generation method of this embodiment includes steps S402 to S406.

[0067] In step S402, multimedia content for the digital clone is generated based on the digital clone and the message.

[0068] In this embodiment, for example, the content of the message, or the content after expanding or responding to the message, can be used as a prompt to generate multimedia content for the digital avatar. Thus, the generated multimedia content can match the message sent by the user.

[0069] In step S404, the theme of the multimedia content of the digital clone is identified.

[0070] Multimedia content processing models can be used to process the multimedia content of digital clones to obtain their themes. For example, descriptive text can be generated based on the multimedia content, and then themes can be extracted from the descriptive text.

[0071] In step S406, the objects associated with the multimedia content are determined based on the topic.

[0072] For example, you can directly use the topic as the search term, or search based on the topic and its synonyms, near-synonyms, etc.

[0073] Through the above embodiments, multimedia content can be generated first, then themes can be extracted from it, and finally, associated objects can be determined. This allows for greater diversity in the generated multimedia content. Although the generated multimedia content is based on user messages, the agent may refer to other information during the generation process, such as the agent's knowledge base. In this way, the generated multimedia content can contain more elements, enabling it to recommend potential, potentially interesting objects to the user while responding to their needs.

[0074] In some cases, the messages sent by the user may or may not explicitly include the intent to generate multimedia content for the digital avatar. Since one of the main functions of an intelligent agent is to interact with the user, i.e., to respond to the user's messages, in some embodiments, the decision to generate multimedia content for the digital avatar may be based on the agent's response to the user. The agent's response message may be text-based, or a combination of text and other types of messages, or other types of messages besides text.

[0075] In some embodiments, a response message is first generated for the agent in response to a message sent by the user. In response to the response message including a specified object, multimedia content for the digital clone is generated based on the digital clone, the message, and the specified object. Before generating the response message, the agent can directly generate a response based on the user-sent message, without considering whether multimedia content for the digital clone will be generated subsequently. For example, before generating a response using a generative model, a prompt message is generated based on the user-sent message, but not based on an instruction to generate multimedia content for the digital clone. After generating the response, it is determined whether the response message includes the specified object, and based on this, it is determined whether to generate multimedia content for the digital clone. If the response message includes the specified object, multimedia content is generated; otherwise, it is not generated. The specified object can be an object suitable for generating multimedia content or a pre-defined object.

[0076] For example, a user sends a message to the agent instructing it to recommend recently popular lipsticks, clothing, etc. The agent can then send responses to these recommendation requests. Furthermore, if the response includes a specific product (e.g., a product from a specific category or brand), multimedia content for the digital avatar is generated based on that product.

[0077] This embodiment enables the generation of multimedia content for a digital avatar based on a specified object when the agent's response includes that object. This allows for a more comprehensive response to the user through various means, including sending text, multimedia content, and object-related information, thereby improving the user's information retrieval efficiency and interactive experience.

[0078] The messages sent by users may clearly describe their needs, such as the elements they want to include in the multimedia content to be generated. However, in some cases, the user's needs are not very clear. The user might instruct the agent to generate multimedia content for the digital avatar in a questioning tone. For example, a user might send the message "What hairstyle suits me?" In this message, the user does not specify what hairstyle to generate. The user might also give a vague instruction, such as "Generate a travel video," but the user does not specify where they want to travel. Therefore, in some embodiments, a response message can be sent to the user first, and then the multimedia content for the digital avatar can be generated based on the response message. For example, in response to a user's message indicating the need to generate multimedia content for the digital avatar, one or more prompt messages are generated based on the message; based on one or more prompt messages, a message is generated for the agent to send to the user; and based on the digital avatar and one or more prompt messages, the multimedia content for the digital avatar is generated. That is, the prompt message for the agent to respond to the user is determined first based on the message sent by the user, and then a response is generated based on the prompt message. For example, the message for the agent to send to the user is generated based on the prompt message and the generative model. Then, a generative model for generating multimedia content is used to process the prompts and the digital clone to generate multimedia content for the digital clone. This allows users to more clearly understand the generation logic or key information of the multimedia content, making the generated content more informative and improving the efficiency of information retrieval for users.

[0079] Figure 5 A schematic diagram of a dialog interface according to other embodiments of this disclosure is shown. For example... Figure 5 As shown, the dialogue interface 5 in this embodiment is a dialogue interface between the user and the intelligent agent A. In this interface, messages sent by the user are exemplarily aligned to the right and displayed, while messages sent by the intelligent agent A are exemplarily aligned to the left and displayed.

[0080] During the conversation, the user sends message 51, "What hairstyle do you think suits me?". Because the message sent by the user is related to their appearance, in some embodiments, this can trigger the generation of multimedia content for the digital avatar. Alternatively, the agent can respond first without considering whether to generate multimedia content, and then generate multimedia content based on that response.

[0081] Based on user-sent messages, it can determine whether the multimedia content to be generated is related to the hairstyle, and generate multimedia content featuring a digital clone with the new hairstyle. Alternatively, it can also be like... Figure 5As shown, the agent first responds to message 51 with message 52, namely, "I think the curly hairstyle suits your face shape quite well, so I generated a photo of you with curly hair and wearing a dress. What do you think?" Then, based on the specific hairstyle and clothing elements in the response, it generates an image (or video, etc.) of the digital clone and sends it to the user via message 53.

[0082] Furthermore, the intelligent agent also sends the user objects associated with the multimedia content of the generated digital avatar. Figure 5 In the example, the associated object is "XX Hair Salon," and information about this associated object is sent to the user via message 54. This message can be a text / image description or a control. Thus, if the user is satisfied with the digital clone effect, they can learn more about the related object through message 54.

[0083] Of course, depending on the needs, the intelligent agent may also choose not to send a response message to the user, but only send multimedia content and object information. Those skilled in the art can make this choice as needed.

[0084] As mentioned earlier, digital clones can be three-dimensional clones to facilitate the generation of multimedia content from various angles of the digital clone. Figure 6 A flowchart illustrating a method for generating multimedia content for a digital clone according to some embodiments of the present disclosure is shown. Figure 6 As shown, the generation method of this embodiment includes steps S602 to S606.

[0085] In step S602, the theme of the multimedia content to be generated is determined based on the message sent by the user. Specific implementation details of this step can be found in the foregoing embodiments and will not be repeated here.

[0086] In step S604, the target part of the three-dimensional digital clone is determined based on the topic.

[0087] For example, if the topic includes a description of a part of the body, then that part is identified as the target part; if the topic is associated with a part of the body, then the associated part is identified as the target part.

[0088] In some embodiments, the target body part can be determined based on image assets associated with a theme. First, image assets associated with the theme are identified; then, the area of ​​effect of the image assets in the 3D digital avatar is determined; and finally, the target body part of the 3D digital avatar is determined based on the area of ​​effect. Image assets can be pre-identified to clearly indicate their area of ​​effect. For example, if the asset is a hat, the area of ​​effect is the head; if the asset is a shoe, the area of ​​effect is the feet. Of course, the area of ​​effect can also be indicated by the name, category, etc., of the image asset. The area of ​​effect of the image asset can be equivalent to the target body part of the digital avatar, or the target body part of the digital avatar can include the area of ​​effect. For example, when the area of ​​effect is the feet, the target body part can be the user's legs and feet, or it can be the user's entire body.

[0089] In step S606, multimedia content for the digital clone is generated based on the target area, the digital clone, and the message sent by the user. The generated multimedia content includes the target area of ​​the digital clone, and the appearance of the digital clone can match or be associated with the message.

[0090] The above embodiments generate multimedia content based on a three-dimensional digital avatar and determine the display location based on the theme. Therefore, the display method of the digital avatar can be automatically determined so that the generated multimedia content can match the messages sent by the user.

[0091] The multimedia content generation method of the embodiments of this disclosure can be executed entirely or partially on a user device. For example, steps related to display and user interaction can be executed on the user device. Steps involving processing using a generative model can be executed on the user device, on a server, or partially on the user device and partially on the server.

[0092] The methods of various embodiments of this disclosure have been described above. The apparatus for implementing the various methods is further described below.

[0093] Figure 7 A schematic diagram of a multimedia content generation apparatus according to some embodiments of the present disclosure is shown. For example... Figure 7 As shown, the generation device 70 of this embodiment includes: a first display module 701 configured to display a user's digital clone in a dialog interface, the digital clone being generated based on an image from the user; a first display module 702 configured to display a message sent by the user in the dialog interface; a generation module 703 configured to generate multimedia content of the digital clone based on the digital clone and the message; and a third display module 704 configured to display the multimedia content sent to the user and information about objects associated with the multimedia content in the dialog interface.

[0094] In some embodiments, the generation module 703 is further configured to determine the theme of the multimedia content to be generated based on the message; search for image materials based on the theme; and generate the multimedia content of the digital clone based on the digital clone and the image materials.

[0095] In some embodiments, the generation module 703 is further configured to search for objects associated with the topic and determine image materials associated with the objects.

[0096] In some embodiments, the image material associated with an object includes at least one of the following: image material of the object, image material of the item provided by the object, and image material of the object's usage effect.

[0097] In some embodiments, the generating apparatus 70 includes a determining module 705 configured to: identify the theme of the multimedia content of the digital clone; and determine an object associated with the multimedia content based on the theme.

[0098] In some embodiments, the generation module 703 is further configured to: generate a response message from the agent to the message; and in response to the response message including a specified object, generate multimedia content for the digital clone based on the digital clone, the message, and the specified object.

[0099] In some embodiments, the generation module 703 is further configured to: generate one or more prompt messages based on the message in response to an indication to generate multimedia content; generate a message sent by the agent to the user based on the one or more prompt messages; and generate multimedia content for the digital clone based on the digital clone and the one or more prompt messages.

[0100] In some embodiments, the generation module 703 is further configured to: determine the theme of the multimedia content to be generated based on the message; determine the target part of the three-dimensional digital clone based on the theme; and generate the multimedia content of the digital clone based on the target part, the digital clone, and the message.

[0101] In some embodiments, the generation module 703 is further configured to: determine image material associated with the theme; determine the area of ​​effect of the image material on the 3D digital clone; and determine the target area of ​​the 3D digital clone based on the area of ​​effect.

[0102] In some embodiments, the generation module 703 is further configured to: identify the intent of the message; and generate multimedia content for the digital clone in response to the message having the intent to create multimedia content based on the digital clone.

[0103] In some embodiments, the information of the object is a jump control, and the generating device 70 further includes: a fourth display module 706, configured to: display the object's display interface or transaction interface in response to a trigger operation on the object's jump control.

[0104] It should be noted that the above-described units are merely logical modules divided according to their specific functions, and are not intended to limit the specific implementation method. For example, they can be implemented in software, hardware, or a combination of both. In actual implementation, the above-described units can be implemented as independent physical entities, or they can be implemented by a single entity (e.g., a processor (CPU or DSP, etc.), integrated circuit, etc.). Furthermore, the units shown in the accompanying drawings with dashed lines indicate that these units may not actually exist, and the operations / functions they perform can be implemented by the processing circuitry itself.

[0105] In addition, although not shown, the device may also include a memory that can store various information generated by the device and its constituent units during operation, programs and data used for operation, data to be transmitted by the communication unit, etc. The memory can be volatile memory and / or non-volatile memory. For example, the memory may include, but is not limited to, random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Of course, the memory may also be located outside the device. Optionally, although not shown, the device may also include a communication unit that can be used to communicate with other devices. In one example, the communication unit can be implemented in a manner known in the art, such as including communication components like antenna arrays and / or radio frequency links, various types of interfaces, communication units, etc. These will not be described in detail here. Furthermore, the device may also include other components not shown, such as radio frequency links, baseband processing units, network interfaces, processors, controllers, etc. These will not be described in detail here.

[0106] Some embodiments of this disclosure also provide an electronic device. Figure 8 A schematic diagram of the structure of an electronic device according to some embodiments of the present disclosure is shown. For example, in some embodiments, the electronic device 8 can be various types of devices, including but not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. For example, the electronic device 8 may include a display panel for displaying data and / or execution results utilized in the scheme according to the present disclosure. For example, the display panel can be of various shapes, such as a rectangular panel, an elliptical panel, or a polygonal panel. In addition, the display panel can be not only a planar panel, but also a curved panel, or even a spherical panel.

[0107] like Figure 8As shown, the electronic device 8 of this embodiment includes a memory 81 and a processor 82 coupled to the memory 81. It should be noted that... Figure 8 The components of the electronic device 8 shown are merely exemplary and not limiting; the electronic device 8 may have other components as needed for actual applications. The processor 82 can control other components in the electronic device 8 to perform desired functions.

[0108] In some embodiments, memory 81 is used to store one or more computer-readable instructions. When processor 82 executes the computer-readable instructions, the computer-readable instructions are executed by processor 82 to implement the method according to any of the above embodiments. For specific implementations and related explanations of the various steps of the method, please refer to the above embodiments; repeated details will not be elaborated here.

[0109] For example, processor 82 and memory 81 can communicate with each other directly or indirectly. For example, processor 82 and memory 81 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. Processor 82 and memory 81 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0110] For example, processor 82 can be embodied in various suitable processors, processing devices, such as central processing unit (CPU), graphics processing unit (GPU), network processor (NP), etc.; it can also be digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The central processing unit (CPU) can be an x86 or ARM architecture, etc. For example, memory 81 can include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Memory 81 can include, for example, system memory, which stores, for example, the operating system, application programs, boot loader, database, and other programs. Various application programs and various data can also be stored in the storage medium.

[0111] Furthermore, according to some embodiments of this disclosure, various operations / processes according to this disclosure, when implemented via software and / or firmware, can be transmitted from a storage medium or network to a computer system with a dedicated hardware architecture, such as... Figure 9 The computer system 90 shown is equipped with the programs that constitute the software. When various programs are installed, the computer system is able to perform various functions, including functions such as those described above. Figure 9A schematic diagram of the structure of a computer system according to some embodiments of the present disclosure is shown.

[0112] exist Figure 9 In this configuration, the Central Processing Unit (CPU) 901 performs various processes based on a program stored in the Read-Only Memory (ROM) 902 or a program loaded into the Random Access Memory (RAM) 903 from the Storage Section 908. The RAM 903 also stores data required when the CPU 901 performs various processes, etc. The CPU is merely exemplary; it can also be other types of processors, such as the various processors described above. The ROM 902, RAM 903, and Storage Section 908 can be various forms of computer-readable storage media, as described below. It should be noted that although... Figure 9 The image shows ROM 902, RAM 903 and storage device 908, but one or more of them may be combined or located in the same or different memory or storage modules.

[0113] CPU 901, ROM 902 and RAM 903 are interconnected via bus 904. Input / output interface 905 is also connected to bus 904.

[0114] The following components are connected to the input / output interface 905: input section 909, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 907, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 908, including hard disk, magnetic tape, etc.; and communication section 909, including network interface cards such as LAN cards, modems, etc. The communication section 909 allows communication processing via a network such as the Internet. It is easy to understand that, although... Figure 9 The various devices or modules in the computer system 90 shown communicate via bus 904, but they may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless and wired networks.

[0115] As needed, drive 910 is also connected to input / output interface 905. Removable media 911, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 910 as needed, so that computer programs read from them can be installed into storage section 908 as needed.

[0116] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or from a storage medium such as a removable medium 911.

[0117] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the CPU 901, it performs the functions defined in the methods of embodiments of this disclosure.

[0118] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to, an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium, capable of sending, propagating, or transmitting a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0119] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0120] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the method of any of the above embodiments. For example, the instructions may be embodied in computer program code.

[0121] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0122] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0123] The modules, components, or units described in the embodiments of this disclosure can be implemented in software or hardware. The names of the modules, components, or units do not necessarily constitute a limitation on the module, component, or unit itself.

[0124] The functions described above in this document can be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), complex programmable logic devices (CPLDs), and so on.

[0125] The above description is merely an illustration of some embodiments of this disclosure and the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0126] Many specific details are set forth in the description provided herein. However, it is understood that embodiments of the invention may be practiced without these specific details. In other instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of the description.

[0127] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0128] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for generating multimedia content, comprising: displaying, in a conversation interface, a digital avatar of a user, the digital avatar being generated based on an image from the user; displaying, in the conversation interface, a message sent by the user; generating, based on the digital avatar and the message, a multimedia content of the digital avatar, including: determining a theme of the multimedia content to be generated according to the message; searching for an object associated with the theme; determining image material associated with the object; and fusing the digital avatar and the image material based on the digital avatar and the image material to generate the multimedia content of the digital avatar; displaying, in the conversation interface, one or more messages sent by an agent to the user, the messages including the multimedia content and information of the object associated with the theme of the multimedia content, wherein the object is an object appearing in the multimedia content or an object related to an element of the multimedia content.

2. The generation method of claim 1, wherein, The image material associated with the object includes at least one of image material of the object, image material of an article provided by the object, and image material of a use effect of the object. 3.The method of claim 1, further comprising: identifying the theme of the multimedia content of the digital avatar; and determining the object associated with the multimedia content according to the theme.

4. The generation method of claim 1, wherein, The generating, based on the digital avatar and the message, of the multimedia content of the digital avatar includes: generating a response message of the agent to the message; and in response to the response message including a specified object, generating the multimedia content of the digital avatar based on the digital avatar, the message, and the specified object.

5. The generation method of claim 1, wherein, The generating, based on the digital avatar and the message, of the multimedia content of the digital avatar includes: in response to the message including an indication of generating the multimedia content, generating one or more prompt messages based on the message; and generating, based on the one or more prompt messages, a message sent by the agent to the user; and generating, based on the digital avatar and the one or more prompt messages, the multimedia content of the digital avatar.

6. The generation method of claim 1, wherein, The digital avatar is a three-dimensional digital avatar, and the generating, based on the digital avatar and the message, of the multimedia content of the digital avatar includes: determining the theme of the multimedia content to be generated based on the message; and determining a target part of the three-dimensional digital avatar based on the theme; and generating the multimedia content of the digital avatar based on the target part, the digital avatar, and the message.

7. The generation method of claim 6, wherein, The determining, based on the theme, of the target part of the three-dimensional digital avatar includes: determining image material associated with the theme; and determining an action part of the three-dimensional digital avatar for the image material; and determining the target part of the three-dimensional digital avatar according to the action part.

8. The generation method of claim 1, wherein, The generating, based on the digital avatar and the message, of the multimedia content of the digital avatar includes: performing intent recognition on the message; and in response to the message having an intent of creating a multimedia content based on the digital avatar, generating the multimedia content of the digital avatar.

9. The generation method of claim 1, wherein, The information of the object is a jump control, and the generation method further includes: In response to a triggering operation on the jump control of the object, displaying a display interface or a transaction interface of the object. 10.An electronic device, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute a multimedia content generation method according to any one of claims 1 to 9 based on instructions stored in the memory. 11.A computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the multimedia content generation method according to any one of claims 1 to 9. 12.A computer program product which, when running on a computer, causes the computer to implement the multimedia content generation method according to any one of claims 1 to 9.

Images