Virtual object generation method, apparatus, and computer-readable storage medium

Abstract

The present disclosure relates to a virtual object generation method, device and computer readable storage medium, and relates to the technical field of computers. The virtual object generation method comprises: generating, according to related information of a display interface, description information of a virtual object to be displayed on the display interface by using a first machine learning model, the related information comprising display style information of the display interface; and generating a virtual object matching the display style of the display interface according to the description information. The technical solution of the present disclosure can improve the generation efficiency of virtual objects.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a method for generating virtual objects, a device for generating virtual objects, a computer-readable storage medium, and a computer program product. Background Technology

[0002] In real-time interactive scenarios, such as chat rooms and live streaming rooms, it has become a popular way for viewers to send virtual items as gifts to the streamer. Therefore, developers need to pre-generate a variety of virtual gifts that are compatible with the interactive scenarios.

[0003] In related technologies, the design of virtual items needs to be conceived manually based on the interactive scenario. Summary of the Invention

[0004] The inventors of this disclosure have discovered the following problem in the aforementioned related technologies: the generation efficiency of virtual objects is low.

[0005] In view of this, this disclosure proposes a virtual object generation technology solution that can improve the generation efficiency of virtual objects.

[0006] According to some embodiments of this disclosure, a method for generating virtual objects is provided, comprising: generating description information of a virtual object to be displayed on the display interface using a first machine learning model based on relevant information of the display interface, wherein the relevant information includes display style information of the display interface; and generating a virtual object that matches the display style of the display interface based on the description information.

[0007] In some embodiments, the descriptive information includes feature description information of the virtual object, which includes display style information of the virtual object described using natural language, and also includes special effects information and / or movement method information of the virtual object.

[0008] In some embodiments, the description information includes material information for generating virtual objects, the material information being generated by a first machine learning model based on the feature description information, the material information including main material information, and the material information further including at least one of foreground material information, background material information, or special effects material information.

[0009] In some embodiments, generating a virtual object that matches the display style of the display interface based on the description information includes: generating a style sample image that matches the display style of the display interface using a second machine learning model; and generating a virtual object based on the style sample image.

[0010] In some embodiments, generating a style sample image that matches the display style of the display interface using a second machine learning model includes: generating a first prompt message using a first machine learning model based on description information, the first prompt message including style prompt words; and generating a style sample image using a second machine learning model based on the first prompt message.

[0011] In some embodiments, the first prompt information includes material prompt words, and generating a virtual object based on a style sample image includes: generating materials included in the virtual object using a third machine learning model based on the style sample image and the material prompt words.

[0012] In some embodiments, the style cue words include background image cue words, and generating a style sample image using a second machine learning model based on the first cue information includes: generating a background image as a style sample image using the second machine learning model based on the background image cue words.

[0013] In some embodiments, generating a virtual object based on a style sample image includes: generating a static image of the material included in the virtual object using a third machine learning model based on the style sample image; and generating a dynamic image of the material based on the static image using the third machine learning model.

[0014] In some embodiments, the material includes the main material of a virtual object. Generating a virtual object based on the style sample image includes: generating a two-dimensional static model of the main material using a third machine learning model based on the style sample image; and adding skeletal animation to the two-dimensional static model using the third machine learning model to generate a three-dimensional dynamic model of the main material.

[0015] In some embodiments, generating a virtual object that matches the display style of the display interface according to the description information includes: selecting a second machine learning model corresponding to the display style information according to the display style information in the description information; selecting a processing node for generating materials corresponding to the material information according to the material information in the description information; generating a style sample image using the second machine learning model; and generating a virtual object using the processing node based on the style sample image.

[0016] In some embodiments, the display interface includes an identifier for the broadcaster and an identifier for the consumer participating in the interaction. The description information includes description information of the virtual object moving from the consumer identifier to the broadcaster identifier. Generating a virtual object that matches the display style of the display interface based on the description information includes generating a dynamic image of the virtual object moving from the consumer identifier to the broadcaster identifier.

[0017] In some embodiments, generating a dynamic image of a virtual object moving from a consumer identifier to a broadcaster identifier includes: generating a dynamic image of the virtual object displaying a specified action after moving to the broadcaster identifier.

[0018] In some embodiments, the relevant information includes layout information of the display interface and / or type information of the display interface. Based on the relevant information of the display interface of the virtual object, generating the description information of the virtual object using the first machine learning model includes: generating the description information of the virtual object using the first machine learning model based on the second prompt information, wherein the second prompt information includes at least one of atmosphere information, scene information, or display style of the virtual object.

[0019] According to some other embodiments of this disclosure, a virtual object generation apparatus is provided, comprising: an information generation unit, configured to generate description information of a virtual object to be displayed on the display interface using a first machine learning model based on relevant information of the display interface, the relevant information including display style information of the display interface; and an object generation unit, configured to generate a virtual object matching the display style of the display interface based on the description information.

[0020] In some embodiments, the descriptive information includes feature description information of the virtual object, which includes display style information of the virtual object described using natural language, and also includes special effects information and / or movement method information of the virtual object.

[0021] In some embodiments, the description information includes material information for generating virtual objects, the material information being generated by a first machine learning model based on the feature description information, the material information including main material information, and the material information further including at least one of foreground material information, background material information, or special effects material information.

[0022] In some embodiments, the object generation unit uses a second machine learning model to generate a style sample image that matches the display style of the display interface; and generates a virtual object based on the style sample image.

[0023] In some embodiments, the object generation unit generates first prompt information based on description information using a first machine learning model, the first prompt information including style prompt words; and generates style sample images based on the first prompt information using a second machine learning model.

[0024] In some embodiments, the first prompt information includes material prompts, and the object generation unit generates materials including virtual objects based on style sample images and material prompts using a third machine learning model.

[0025] In some embodiments, style cue words include background image cue words, and the object generation unit generates a background image as a style sample image based on the background image cue words using a second machine learning model.

[0026] In some embodiments, the object generation unit generates static images of materials included in the virtual object using a third machine learning model based on style sample images; and generates dynamic images of the materials using the third machine learning model based on the static images.

[0027] In some embodiments, the material includes the main material of a virtual object. The object generation unit generates a two-dimensional static model of the main material using a third machine learning model based on a style sample image. Based on the two-dimensional static model, skeletal animation is added using the third machine learning model to generate a three-dimensional dynamic model of the main material.

[0028] In some embodiments, the object generation unit selects a second machine learning model corresponding to the display style information based on the display style information in the description information, selects a processing node for generating materials corresponding to the material information based on the material information in the description information, generates a style sample image using the second machine learning model, and generates a virtual object using the processing node based on the style sample image.

[0029] In some embodiments, the display interface includes an identifier for the broadcaster and an identifier for the consumer participating in the interaction, and the description information includes description information of the virtual object moving from the consumer identifier to the broadcaster identifier. The object generation unit generates a dynamic image of the virtual object moving from the consumer identifier to the broadcaster identifier.

[0030] In some embodiments, the object generation unit generates a dynamic image that displays a specified action after the virtual object moves to the anchor's identifier.

[0031] In some embodiments, the relevant information includes layout information of the display interface and / or type information of the display interface. The information generation unit generates description information of the virtual object using a first machine learning model based on the second prompt information. The second prompt information includes at least one of atmosphere information, scene information, or display style of the virtual object.

[0032] According to further embodiments of this disclosure, a virtual object generation apparatus is provided, comprising: a memory; and a processor coupled to the memory, the processor being configured to execute the virtual object generation method of any of the above embodiments based on instructions stored in the memory device.

[0033] According to further embodiments of the present disclosure, a computer-readable storage medium is provided having computer instructions stored thereon that, when executed by a processor, implement the virtual object generation method of any of the above embodiments.

[0034] According to further embodiments of this disclosure, a computer program product is also provided, including instructions that, when executed by a processor, cause the processor to perform a virtual object generation method according to any of the foregoing embodiments.

[0035] In the above embodiments, based on the display style of the virtual object's interface, machine learning is used to automatically generate descriptive information as the basis for virtual object generation, thereby generating virtual objects with a unified style with the display interface. This reduces the generation cost of virtual objects, thereby improving their generation efficiency.

[0036] Other features, aspects, and advantages of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0037] Embodiments of this disclosure are described below with reference to the accompanying drawings. It should be understood that the drawings described below are merely illustrative of some embodiments of this disclosure and are not intended to limit the scope of this disclosure. In the drawings:

[0038] Figure 1 Flowcharts illustrating some embodiments of the virtual object generation method of this disclosure;

[0039] Figure 2a , 2b Schematic diagrams illustrating some embodiments of the description information generation method of this disclosure;

[0040] Figure 3a , 3b Schematic diagrams illustrating some embodiments of the dynamic image generation method of this disclosure;

[0041] Figure 4a , 4b Schematic diagrams illustrating some embodiments of the 3D dynamic model generation method of this disclosure;

[0042] Figure 5a , 5b Schematic diagrams illustrating some embodiments of the virtual object generation method of this disclosure;

[0043] Figure 6a , 6b Schematic diagrams illustrating other embodiments of the virtual object generation method of this disclosure;

[0044] Figure 7 Block diagrams illustrating some embodiments of the virtual object generation apparatus of this disclosure;

[0045] Figure 8 Block diagrams illustrating other embodiments of the virtual object generation apparatus of this disclosure;

[0046] Figure 9 Block diagrams illustrating further embodiments of the virtual object generation apparatus of this disclosure are shown.

[0047] It should be understood that, for ease of description, the dimensions of the various parts shown in the accompanying drawings are not necessarily drawn to actual scale. The same or similar reference numerals are used in the various drawings to denote the same or similar parts. Therefore, once an item is defined in one drawing, it may not be discussed further in subsequent drawings. Detailed Implementation

[0048] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. It should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein.

[0049] It should be understood that the various steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect. Unless otherwise specifically stated, the relative arrangement, numerical expressions, and values ​​of components and steps set forth in these embodiments should be interpreted as merely exemplary and do not limit the scope of this disclosure.

[0050] As used in this disclosure, the term "comprising" and its variations are open-ended terms that include at least the following elements / features but do not exclude other elements / features, i.e., "including but not limited to". The term "based on" means "at least partially based on".

[0051] It should be noted that the concepts of "first," "second," etc., used in this disclosure are used only to distinguish different devices, modules, or units, and are not intended to define the order of functions performed by these devices, modules, or units or their interdependencies. Unless otherwise specified, the concepts of "first," "second," etc., are not intended to imply that the objects described herein must be in a given temporal, spatial, rank, or any other given order.

[0052] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0053] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0054] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, use and processing of the relevant data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0055] The embodiments of this disclosure are described in detail below with reference to the accompanying drawings; however, this disclosure is not limited to these specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. Furthermore, in one or more embodiments, specific features, structures, or characteristics can be combined in any suitable manner that will be apparent to those skilled in the art from this disclosure.

[0056] It should be understood that this disclosure does not limit how the image to be applied / processed is obtained. In some embodiments of this disclosure, it can be obtained from a storage device, such as internal memory or external storage device. In other embodiments of this disclosure, a camera component can be invoked to capture an image. It should be noted that the acquired image can be a captured image or a frame from a captured video, and is not particularly limited to these.

[0057] In the context of this disclosure, "image" can refer to any of a variety of images, such as color images, grayscale images, etc. It should be noted that the type of image is not specifically limited in the context of this specification. Furthermore, an image can be any suitable image, such as a raw image obtained by a camera device, or an image from which specific processing has been performed, such as preliminary filtering, dealiasing, color adjustment, contrast adjustment, normalization, etc. It should be noted that preprocessing operations may also include other types of preprocessing operations known in the art, which will not be described in detail here.

[0058] As mentioned earlier, in real-time interactive scenarios, viewers often need to gift virtual items to the streamer. In scenarios with multiple streamers, such as multi-person chat rooms, viewers can choose at least one streamer as the recipient. For example, it's necessary to generate various materials for the virtual object, such as its background image, its trajectory when moving to the recipient in the display interface, and special effects upon reaching the recipient. These materials need to match the style of the display interface.

[0059] However, if all these materials require manual design and copywriting, and all of them need to be matched with the display interface, even if machine learning is used to generate these materials based on the design copywriting, a lot of human and time resources are needed for design, resulting in high development and debugging costs and low generation efficiency.

[0060] To address the aforementioned technical problems, the technical solution disclosed herein automatically generates virtual object designs based on an understanding of the characteristics of the current display interface, and then generates various materials for the virtual objects based on the automatically generated designs. For example, the design of virtual objects can be further generated automatically by combining an understanding of the characteristics of virtual objects and their corresponding special effects with auxiliary prompts.

[0061] In this way, by using machine learning technology to replace manual design, the cost and time of generating virtual objects are reduced, and the generation efficiency is improved.

[0062] For example, the technical solution of this disclosure can be implemented through the following embodiments.

[0063] Figure 1 Flowcharts illustrating some embodiments of the virtual object generation method of this disclosure are shown.

[0064] like Figure 1 As shown, in step 110, based on the relevant information of the display interface, a first machine learning model is used to generate descriptive information of the virtual object to be displayed on the display interface. This relevant information includes the display style information of the display interface. For example, the first machine learning model can be a machine learning model with text generation capabilities.

[0065] For example, the display interface of a virtual object can be the interactive interface of a real-time interactive scenario involving multiple participants (such as a multi-person chat room, a live broadcast room, or other multi-person live streaming scenarios); the relevant information of the display interface can be the understanding information of the display interface by the first machine learning model, such as display style information, layout information, type information (such as radio, video chat room, audio chat room, etc.).

[0066] For example, the descriptive information of virtual objects may include display style, design concept, appearance description, special effects, and movement method expressed in natural language.

[0067] In step 120, a virtual object matching the display style of the display interface is generated based on the description information. For example, the display style may include style types such as cartoon style, realistic style, ancient style, and modern style.

[0068] In the above embodiments, based on the display style of the virtual object's interface, machine learning is used to automatically generate descriptive information as the basis for virtual object generation, thereby generating virtual objects with a unified style with the display interface. This reduces the generation cost of virtual objects, thereby improving their generation efficiency.

[0069] The following examples illustrate the technical solution for generating descriptive information using a first machine learning model.

[0070] In some embodiments, the descriptive information includes feature description information of the virtual object, which includes display style information of the virtual object described using natural language; for example, the feature description information may also include special effects information and / or movement method information of the virtual object. For example, the feature description information may also include appearance information of the virtual object, used to describe the shape, color, and other attributes of the virtual object, such as heart shape, colorful clouds, etc.

[0071] In some embodiments, the relevant information includes layout information and / or type information of the display interface, and description information of the virtual object is generated using a first machine learning model based on the second prompt information. The second prompt information may include at least one of atmosphere information, scene information, or display style of the virtual object. For example, scene information may include festival information, venue information, etc.

[0072] For example, a first machine learning model can be used to deeply understand the current real-time interactive scenario to determine relevant information of the corresponding display interface, such as display style information, layout information, type information, etc.; combined with auxiliary prompts in the second prompt information (such as atmosphere information, current activity information, current scene information, and information of the objects involved), at least one content description text of a virtual object that matches the current real-time interactive scenario can be generated as description information.

[0073] In some embodiments, the display interface includes an identifier for the broadcaster and an identifier for the consumer participating in the interaction, and the description information includes description information of the virtual object moving from the consumer identifier to the broadcaster identifier; a dynamic image is generated showing the virtual object moving from the consumer identifier to the broadcaster identifier. For example, a dynamic image is generated showing a specified action after the virtual object moves to the broadcaster identifier.

[0074] For example, the current real-time interactive scenario is a multi-person chat room, with a mysterious and dreamlike mythological style. The auxiliary prompts can be related to a specific character from a mythological context. Based on this style and the auxiliary prompts, the first machine learning model generates the following description: Creative Name: A Fantastic Journey Through a Mythological World. Creative Content: The background transforms into a mysterious sea of ​​clouds and sky, with blue skies and white clouds faintly visible in the distance. A weapon belonging to a character surrounded by a glowing aura flies in from afar, gradually approaching and circling the target streamer's location, creating a colorful (platinum glitter) cloud trail. The trail forms a heart-shaped 3D halo. The weapon disappears after reaching the target streamer's location. Then, a flash of golden light appears, and the character leaps from the ground, brandishing their weapon. A treasure chest rises from the ground and opens, releasing pearls illuminated by brilliant light before disappearing. The background returns to the original chat room background, and the gift-giving ends. Style Summary: Mysterious, dreamlike, Eastern mythology.

[0075] In this way, based on the relevant information displayed on the interface, combined with auxiliary prompts, descriptive information that better matches the current scene can be automatically generated, thereby reducing the generation cost of virtual objects and improving the generation efficiency of virtual objects.

[0076] In some embodiments, the descriptive information includes material information for generating virtual objects. This material information is generated by a first machine learning model based on the feature description information. The material information includes main body material information, and further includes at least one of foreground material information, background material information, or special effects material information. For example, the first machine learning model can be used to summarize relevant information about the material to be generated from the feature description information.

[0077] For example, the following material information can be summarized from the description information of the above embodiments: Background material: Mysterious sea of ​​clouds; Main material flying in the interface: 3D weapon model + clouds; Special effect at the location of the target streamer: Animation of XX character wielding a weapon; Trailing effect: Cloud trail is made of a certain material.

[0078] In this way, based on the automatically generated descriptive information, the relevant information of the materials included in the virtual object is further summarized as the basis for generating the virtual object, thereby improving the generation efficiency of the virtual object.

[0079] In some embodiments, based on the description information, a first prompt is generated using a first machine learning model and used as input to the machine learning model for generating virtual objects, thereby further improving the generation efficiency of virtual objects. For example, the first prompt may include style prompts to describe the style type of the virtual object; the first prompt may also include material prompts to describe the material of the virtual object.

[0080] The following is through Figure 2a, 2b The embodiments described above exemplify the generation of virtual objects based on the description information in the embodiments described above.

[0081] Figure 2a , 2b Schematic diagrams illustrating some embodiments of the description information generation method of this disclosure are shown.

[0082] like Figure 2a As shown, the current real-time interactive scenario is a multi-person chat room scenario. The display interface includes multiple guests participating in the interaction (hosts 21-24). The identifiers of hosts 21-24 are located as follows: one on the top row and three on the bottom row. The display interface has a mysterious and dreamlike mythological style. In response to the audience 25 sending a virtual item (i.e., a virtual object generated according to the information described in the above embodiment) to host 23, the background of the display interface changes to a mysterious cloud and sky. A weapon 26a surrounded by a glow is displayed on the background, accompanied by clouds with colorful trails flying in from a distance (a designated location in the display interface, which could be the audience 25 or another location).

[0083] like Figure 2b As shown, weapon 26a disappears after reaching the location of the anchor 23; then character 25b leaps out from the ground, wields weapon 26a, and displays the special effects described in the above embodiment.

[0084] Based on the above description information, the following examples illustrate the technical solution for generating virtual objects according to the description information.

[0085] In some embodiments, a second machine learning model is used to generate a style sample image that matches the display style of the display interface; and a virtual object is generated based on the style sample image. For example, the style sample image is generated using the second machine learning model based on the style prompt words included in the first prompt information; and the materials included in the virtual object are generated using a third machine learning model based on the style sample image and the material prompt words included in the first prompt information.

[0086] For example, style cues include background image cues. Based on these cues, a second machine learning model generates a background image as a style sample image. If the virtual object to be generated does not include background material, a style sample image can also be generated directly.

[0087] For example, after generating descriptive information, style prompts can be generated based on the display style information included in the descriptive information. Then, a text-based image model can be invoked to generate style sample images based on these style prompts. In this way, the style sample images serve as the basis for the machine learning model to generate materials, ensuring that the style of all materials for the virtual object matches the style of the display interface, thereby improving the generation effect of the virtual object.

[0088] In some embodiments, a second machine learning model corresponding to the display style information is selected based on the display style information in the description information; a processing node for generating materials corresponding to the material information is selected based on the material information in the description information; a style sample image is generated using the second machine learning model; and a virtual object is generated using the processing node based on the style sample image.

[0089] For example, a matching second machine learning model can be selected from the model pool based on style cue words; processing nodes can be selected based on the types of materials included in the virtual object, such as plugins with dynamic image generation capabilities; the above selection process can be completed by machine learning or by manual screening after the selection is completed.

[0090] This allows for the selection of machine learning models and processing nodes that are suitable for the current style and the current material, thereby improving the generation effect of virtual objects.

[0091] Based on the style sample images mentioned above, the following examples illustrate the technical solution for generating virtual objects from style sample images.

[0092] In some embodiments, a static image of the material included in the virtual object is generated using a third machine learning model based on style sample images; and an animated image of the material is generated using the third machine learning model based on the static image. For example, the animated image can be a video, a GIF (Graphics Interchange Format) image, etc.

[0093] For example, static images of various materials, including main materials, foreground materials, and special effects materials, can be generated using static image generation nodes in a third machine learning model (such as style transfer plugins) based on material prompts and style sample images; and static images of background materials can be generated when background materials are available.

[0094] For example, based on static images, dynamic images of various materials can be generated using dynamic image generation nodes in a third machine learning model. For instance, plugins with dynamic image generation capabilities can be used to generate dynamic images of the main subject, foreground material, and special effects material; similarly, in the case of background material, plugins with dynamic image generation capabilities can be used to generate dynamic images of the background material.

[0095] In this way, dynamic images make the generated virtual objects more vivid, thereby improving the generation effect of virtual objects. For example, it can be achieved through... Figure 3a , 3b The embodiments described herein exemplify the generation of dynamic images.

[0096] Figure 3a , 3b Schematic diagrams illustrating some embodiments of the dynamic image generation method of this disclosure are shown.

[0097] like Figure 3a As shown, depending on the type of material, the processing nodes can be selected, including: variational autoencoder plugin, sparse control plugin, animation generation plugin, rank adaptive plugin, checkpoint plugin, control net plugin, sampling plugin, and video joint plugin. For example, the variational autoencoder plugin can be used for decoding, the sparse control model can be used for image preprocessing, the animation generation plugin and rank adaptive plugin can be used for processing based on positive and negative prompt words, and the sampling plugin can be used for sampling processing, etc., to finally obtain a dynamic image.

[0098] like Figure 3b As shown, based on the style sample image, processing nodes can be selected, including: style transfer plugin, variational autoencoder plugin, animation generation plugin, rank adaptive plugin, checkpoint plugin, sampling plugin, and video joint plugin. For example, the style transfer plugin can be used for style customization, the sampling plugin can be used for sampling based on positive and negative cue words, and the variational autoencoder plugin can be used for decoding, ultimately resulting in a dynamic image.

[0099] In some embodiments, the material includes the main material of a virtual object. Based on a style sample image, a 2D static model of the main material is generated using a third machine learning model. Based on the 2D static model, skeletal animation is added using a third machine learning model to generate a 3D dynamic model of the main material.

[0100] For example, by combining material prompts and style sample images, a text-based image model can be called to generate a 2.5D image of a 3D dynamic model (a 2D image with a visual 3D effect, which can be understood as a 2D static model); based on the 2.5D image, a text-based 3D image model can be called to generate the Mesh assets and texture assets of the 3D dynamic model, which can improve the effect of the 3D dynamic model.

[0101] In this way, the generated virtual objects are made more lifelike through 3D dynamic models, thereby improving the generation effect of virtual objects. For example, it can be achieved through... Figure 4a , 4b The embodiments described herein exemplify the generation of 3D dynamic models.

[0102] Figure 4a , 4b Schematic diagrams illustrating some embodiments of the 3D dynamic model generation method of this disclosure are shown.

[0103] like Figure 4aAs shown, without inputting material prompts and style sample images, a 2.5D image is first generated as a whole, and then converted into a 3D dynamic model. For example, by combining material prompts and style sample images, a 2.5D image of a specified style is generated. A model that supports 2.5D images can be used to generate a 3D image to improve the generation effect of the 3D dynamic model. A single 2.5D image is input, and the output is the mesh assets and texture assets of a 3D static model (which can be called a 3D material). After generating the 3D static model, skeletal animation is added to generate the 3D dynamic model.

[0104] like Figure 4b As shown, the above image-to-3D conversion process can utilize different models depending on the specific material requirements, and the generated effects can be compared to select the best model. For example, LayerDiffusion and modeling tools can be used: transparent materials are generated through LayerDiffusion; after processing, the transparent materials are input into the modeling tool for 3D generation; and the 3D-pack plugin is used for previewing and saving.

[0105] In some embodiments, such as in multi-person chat rooms where multiple streamers exist, viewers can choose at least one streamer as the recipient of a virtual item; the virtual item can move towards the target streamer and display special effects. This interactive method prioritizes generating richer effects, requiring the combination of dynamic background materials, 3D dynamic models, 3D movement paths of virtual items, and trailing animations to enrich the effects of virtual items.

[0106] The following is through Figure 5a , 5b The embodiments described herein exemplify methods for rendering virtual objects and display interfaces.

[0107] Figure 5a , 5b Schematic diagrams illustrating some embodiments of the virtual object generation method of this disclosure.

[0108] like Figure 5a As shown, the rendering layers of the above scene include: the chat room can be rendered as a whole based on a perspective projection 3D scene; a single rendering layer using graphics rendering technology can be used to draw the avatar at the anchor's location; dynamic backgrounds, 3D dynamic models and animation effects, trailing materials, and animation effects at the anchor's location can also be rendered. For example... Figure 5b As shown, the rendering process includes: resource configuration distribution, layout configuration of the display interface, and drawing of animation effects.

[0109] The following is through Figure 6a , 6b The embodiments described herein exemplify a method for generating virtual objects.

[0110] Figure 6a , 6b Schematic diagrams illustrating some other embodiments of the virtual object generation method of this disclosure are shown.

[0111] like Figure 6a As shown, a first machine learning model can be used, combining information about the live chat room scene and the understanding of different forms of virtual object gifts (such as 2D virtual objects, 3D virtual objects, foreground materials, background materials, and different combinations of multiple materials), to generate multiple sets of descriptive information and relevant prompts for generating virtual objects based on auxiliary prompts. For example, it can first understand the content format and various elements displayed in the chat room scene (such as the location and level of the anchor icon); then it can understand the different forms of virtual object gifts, such as the understanding of different combinations of multiple materials, the understanding of auxiliary prompts, material requirements, and material content; then, it can combine the chat room scene and the requirements to generate descriptive information; finally, it can generate relevant prompts for generating virtual objects.

[0112] In this way, the first machine learning model, by understanding the virtual object gifting process in the chat room scenario, determines the matching of foreground and background materials, the association between moving main material and foreground and background materials, the moving main material, the movement path of the moving main material, and the animation association after reaching the target, thereby improving the generation effect of virtual objects.

[0113] like Figure 6b As shown, firstly, descriptive information for virtual objects in a multi-person chat room scenario is generated. Based on auxiliary prompts, a first machine learning model is used to generate feature description information, thereby summarizing material information. Based on the feature description information and material information, the first machine learning model is used to generate first prompt information, including style prompts and material prompts. For example, style prompts may include a first style prompt, used to generate a style sample image when the virtual object has no background material; style prompts may include a second style prompt, used to generate a background image as a style sample image when the virtual object has background material.

[0114] Then, a processing node and a second machine learning model are selected based on the description information. The second machine learning model corresponding to the display style information is selected based on the display style information in the description information; the processing node used to generate the corresponding material is selected based on the material information in the description information; after selection, manual screening is performed.

[0115] Then, a second machine learning model is used to generate style sample images. When the virtual object has no background material, the second machine learning model (text-based image model) is used to generate style sample images based on the first style cue words; when the virtual object has background material, the second machine learning model is used to generate background images as style sample images based on the second style cue words.

[0116] Finally, based on style sample images and material prompts, a third machine learning model is used to generate virtual objects. Static images of each material are generated using style transfer plugins, control nets, etc. For example, based on the static images, a variational autoencoder plugin is used to generate static images of background material, and then another variational autoencoder plugin is used to generate background video; based on the static images, an animation generation plugin is used to generate foreground video, animated GIFs of the anchor's position, and animated GIFs of moving main material, etc.; plugins such as RIFI VFI and Upscale are used to perform frame interpolation and high-definition restoration on the above videos and GIFs to obtain dynamic images of the virtual objects.

[0117] For example, if the moving subject is 2D, an animation generation plugin can be used to generate a 2D animated image of the moving subject; if the moving subject is 3D, a 2D static model is first generated, then the ComfyUI-3D-Pack plugin is used to convert it into a 3D static model, and finally a 3D dynamic model is generated by adding skeletal animation.

[0118] Figure 7 Block diagrams illustrating some embodiments of the virtual object generation apparatus of this disclosure are shown.

[0119] like Figure 7 As shown, the virtual object generation device 7 includes: an information generation unit 71, used to generate description information of a virtual object to be displayed on the display interface using a first machine learning model based on relevant information of the display interface, the relevant information including display style information of the display interface; and an object generation unit 72, used to generate a virtual object that matches the display style of the display interface based on the description information.

[0120] In some embodiments, the descriptive information includes feature description information of the virtual object, which includes display style information of the virtual object described using natural language, and also includes special effects information and / or movement method information of the virtual object.

[0121] In some embodiments, the description information includes material information for generating virtual objects, the material information being generated by a first machine learning model based on the feature description information, the material information including main material information, and the material information further including at least one of foreground material information, background material information, or special effects material information.

[0122] In some embodiments, the object generation unit 72 uses a second machine learning model to generate a style sample image that matches the display style of the display interface; and generates a virtual object based on the style sample image.

[0123] In some embodiments, the object generation unit 72 generates first prompt information using a first machine learning model based on description information, the first prompt information including style prompt words; and generates style sample images using a second machine learning model based on the first prompt information.

[0124] In some embodiments, the first prompt information includes material prompts, and the object generation unit 72 generates materials included in the virtual object using a third machine learning model based on the style sample image and the material prompts.

[0125] In some embodiments, style cue words include background image cue words, and the object generation unit 72 generates a background image as a style sample image based on the background image cue words using a second machine learning model.

[0126] In some embodiments, the object generation unit 72 generates static images of materials included in the virtual object using a third machine learning model based on style sample images; and generates dynamic images of the materials using the third machine learning model based on the static images.

[0127] In some embodiments, the material includes the main material of a virtual object. The object generation unit 72 generates a two-dimensional static model of the main material using a third machine learning model based on a style sample image. Based on the two-dimensional static model, it adds skeletal animation using the third machine learning model to generate a three-dimensional dynamic model of the main material.

[0128] In some embodiments, the object generation unit 72 selects a second machine learning model corresponding to the display style information according to the display style information in the description information, selects a processing node for generating materials corresponding to the material information according to the material information in the description information, generates a style sample image using the second machine learning model, and generates a virtual object using the processing node based on the style sample image.

[0129] In some embodiments, the display interface includes the identifiers of the broadcaster and the consumer, and the description information includes description information of the virtual object moving from the consumer identifier to the broadcaster identifier. The object generation unit 72 generates a dynamic image of the virtual object moving from the consumer identifier to the broadcaster identifier.

[0130] In some embodiments, the object generation unit 72 generates a dynamic image of a virtual object moving to the anchor's identifier and displaying a specified action.

[0131] In some embodiments, the relevant information includes layout information of the display interface and / or type information of the display interface. The information generation unit 71 generates description information of the virtual object using a first machine learning model based on the second prompt information. The second prompt information includes at least one of atmosphere information, scene information, or display style of the virtual object.

[0132] Figure 8 Block diagrams illustrating other embodiments of the virtual object generation apparatus of this disclosure are shown.

[0133] Memory 81 is used to store one or more computer-readable instructions. Memory 81 may include any combination of various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory, including but not limited to random access memory (RAM), dynamic random access memory (DRAM), static random access memory (SRAM), read-only memory (ROM), and flash memory. Memory 81 may, for example, store operating systems, application programs, bootloaders, databases, and other programs, as well as various application programs and various data.

[0134] The processor 82 is configured to execute computer-readable instructions to implement the virtual object generation method described in any of the foregoing embodiments. Specific implementations of each step of the virtual object generation method can be found in the above embodiments; repeated details will not be elaborated upon here.

[0135] The processor 82 can be configured to execute the steps of the virtual object generation method in any of the above embodiments. The processor 82 can be various processing devices, such as a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. The central processing unit (CPU) can be an x86 or ARM architecture, etc.

[0136] The processor 82 and the memory 81 can communicate with each other directly or indirectly. For example, the processor 82 and the memory 81 can communicate via a network. The network can include wireless networks, wired networks, and / or any combination of wireless and wired networks. The processor 82 and the memory 81 can also communicate with each other via a system bus, which is not limited in this disclosure.

[0137] It should be noted that Figure 8 The components of the virtual object generation apparatus 81 shown are merely exemplary and not limiting. The virtual object generation apparatus 8 may have other components depending on the specific application requirements. The processor 82 can control other components in the virtual object generation apparatus 8 to perform desired functions.

[0138] The virtual object generation device 8 can be implemented by software, firmware and / or hardware, and can be integrated into a device with the relevant application installed.

[0139] Figure 9 Block diagrams illustrating further embodiments of the virtual object generation apparatus of this disclosure are shown.

[0140] Figure 9 The virtual object generation device 9 shown can be a computer system with a dedicated hardware structure, which can perform corresponding functions when relevant applications are installed.

[0141] Virtual object generation device 9 includes, but is not limited to, mobile terminals such as smartphones, laptops, personal digital assistants (PDAs), tablet PCs, portable multimedia players (PMPs), in-vehicle terminals (e.g., in-vehicle navigation terminals), wearable devices, and fixed terminals such as digital televisions and desktop computers.

[0142] like Figure 9 As shown, the Central Processing Unit (CPU) 91 performs various processes based on a program stored in the Read-Only Memory (ROM) 92 or a program loaded from the storage section 98 into the Random Access Memory (RAM) 93. The RAM 93 stores data required as needed when the CPU 91 performs various processes, etc. The CPU is merely exemplary; it could also be other types of processors, such as the various processors described above. The ROM 92, RAM 93, and storage section 98 can be various forms of computer-readable storage media. It should be noted that although... Figure 9 The diagram shows ROM 92, RAM 93 and storage section 98, but one or more of them may be combined or located in the same or different memory or storage modules.

[0143] CPU 91, ROM 92 and RAM 93 are interconnected via bus 64. Input / output interface 95 is also connected to bus 94.

[0144] The following components are connected to the input / output interface 95: input section 96, such as a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output section 97, including displays such as cathode ray tube (CRT), liquid crystal display (LCD), speakers, vibrators, etc.; storage section 98, including hard disk, magnetic tape, etc.; and communication section 99, including network interface cards such as LAN cards, modems, etc. The communication section 99 allows communication processing via a network such as the Internet. It is easy to understand that, although... Figure 9 The virtual object generation apparatus 9 shown communicates via bus 94, but it may also communicate via a network or other means, wherein the network may include a wireless network, a wired network, and / or any combination of wireless and wired networks.

[0145] As needed, drive 910 is also connected to input / output interface 95. Removable media 911, such as disks, optical disks, magneto-optical disks, semiconductor memories, etc., are installed on drive 910 as needed, so that computer programs read from them can be installed into storage section 98 as needed.

[0146] When the above series of processes are implemented through software, the program constituting the software can be installed from a network such as the Internet or a storage medium such as a removable medium 911.

[0147] According to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product that, when run on a computer, causes the computer to implement the virtual object generation method described in any of the foregoing embodiments. The computer program product includes computer instructions carried on a computer-readable medium, containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer instructions can be downloaded and installed from a network via communication section 69, or installed from storage section 68, or installed from ROM 62. When the computer program is executed by CPU 61, the virtual object generation method of any embodiment of this disclosure is performed.

[0148] It should be noted that, in the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store programs for use by or in conjunction with an instruction execution system, apparatus, or device.

[0149] A computer-readable medium may be a computer-readable storage medium, a computer-readable signal medium, or any combination thereof.

[0150] Computer-readable storage media include, but are not limited to, systems, apparatuses, or devices that are electrical, magnetic, optical, electromagnetic, infrared, or semiconductor, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to, electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. The computer-readable storage medium stores computer instructions that, when executed by a processor, implement the virtual object generation method described in any of the foregoing embodiments.

[0151] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media may also be any computer-readable medium other than computer-readable storage media, capable of sending, propagating, or transmitting programs for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0152] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0153] In some embodiments, a computer program is also provided, comprising: instructions that, when executed by a processor, cause the processor to perform the virtual object generation method described in any of the foregoing embodiments. For example, the instructions may be embodied in computer program code.

[0154] In embodiments of this disclosure, computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof. These programming languages ​​include, but are not limited to, object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network (including a local area network (LAN) or a wide area network (WAN)), or it can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0155] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0156] The functions described above can be performed, at least in part, by one or more hardware logic components. For example, without limitation, exemplary hardware logic components that can be used include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0157] While specific embodiments of this disclosure have been described in detail by way of example, those skilled in the art should understand that the examples are for illustrative purposes only and not intended to limit the scope of this disclosure. Those skilled in the art should understand that modifications can be made to the above embodiments without departing from the scope and spirit of this disclosure. The scope of this disclosure is defined by the appended claims.

Claims

1. A method for generating virtual objects, comprising: Based on the relevant information of the display interface, a first machine learning model is used to generate descriptive information of the virtual object to be displayed on the display interface, the relevant information including the display style information of the display interface; Based on the description information, a virtual object matching the display style of the display interface is generated.

2. The virtual object generation method according to claim 1, wherein, The description information includes feature description information of the virtual object, which includes display style information of the virtual object described using natural language, and also includes special effects information and / or movement method information of the virtual object.

3. The virtual object generation method according to claim 2, wherein, The description information includes material information used to generate the virtual object. The material information is generated by the first machine learning model based on the feature description information. The material information includes main material information and also includes at least one of foreground material information, background material information, or special effects material information.

4. The virtual object generation method according to claim 1, wherein, The step of generating the virtual object that matches the display style of the display interface based on the description information includes: Using a second machine learning model, style sample images that match the display style of the display interface are generated; The virtual object is generated based on the style sample image.

5. The virtual object generation method according to claim 4, wherein, The step of generating style sample images that match the display style of the display interface using a second machine learning model includes: Based on the description information, the first machine learning model is used to generate first prompt information, which includes style prompt words; Based on the first prompt information, the style sample image is generated using the second machine learning model.

6. The virtual object generation method according to claim 5, wherein, The first prompt information includes material prompt words, Generating the virtual object based on the style sample image includes: Based on the style sample images and the material prompts, the materials included in the virtual object are generated using a third machine learning model.

7. The virtual object generation method according to claim 5, wherein, The style cue words include background image cue words. The step of generating the style sample image using the second machine learning model based on the first prompt information includes: Based on the background image prompts, the second machine learning model generates a background image as the style sample image.

8. The virtual object generation method according to claim 4, wherein, Generating the virtual object based on the style sample image includes: Based on the style sample images, a third machine learning model is used to generate static images of the materials included in the virtual object; Based on the static image, the third machine learning model is used to generate a dynamic image of the material.

9. The virtual object generation method according to claim 8, wherein, The materials include the main materials of the virtual object. Generating the virtual object based on the style sample image includes: Based on the style sample image, a two-dimensional static model of the main material is generated using the third machine learning model; Based on the two-dimensional static model, skeletal animation is added using the third machine learning model to generate a three-dimensional dynamic model of the main material.

10. The virtual object generation method according to claim 4, wherein, The step of generating the virtual object that matches the display style of the display interface based on the description information includes: Based on the display style information in the description information, select the second machine learning model corresponding to the display style information; Based on the material information in the description information, select a processing node to generate the material corresponding to the material information; The style sample image is generated using the second machine learning model; The virtual object is generated using the processing node based on the style sample image.

11. The virtual object generation method according to any one of claims 1-10, wherein, The display interface includes identifiers for the broadcaster and the consumer, and the description information includes information about the virtual object moving from the consumer identifier to the broadcaster identifier. The step of generating the virtual object that matches the display style of the display interface based on the description information includes: Generate a dynamic image of the virtual object moving from the consumer identifier to the broadcaster identifier.

12. The virtual object generation method according to claim 11, wherein, The generation of the dynamic image of the virtual object moving from the consumer identifier to the broadcaster identifier includes: A dynamic image is generated that displays a specified action after the virtual object moves to the anchor's identifier.

13. The virtual object generation method according to any one of claims 1-10, wherein, The relevant information includes the layout information of the display interface and / or the type information of the display interface. The step of generating descriptive information of the virtual object using a first machine learning model based on relevant information from the virtual object's display interface includes: Based on the second prompt information, the first machine learning model is used to generate description information of the virtual object. The second prompt information includes at least one of atmosphere information, scene information, or the display style of the virtual object.

14. A virtual object generation apparatus, comprising: An information generation unit is used to generate description information of a virtual object to be displayed on the display interface using a first machine learning model based on relevant information of the display interface. The relevant information includes the display style information of the display interface. The object generation unit is used to generate the virtual object that matches the display style of the display interface based on the description information.

15. A virtual object generation apparatus, comprising: Memory; and A processor coupled to the memory, the processor being configured to execute the virtual object generation method of any one of claims 1-13 based on instructions stored in the memory.

16. A computer-readable storage medium having stored thereon computer instructions that, when executed by a processor, implement the virtual object generation method according to any one of claims 1-13.

17. A computer program product comprising instructions that, when executed by a processor, cause the processor to perform the virtual object generation method according to any one of claims 1-13.

Images