Virtual image generation method and device based on large model, intelligent agent, electronic equipment and storage medium

Abstract

The invention provides a virtual image generation method and device based on a large model, an intelligent agent, electronic equipment and a storage medium, relates to the technical field of artificial intelligence, in particular to the technical fields of computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC, digital humans, intelligent e-commerce and the like. The method comprises the steps that a large model is used for processing a target image comprising a target object, object description information is obtained, and the target object has texture information; the target image and a to-be-processed image representing the object form of the three-dimensional object are processed through a texture generation type large model, a target three-dimensional object with target texture information is obtained, the three-dimensional object is determined based on object description information, and the target texture information is matched with the texture information; and generating a virtual image based on the target three-dimensional object.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to technical fields such as computer vision, deep learning, and large models, and can be applied to scenarios such as AIGC (Artificial Intelligence Generative Content), digital humans, and smart e-commerce. Background Art

[0002] With the development of Internet e-commerce, animation games, video production and other fields, it is possible to interact with users by designing virtual images. For example, in the field of Internet e-commerce, three-dimensional virtual images can be used to introduce product functions to enhance the display effect of products. Summary of the invention

[0003] The present invention provides a method, device, intelligent body, electronic device, storage medium and program product for generating a virtual image based on a large model.

[0004] According to one aspect of the present disclosure, a method for generating a virtual image based on a large model is provided, comprising: using the large model to process a target image including a target object to obtain object description information, wherein the target object has texture information; using the texture generation large model to process the target image and an image to be processed that represents the object morphology of a three-dimensional object to obtain a target three-dimensional object having target texture information, wherein the three-dimensional object is determined based on the object description information, and the target texture information matches the texture information; and generating a virtual image based on the target three-dimensional object.

[0005] According to another aspect of the present disclosure, an artificial intelligence agent is provided, which is configured to execute the method provided according to the embodiment of the present disclosure.

[0006] According to another aspect of the present disclosure, a virtual image generation device based on a large model is provided, including: an object description information acquisition module, used to use the large model to process a target image including a target object to obtain object description information, and the target object has texture information; a target three-dimensional object acquisition module, used to use the texture generation large model to process the target image and the to-be-processed image representing the object morphology of the three-dimensional object to obtain a target three-dimensional object with target texture information, the three-dimensional object is determined based on the object description information, and the target texture information matches the texture information; and a virtual image generation module, used to generate a virtual image based on the target three-dimensional object.

[0007] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to an embodiment of the present disclosure.

[0008] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method provided according to an embodiment of the present disclosure.

[0009] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, the method provided according to the embodiment of the present disclosure is implemented.

[0010] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0012] Figure 1 An exemplary system architecture to which a method and device for generating a virtual image based on a macro model can be applied according to an embodiment of the present disclosure is schematically shown;

[0013] Figure 2 The flowchart of the method for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 3 The principle diagram of the texture generation type large model according to the embodiment of the present disclosure is schematically shown;

[0015] Figure 4 The schematic diagram of a texture generation type large model according to another embodiment of the present disclosure is shown;

[0016] Figure 5 The principle diagram of a texture generation type large model according to another embodiment of the present disclosure is schematically shown;

[0017] Figure 6 The schematic diagram shows a principle diagram of a texture generation type large model according to another embodiment of the present disclosure;

[0018] Figure 7 The application scenario diagram of the method for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 8 A block diagram of a virtual image generation device based on a large model according to an embodiment of the present disclosure is schematically shown;

[0020] Fig. 9 A block diagram schematically shows a structure of an artificial intelligence agent according to an embodiment of the present disclosure; and

[0021] Fig.10 A block diagram of an electronic device suitable for implementing a method for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0022] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0023] In the technical solution disclosed in the present invention, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations, necessary confidentiality measures are taken, and do not violate public order and good morals.

[0024] The inventors have found that in the fields of Internet e-commerce, film and television animation, products and video plots are displayed by driving three-dimensional virtual images to perform specified action tasks. In addition, users can also create three-dimensional virtual images based on personal needs. However, it usually takes a lot of time to create a virtual image that matches the user's needs, and the matching degree between the generated virtual image and the user's actual needs is low, thereby reducing the display effect of the virtual image.

[0025] The embodiments of the present disclosure provide a method, device, intelligent agent, electronic device, storage medium and program product for generating a virtual image based on a large model. The method for generating a virtual image based on a large model includes: using a large model to process a target image including a target object to obtain object description information, wherein the target object has texture information; using a texture generation-type large model to process the target image and an image to be processed that represents the object morphology of a three-dimensional object to obtain a target three-dimensional object with target texture information, wherein the three-dimensional object is determined based on the object description information, and the target texture information matches the texture information; and generating a virtual image based on the target three-dimensional object.

[0026] According to the embodiments of the present disclosure, by processing the target image with a large model, the object attribute information such as the outline, morphology, style, etc. of the target object in the target image can be learned based on the relatively powerful visual understanding ability of the large model, so that the output object description information can more accurately represent the object attributes of the target object. The three-dimensional object determined based on the object description information can match the object attribute information represented by the object description information, so the image to be processed that represents the object morphology of the three-dimensional object can more accurately represent the object morphology of the target object. By using the texture generation type large model to process the target image and the image to be processed, the texture information of the target object in the target image and the object morphology represented by the image to be processed can be more accurately fused, so that the generated target three-dimensional object can more accurately represent the matching relationship between the object morphology and texture information of the target object in three-dimensional space, so that the virtual image generated according to the target three-dimensional object can accurately represent the morphology and texture of each target object in the target image in three-dimensional space, improve the matching degree between the virtual image and the target image, and then realize the automatic and accurate generation of a three-dimensional virtual image that matches the user's needs.

[0027] Figure 1 An exemplary system architecture to which a method and apparatus for generating a virtual image based on a large model can be applied according to an embodiment of the present disclosure is schematically shown.

[0028] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, an exemplary system architecture to which the method and apparatus for generating a virtual image based on a large model can be applied may include a terminal device, but the terminal device may implement the method and apparatus for generating a virtual image based on a large model provided by the embodiments of the present disclosure without interacting with the server.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0031] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0032] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0033] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0034] It should be noted that the virtual image generation method based on the large model provided in the embodiment of the present disclosure can generally be executed by the server 105. Accordingly, the virtual image generation device based on the large model provided in the embodiment of the present disclosure can generally be set in the server 105. The virtual image generation method based on the large model provided in the embodiment of the present disclosure can also be executed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the virtual image generation device based on the large model provided in the embodiment of the present disclosure can also be set in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0035] For example, any one of the terminal devices 101, 102, and 103 can obtain a target image input by a user, and then send the obtained target image to the server 105, which processes the target image using a large model to obtain object description information; uses a texture generation large model to process the target image and the image to be processed that represents the object morphology of the three-dimensional object to obtain a target three-dimensional object with target texture information; and generates a virtual image based on the target three-dimensional object. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 processes the target image and the image to be processed, and finally generates a virtual image.

[0036] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0037] Figure 2 The flowchart of the method for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown.

[0038] like Figure 2 As shown, the method for generating a virtual image based on a large model includes operations S210 to S230.

[0039] In operation S210, a target image including a target object is processed using a large model to obtain object description information, wherein the target object has texture information.

[0040] In operation S220, the target image and the to-be-processed image representing the object morphology of the three-dimensional object are processed using the texture generation type large model to obtain a target three-dimensional object having target texture information.

[0041] In operation S230, a virtual image is generated based on the target three-dimensional object.

[0042] According to an embodiment of the present disclosure, the target object in the target image may be any type of object such as a person, clothing, accessories, tools, buildings, vehicles, etc. in the target image. The texture information of the target object may be characterized based on the pixels of the image area corresponding to the target object in the target image. For example, the texture information may be characterized based on the RGB (RedGreenBlue) information of the pixels in the target image. It should be noted that the target object may represent a real object such as a person or an animal, or the target object may also represent a virtual object such as an animated character.

[0043] According to an embodiment of the present disclosure, a large model may refer to a deep learning model with large-scale model parameters, and a large model generally contains hundreds of millions, tens of billions, hundreds of billions, trillions, or even more than ten trillion model parameters. The large model may include a multimodal large model with visual understanding capabilities such as a VideoChat model and a Video-LlaMA model. The large model involved in the embodiment of the present disclosure may be a general large model, or it may be an expert large model obtained after fine-tuning (Fine Tune) based on sample object description information and sample images, and the embodiment of the present disclosure does not limit this. The large model can be used to process information of any modality such as images, text, video, and audio.

[0044] According to the embodiments of the present disclosure, by processing the target image with a large model, the object attribute information such as the outline, form, style, etc. of the target object in the target image can be learned based on the relatively powerful visual understanding ability of the large model, so that the output object description information can more accurately represent the object attributes of the target object. Determining the three-dimensional object based on the object description information may include searching based on the object description information to obtain the three-dimensional object. The three-dimensional object form of the three-dimensional object may be adapted to the object attributes represented by the object description information.

[0045] It should be understood that the target object can be represented based on a two-dimensional image region in the target image. A three-dimensional object can be represented based on a three-dimensional object element (eg, a three-dimensional space point, a three-dimensional grid element) in a three-dimensional space.

[0046] According to an embodiment of the present disclosure, a three-dimensional object is determined based on object description information.

[0047] In one example, the three-dimensional object can be retrieved from a preset three-dimensional object library based on the object description information. The preset three-dimensional object in the preset three-dimensional object library can be associated with the preset description information. The three-dimensional object can be determined from the preset three-dimensional object library based on the similarity between the object description information and the preset description information.

[0048] It should be understood that, under the condition of obtaining relevant authorization, a search can also be performed in any database based on the object description information to obtain a three-dimensional object that matches the object description information. For example, a search can be performed in an open source three-dimensional model database to obtain a three-dimensional object. The embodiments of the present disclosure do not limit the specific method of determining the three-dimensional object, as long as the three-dimensional object matches the object attribute information represented by the object description information.

[0049] According to an embodiment of the present disclosure, the image to be processed may include a two-dimensional image, such as a grayscale image, a binary image, a UV map or other two-dimensional image. The pixels of the image to be processed may have a spatial position mapping relationship with the object elements of the three-dimensional object. The image to be processed may be obtained by performing two-dimensional spatial mapping on the object pixels of the three-dimensional object. Alternatively, the image to be processed may also be determined based on a two-dimensional UV map used to render the three-dimensional object.

[0050] According to an embodiment of the present disclosure, the target texture information matches the texture information. The target texture information of the target three-dimensional object may have a mapping relationship with the texture information of the target object in space.

[0051] According to the embodiments of the present disclosure, by using a large texture generative model to process the image to be processed and the target image, the semantic attribute relationship between the object morphology represented by the image to be processed and the texture information of the specified position of the target object in the target image can be learned based on the relatively powerful image semantic understanding ability of the large texture generative model, thereby realizing the texture semantic attribute migration of the object morphology represented by the image to be processed based on the texture information of the target object. In this way, the generated target three-dimensional object can more accurately represent the texture semantic attributes of the target object, and the target texture information of the target three-dimensional object can match the texture information of the target object, thereby improving the representation accuracy of the target three-dimensional object in representing the target object.

[0052] According to an embodiment of the present disclosure, generating a virtual image based on a target three-dimensional object may include fusing multiple target three-dimensional objects according to the positional relationship represented by the target image to obtain a virtual image. Alternatively, it may also include fusing the target three-dimensional object with other preset three-dimensional virtual objects to obtain a virtual image that matches the user's needs. The embodiment of the present disclosure does not limit the specific method of generating a virtual image.

[0053] According to an embodiment of the present disclosure, the target object includes at least one of the following: a clothing object, a body part object, a vehicle object, and a building object.

[0054] According to an embodiment of the present disclosure, a clothing object may represent clothing displayed in a target image, such as a suit, a long skirt, or other clothing worn by a person. The target three-dimensional object corresponding to the clothing object may be a three-dimensional clothing model that matches the texture information of the clothing object.

[0055] According to the embodiments of the present disclosure, the body part object can represent any body part such as the head, hand, arm, etc. The target three-dimensional object corresponding to the body part object can be a three-dimensional body part model representing the body part. By fusing the three-dimensional body part model with the target texture information according to the posture and position of the person in the target image, the generated virtual image can more accurately represent the posture and texture of the person in the target image, so that the virtual image can simulate the target image more accurately, thereby improving the matching degree between the virtual image and the user's needs.

[0056] According to an embodiment of the present disclosure, the vehicle object may represent any type of movable vehicle such as a car, a ship, an airplane, etc. The target three-dimensional object corresponding to the vehicle object may be a three-dimensional vehicle model having target texture information.

[0057] According to an embodiment of the present disclosure, a building object may represent any type of building such as a house, a bridge, a warehouse, etc. A target three-dimensional object corresponding to the building object may be a three-dimensional house model, a three-dimensional bridge model, etc.

[0058] According to an embodiment of the present disclosure, a texture generative large model includes a texture generation network. The texture generation network can be constructed based on a generative model algorithm. For example, it can be constructed based on any type of generative model algorithm such as a GAN (Generative Adversaria Networks) model, a VAE (Variational auto-encoder) model, a Flow-based Models, a Diffusion Models, etc. However, it is not limited to this, and a texture generation network can also be constructed based on other types of deep learning algorithms. For example, a texture generation network can be constructed based on a convolutional neural network algorithm.

[0059] According to an embodiment of the present disclosure, the image to be processed includes a position mapping image. The position pixels of the position mapping image represent the three-dimensional coordinates of the object elements in the three-dimensional object. For example, the position pixels can store the three-dimensional space coordinates (x, y, z) of the object mesh elements of the three-dimensional object.

[0060] According to an embodiment of the present disclosure, the position pixels represent the three-dimensional coordinates of the object elements in the three-dimensional object, which can be understood as the position pixels in the position mapping image can store the three-dimensional coordinates of the object elements. The position mapping image can be determined based on the three-dimensional coordinates of the object elements in the three-dimensional object.

[0061] It should be noted that a corresponding position mapping image can be constructed for a preset three-dimensional object in the preset three-dimensional object library. For example, the preset UV map used for rendering the preset three-dimensional object can be updated with map pixel information, and the preset texture information stored in the map pixels of the preset UV map can be updated to the three-dimensional coordinates of the object elements of the preset three-dimensional object, thereby obtaining a position mapping image.

[0062] According to an embodiment of the present disclosure, using a texture generation large model to process a target image and an image to be processed that represents the object morphology of a three-dimensional object may include: based on a texture generation network, performing feature fusion according to object style features and position mapping images to obtain a target texture map that matches the texture information; and updating the three-dimensional object based on the target texture information of the target texture map to obtain a target three-dimensional object.

[0063] According to an embodiment of the present disclosure, the object style feature is determined based on the target image. For example, the target image can be feature extracted based on an image feature extraction algorithm to obtain the object style feature. The image feature extraction algorithm can include any type of deep learning algorithm such as a convolutional neural network algorithm, and the embodiment of the present disclosure does not limit the specific type of the image feature extraction algorithm.

[0064] According to the embodiments of the present disclosure, the object style feature characterizes the texture semantic attributes, shape detail semantic attributes, pattern semantic attributes, etc. and image semantic attributes of the target object. Based on the texture generation network, the object style features and the position mapping image are feature fused. Based on the three-dimensional coordinates stored in the position pixels as prior information, the texture generation network can be used to use the larger-scale model parameters to learn the correspondence between the three-dimensional coordinates of the object elements and the object style features, so that the image style semantics related to the target object in the target image is mapped to the three-dimensional coordinates corresponding to the object elements in the three-dimensional space, so that the generated target texture map can store the target texture information more accurately according to the position of the object elements of the three-dimensional object in the three-dimensional space. In this way, the target texture map can more accurately position the texture information of the target object in the three-dimensional space, so that the generated target three-dimensional object can more accurately match the texture information of the target object, thereby improving the characterization accuracy of the target object.

[0065] It should be understood that the map pixels of the target texture map may have a spatial mapping relationship with the object elements of the three-dimensional object, and the target three-dimensional object with target texture information may be obtained by unfolding the target texture map in three-dimensional space.

[0066] In one example, the target texture map may be processed based on a rendering engine to obtain a target three-dimensional object having target texture information.

[0067] According to an embodiment of the present disclosure, based on a texture generation network, feature fusion based on object style features and a position mapping image may include: performing feature fusion on the position mapping image and a shape mask image in the image to be processed to obtain an initial fusion feature; performing feature fusion based on the initial fusion feature and the object style feature to obtain a target fusion feature.

[0068] According to an embodiment of the present disclosure, the mask pixels of the shape mask image represent whether the object elements of the three-dimensional object can store texture information, and there is a position mapping relationship between the mask pixels and the object elements.

[0069] According to an embodiment of the present disclosure, the shape mask image can be determined based on a UV map corresponding to a three-dimensional object. For example, the initial texture information of the map pixels in the UV map can be updated to mask information, and the mask information can characterize whether the object element can store texture information based on a preset code, value or representation.

[0070] In one example, the pixel value of the mask pixel of the shape mask image can be represented as 1 or 0. A mask pixel of 1 indicates that the object pixel corresponding to the mask pixel needs to have target texture information, and a mask pixel of 0 indicates that the object pixel corresponding to the mask pixel does not have target texture information.

[0071] According to an embodiment of the present disclosure, feature fusion of the position mapping image and the shape mask image in the image to be processed may include processing the position mapping image and the shape mask image based on an attention network algorithm to obtain initial fusion features.

[0072] In one embodiment, the position mapping image and the shape mask image can be processed based on the cross attention network algorithm to fully integrate the three-dimensional coordinates of the object pixels represented by the position pixels and the semantic attributes of whether the texture information needs to be stored represented by the mask pixels. In this way, the initial fusion feature can be used to perform feature fusion with the object style feature to control the correspondence between the pixel position of the target object and the three-dimensional spatial position of the object element of the three-dimensional object, and accurately integrate the style semantic attributes of the object style feature into the object element of the three-dimensional object to generate a target fusion feature that can more accurately represent the texture information of the target object.

[0073] According to the embodiments of the present disclosure, based on the initial fusion features obtained by fusing the mask pixels and position pixels of the shape mask image, the texture generation network can be controlled to update the texture semantic attributes of the object elements of the three-dimensional object that need to add the target texture information, and the texture generation network can be controlled to avoid updating the texture semantic attributes of the object elements in the three-dimensional object that do not need to add the target texture information, so that the target fusion features can more accurately control the target texture information of the object elements of the target three-dimensional object, avoid style feature migration errors caused by errors in the position mapping relationship between the object elements of the three-dimensional object and the pixels of the target object, and improve the representation accuracy of the texture semantic attributes of the target three-dimensional object.

[0074] According to an embodiment of the present disclosure, the target texture map may be determined based on the target fusion feature. For example, the target fusion feature may be processed based on any type of machine learning algorithm such as a fully connected layer to obtain the target texture map.

[0075] According to an embodiment of the present disclosure, the texture generative large model also includes a first style feature extraction network, which includes a downsampling layer and an upsampling layer having a U-shaped network structure. The object style feature may include: at least one level of downsampling style features and at least one level of upsampling style features obtained by processing the target image through the downsampling layer and the upsampling layer. The at least one level of downsampling style features may be output by the downsampling layer, and the at least one level of upsampling style features may be output by the upsampling layer.

[0076] According to an embodiment of the present disclosure, the downsampling layer or the upsampling layer can be constructed based on a convolutional neural network algorithm. However, it is not limited thereto, and the downsampling layer or the upsampling layer can also be constructed based on other types of neural network algorithms such as an attention network algorithm. The embodiment of the present disclosure does not limit the specific algorithm type for constructing the downsampling layer or the upsampling layer.

[0077] In one example, the downsampling layer or the upsampling layer can be constructed based on a convolutional neural network algorithm and a pooling algorithm.

[0078] According to an embodiment of the present disclosure, feature fusion is performed based on the initial fusion feature and the object style feature to obtain the target fusion feature, including: using a texture encoder of a texture generation network to perform a feature encoding operation on the initial fusion feature and at least one level of down-sampled style feature to obtain a first intermediate fusion feature; and using a texture decoder of the texture generation network to perform a feature decoding operation on the first intermediate fusion feature and at least one level of up-sampled style feature to obtain the target fusion feature.

[0079] According to an embodiment of the present disclosure, a texture generation network may include a texture encoder and a texture decoder. The texture encoder and the texture decoder may be constructed based on the network structure of an encoder and a decoder in a diffusion model.

[0080] According to an embodiment of the present disclosure, the feature encoding operation can be understood as a feature encoding process performed based on the initial fusion feature and the downsampled style feature. The feature encoding operation can perform feature fusion on the initial fusion feature and the downsampled style feature by introducing noise information, so that the first intermediate fusion feature can fully fuse the high-level style semantic attributes in the target image according to the spatial position of the three-dimensional object represented by the initial fusion feature. The feature decoding operation can be understood as performing feature fusion and feature decoding processing based on the first intermediate fusion feature and the upsampled style feature, so that the target fusion feature can fuse the style semantic attributes and restore the resolution according to the spatial position of the object element of the three-dimensional object. In this way, the target fusion feature can more accurately represent the style details of the three-dimensional object and improve the matching degree between the target texture information of the target three-dimensional object and the texture information of the target object.

[0081] In one example, preset noise information may be introduced during feature encoding and decoding operations, initial fusion features and object style features may be fused through feature encoding and decoding operations, and feature denoising may be performed to obtain target fusion features.

[0082] According to an embodiment of the present disclosure, the texture encoder may include a multi-level encoder layer, and the texture decoder may include a multi-level decoder layer. The encoder layer and the decoder layer may be used to perform feature encoding operations and feature decoding operations.

[0083] In one example, the encoder layer or the decoder layer can be constructed based on an attention network algorithm, for example, can be constructed based on a cross-attention network algorithm or a self-attention network algorithm.

[0084] In one example, performing a feature encoding operation includes: fusing at least one level of first encoding features with at least one level of down-sampled style features to obtain a next level of first encoding features, wherein the first level first encoding features may be determined by performing a feature encoding operation on the initial fused features and the first level of down-sampled style features.

[0085] In one example, performing a feature decoding operation includes: fusing at least one level of first decoding features with at least one level of up-sampled style features to obtain a next level of first decoding features. The first level first decoding features are determined by performing a feature decoding operation on the first intermediate fusion features and the first level up-sampled style features. The target fusion feature is determined based on the decoding features of the last level.

[0086] Figure 3 The principle diagram of a texture generative large model according to an embodiment of the present disclosure is schematically shown.

[0087] like Figure 3 As shown, the texture generative model 300 may include a first style feature extraction network 310 and a texture generation network, and the texture generation network includes a texture encoder 321 and a texture decoder 322. The texture generation network can be constructed based on a diffusion model algorithm. The first style feature extraction network 310 may include a downsampling layer and an upsampling layer determined based on a U-shaped network structure. The downsampling layer and the upsampling layer may include a first-level downsampling layer 3111 and a third-level upsampling layer 3112, a second-level downsampling layer 3121 and a second-level upsampling layer 3122, a third-level downsampling layer 3131 and a first-level upsampling layer 3132. The downsampling layer or the upsampling layer may be constructed based on a convolutional neural network algorithm. The first U-shaped substructure composed of the first downsampling layer 3111 and the third upsampling layer 3112, the second U-shaped substructure composed of the second downsampling layer 3121 and the second upsampling layer 3122, and the third U-shaped substructure composed of the third downsampling layer 3131 and the first upsampling layer 3132 can be used to extract style features of different scales. The texture encoder 321 includes a first encoder layer 3211, a second encoder layer 3212, and a third encoder layer 3213 constructed based on the attention network algorithm. The texture decoder 322 includes a first decoder layer 3221, a second decoder layer 3222, and a third decoder layer 3223 constructed based on the attention network algorithm.

[0088] like Figure 3 As shown, the target image 301 is input into the first-level downsampling layer 3111 to obtain the first-level downsampling style features. The first-level downsampling style features are input into the second-level downsampling layer 3121 to obtain the second-level downsampling style features. The second-level downsampling style features are input into the third-level downsampling layer 3131 to obtain the third-level downsampling style features. The third-level downsampling style features and the feature map determined based on the third-level downsampling style features are input into the first-level upsampling layer 3132 to output the first-level upsampling style features. The first-level upsampling style features and the feature map determined based on the second-level downsampling style features are input into the second-level upsampling layer 3122 to output the second-level upsampling style features. The second-level upsampling style features and the feature map determined based on the first-level downsampling style features are input into the third-level upsampling layer 3112 to output the third-level upsampling style features.

[0089] The initial fused features 302 and the first-level down-sampled style features obtained by feature fusion based on the position mapping image and the shape mask image are input into the first encoder layer 3211. The first encoder layer 3211 fuses the initial fused features 302 and the first-level down-sampled style features based on the attention network algorithm, and performs the first-level feature encoding operation based on the noise information to obtain the first-level first encoded features. The first-level first encoded features and the second-level down-sampled style features are input into the second encoder layer 3212. The second encoder layer 3212 fuses the first-level first encoded features and the second-level down-sampled style features based on the attention network algorithm, and performs the second-level feature encoding operation based on the noise information to obtain the second-level first encoded features. The second-level first encoded features and the third-level down-sampled style features are input into the third encoder layer 3213. The third encoder layer 3213 fuses the second-level first encoded features and the third-level down-sampled style features based on the attention network algorithm, and performs the third-level feature encoding operation based on the noise information to obtain the third-level first encoded features. The first encoding feature of level 3 is determined as the first intermediate fusion feature.

[0090] like Figure 3 As shown, the first encoding feature of the third level is used as the first intermediate fusion feature, and the first intermediate fusion feature and the first level up-sampled style feature are input into the first decoder layer 3221. The first decoder layer 3221 can process the first intermediate fusion feature and the first level up-sampled style feature based on the attention mechanism to perform a feature decoding operation to obtain the first level first decoding feature. The first level first decoding feature and the second level up-sampled style feature are input into the second decoder layer 3222. The second decoder layer 3222 can process the first level first decoding feature and the second level up-sampled style feature based on the attention mechanism to perform a feature decoding operation to obtain the second level first decoding feature. The second level first decoding feature and the third level up-sampled style feature are input into the third decoder layer 3223. The third decoder layer 3223 can process the second level first decoding feature and the third level up-sampled style feature based on the attention mechanism to perform a feature decoding operation to obtain the third level first decoding feature. The target fusion feature 303 can be determined based on the third level first decoding feature. Image rendering based on the target fusion feature 303 can obtain a target texture map 304 .

[0091] According to an embodiment of the present disclosure, using a texture decoder of a texture generation network to perform a feature decoding operation on a first intermediate fusion feature and at least one level of upsampled style features includes: based on an attention mechanism, using a texture decoder to perform a feature decoding operation on the first intermediate fusion feature, at least one level of upsampled style features and at least one level of position features.

[0092] According to an embodiment of the present disclosure, the position feature is determined based on the position mapping image. The position feature can be obtained by extracting features from the position mapping image. For example, the feature extraction can be performed on the position mapping image based on any type of algorithm such as a convolutional neural network. Alternatively, the position feature can be obtained by performing feature fusion on the initial fusion feature and the position mapping image. For example, the initial fusion feature and the position mapping image can be fused based on any type of deep learning algorithm such as a convolutional neural network algorithm and an attention network algorithm to obtain primary or multi-level position features. The embodiment of the present disclosure does not limit the specific method for determining the position feature.

[0093] In one example, the initial fused features and the position mapping image can be processed based on a cascade of multiple convolutional neural network layers to obtain multi-level position features. The texture encoder can be used to process the first intermediate fused features, at least one level of upsampled style features, and at least one level of position features to perform feature decoding operations to obtain the target fused features.

[0094] According to an embodiment of the present disclosure, the texture generative large model may further include a position feature extraction network, and the position feature extraction network may include a cascaded multi-level position feature extraction layer. The position feature extraction layer may be constructed based on a convolutional neural network, or the position feature extraction layer may also be constructed based on other types of neural network algorithms. The position feature includes multiple levels, and the multi-level position features are determined by processing the target image using the multi-level position feature extraction layer.

[0095] According to an embodiment of the present disclosure, the cascaded multi-level position feature extraction layer can extract features from a position mapping image at multiple scales, so that the multi-level position feature can characterize the position mapping relationship between the three-dimensional spatial position of the object element and the two-dimensional pixel based on the position mapping semantic attributes of the position mapping image. By using the texture encoder to fuse the first intermediate fusion feature, the multi-level up-sampling style feature and the multi-level position feature during the feature decoding operation, fine-grained fusion of the position mapping relationship and the style semantic attribute can be achieved to improve the characterization accuracy of the target fusion feature regarding the mapping relationship between the target texture information and the coordinates of the object element, thereby improving the matching degree between the target three-dimensional object and the target object in the target image.

[0096] Figure 4 The principle diagram of a texture generation-based large model according to another embodiment of the present disclosure is schematically shown.

[0097] like Figure 4As shown, the texture generative model 400 may include a first style feature extraction network 410, a texture generation network and a position feature extraction network 430, and the texture generation network includes a texture encoder 421 and a texture decoder 422. The texture generation network can be constructed based on a diffusion model algorithm. The first style feature extraction network 410 may include a downsampling layer and an upsampling layer determined based on a U-shaped network structure. The downsampling layer and the upsampling layer may include a first-level downsampling layer 4111 and a third-level upsampling layer 4112, a second-level downsampling layer 4121 and a second-level upsampling layer 4122, a third-level downsampling layer 4131 and a first-level upsampling layer 4132. The downsampling layer or the upsampling layer may be constructed based on a convolutional neural network algorithm. The first downsampling layer 4111 and the third upsampling layer 4112 form a U-shaped substructure, the second downsampling layer 4121 and the second upsampling layer 4122 form another U-shaped substructure, and the third downsampling layer 4131 and the first upsampling layer 4132 form a third U-shaped substructure. The three nested U-shaped substructures can be used to extract style features of different scales. The texture encoder 421 includes a first encoder layer 4211, a second encoder layer 4212, and a third encoder layer 4213 constructed based on an attention network algorithm. The texture decoder 422 includes a first decoder layer 4221, a second decoder layer 4222, and a third decoder layer 4223 constructed based on an attention network algorithm. The position feature extraction network 430 includes a cascaded first position feature extraction layer 431, a second position feature extraction layer 432, a third position feature extraction layer 433, and a fourth position feature extraction layer 434. The first position feature extraction layer 431 , the second position feature extraction layer 432 , the third position feature extraction layer 433 and the fourth position feature extraction layer 434 are respectively constructed based on convolutional neural network algorithms of different scales.

[0098] like Figure 4 As shown, the target image 401 is input into the first-level downsampling layer 4111 to obtain the first-level downsampling style features. The first-level downsampling style features are input into the second-level downsampling layer 4121 to obtain the second-level downsampling style features. The second-level downsampling style features are input into the third-level downsampling layer 4131 to obtain the third-level downsampling style features. The third-level downsampling style features and the feature map determined based on the third-level downsampling style features are input into the first-level upsampling layer 4132 to output the first-level upsampling style features. The first-level upsampling style features and the feature map determined based on the second-level downsampling style features are input into the second-level upsampling layer 4122 to output the second-level upsampling style features. The second-level upsampling style features and the feature map determined based on the first-level downsampling style features are input into the third-level upsampling layer 4112 to output the third-level upsampling style features.

[0099] like Figure 4As shown, the position mapping image 403 is added to the initial fusion feature 402 and input into the first position feature extraction layer 431 to obtain the first level position feature. The first level position feature is input into the second position feature extraction layer 432 to obtain the second level position feature. The second level position feature is input into the third level position feature extraction layer 433 to obtain the third level position feature. The third level position feature is input into the fourth level position feature extraction layer 434 to obtain the fourth level position feature.

[0100] like Figure 4 As shown, the initial fused feature 402 and the first-level down-sampled style feature obtained by feature fusion based on the position mapping image and the shape mask image are input into the first encoder layer 4211. The first encoder layer 4211 fuses the initial fused feature 402 and the first-level down-sampled style feature based on the attention network algorithm, and performs the first-level feature encoding operation based on the noise information to obtain the first-level first encoded feature. The first-level first encoded feature and the second-level down-sampled style feature are input into the second encoder layer 4212. The second encoder layer 4212 fuses the first-level first encoded feature and the second-level down-sampled style feature based on the attention network algorithm, and performs the second-level feature encoding operation based on the noise information to obtain the second-level first encoded feature. The second-level first encoded feature and the third-level down-sampled style feature are input into the third encoder layer 4213. The third encoder layer 4213 fuses the second-level first encoded feature and the third-level down-sampled style feature based on the attention network algorithm, and performs the third-level feature encoding operation based on the noise information to obtain the third-level first encoded feature. The first encoding feature of level 3 is determined as the first intermediate fusion feature.

[0101] like Figure 4As shown, the first encoded feature of the third level is used as the first intermediate fusion feature, and the first intermediate fusion feature, the first level upsampled style feature, and the fourth level position feature are input into the first decoder layer 4221. The first decoder layer 4221 can process the first intermediate fusion feature, the first level upsampled style feature, and the fourth level position feature based on the attention mechanism to perform a feature decoding operation to obtain the first level first decoded feature. The first level first decoded feature, the second level upsampled style feature, and the third level position feature are input into the second decoder layer 4222. The second decoder layer 4222 can process the first level first decoded feature, the second level upsampled style feature, and the third level position feature based on the attention mechanism to perform a feature decoding operation to obtain the second level first decoded feature. The second level first decoded feature, the second level position feature, and the third level upsampled style feature are input into the third decoder layer 4223. The third decoder layer 4223 can process the second-level first decoding feature, the second-level position feature, and the third-level upsampled style feature based on the attention mechanism to perform a feature decoding operation to obtain the third-level first decoding feature. Based on the third-level first decoding feature and the first-level position feature, feature fusion can be performed to determine the target fusion feature 404. Image rendering based on the target fusion feature 404 can obtain the target texture map 405.

[0102] According to an embodiment of the present disclosure, the texture generative large model also includes a second style feature extraction network, which includes a cascaded multi-level style feature extraction layer. The style feature extraction layer can be constructed based on any type of neural network algorithm such as a convolutional neural network algorithm and an attention network algorithm.

[0103] According to an embodiment of the present disclosure, the object style feature includes: a multi-level style feature obtained by processing the object style feature by a multi-level style extraction layer. The multi-level style feature can characterize the style semantic attributes of the target image at different scale attributes.

[0104] According to an embodiment of the present disclosure, feature fusion is performed based on the initial fusion feature and the object style feature to obtain the target fusion feature, including: using a texture encoder of a texture generation network to perform a feature encoding operation on the initial fusion feature to obtain a second intermediate fusion feature; and using a texture decoder of the texture generation network to perform a feature decoding operation on the second intermediate fusion feature and at least one level of style feature to obtain the target fusion feature.

[0105] According to an embodiment of the present disclosure, the texture encoder may include a cascade of multi-level encoder layers, and the encoder layers may be constructed based on an attention network algorithm. The first-level encoder layer may process initial fusion features and noise information based on the attention network algorithm to implement feature encoding operations on the initial fusion features to obtain first-level encoding features. The second-level encoder layer may process first-level encoding features and noise information based on the attention network algorithm to implement feature encoding operations on first-level encoding features to obtain second-level encoding features. The second-level intermediate fusion features may be determined based on the last-level encoding features output by the last-level encoder layer. For example, the last-level encoding features may be autonomously fused to obtain second intermediate fusion features.

[0106] According to an embodiment of the present disclosure, a feature decoding operation can be performed on the second intermediate fusion feature and at least one level of style feature based on a multi-level decoder layer of a texture decoder. For example, based on an attention network algorithm, the second intermediate fusion feature and at least one level of style feature are processed by a first-level decoder layer to implement a feature decoding operation on the second intermediate fusion feature, and a first-level decoding feature is obtained. Based on an attention network algorithm, the first-level decoding feature and at least one level of style feature are processed by a second-level decoder layer to implement a feature decoding operation on the first-level decoding feature, and a second-level decoding feature is obtained. The target fusion feature is determined based on the last-level decoding feature obtained by performing a feature decoding operation on the last-level decoder layer.

[0107] According to an embodiment of the present disclosure, by utilizing a texture decoder to fuse the second intermediate fusion feature and at least one level of style feature in the process of performing a feature decoding operation, the correlation between the position semantic attributes and the style semantic attributes of the object elements can be enhanced in the stage of restoring the semantic attributes of the target map, so as to improve the accuracy of the correlation between the style semantic attributes and the three-dimensional spatial coordinates of the object elements in the target fusion feature, and can also reduce the amount of data processing for generating the target fusion feature from a large texture generation model, save computational overhead, and improve the accuracy of the target fusion feature in representing the target texture information of the three-dimensional object.

[0108] Figure 5 The principle diagram of a texture generative large model according to another embodiment of the present disclosure is schematically shown.

[0109] like Figure 5As shown, the texture generative model 500 may include a second style feature extraction network 510 and a texture generation network, and the texture generation network includes a texture encoder 521 and a texture decoder 522. The texture generation network can be constructed based on a diffusion model algorithm. The second style feature extraction network 510 may include multiple style feature extraction layers determined based on a cascade network structure. The multiple style feature extraction layers may include a first style feature extraction layer 511, a second style feature extraction layer 512, a third style feature extraction layer 513, and a fourth style feature extraction layer 514. The style feature extraction layer may be constructed based on a convolutional neural network algorithm, and the multiple style feature extraction layers may be used to extract style feature maps of different scales of the target image. The texture encoder 521 includes a first encoder layer 5211, a second encoder layer 5212, and a third encoder layer 5213 constructed based on an attention network algorithm. The texture decoder 522 includes a first decoder layer 5221, a second decoder layer 5222, and a third decoder layer 5223 constructed based on an attention network algorithm.

[0110] like Figure 5 As shown, the target image 501 is processed by the initial style feature extraction layer to obtain the initial style feature. The initial style feature is added to the initial fusion feature 502 obtained by feature fusion based on the position mapping image and the shape mask image to achieve feature fusion, and the target initial style feature is obtained. The target initial style feature is input into the first style feature extraction layer 511 to extract the style semantic feature of the first scale to obtain the first level style feature. The first level style feature is input into the second style feature extraction layer 512 to extract the style semantic feature of the second scale to obtain the second level style feature. The second level style feature is input into the third style feature extraction layer 513 to extract the style semantic feature of the third scale to obtain the third level style feature. The third level style feature is input into the fourth style feature extraction layer 514 to extract the style semantic feature of the fourth scale to obtain the fourth level style feature.

[0111] In the feature encoding stage, the texture encoder of the texture generation network is used to perform a feature encoding operation on the initial fused features, and the second intermediate fused features can be obtained, which may include: the first encoder layer 5211 can be used as a first-level encoder layer to process the initial fused features 502 and noise information based on the attention network algorithm to perform a feature encoding operation on the initial fused features to obtain the first-level encoded features. The second encoder layer 5212 is used as a second-level encoder layer, which can be used to process the first-level encoded features and noise information based on the attention network algorithm to perform a feature encoding operation on the first-level encoded features to obtain the second-level encoded features. The third encoder layer 5213 is used as a third-level encoder layer, which can be used to process the second-level encoded features and noise information based on the attention network algorithm to perform a feature encoding operation on the second-level encoded features to obtain the third-level encoded features. The third-level encoded features can be used as the second-level intermediate fused features.

[0112] In the feature decoding stage, the first decoder layer 5221 of the texture encoder 522 performs a first-level feature decoding operation by fusing the second intermediate fusion feature and the fourth-level style feature based on the attention mechanism to obtain the first-level decoded feature. The second decoder layer 5222 performs a second-level feature decoding operation by fusing the first-level decoding feature and the third-level style feature based on the attention mechanism to obtain the second-level decoded feature. The third decoder layer 5223 performs a third-level feature decoding operation by fusing the second-level decoding feature and the second-level style feature based on the attention mechanism to obtain the third-level decoded feature. The third-level decoded feature is feature-fused with the first-level style feature to obtain the target fused feature 503. Image rendering is performed based on the target fused feature 503 to obtain the target texture map 504.

[0113] According to an embodiment of the present disclosure, using a texture decoder of a texture generation network to perform a feature decoding operation on a second intermediate fusion feature and at least one level of style feature includes: based on an attention mechanism, using a texture decoder to perform a feature decoding operation on the second intermediate fusion feature, at least one level of style feature and at least one level of position feature.

[0114] According to an embodiment of the present disclosure, a feature decoding operation can be performed on the second intermediate fusion feature, at least one level of style feature, and at least one level of position feature based on a multi-level decoder layer of a texture decoder. For example, based on an attention network algorithm, a first-level decoder layer is used to process the second intermediate fusion feature, at least one level of style feature, and at least one level of position feature to implement a feature decoding operation on the second intermediate fusion feature, and obtain a first-level decoded feature. Based on an attention network algorithm, a second-level decoder layer is used to process the first-level decoded feature, at least one level of style feature, and at least one level of position feature to implement a feature decoding operation on the first-level decoded feature, and obtain a second-level decoded feature. The target fusion feature is determined based on the last-level decoded feature obtained by performing a feature decoding operation on the last-level decoder layer.

[0115] According to an embodiment of the present disclosure, by utilizing a texture decoder to perform a feature decoding operation to fuse a second intermediate fusion feature, at least one level of style feature and at least one level of position feature, it is possible to enhance the correlation between the position semantic attributes and the style semantic attributes of the object elements based on the coordinates of the object elements in three-dimensional space represented by the position features in the stage of restoring the semantic attributes of the target map, so as to improve the accuracy of the correlation between the style semantic attributes and the three-dimensional space coordinates of the object elements with respect to the target fusion feature, and it is also possible to reduce the amount of data processing for generating the target fusion feature from a large texture generation model, save computational overhead, and improve the accuracy of the representation of the target texture information of the three-dimensional object represented by the target fusion feature.

[0116] Figure 6 The principle diagram of a texture generation large model according to another embodiment of the present disclosure is schematically shown.

[0117] like Figure 6As shown, the texture generative model 600 may include a second style feature extraction network 610, a texture generation network and a position feature extraction network 630. The texture generation network includes a texture encoder 621 and a texture decoder 622. The texture generation network can be constructed based on a diffusion model algorithm. The second style feature extraction network 610 may include multiple style feature extraction layers determined based on a cascade network structure. The multiple style feature extraction layers may include a first style feature extraction layer 611, a second style feature extraction layer 612, a third style feature extraction layer 613 and a fourth style feature extraction layer 614. The style feature extraction layer may be constructed based on a convolutional neural network algorithm, and the multiple style feature extraction layers may be used to extract style feature maps of different scales of the target image. The texture encoder 621 includes a first encoder layer 6211, a second encoder layer 6212 and a third encoder layer 6213 constructed based on an attention network algorithm. The texture decoder 622 includes a first decoder layer 6221, a second decoder layer 6222 and a third decoder layer 6223 constructed based on an attention network algorithm. The position feature extraction network 630 includes a cascaded first position feature extraction layer 631, a second position feature extraction layer 632, a third position feature extraction layer 633, and a fourth position feature extraction layer 634. The first position feature extraction layer 631, the second position feature extraction layer 632, the third position feature extraction layer 633, and the fourth position feature extraction layer 634 are constructed based on convolutional neural network algorithms of different scales.

[0118] like Figure 6 As shown, the target image 501 is processed by the initial style feature extraction layer to obtain the initial style feature. The initial style feature is added to the initial fusion feature 502 obtained by feature fusion based on the position mapping image and the shape mask image to achieve feature fusion, and the target initial style feature is obtained. The target initial style feature is input into the first style feature extraction layer 511 to extract the style semantic feature of the first scale to obtain the first level style feature. The first level style feature is input into the second style feature extraction layer 512 to extract the style semantic feature of the second scale to obtain the second level style feature. The second level style feature is input into the third style feature extraction layer 513 to extract the style semantic feature of the third scale to obtain the third level style feature. The third level style feature is input into the fourth style feature extraction layer 514 to extract the style semantic feature of the fourth scale to obtain the fourth level style feature.

[0119] like Figure 6As shown, the position mapping image 603 is added to the initial fusion feature 602 to achieve feature fusion, and the fused initial position fusion feature is input into the first position feature extraction layer 631 to obtain the first level position feature. The first level position feature is input into the second position feature extraction layer 632 to obtain the second level position feature. The second level position feature is input into the third level position feature extraction layer 633 to obtain the third level position feature. The third level position feature is input into the fourth level position feature extraction layer 634 to obtain the fourth level position feature.

[0120] In the feature encoding stage, the texture encoder of the texture generation network is used to perform a feature encoding operation on the initial fusion feature, and the second intermediate fusion feature can be obtained, which may include: the first encoder layer 6211 can be used as a first-level encoder layer to process the initial fusion feature 602 and noise information based on the attention network algorithm to perform a feature encoding operation on the initial fusion feature to obtain the first-level encoding feature. The second encoder layer 6212 is used as a second-level encoder layer, which can be used to process the first-level encoding feature and noise information based on the attention network algorithm to perform a feature encoding operation on the first-level encoding feature to obtain the second-level encoding feature. The third encoder layer 6213 is used as a third-level encoder layer, which can be used to process the second-level encoding feature and noise information based on the attention network algorithm to perform a feature encoding operation on the second-level encoding feature to obtain the third-level encoding feature. The third-level encoding feature can be used as the second-level intermediate fusion feature.

[0121] In the feature decoding stage, the first decoder layer 6221 of the texture encoder 622 performs a first-level feature decoding operation by fusing the second intermediate fusion feature, the fourth-level position feature, and the fourth-level style feature based on the attention mechanism to obtain the first-level decoded feature. The second decoder layer 6222 performs a second-level feature decoding operation by fusing the first-level decoding feature, the third-level position feature, and the third-level style feature based on the attention mechanism to obtain the second-level decoded feature. The third decoder layer 6223 performs a third-level feature decoding operation by fusing the second-level decoding feature, the second-level position feature, and the second-level style feature based on the attention mechanism to obtain the third-level decoded feature. The third-level decoded feature is feature-fused with the first-level position feature and the first-level style feature to obtain the target fused feature 604. Image rendering is performed based on the target fused feature 603 to obtain the target texture map 605.

[0122] According to an embodiment of the present disclosure, using a texture generative large model to process a target image and an image to be processed that represents the object morphology of a three-dimensional object may include: processing the target image and an object depth map of the image to be processed based on the texture generative large model to obtain a target texture image; and updating texture properties of the three-dimensional object based on the target texture image to obtain a target three-dimensional object.

[0123] According to an embodiment of the present disclosure, an object depth map represents an image of a three-dimensional object at a specified viewing angle. The specified viewing angle may represent a front viewing angle, a back viewing angle, a top viewing angle, etc. of a three-dimensional object. However, the specified viewing angle is not limited thereto, and may also include any other viewing angle for observing a three-dimensional object, such as a viewing angle for observing a three-dimensional object at a 45-degree angle from above. The embodiment of the present disclosure does not limit the specific type of the specified viewing angle.

[0124] According to an embodiment of the present disclosure, a depth map pixel of an object depth map may include depth information representing an object element of a three-dimensional object at a specified viewing angle, and the pixel coordinates of the depth map pixel may correspond to the coordinate information of the projection plane corresponding to the object element at the specified viewing angle. The coordinates of the object element in three-dimensional space may be represented based on the pixel coordinates and depth information of the object depth map pixel. For example, the pixel coordinates and depth information of the depth map may be converted to a coordinate system based on the camera parameters corresponding to the specified viewing angle to obtain the coordinates of the object element that can be observed at the specified viewing angle.

[0125] According to an embodiment of the present disclosure, updating the texture attributes of a three-dimensional object according to a target texture image to obtain a target three-dimensional object may include updating the texture information of the three-dimensional object using pixel texture information of the target texture image based on a position mapping relationship between image pixels of the target texture image and object elements of the three-dimensional object, thereby obtaining a target three-dimensional object having the target texture information.

[0126] According to the embodiments of the present disclosure, the texture generative large model can learn the mapping relationship between the object depth map and the object elements of the specified viewing angle by fine-tuning the model. Therefore, by processing the object depth map of the target image and the image to be processed by the texture generative large model, the texture generative large model can be controlled based on the depth map pixels and depth information of the object depth map to fuse the texture information of the target object in the target image according to the coordinates of the object elements represented in the three-dimensional space to perform texture semantic attribute fusion, thereby accurately updating the texture semantic attributes of the object elements of the three-dimensional object at the specified viewing angle. Therefore, the target texture image obtained can more accurately represent the result of the three-dimensional object being displayed according to the texture semantic attributes of the target object at the specified viewing angle. In this way, the target three-dimensional object generated based on the target texture image can more accurately represent the texture semantic attributes of the target object in the three-dimensional space, and improve the matching degree between the target three-dimensional object and the target object in the target image.

[0127] According to an embodiment of the present disclosure, updating the texture properties of a three-dimensional object based on a target texture image to obtain a target three-dimensional object may include: determining a pixel mapping relationship between a first pixel of the target texture image and a second pixel of an initial object map based on the target texture image and an object depth map; updating the initial object map according to target texture information of the target texture image based on the pixel mapping relationship to obtain a target object map; and rendering an object based on the target object map to obtain a target three-dimensional object.

[0128] According to an embodiment of the present disclosure, the initial object map is determined based on the three-dimensional object. The initial object map may be, for example, a UV map for rendering the three-dimensional object. The pixel coordinates of the second pixel of the initial object map may have a position mapping relationship with the three-dimensional coordinates of the object element in the three-dimensional space.

[0129] According to an embodiment of the present disclosure, a first pixel and a second pixel having a pixel mapping relationship can represent an object element at the same position in a three-dimensional space. The first pixel of a target texture image and the depth map pixel of an object depth map can be aligned in a coordinate system based on preset camera parameters, thereby obtaining an aligned target texture image and an object depth map. Based on the aligned first pixel and depth map pixel, the aligned first pixel can be enabled to determine the second pixel representing the object element at the same position from the initial object map based on the depth information of the associated depth map pixel, thereby determining the pixel mapping relationship between the first pixel and the second pixel.

[0130] According to an embodiment of the present disclosure, updating the initial object map according to target texture information of a target texture image based on a pixel mapping relationship may include: updating the initial object map using target texture information of target texture images at multiple specified viewing angles based on a pixel mapping relationship.

[0131] According to the embodiments of the present disclosure, the target texture images under multiple specified viewing angles can represent the target texture information of the object elements of the three-dimensional object observed from multiple specified viewing angles. The second pixels having the pixel mapping relationship are respectively updated by the target texture information of the first pixel under multiple specified viewing angles, so that the map pixels of the obtained target object map can represent the target texture information of the object elements of multiple specified viewing angles. Therefore, the target three-dimensional object is rendered based on the target object map, so that the target texture information of the target three-dimensional object can more accurately match the texture information of the target object, so as to improve the matching degree between the target three-dimensional object and the target object.

[0132] Figure 7 The application scenario diagram of the method for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown.

[0133] like Figure 7As shown, in this application scenario, the user can input a two-dimensional target image 110, and a text command "Please describe the hairstyle, bangs, face shape, and clothing style of the two-dimensional character in the picture". The target image 701 can represent a virtual character wearing a purple skirt. The text command is used as a prompt word for the large model, and the large model is used to process the target image 701 and the text name and benefit to obtain object description information 710. The object description information 710 may include character description information for the virtual character and clothing description information for the clothing object. The character description information may be, for example: 1) Gender: Female; 2) Hairstyle: Cute short hair; 3) Bangs type: Bangs; 4) Glasses shape: ... etc. The clothing description information may be, for example: 9) Clothing style: Short-sleeved dress; 10) Clothing length: ... etc.

[0134] Based on the character description information and the clothing description information, a search is performed in the preset three-dimensional object library, and the search result 720 can be obtained as a virtual character three-dimensional object 721 and a clothing three-dimensional object 722. Among them, the clothing three-dimensional object 722 can be a three-dimensional model without texture information.

[0135] The clothing 3D object 722 and the target image 701 are input into the texture generative macro model to obtain the target 3D object 702. The target 3D object 702 may be a texture clothing 3D object having the same or similar texture information as the clothing in the target image 701. The target 3D object 702 and the character 3D object 721 are rendered in a 3D space to obtain a 3D virtual image that matches the virtual character wearing purple clothing in the target image 701. The 3D virtual image can be used in scenes such as animation production and intelligent customer service.

[0136] According to an embodiment of the present disclosure, the method for generating a virtual image based on a large model also includes: in response to a modification request for a displayed virtual image, determining a modification prompt word that matches the modification request; processing a target image, an image to be processed, and a modification prompt word based on the texture-generated large model to obtain an updated target three-dimensional object; and generating an updated virtual image based on the updated target three-dimensional object.

[0137] According to an embodiment of the present disclosure, the modification request may be determined based on a modification operation performed by a user on the currently displayed avatar. For example, a user may input text "please lengthen the sleeve length of the clothes" based on any type of interactive operation, and a modification request may be generated based on the input text.

[0138] According to an embodiment of the present disclosure, a modification prompt word that matches a modification request can be used to characterize the modification requirement for the currently displayed target three-dimensional object represented by the modification request. For example, the modification prompt word can be: "Lengthen the sleeve length of the heroine's clothes". The modification prompt word can be used to control the texture generation model to process the target image and the image to be processed, so as to lengthen the sleeves of the current three-dimensional clothing model according to the modification intention represented by the modification prompt word, and generate an updated three-dimensional clothing model as the target three-dimensional object. An updated virtual image can be obtained by fusing the updated target three-dimensional object with other target three-dimensional objects that do not need to be modified.

[0139] According to an embodiment of the present disclosure, the method for generating a virtual image based on a large model may further include: in response to a target driving instruction, driving the virtual image to perform an action related to the target driving instruction.

[0140] According to an embodiment of the present disclosure, the target drive instruction may be generated based on a user's operation request, or may be generated based on other methods. For example, the target drive instruction may be determined in response to the generation of a target three-dimensional object.

[0141] According to an embodiment of the present disclosure, the target drive instruction may include action parameters representing any action such as lip movements, expression change movements, and posture change movements. The avatar can be controlled to perform actions based on the action parameters to meet the action requirements represented by the target drive instruction. In this way, the avatar generation method based on a large model provided by the embodiment of the present disclosure can be used in any AIGC scenario such as game animation, video production, and smart e-commerce.

[0142] Based on the method for generating a virtual image based on a large model provided by the present disclosure, the present disclosure also provides a device for generating a virtual image based on a large model. Figure 8 The device is described in detail.

[0143] Figure 8 A block diagram of a device for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown.

[0144] like Figure 8 As shown, the large model-based virtual image generation device 800 may include: an object description information acquisition module 810, a target three-dimensional object acquisition module 820 and a virtual image generation module 830.

[0145] An object description information obtaining module 810 is used to process a target image including a target object using a large model to obtain object description information, wherein the target object has texture information;

[0146] a target three-dimensional object obtaining module 820, configured to process the target image and the to-be-processed image representing the object morphology of the three-dimensional object using a texture generation-based large model, to obtain a target three-dimensional object having target texture information, wherein the three-dimensional object is determined based on the object description information, and the target texture information matches the texture information; and

[0147] The virtual image generation module 830 is used to generate a virtual image based on the target three-dimensional object.

[0148] According to an embodiment of the present disclosure, the texture generation large model includes a texture generation network, and the image to be processed includes a position mapping image, and the position pixels of the position mapping image represent the three-dimensional coordinates of the object elements in the three-dimensional object.

[0149] According to an embodiment of the present disclosure, the target three-dimensional object acquisition module includes: a target texture map acquisition submodule and a first acquisition submodule.

[0150] The target texture map acquisition submodule is used to perform feature fusion based on the texture generation network according to the object style features and the position mapping image to obtain a target texture map that matches the texture information, wherein the object style features are determined based on the target image.

[0151] The first acquisition submodule is used to update the three-dimensional object based on the target texture information of the target texture map to obtain the target three-dimensional object.

[0152] According to an embodiment of the present disclosure, the target texture map obtaining submodule includes: an initial fusion feature obtaining unit and a target fusion feature obtaining unit.

[0153] The initial fusion feature acquisition unit is used to perform feature fusion on the position mapping image and the shape mask image in the image to be processed to obtain the initial fusion feature, wherein the mask pixels of the shape mask image represent whether the object elements of the three-dimensional object can store texture information, and there is a position mapping relationship between the mask pixels and the object elements.

[0154] The target fusion feature acquisition unit is used to perform feature fusion based on the initial fusion feature and the object style feature to obtain the target fusion feature, wherein the target texture map is determined based on the target fusion feature.

[0155] According to an embodiment of the present disclosure, the texture generative large model also includes a first style feature extraction network, which includes a downsampling layer and an upsampling layer with a U-shaped network structure, and the object style feature includes: at least one level of downsampling style features obtained by processing the target image through the downsampling layer and the upsampling layer, and at least one level of upsampling style features.

[0156] According to an embodiment of the present disclosure, the target fusion feature obtaining unit includes: a first intermediate fusion feature obtaining subunit and a first target fusion feature obtaining subunit.

[0157] The first intermediate fusion feature obtaining subunit is used to use the texture encoder of the texture generation network to perform a feature encoding operation on the initial fusion feature and at least one level of down-sampled style features to obtain the first intermediate fusion feature.

[0158] The first target fusion feature acquisition subunit is used to use the texture decoder of the texture generation network to perform a feature decoding operation on the first intermediate fusion feature and at least one level of up-sampled style features to obtain a target fusion feature.

[0159] According to an embodiment of the present disclosure, the first target fusion feature acquisition subunit is configured as: based on the attention mechanism, using a texture decoder to perform a feature decoding operation on the first intermediate fusion feature, at least one level of upsampled style features and at least one level of position features, and the position feature is determined based on the position mapping image.

[0160] According to an embodiment of the present disclosure, the texture generative large model also includes a second style feature extraction network, the second style feature extraction network includes a cascaded M-level style feature extraction layer, and the object style feature includes: a multi-level style feature obtained by processing the object style feature by the multi-level style extraction layer.

[0161] According to an embodiment of the present disclosure, the target fusion feature obtaining unit includes: a second intermediate fusion feature obtaining subunit and a second target fusion feature obtaining subunit.

[0162] A second intermediate fusion feature obtaining subunit is used to perform a feature encoding operation on the initial fusion feature using a texture encoder of a texture generation network to obtain a second intermediate fusion feature; and

[0163] The second target fusion feature acquisition subunit is used to use the texture decoder of the texture generation network to perform a feature decoding operation on the second intermediate fusion feature and at least one level of style feature to obtain the target fusion feature.

[0164] According to an embodiment of the present disclosure, the second target fusion feature acquisition subunit is configured as: based on the attention mechanism, using the texture decoder to perform feature decoding operations on the second intermediate fusion feature, at least one level of style feature and at least one level of position feature, and the position feature is determined according to the position mapping image.

[0165] According to an embodiment of the present disclosure, the texture generative large model also includes a position feature extraction network, and the position feature extraction network includes a cascaded multi-level position feature extraction layer.

[0166] According to an embodiment of the present disclosure, the position feature includes multiple levels, and the multi-level position feature is determined by processing the target image using a multi-level position feature extraction layer.

[0167] According to an embodiment of the present disclosure, the target three-dimensional object acquisition module includes: a target texture image acquisition submodule and a second acquisition submodule.

[0168] The target texture image acquisition submodule is used to process the target image and the object depth map of the image to be processed based on the texture generation model to obtain the target texture image. The object depth map represents the image of the three-dimensional object at a specified viewing angle.

[0169] The second acquisition submodule is used to update the texture attributes of the three-dimensional object based on the target texture image to obtain the target three-dimensional object.

[0170] According to an embodiment of the present disclosure, the second obtaining submodule includes: a pixel mapping relationship determining unit, a target object map determining unit and a target three-dimensional object obtaining unit.

[0171] A pixel mapping relationship determining unit, configured to determine, based on the target texture image and the object depth map, a pixel mapping relationship between a first pixel of the target texture image and a second pixel of an initial object map, wherein the initial object map is determined based on the three-dimensional object;

[0172] A target object map determining unit is used to update the initial object map according to the target texture information of the target texture image based on the pixel mapping relationship to obtain the target object map; and

[0173] The target three-dimensional object obtaining unit is used to perform object rendering based on the target object map to obtain the target three-dimensional object.

[0174] According to an embodiment of the present disclosure, the target object map determination unit includes an updating subunit.

[0175] The updating subunit is used to update the initial object map based on the pixel mapping relationship using the target texture information of the target texture images under multiple specified viewing angles.

[0176] According to an embodiment of the present disclosure, the virtual image generation device based on the large model also includes: a modification prompt word determination module, a target three-dimensional object modification module and a virtual image modification module.

[0177] The modification prompt word determination module is used to respond to a modification request for a displayed virtual image and determine a modification prompt word that matches the modification request.

[0178] The target three-dimensional object modification module is used to process the target image, the image to be processed and the modification prompt word based on the texture generation large model to obtain an updated target three-dimensional object.

[0179] The virtual image modification module is used to generate an updated virtual image according to the updated target three-dimensional object.

[0180] According to an embodiment of the present disclosure, the target object includes at least one of the following: a clothing object, a body part object, a vehicle object, and a building object.

[0181] According to an embodiment of the present disclosure, the apparatus for generating a virtual image based on a large model further includes a driving module.

[0182] The driving module is used to drive the virtual image to perform actions related to the target driving instruction in response to the target driving instruction.

[0183] Based on the above-mentioned method for generating a virtual image based on a large model, the embodiment of the present disclosure further provides an artificial intelligence agent, which is configured to execute the method for generating a virtual image based on a large model provided in the above-mentioned embodiment. The agent will be described in detail below in conjunction with the figure.

[0184] Fig. 9 The structural block diagram of an artificial intelligence agent according to an embodiment of the present disclosure is schematically shown.

[0185] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig. 9 As shown, the AI ​​agent 900 may include five core modules: an input module 910 , a control module 920 , a storage module 930 , a calculation module 940 and an output module 950 .

[0186] The input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world. It enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0187] In an example, the input module 910 may input information such as the target image, the image to be processed, or the three-dimensional object described above.

[0188] In the example, the control module 920 is the core support for the AI ​​agent 900 to handle complex tasks. The control module 920 can execute the virtual image generation method based on the large model described above.

[0189] In the example, the control module 920 will continuously interact with the storage module 930, the computing module 940 and / or the output module 950 during operation. However, it should be noted that in the embodiment of the present disclosure, the control module 920 acts as a single initiator to initiate communication with the storage module 930, the computing module 940 and / or the output module 950, and there is no communication coupling between the storage module 930, the computing module 940 and the output module 950.

[0190] In the example, the performance of the control module 920 may be closely related to the large model on which the AI ​​agent 900 is based. In order to give full play to the capabilities of the large model, the internal structure of the control module 920 may be designed to be highly configurable and extensible so as to cope with various types of tasks and requirements in real scenarios.

[0191] The storage module 930 may be responsible for memorizing information such as historical conversations, event flows, etc. The prompt information and data resources mentioned above may be included in the storage module 930 .

[0192] In the example, after the AI ​​agent 900 obtains the search request, the AI ​​agent 900 can use the large model to process the target image, or can also use the texture generation large model to process the target image to obtain the target three-dimensional object. The target image can be stored in the storage module 930. The AI ​​agent 900 can determine the three-dimensional object, the image to be processed and other information from the storage module 930, and feed it back to the control module 920. Then, the control module 920 can use the feedback prompt information and data resources to obtain the target three-dimensional object corresponding to the target image, and pass the target three-dimensional object to the output module 950.

[0193] The operation module 940 can be regarded as a predefined tool library. The rendering engine and the like as mentioned above can be included in the operation module 940 .

[0194] In an example, the output module 950 may output the target three-dimensional object or virtual image described above.

[0195] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0196] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0197] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0198] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the aforementioned method for generating a virtual image based on a large model.

[0199] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements a method for generating a virtual image based on a large model.

[0200] Fig.10 A block diagram of an electronic device suitable for implementing a method for generating a virtual image based on a large model according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0201] like Fig.10 As shown, the device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0202] A number of components in the device 1000 are connected to the I / O interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0203] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as a virtual image generation method based on a large model. For example, in some embodiments, the virtual image generation method based on a large model may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the virtual image generation method based on the large model described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to execute the macro-model-based virtual image generation method in any other appropriate manner (eg, by means of firmware).

[0204] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0205] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0206] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0207] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0208] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0209] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0210] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0211] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A method for generating a virtual image based on a large model, comprising: Processing a target image including a target object using the large model to obtain object description information, wherein the target object has texture information; Processing the target image and the to-be-processed image representing the object morphology of the three-dimensional object using the texture generative large model to obtain a target three-dimensional object having target texture information, wherein the three-dimensional object is determined based on the object description information, and the target texture information matches the texture information; as well as A virtual image is generated based on the target three-dimensional object.

2. The method according to claim 1, wherein: The texture generation large model includes a texture generation network, the image to be processed includes a position mapping image, and the position pixels of the position mapping image represent the three-dimensional coordinates of the object elements in the three-dimensional object; Wherein, the process of processing the target image and the to-be-processed image representing the object morphology of the three-dimensional object by using the texture generative large model comprises: Based on the texture generation network, feature fusion is performed according to the object style feature and the position mapping image to obtain a target texture map that matches the texture information, wherein the object style feature is determined based on the target image; and The three-dimensional object is updated based on the target texture information of the target texture map to obtain the target three-dimensional object.

3. The method according to claim 2, wherein: The performing feature fusion based on the texture generation network and the object style feature and the position mapping image comprises: Performing feature fusion on the position mapping image and the shape mask image in the image to be processed to obtain an initial fusion feature, wherein the mask pixels of the shape mask image represent whether the object elements of the three-dimensional object can store texture information, and there is a position mapping relationship between the mask pixels and the object elements; Feature fusion is performed based on the initial fusion feature and the object style feature to obtain a target fusion feature, wherein the target texture map is determined based on the target fusion feature.

4. The method according to claim 3, wherein: The texture generative large model also includes a first style feature extraction network, the first style feature extraction network includes a downsampling layer and an upsampling layer having a U-shaped network structure, and the object style feature includes: at least one level of downsampling style features obtained by processing the target image through the downsampling layer and the upsampling layer, and at least one level of upsampling style features; The step of performing feature fusion based on the initial fusion feature and the object style feature to obtain a target fusion feature includes: Using a texture encoder of the texture generation network to perform a feature encoding operation on the initial fused feature and at least one level of the downsampled style feature to obtain a first intermediate fused feature; and A texture decoder of the texture generation network is used to perform a feature decoding operation on the first intermediate fusion feature and at least one level of the upsampled style feature to obtain the target fusion feature.

5. The method according to claim 4, wherein: The performing of a feature decoding operation on the first intermediate fusion feature and at least one level of the upsampled style feature by using the texture decoder of the texture generation network comprises: Based on the attention mechanism, the texture decoder is used to perform a feature decoding operation on the first intermediate fusion feature, at least one level of the upsampled style feature and at least one level of position feature, where the position feature is determined based on the position mapping image.

6. The method according to claim 3, wherein: The texture generative large model also includes a second style feature extraction network, the second style feature extraction network includes a cascaded M-level style feature extraction layer, and the object style feature includes: a multi-level style feature obtained by the multi-level style extraction layer processing the object style feature; The step of performing feature fusion based on the initial fusion feature and the object style feature to obtain a target fusion feature includes: Using a texture encoder of the texture generation network to perform a feature encoding operation on the initial fused feature to obtain a second intermediate fused feature; and A texture decoder of the texture generation network is used to perform a feature decoding operation on the second intermediate fusion feature and at least one level of the style feature to obtain the target fusion feature.

7. The method according to claim 6, wherein: The performing of a feature decoding operation on the second intermediate fusion feature and at least one level of the style feature by using the texture decoder of the texture generation network comprises: Based on the attention mechanism, the texture decoder is used to perform a feature decoding operation on the second intermediate fusion feature, at least one level of the style feature and at least one level of the position feature, where the position feature is determined based on the position mapping image.

8. A method according to claim 5 or 7, wherein The texture generative large model also includes a position feature extraction network, and the position feature extraction network includes a cascaded multi-level position feature extraction layer; The position features include multiple levels, and the multiple levels of position features are determined by processing the target image using the multiple levels of position feature extraction layers.

9. The method according to claim 1, wherein: The method of processing the target image and the image to be processed representing the object morphology of the three-dimensional object by using the texture generative large model includes: Processing the target image and the object depth map of the image to be processed based on the texture generative large model to obtain a target texture image, wherein the object depth map represents the image of the three-dimensional object at a specified viewing angle; and The texture attributes of the three-dimensional object are updated based on the target texture image to obtain the target three-dimensional object.

10. The method according to claim 9, wherein: Updating the texture attribute of the three-dimensional object based on the target texture image to obtain the target three-dimensional object includes: Determining, based on the target texture image and the object depth map, a pixel mapping relationship between a first pixel of the target texture image and a second pixel of an initial object map, wherein the initial object map is determined based on the three-dimensional object; Based on the pixel mapping relationship, updating the initial object map according to the target texture information of the target texture image to obtain a target object map; and Object rendering is performed based on the target object map to obtain the target three-dimensional object.

11. The method according to claim 9, wherein: The updating of the initial object map according to the target texture information of the target texture image based on the pixel mapping relationship comprises: Based on the pixel mapping relationship, the initial object map is updated using target texture information of the target texture images at the specified viewing angles.

12. The method according to claim 1, 2 or 9, further comprising: In response to a modification request for the displayed virtual image, determining a modification prompt word matching the modification request; Processing the target image, the image to be processed and the modification prompt word based on the texture generative large model to obtain an updated target three-dimensional object; as well as An updated virtual image is generated according to the updated target three-dimensional object.

13. The method according to claim 1, wherein: The target object includes at least one of the following: Clothing objects, body part objects, vehicle objects, building objects.

14. The method according to claim 1, further comprising: In response to the target driving instruction, the virtual image is driven to perform an action related to the target driving instruction.

15. An artificial intelligence agent configured to perform the method according to any one of claims 1 to 14.

16. A virtual image generation device based on a large model, comprising: An object description information obtaining module, used to process a target image including a target object using a large model to obtain object description information, wherein the target object has texture information; a target three-dimensional object acquisition module, for processing the target image and the to-be-processed image representing the object morphology of the three-dimensional object using a texture generative large model, to obtain a target three-dimensional object having target texture information, wherein the three-dimensional object is determined based on the object description information, and the target texture information matches the texture information; as well as A virtual image generation module is used to generate a virtual image based on the target three-dimensional object.

17. The device according to claim 16, wherein: The texture generation large model includes a texture generation network, the image to be processed includes a position mapping image, and the position pixels of the position mapping image represent the three-dimensional coordinates of the object elements in the three-dimensional object; Wherein, the target three-dimensional object acquisition module includes: a target texture map acquisition submodule, configured to perform feature fusion based on the texture generation network according to the object style features and the position mapping image to obtain a target texture map matching the texture information, wherein the object style features are determined based on the target image; and The first obtaining submodule is used to update the three-dimensional object based on the target texture information of the target texture map to obtain the target three-dimensional object.

18. The device according to claim 17, wherein: The target texture map acquisition submodule includes: an initial fusion feature obtaining unit, configured to perform feature fusion on the position mapping image and the shape mask image in the image to be processed to obtain an initial fusion feature, wherein the mask pixels of the shape mask image represent whether the object elements of the three-dimensional object can store texture information, and there is a position mapping relationship between the mask pixels and the object elements; The target fusion feature obtaining unit is used to perform feature fusion based on the initial fusion feature and the object style feature to obtain the target fusion feature, wherein the target texture map is determined based on the target fusion feature.

19. The device according to claim 18, wherein: The texture generative large model also includes a first style feature extraction network, the first style feature extraction network includes a downsampling layer and an upsampling layer having a U-shaped network structure, and the object style feature includes: at least one level of downsampling style features obtained by processing the target image through the downsampling layer and the upsampling layer, and at least one level of upsampling style features; Wherein, the target fusion feature obtaining unit includes: A first intermediate fusion feature obtaining subunit is used to perform a feature encoding operation on the initial fusion feature and at least one level of the down-sampled style feature using a texture encoder of the texture generation network to obtain a first intermediate fusion feature; and The first target fusion feature acquisition subunit is used to use the texture decoder of the texture generation network to perform a feature decoding operation on the first intermediate fusion feature and at least one level of the up-sampled style feature to obtain the target fusion feature.

20. The device according to claim 19, wherein The first target fusion feature acquisition subunit is configured as follows: Based on the attention mechanism, the texture decoder is used to perform a feature decoding operation on the first intermediate fusion feature, at least one level of the upsampled style feature and at least one level of position feature, where the position feature is determined based on the position mapping image.

21. The apparatus according to claim 18, wherein: The texture generative large model also includes a second style feature extraction network, the second style feature extraction network includes a cascaded M-level style feature extraction layer, and the object style feature includes: a multi-level style feature obtained by the multi-level style extraction layer processing the object style feature; Wherein, the target fusion feature obtaining unit includes: A second intermediate fusion feature obtaining subunit is used to perform a feature encoding operation on the initial fusion feature using a texture encoder of the texture generation network to obtain a second intermediate fusion feature; and The second target fusion feature acquisition subunit is used to use the texture decoder of the texture generation network to perform a feature decoding operation on the second intermediate fusion feature and at least one level of the style feature to obtain the target fusion feature.

22. The device according to claim 21, wherein The second target fusion feature acquisition subunit is configured as follows: Based on the attention mechanism, the texture decoder is used to perform a feature decoding operation on the second intermediate fusion feature, at least one level of the style feature and at least one level of the position feature, where the position feature is determined based on the position mapping image.

23. The device according to claim 20 or 22, wherein: The texture generative large model also includes a position feature extraction network, and the position feature extraction network includes a cascaded multi-level position feature extraction layer; The position features include multiple levels, and the multiple levels of position features are determined by processing the target image using the multiple levels of position feature extraction layers.

24. The device according to claim 16, wherein: The target three-dimensional object acquisition module includes: a target texture image acquisition submodule, configured to process the target image and the object depth map of the image to be processed based on the texture generative large model to obtain a target texture image, wherein the object depth map represents the image of the three-dimensional object at a specified viewing angle; and The second acquisition submodule is used to update the texture attributes of the three-dimensional object based on the target texture image to obtain the target three-dimensional object.

25. The device according to claim 24, wherein: The second obtaining submodule comprises: a pixel mapping relationship determining unit, configured to determine, based on the target texture image and the object depth map, a pixel mapping relationship between a first pixel of the target texture image and a second pixel of an initial object map, wherein the initial object map is determined based on the three-dimensional object; a target object map determining unit, configured to update the initial object map based on the pixel mapping relationship and target texture information of the target texture image to obtain a target object map; and A target three-dimensional object obtaining unit is used to perform object rendering based on the target object map to obtain the target three-dimensional object.

26. The device according to claim 24, wherein: The target object map determination unit comprises: An updating subunit is used to update the initial object map based on the pixel mapping relationship using target texture information of the target texture images under the specified viewing angles.

27. The apparatus of claim 16, 17 or 24, further comprising: A modification prompt word determination module, configured to respond to a modification request for the displayed virtual image and determine a modification prompt word that matches the modification request; A target three-dimensional object modification module, used for processing the target image, the image to be processed and the modification prompt word based on the texture generation large model to obtain an updated target three-dimensional object; as well as The virtual image modification module is used to generate an updated virtual image according to the updated target three-dimensional object.

28. The device according to claim 16, wherein: The target object includes at least one of the following: Clothing objects, body part objects, vehicle objects, building objects.

29. The apparatus of claim 16, further comprising: The driving module is used to drive the virtual image to perform actions related to the target driving instruction in response to the target driving instruction.

30. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 14.

31. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 14.

32. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 14.

Images