Virtual human asset generation method and device based on 3D supervision, equipment and medium

Abstract

The invention relates to the technical field of image modeling, can be applied to business system platforms of financial science and technology, medical treatment and health and the like, and discloses a virtual human asset generation method, device, equipment and medium based on 3D supervision, and the method comprises the steps: obtaining a two-dimensional image, carrying out the feature extraction and noise addition of the two-dimensional image, and obtaining an image feature noise matrix; encoding the two-dimensional image to obtain a matrix parameter, and fusing the matrix parameter with the attention layer parameter to obtain a semantic enhancement feature; determining view angle parameters according to external parameters of the camera, and obtaining a multi-view-angle picture video according to the semantic enhancement features, the view angle parameters and the image feature noise matrix; performing information extraction on the multi-view picture video to obtain a depth information feature map, and taking the depth information feature map and view parameters as supervision information; and generating a diffusion result according to the multi-view picture video and the supervision information, and reconstructing the diffusion result to obtain virtual human assets. According to the invention, the accuracy of generating the 3D virtual human by using the two-dimensional image can be improved.

Description

Technical Field

[0001] The present invention relates to the field of image modeling technology, and in particular to a method, device, equipment and medium for generating virtual human assets based on 3D supervision. Background Art

[0002] In the field of computer vision, generating 3D virtual humans with the help of two-dimensional images is a crucial task. Its application scenarios have extended to multiple fields such as video game character modeling, metaverse digital identity construction, and virtual reality interaction scenarios. The current mainstream virtual human generation technology paths mainly include three categories: a technical route based on 2D image generation and three-dimensional reconstruction, a stereo modeling solution that relies on multi-view data acquisition, and a technical system that realizes three-dimensional generation through algorithmic reasoning.

[0003] In healthcare, 3D virtual human generation technology is used to construct human liver models to assist in preoperative planning. However, due to the lack of 3D perception when reconstructing liver vessels from 2D images, the generated liver vascular branches vary significantly from different viewing angles. Some small bile duct structures may even appear broken or misaligned, resulting in low accuracy and compromising surgical planning.

[0004] In the fintech sector, 3D virtual human generation technology is used to create digital customer service avatars. When this technology generates a multi-view model based on a single portrait photo, due to a lack of 3D monitoring information, facial contours may be distorted when the virtual human's head turns, finger joints may be distorted during hand interactions, and even gesture logic errors may occur. This results in low accuracy in the generated digital customer service avatars and makes it impossible to accurately implement functions such as gesture guidance required for financial services.

[0005] Current 3D virtual human generation technology faces multiple challenges: Methods based on 2D image generation and reconstruction lack 3D perception, making it difficult for virtual humans to maintain consistency from different perspectives; multi-perspective acquisition requires complex hardware, which is costly and has limited practicality; and inference-based 3D generation relies on large, difficult-to-obtain 3D datasets and requires significant computing resources. Furthermore, existing algorithms for generating virtual humans lack accuracy in details like arms and heads, and the 2D results generated by diffusion models are poorly controllable, often requiring multiple experiments and even model retraining, making it difficult to meet high-precision requirements. These challenges ultimately lead to low accuracy in the generated 3D virtual human assets. Summary of the Invention

[0006] The present invention provides a method, device, equipment and medium for generating virtual human assets based on 3D supervision, so as to solve the problem of low accuracy of 3D virtual human asset generation results.

[0007] In a first aspect, a method for generating virtual human assets based on 3D supervision is provided, comprising:

[0008] Acquire two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix;

[0009] Performing a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fusing the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features;

[0010] Determining the viewing angle parameters of the target virtual human according to preset camera extrinsics, and generating a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement features, the viewing angle parameters and the image feature noise matrix;

[0011] Extracting depth information from the multi-view image video to obtain a depth information feature map, and using the depth information feature map and the viewing angle parameter as supervision information for the two-dimensional image data;

[0012] A two-dimensional diffusion result of the two-dimensional image data is generated according to the multi-view picture video and the supervision information, and the two-dimensional diffusion result is geometrically reconstructed to obtain a virtual human asset corresponding to the two-dimensional image data.

[0013] In a second aspect, a device for generating virtual human assets based on 3D supervision is provided, comprising:

[0014] An image feature noise matrix extraction module is used to obtain two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix;

[0015] A semantic enhancement feature fusion module is used to perform a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fuse the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features;

[0016] a multi-view image and video generation module, configured to determine the view parameters of the target virtual human according to preset camera extrinsics, and generate a multi-view image and video corresponding to the two-dimensional image data according to the semantic enhancement features, the view parameters, and the image feature noise matrix;

[0017] a supervisory information analysis module, configured to extract depth information from the multi-view image or video to obtain a depth information feature map, and use the depth information feature map and the viewing angle parameter as supervisory information for the two-dimensional image data;

[0018] A virtual human asset reconstruction module is used to generate a two-dimensional diffusion result of the two-dimensional image data based on the multi-view image video and the supervision information, and geometrically reconstruct the two-dimensional diffusion result to obtain the virtual human assets corresponding to the two-dimensional image data.

[0019] In a third aspect, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the above-mentioned method for generating virtual human assets based on 3D supervision are implemented.

[0020] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned method for generating virtual human assets based on 3D supervision are implemented.

[0021] In the scheme implemented by the above-mentioned virtual human asset generation method, device, equipment and medium based on 3D supervision, the two-dimensional image data of the target virtual human can be obtained through the client, the feature extraction of the two-dimensional image data is performed to obtain the image feature matrix, the preset structured noise is added to the image feature matrix to obtain the image feature noise matrix; the feature encoding operation is performed on the two-dimensional image data to obtain the semantic matrix parameters, the semantic matrix parameters are fused with the preset cross-attention layer parameters through the attention mechanism to obtain the semantic enhancement features; the perspective parameters of the target virtual human are determined according to the preset camera extrinsic parameters, and the semantic enhancement features are obtained according to the semantic enhancement features. The present invention uses the 3D supervision information injection and optimized diffusion model training to significantly improve the efficiency of virtual human customization in Ping An services. The two-stage generation model not only uses depth information to enhance 3D perception capabilities and ensure consistency of virtual human details (such as arms and head) from multiple perspectives, but also improves generation stability through fixed perspective parameter training, thus avoiding the tedious process of retraining the model multiple times in traditional methods. The extremely simple process of single image input lowers the usage threshold, and the fast generation speed adapts to the rapid iteration requirements of marketing scenarios. It does not require complex hardware and massive data sets, and can efficiently meet the high-precision virtual human customization needs of multiple business scenarios such as metaverse asset generation and creative marketing. It achieves a deep integration of technology implementation and business applications, and can solve the problem of low accuracy of 3D virtual human asset generation results. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments of the present invention. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0023] Figure 1 2 is a schematic diagram of an application environment of a method for generating virtual human assets based on 3D supervision in one embodiment of the present invention;

[0024] Figure 2 1 is a flow chart of a method for generating virtual human assets based on 3D supervision according to an embodiment of the present invention;

[0025] Figure 3 yes Figure 2 A schematic flow chart of a specific implementation of step S2;

[0026] Figure 4 yes Figure 2 A schematic flow chart of a specific implementation of step S4;

[0027] Figure 5 1 is a schematic structural diagram of a virtual human asset generation device based on 3D supervision in one embodiment of the present invention;

[0028] Figure 6 is a structural diagram of a computer device in one embodiment of the present invention;

[0029] Figure 7 FIG. 2 is another structural diagram of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0031] The virtual human asset generation method based on 3D supervision provided by the embodiment of the present invention can be applied in Figure 1In an application environment, the client communicates with the server through a network. The server can obtain two-dimensional image data of a target virtual person through the client, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, add preset structured noise to the image feature matrix to obtain an image feature noise matrix; perform feature encoding on the two-dimensional image data to obtain semantic matrix parameters, fuse the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features; determine the perspective parameters of the target virtual person according to preset camera extrinsics, and generate a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement features, the perspective parameters, and the image feature noise matrix; perform depth information extraction on the multi-view picture video to obtain a depth information feature map, and use the depth information feature map and the perspective parameters as supervision information for the two-dimensional image data; generate a two-dimensional diffusion result of the two-dimensional image data according to the multi-view picture video and the supervision information, perform geometric reconstruction on the two-dimensional diffusion result, and obtain the virtual person asset corresponding to the two-dimensional image data. In the present invention, by injecting 3D supervision information and optimizing diffusion model training, the efficiency of virtual person customization in Ping An business can be significantly improved. The two-stage generation mode not only uses depth information to enhance 3D perception capabilities and ensure the consistency of virtual human details (such as arms and heads) under multiple perspectives, but also improves generation stability through fixed perspective parameter training, avoiding the tediousness of multiple retraining models of traditional methods. The extremely simple process of single image input lowers the usage threshold, and the fast generation speed adapts to the rapid iteration requirements of marketing scenarios, and does not require complex hardware and massive data sets. It can efficiently meet the high-precision virtual human customization needs of multiple business scenarios such as metaverse asset generation and creative marketing, and achieve a deep fit between technology implementation and business applications. Among them, the client can be but is not limited to various personal computers, laptops, smart phones, tablets and portable wearable devices. The server can be implemented with an independent server or a server cluster composed of multiple servers. The present invention is described in detail below through specific embodiments.

[0032] See also Figure 2 As shown, Figure 2 A flowchart of a method for generating virtual human assets based on 3D supervision provided in an embodiment of the present invention includes the following steps:

[0033] S1. Acquire two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix.

[0034] In an embodiment of the present invention, the target virtual person refers to a specific virtual person object that needs to be generated in business scenarios such as Ping An Lufax, Ping An Bank, and Ping An Property & Casualty Insurance to meet the needs of diversified marketing scenario design (such as creative marketing, metaverse design, etc.); the two-dimensional image data is used to generate single or multiple planar image information of the virtual person, which can be obtained through camera shooting, existing pictures, etc.; the image feature matrix is ​​after feature extraction of the two-dimensional image data, converting the key information in the image (such as edges, textures, shapes, etc.) into a numerical matrix form that can be processed by a computer, for subsequent injection of diffusion models and other operations.

[0035] Specifically, when obtaining the two-dimensional image data of the target virtual human, a single reference image can be obtained by taking a single-perspective shot of a real person or an existing virtual image through a camera or other equipment, or a two-dimensional image that matches the characteristics of the target virtual human (such as posture, clothing, facial features, etc.) can be selected from an existing image resource library and used as the original input data for generating high-quality virtual human assets.

[0036] Specifically, after two-dimensional image data is input into Vae (a variational autoencoder), the encoder portion of Vae performs a series of mathematical transformations on the image. It analyzes the image's pixel values, color, texture, and other information, utilizing the multi-layered structure of a neural network to abstract and compress the image, transforming the originally high-dimensional image data containing a large amount of redundant information into low-dimensional feature vectors. In this process, Vae learns the underlying distribution patterns of the image, enabling these low-dimensional features to effectively represent the image's key information, reducing the amount of data for subsequent processing while retaining important features.

[0037] For example, in a medical scenario, two-dimensional image data of a target virtual person can be obtained by performing a tomographic scan of specific parts of the patient (such as the lungs and brain) using CT or MRI equipment to obtain two-dimensional medical imaging data containing lesion information. When these two-dimensional images are input into VAE for feature extraction, the VAE encoder compresses the high-dimensional medical imaging data into a low-dimensional feature vector by analyzing information such as image pixel values, grayscale distribution, and tissue boundaries, learns the potential distribution law of the image, and generates an image feature matrix containing anatomical structure and pathological characteristics.

[0038] In an embodiment of the present invention, the preset structured noise is pre-set noise data with a specific distribution or regularity. This noise is not random and disordered, but generated according to certain rules, such as regular addition in dimensions such as specific frequency and spatial position to simulate interference factors in real scenes. The image feature noise matrix is ​​a new matrix obtained by adding the preset structured noise to the original image feature matrix, which includes the original image features and the introduced noise information.

[0039] In detail, the preset structured noise is generated according to specific rules and has a distribution pattern. After adapting it to the dimension and structure of the image feature matrix, it is fused by element-by-element addition or a specific mapping method, so that the noise and the matrix values ​​interact with each other, thereby obtaining an image feature noise matrix that has both the original image features and structured noise information.

[0040] For example, in a financial scenario, when constructing a virtual customer model for risk assessment, the two-dimensional image data of the customer's face is collected through a camera, and the image feature matrix containing facial features is obtained through VAE encoding. The preset structured noise (such as regular noise that simulates different lighting conditions and slight changes in expression) is adapted according to the matrix dimension and integrated element by element, so that the original feature matrix is ​​superimposed with noise that conforms to the interference rules of the financial business scenario to form an image feature noise matrix.

[0041] S2. Perform feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fuse the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features.

[0042] In the embodiment of the present invention, the semantic matrix parameters refer to matrix data that is generated by performing feature encoding operations on two-dimensional image data and processed by the CLIP model and that can represent image semantic information.

[0043] In detail, when feature encoding is performed on two-dimensional image data to obtain semantic matrix parameters, the image data is first processed through the CLIP model, and its text and image alignment capability is used to extract semantic features in the image, and these features are converted into parameters in matrix form, thereby completing the encoding process from two-dimensional image data to semantic matrix parameters.

[0044] In an embodiment of the present invention, the preset cross-attention layer parameters refer to the pre-set learnable parameters of the cross-attention layer in the Transformer module in the diffusion model (Stable VideoModel), which are used to process the input feature information; the attention mechanism refers to an operation mechanism that allows the model to focus on semantically related key information for feature fusion by calculating the correlation weights between the semantic matrix parameters (as Key and Value) and the cross-attention layer features (as Query); the semantic enhancement feature is a new feature generated after the fusion of the above-mentioned attention mechanism, which contains richer semantic information and can guide the diffusion model to generate high-quality virtual human assets that are more in line with semantic requirements.

[0045] In the embodiment of the present invention, referring to Figure 3 As shown, the semantic matrix parameters are fused with the preset cross attention layer parameters through the attention mechanism to obtain semantic enhancement features, including:

[0046] S31, performing dimension alignment processing on the semantic matrix parameters to obtain standardized matrix parameters;

[0047] S32, performing a first linear transformation and a second linear transformation on the standardized matrix parameters respectively to obtain key matrix parameters and value matrix parameters in a preset cross attention layer;

[0048] S33: Using the feature vector in the preset cross-modal interaction layer as the query vector of the preset cross-attention layer, and performing similarity calculation on the key matrix parameters according to the query vector to obtain a semantic association weight;

[0049] S34. Perform feature weighted fusion on the value matrix parameters according to the semantic association weight to obtain semantic enhancement features.

[0050] In detail, the standardized matrix parameters refer to the matrix parameters after the semantic matrix parameters are dimensionally aligned so that their size, shape and other specifications match the input requirements of the cross-attention layer in the diffusion model to ensure compatibility with subsequent operations; the key matrix parameters are the matrix parameters used as the key in the cross-attention layer after the standardized matrix parameters undergo the first linear transformation, which are used to calculate the relevance weight with the query; the value matrix parameters are the matrix parameters obtained as the value after the standardized matrix parameters undergo the second linear transformation, which are used to aggregate semantic information according to the weight.

[0051] Specifically, when dimensionally aligning the semantic matrix parameters to obtain standardized matrix parameters, it is necessary to first determine the input dimension requirements of the cross-attention layer in the diffusion model, and then adjust the number of rows, columns or channels of the semantic matrix parameters through operations such as padding, cropping or linear transformation to match the input specifications of the cross-attention layer in terms of dimension size and shape specifications; when performing the first linear transformation and the second linear transformation on the standardized matrix parameters to obtain key matrix parameters and value matrix parameters, first, according to the parameter dimension requirements of Key and Value in the cross-attention layer, two sets of different linear transformation matrices (i.e., weight matrices) are designed; then, the standardized matrix parameters are multiplied with the first linear transformation matrix respectively, and mapped into key matrix parameters that meet the Key dimension specifications through matrix multiplication operations, and then multiplied with the second linear transformation matrix to map them into value matrix parameters that meet the Value dimension requirements.

[0052] For example, in a medical scenario, after encoding a lung CT image to obtain a semantic matrix, the matrix dimensions are adjusted through padding or interpolation according to the input requirements of the analysis model's cross-attention layer to match the lesion feature input specifications, resulting in standardized matrix parameters. This matrix then undergoes two sets of linear transformations: the first sets maps it into key matrix parameters to capture lesion spatial associations, while the second sets converts it into value matrix parameters to aggregate semantic information such as nodule density. The resulting parameters are then incorporated into the model's cross-attention layer to assist doctors in accurately analyzing lung lesions.

[0053] Furthermore, the preset cross-modal interaction layer refers to a functional layer pre-set in the model for processing the interactive fusion of different modal data (such as images, text, etc.). The feature vector it outputs contains cross-modal semantic information and can be used as the query vector of the cross-attention layer to guide semantic association calculations; the semantic association weight is the weight value obtained by performing similarity calculations (such as dot product, cosine similarity, etc.) on the query vector and the key matrix parameters, reflecting the degree of semantic relevance between the query vector and each key matrix parameter; the semantic enhancement feature is the output result of feature weighted fusion of the value matrix parameters based on the semantic association weight. This feature strengthens the content that semantically matches the query vector by aggregating highly correlated value matrix information, thereby achieving semantic enhancement of the original feature.

[0054] Furthermore, the feature vector of multimodal information such as fused images and texts is extracted from the preset cross-modal interaction layer and used as the query vector of the cross-attention layer; the similarity between the query vector and each feature vector in the key matrix parameter is calculated through the dot product operation or the cosine similarity algorithm to generate a weight matrix representing the degree of semantic association; according to the semantic association weight, the features in the value matrix parameter are weighted and summed, so that semantic features with high association degree obtain greater weights, thereby fusion generating semantic enhancement features containing more accurate semantic information.

[0055] For example, in a financial scenario, the user consultation text and customer service image are input into the cross-modal interaction layer, and the feature vector containing product features is extracted as the query vector; the two-dimensional image of the customer service image is encoded and transformed to obtain a key-value matrix that stores facial expression and action features; the similarity between the query vector and the key matrix is ​​calculated to generate semantic weights (such as the weight of smiling words corresponding to high-yield consultation), and the weighted fusion value matrix is ​​used to generate semantic enhancement features.

[0056] S3. Determine the viewing angle parameters of the target virtual human according to the preset camera extrinsic parameters, and generate a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement features, the viewing angle parameters and the image feature noise matrix.

[0057] In an embodiment of the present invention, the preset camera extrinsic parameters refer to pre-set parameters that describe the position and orientation of the camera in the world coordinate system, including the camera's translation vector (indicating the position of the camera relative to the origin of the world coordinate system) and the rotation matrix (indicating the orientation of the camera), which are used to determine the spatial relative relationship between the camera and the target virtual human; the viewing angle parameters are parameters determined based on the camera extrinsic parameters that characterize the presentation effect of the virtual human under the camera's viewing angle, specifically including viewing angle direction, viewing angle range, pitch angle, yaw angle, etc. These parameters determine the viewing angle position, observation angle and visual range of the virtual human displayed in the image or scene, so that the virtual human can be presented in the picture with an expected viewing angle.

[0058] In the embodiment of the present invention, determining the viewing angle parameters of the target virtual human according to preset camera extrinsics includes:

[0059] Normalizing the rotation matrix and translation vector in the preset camera extrinsic parameters to obtain a normalized rotation matrix and a normalized translation vector;

[0060] Performing feature splicing on the normalized rotation matrix and the normalized translation vector to obtain normalized parameter data;

[0061] Performing dimension conversion on the normalized parameter data to obtain a high-dimensional viewing angle feature vector, and using the high-dimensional viewing angle feature vector as the encoding viewing angle feature of the target virtual person;

[0062] Performing a splicing operation on the coded view feature and the preset multi-view image coding feature to obtain a fusion feature parameter;

[0063] Perform feature correlation analysis on the fused feature parameters to obtain viewing angle parameters.

[0064] In detail, the rotation matrix is ​​a 3×3 matrix that describes the rotation direction of the camera in three-dimensional space, and is used to represent the rotation relationship of the camera coordinate system relative to the world coordinate system; the translation vector is a 3×1 vector that represents the position offset of the camera in three-dimensional space, and is used to describe the position of the origin of the camera coordinate system in the world coordinate system; the normalized rotation matrix is ​​a matrix that normalizes the elements of the rotation matrix (such as dividing by the matrix norm, etc.) so that it meets a specific numerical range or constraint condition; the normalized translation vector is a vector after the components of the translation vector are normalized (such as dividing by the vector modulus or mapping to a specific interval); the normalized parameter data is a unified parameter sequence formed by splicing the normalized rotation matrix and the normalized translation vector in a certain order (such as expanding by rows or columns and then connecting).

[0065] Specifically, the rotation matrix (a 3×3 matrix representing the three-dimensional rotation of the camera) and the translation vector (a 3×1 vector representing the position offset of the camera) in the preset camera extrinsics are normalized. By dividing the rotation matrix elements by the matrix norm and the translation vector components by the vector modulus, the values ​​of the two are limited to a specific range to obtain the normalized rotation matrix and the normalized translation vector. The normalized rotation matrix is ​​then expanded into a one-dimensional vector by row and sequentially spliced ​​with the one-dimensional form of the normalized translation vector to form normalized parameter data that integrates the rotation and translation information to meet the feature fusion requirements of the network input.

[0066] Furthermore, the high-dimensional perspective feature vector is a high-dimensional vector obtained by performing dimensionality enhancement conversion on the normalized parameter data through a fully connected layer or other means; the encoded perspective feature is the high-dimensional perspective feature vector, which is a feature representation after encoding the perspective information; the preset multi-perspective image encoding feature is a feature vector obtained after processing the input multi-perspective image through a feature extraction network, which contains the semantic and visual information of the image; the fusion feature parameter is a comprehensive feature vector formed by splicing the encoded perspective feature and the multi-perspective image encoding feature according to certain rules.

[0067] Furthermore, the normalized parameter data is transformed into a higher-dimensional image through operations such as a fully connected layer to obtain a high-dimensional view feature vector representing the view information of the target virtual person and used as the encoding view feature. At the same time, a feature extraction network (such as CLIP) is used to extract encoding features containing semantic and visual information from the preset multi-view image. The encoding view features and the multi-view image encoding features are spliced ​​in dimensions such as channels to form fused feature parameters. The correlation between the view and image features in the fused feature parameters is then analyzed through the attention mechanism. The final view parameter is decoupled and extracted from the fused feature, realizing the process of feature association to parameter output.

[0068] In an embodiment of the present invention, the multi-view picture video refers to a video composed of a sequence of pictures of the same subject or scene that continuously changes from different perspectives, generated based on two-dimensional image data and combined with semantic enhancement features (feature representation after enhancing the semantic information of the image), perspective parameters (camera perspective related parameters obtained through feature association analysis), and image feature noise matrix (a matrix used to introduce randomness to enrich image details).

[0069] In an embodiment of the present invention, generating a multi-view picture video corresponding to the two-dimensional image data based on the semantic enhancement feature, the viewing angle parameter, and the image feature noise matrix includes:

[0070] Performing noise reduction processing on the image feature noise matrix to obtain a filtered image feature matrix;

[0071] Analyzing key semantic information of the semantic enhancement feature;

[0072] Using the viewing angle parameter to adjust the viewing angle of the filtered image feature matrix to obtain a target viewing angle feature matrix;

[0073] The key semantic information is subjected to feature fusion with the target perspective feature matrix to obtain a multi-perspective picture video corresponding to the two-dimensional image data.

[0074] In detail, the filtered image feature matrix is ​​the matrix obtained by performing noise reduction processing on the image feature noise matrix, filtering out noise interference and retaining effective feature information; the key semantic information is the core semantic elements that play a decisive role in understanding the image content, analyzed and extracted from the semantic enhancement features, such as the category, attributes, actions, and mutual relationships of the target objects in the image, which are representative information at the semantic level. This information can represent the main content and meaning of the image.

[0075] Specifically, the image feature noise matrix is ​​subjected to noise suppression and feature retention through smoothing algorithms such as Gaussian filtering and median filtering, or the noise reduction module in the convolutional neural network, to generate a filtered image feature matrix. At the same time, the attention mechanism or semantic parsing network is used to perform weight assignment and feature screening on the semantic enhancement features, and the core elements such as object categories, attributes, actions, etc. that are semantically representative are identified to complete the noise reduction processing of the image features and the key extraction of semantic information.

[0076] Furthermore, the target perspective feature matrix refers to a feature matrix that conforms to a specific observation angle or spatial position, obtained by performing a spatial transformation operation on the filtered image feature matrix with the help of perspective parameters, so that the feature matrix can match the visual representation under the target perspective. This matrix carries the adjusted perspective information and image features.

[0077] Furthermore, the view parameters are used to perform spatial transformation on the filtered image feature matrix to adapt it to the visual representation of the target view, generating a target view feature matrix. At the same time, the key semantic information extracted from the semantic enhancement features is integrated into the target view feature matrix through an attention mechanism or feature splicing, realizing the fusion of semantic information and image features after view adjustment. Finally, a diffusion model (Stable Video Model) is used to generate a sequence of images containing continuously changing multiple viewpoints, forming a multi-view image video corresponding to the two-dimensional image data.

[0078] S4. Extract depth information from the multi-view image video to obtain a depth information feature map, and use the depth information feature map and the viewing angle parameter as supervision information for the two-dimensional image data.

[0079] In an embodiment of the present invention, the depth information feature map is a feature expression presented in the form of a matrix or tensor, which records the spatial position and depth relationship of objects in the image or video by using the depth extraction function of a model such as ControlNet and the Midas function to calculate and encode the distance from each pixel point to the camera or the scene depth in a multi-view image or video.

[0080] In the embodiment of the present invention, referring to Figure 4 As shown, the depth information is extracted from the multi-view picture video to obtain a depth information feature map, including:

[0081] S41, performing frame sequence analysis on the multi-view picture video to obtain single-frame image data;

[0082] S42, extracting depth information from the single-frame image data using a preset depth extraction model to obtain an initial depth map;

[0083] S43: Perform noise filtering on the pixel distribution features in the initial depth map to obtain a depth information feature map.

[0084] In detail, single-frame image data is each frame of independent image data parsed from multi-view pictures and videos, which contains pixel color, brightness and other information of the frame; the preset depth extraction model is a model for calculating image depth, such as the Midas function integrated in ControlNet, which can process image pixels through algorithms to estimate depth; the initial depth map is the preliminary depth distribution result obtained after the depth extraction model performs feature calculation on the single-frame image data, and represents the depth value of each pixel in the form of a matrix; the pixel distribution feature is the distribution law, density and depth difference of adjacent pixels of the pixel depth value in the initial depth map.

[0085] Specifically, frame sequence analysis is performed on multi-view images and videos, and continuous videos are decomposed into independent single-frame image data. Each frame contains information such as pixel color and brightness under that view. Then, a preset depth extraction model, such as the ControlNet model that integrates the Midas function, is used to perform feature calculations on the single-frame image data. The Midas function estimates the depth value of each pixel by analyzing visual clues such as the image's texture, brightness, and occlusion relationship, thereby obtaining an initial depth map, which is a preliminary depth distribution matrix. Since the initial depth map may contain noise, a filtering algorithm is used to filter out the noise based on the pixel distribution characteristics such as the distribution pattern, density, and adjacent differences of the pixel depth values ​​to obtain a more accurate depth information feature map. In this process, the Midas function is used to perform multiple depth information extractions from different angles on the first generated multi-view image video to capture more comprehensive depth features and reduce errors. The extracted depth information is then normalized to a specific range and converted into the same number of channels and data structure as the image, so that its encoding format is consistent with the image.

[0086] In the embodiment of the present invention, the supervisory information refers to a reference for guiding the learning or optimization of two-dimensional image data during model training or processing.

[0087] In detail, the depth information feature map and perspective parameters are used as supervision information here. By comparing with the two-dimensional image data, calculating the loss or constraining the model output, the model can learn the spatial depth relationship and perspective characteristics of the image, thereby improving the accuracy and authenticity of the two-dimensional image in tasks such as three-dimensional reconstruction and perspective conversion.

[0088] Specifically, after multiple depth information extraction is performed on the multi-view image video generated for the first time and encoded into a format consistent with the image, it is fused and spliced ​​with the encoded multi-view image data and perspective information, so that the depth information feature map and perspective parameters are integrated into the processing flow of the two-dimensional image data in the form of supervision information.

[0089] S5. Generate a two-dimensional diffusion result of the two-dimensional image data according to the multi-view image video and the supervision information, and geometrically reconstruct the two-dimensional diffusion result to obtain a virtual human asset corresponding to the two-dimensional image data.

[0090] In an embodiment of the present invention, the two-dimensional diffusion result refers to two-dimensional image data containing high-precision details (such as the arms and head of a virtual human) and strong semantic consistency, which is generated by the diffusion model after being processed by a two-stage diffusion model and combined with 3D supervision information such as depth information extracted from multi-view images and videos.

[0091] In the embodiment of the present invention, generating a two-dimensional diffusion result of the two-dimensional image data according to the multi-view image video and the supervision information includes:

[0092] Extracting features from the multi-view images and videos to obtain multi-view feature vectors;

[0093] Using the supervision information to constrain the multi-view feature vector to obtain a target constraint feature;

[0094] A two-dimensional diffusion result of the two-dimensional image data is generated according to the target constraint feature.

[0095] In detail, the multi-view feature vector is a feature vector extracted from multi-view pictures and videos that can represent the image content under different viewpoints, and contains the visual information of each viewpoint; the target constraint feature is a feature obtained by constraining the multi-view feature vector using 3D supervision information (such as depth information, fixed view parameters, etc.). This feature incorporates constraints such as three-dimensional spatial relationships and viewpoint consistency, and is used to guide the diffusion model to generate high-precision two-dimensional diffusion results.

[0096] Specifically, when extracting features from multi-view images and videos, a deep learning model is used to extract multi-view feature vectors containing visual information such as color, texture, and structure from image sequences from different viewpoints. This vector represents the image content at each viewpoint. 3D supervision information is used to constrain the multi-view feature vectors, and constraints such as three-dimensional spatial relationships and viewpoint consistency are integrated into the feature vectors to form target constraint features. The target constraint features are input into the diffusion model, and through operations such as the cross-attention mechanism and self-attention mechanism within the model, combined with semantic information embedding processing, a two-dimensional diffusion result containing two-dimensional image data with high-precision details and consistent semantics is generated.

[0097] In this embodiment of the present invention, the virtual human asset refers to a digital asset of a virtual character with a three-dimensional spatial form and high-fidelity details obtained by geometrically reconstructing the two-dimensional diffusion results into a 3D Gaussian Splatting model. It can be applied to various scenarios such as marketing scenario design and metaverse asset generation in Ping An business.

[0098] In an embodiment of the present invention, geometrically reconstructing the two-dimensional diffusion result to obtain a virtual human asset corresponding to the two-dimensional image data includes:

[0099] Performing three-dimensional coordinate mapping on the two-dimensional diffusion result to obtain a spatial depth point cloud set;

[0100] Performing surface modeling on the spatial depth point cloud set to obtain Gaussian sphere surface elements;

[0101] Multi-view fusion rendering is performed on the Gaussian sphere surface element to obtain a virtual human asset corresponding to the two-dimensional image data.

[0102] In detail, the spatial depth point cloud set is a set of a large number of discrete points formed by converting each pixel point in the two-dimensional diffusion result into three-dimensional space through three-dimensional coordinate mapping, combined with camera parameters and depth information. Each point contains X, Y, Z three-dimensional coordinates and corresponding color, depth and other attributes, representing the position and appearance information of the virtual person in three-dimensional space; Gaussian sphere surface element is when surface modeling of the spatial depth point cloud set, with each point as the center, using a Gaussian sphere with a specific radius, direction and color attributes as the basic surface element to fit the three-dimensional surface, and by adjusting the parameters of the Gaussian sphere (such as weight, covariance matrix), the superposition of multiple Gaussian spheres can accurately represent the three-dimensional geometric shape and surface features of the virtual person.

[0103] Specifically, when performing three-dimensional coordinate mapping on the two-dimensional diffusion results, the pixel points are converted into discrete points in three-dimensional space by combining camera parameters and depth information, forming a spatial depth point cloud set containing attributes such as coordinates and color. Then, with the point in the point cloud as the center, a Gaussian sphere with attributes such as radius and direction is used to fit the surface to obtain Gaussian sphere face elements. These face elements are then rendered with multi-perspective fusion. By adjusting the Gaussian sphere parameters, the rendering effects under different perspectives are superimposed to generate virtual human assets that can be used in Ping An business scenarios.

[0104] Furthermore, by performing three-dimensional coordinate mapping on the two-dimensional diffusion results to generate a spatial depth point cloud, then obtaining Gaussian sphere facets through surface modeling, and finally obtaining virtual human assets through multi-perspective fusion rendering. This process uses 3D Gaussian Splatting technology to convert two-dimensional images into three-dimensional models. It not only uses the high-precision two-dimensional results generated by the diffusion model to ensure the integrity of details, but also achieves efficient three-dimensional reconstruction through the conversion of point clouds to Gaussian facets. It can quickly generate high-fidelity virtual human assets that are suitable for Ping An's marketing scenarios and metaverse designs, without the need for complex hardware and a large amount of training data, reducing the threshold and cost of asset generation and improving business application efficiency.

[0105] It can be seen that in the above scheme, by injecting 3D supervision information and optimizing diffusion model training, the efficiency of virtual human customization in Ping An business can be significantly improved. The two-stage generation model not only uses depth information to enhance 3D perception capabilities and ensure the consistency of virtual human details (such as arms and heads) under multiple perspectives, but also improves generation stability through fixed perspective parameter training, avoiding the tediousness of multiple retraining models of traditional methods. The extremely simple process of single image input lowers the usage threshold, and the fast generation speed adapts to the rapid iteration requirements of marketing scenarios. It does not require complex hardware and massive data sets, and can efficiently meet the high-precision virtual human customization needs of multiple business scenarios such as metaverse asset generation and creative marketing, achieving a deep fit between technology implementation and business applications.

[0106] It should be understood that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0107] In one embodiment, a virtual human asset generation device based on 3D supervision is provided, and the virtual human asset generation device based on 3D supervision corresponds to the virtual human asset generation method based on 3D supervision in the above embodiment. Figure 5 As shown, the 3D supervision-based virtual human asset generation device 100 includes an image feature noise matrix extraction module 101, a semantic enhancement feature fusion module 102, a multi-view image and video generation module 103, a supervision information analysis module 104, and a virtual human asset reconstruction module 105. The functional modules are described in detail as follows:

[0108] The image feature noise matrix extraction module 101 is used to obtain two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix;

[0109] The semantic enhancement feature fusion module 102 is used to perform a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fuse the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features;

[0110] A multi-view image and video generation module 103 is configured to determine the view parameters of the target virtual human according to preset camera extrinsics, and to generate a multi-view image and video corresponding to the two-dimensional image data according to the semantic enhancement features, the view parameters, and the image feature noise matrix;

[0111] A supervisory information analysis module 104 is configured to extract depth information from the multi-view image or video to obtain a depth information feature map, and use the depth information feature map and the viewing angle parameter as supervisory information for the two-dimensional image data;

[0112] The virtual human asset reconstruction module 105 is configured to generate a two-dimensional diffusion result of the two-dimensional image data based on the multi-view image video and the supervision information, and geometrically reconstruct the two-dimensional diffusion result to obtain the virtual human asset corresponding to the two-dimensional image data.

[0113] In one embodiment, the semantic enhancement feature fusion module 102, when fusing the semantic matrix parameters with the preset cross attention layer parameters through the attention mechanism to obtain the semantic enhancement feature, is configured to:

[0114] Performing dimension alignment processing on the semantic matrix parameters to obtain standardized matrix parameters;

[0115] Performing a first linear transformation and a second linear transformation on the standardized matrix parameters respectively to obtain key matrix parameters and value matrix parameters in a preset cross attention layer;

[0116] Using the feature vector in the preset cross-modal interaction layer as the query vector of the preset cross-attention layer, and performing similarity calculation on the key matrix parameters according to the query vector to obtain a semantic association weight;

[0117] The value matrix parameters are subjected to feature weighted fusion according to the semantic association weights to obtain semantic enhancement features.

[0118] In one embodiment, the multi-view image and video generation module 103, when determining the view parameters of the target virtual human according to preset camera extrinsics, is configured to:

[0119] Normalizing the rotation matrix and translation vector in the preset camera extrinsic parameters to obtain a normalized rotation matrix and a normalized translation vector;

[0120] Performing feature splicing on the normalized rotation matrix and the normalized translation vector to obtain normalized parameter data;

[0121] Performing dimension conversion on the normalized parameter data to obtain a high-dimensional viewing angle feature vector, and using the high-dimensional viewing angle feature vector as the encoding viewing angle feature of the target virtual person;

[0122] Performing a splicing operation on the coded view feature and the preset multi-view image coding feature to obtain a fusion feature parameter;

[0123] Perform feature correlation analysis on the fused feature parameters to obtain viewing angle parameters.

[0124] In one embodiment, the multi-view image and video generation module 103, when executing and generating the multi-view image and video corresponding to the two-dimensional image data based on the semantic enhancement feature, the viewing angle parameter, and the image feature noise matrix, is further configured to:

[0125] Performing noise reduction processing on the image feature noise matrix to obtain a filtered image feature matrix;

[0126] Analyzing key semantic information of the semantic enhancement feature;

[0127] Using the viewing angle parameter to adjust the viewing angle of the filtered image feature matrix to obtain a target viewing angle feature matrix;

[0128] The key semantic information is subjected to feature fusion with the target perspective feature matrix to obtain a multi-perspective picture video corresponding to the two-dimensional image data.

[0129] In one embodiment, the supervisory information analysis module 104, when performing depth information extraction on the multi-view image or video to obtain a depth information feature map, is configured to:

[0130] Performing frame sequence analysis on the multi-view picture video to obtain single-frame image data;

[0131] Extracting depth information from the single-frame image data using a preset depth extraction model to obtain an initial depth map;

[0132] Noise filtering is performed on the pixel distribution features in the initial depth map to obtain a depth information feature map.

[0133] In one embodiment, the virtual human asset reconstruction module 105, when generating a two-dimensional diffusion result of the two-dimensional image data based on the multi-view image video and the supervision information, is configured to:

[0134] Extracting features from the multi-view images and videos to obtain multi-view feature vectors;

[0135] Using the supervision information to constrain the multi-view feature vector to obtain a target constraint feature;

[0136] A two-dimensional diffusion result of the two-dimensional image data is generated according to the target constraint feature.

[0137] In one embodiment, the virtual human asset reconstruction module 105, when performing geometric reconstruction on the two-dimensional diffusion result to obtain the virtual human asset corresponding to the two-dimensional image data, is further configured to:

[0138] Performing three-dimensional coordinate mapping on the two-dimensional diffusion result to obtain a spatial depth point cloud set;

[0139] Performing surface modeling on the spatial depth point cloud set to obtain Gaussian sphere surface elements;

[0140] Multi-view fusion rendering is performed on the Gaussian sphere surface element to obtain a virtual human asset corresponding to the two-dimensional image data.

[0141] The present invention provides a virtual human asset generation device based on 3D supervision, which can significantly improve the efficiency of virtual human customization in Ping An business by injecting 3D supervision information and optimizing diffusion model training. The two-stage generation mode not only uses depth information to enhance 3D perception capabilities and ensure the consistency of virtual human details (such as arms and heads) under multiple perspectives, but also improves generation stability through fixed perspective parameter training, avoiding the tediousness of multiple retraining models of traditional methods. The extremely simple process of single image input lowers the usage threshold, and the fast generation speed adapts to the rapid iteration requirements of marketing scenarios. It does not require complex hardware and massive data sets, and can efficiently meet the high-precision virtual human customization requirements of multiple business scenarios such as metaverse asset generation and creative marketing, achieving a deep fit between technology implementation and business applications.

[0142] The specific limitations of the 3D-supervised virtual human asset generation device can be found in the limitations of the 3D-supervised virtual human asset generation method described above and will not be further elaborated here. Each module within the 3D-supervised virtual human asset generation device can be implemented in whole or in part via software, hardware, or a combination thereof. Each of these modules can be embedded in or independent of a processor within a computer device in hardware form, or stored in a computer device memory in software form, allowing the processor to invoke and execute the corresponding operations of each module.

[0143] In one embodiment, a computer device is provided. The computer device may be a server, and its internal structure diagram may be as follows: Figure 6 As shown. The computer device includes a processor, a memory, a network interface and a database connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile and / or volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external client via a network connection. When the computer program is executed by the processor, it realizes the functions or steps on the server side of a virtual human asset generation method based on 3D supervision.

[0144] In one embodiment, a computer device is provided. The computer device may be a client, and its internal structure diagram may be as follows: Figure 7As shown. The computer device includes a processor, memory, network interface, display screen and input device connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps on the client side of a virtual human asset generation method based on 3D supervision.

[0145] In one embodiment, a computer device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are performed:

[0146] Acquire two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix;

[0147] Performing a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fusing the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features;

[0148] Determining the viewing angle parameters of the target virtual human according to preset camera extrinsics, and generating a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement features, the viewing angle parameters and the image feature noise matrix;

[0149] Extracting depth information from the multi-view image video to obtain a depth information feature map, and using the depth information feature map and the viewing angle parameter as supervision information for the two-dimensional image data;

[0150] A two-dimensional diffusion result of the two-dimensional image data is generated according to the multi-view picture video and the supervision information, and the two-dimensional diffusion result is geometrically reconstructed to obtain a virtual human asset corresponding to the two-dimensional image data.

[0151] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the following steps are implemented:

[0152] Acquire two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix;

[0153] Performing a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fusing the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features;

[0154] Determining the viewing angle parameters of the target virtual human according to preset camera extrinsics, and generating a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement features, the viewing angle parameters and the image feature noise matrix;

[0155] Extracting depth information from the multi-view image video to obtain a depth information feature map, and using the depth information feature map and the viewing angle parameter as supervision information for the two-dimensional image data;

[0156] A two-dimensional diffusion result of the two-dimensional image data is generated according to the multi-view picture video and the supervision information, and the two-dimensional diffusion result is geometrically reconstructed to obtain a virtual human asset corresponding to the two-dimensional image data.

[0157] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or computer device can be found in the relevant descriptions of the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0158] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0159] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0160] It should be noted that if software tools or components other than those of our company appear in the embodiments of this application, they are only used for illustration and do not represent actual use.

[0161] The embodiments described above are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for generating virtual human assets based on 3D supervision, characterized in that: include: Acquire two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix; Performing a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fusing the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features; Determining the viewing angle parameters of the target virtual human according to preset camera extrinsics, and generating a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement features, the viewing angle parameters and the image feature noise matrix; Extracting depth information from the multi-view image video to obtain a depth information feature map, and using the depth information feature map and the viewing angle parameter as supervision information for the two-dimensional image data; A two-dimensional diffusion result of the two-dimensional image data is generated according to the multi-view picture video and the supervision information, and the two-dimensional diffusion result is geometrically reconstructed to obtain a virtual human asset corresponding to the two-dimensional image data.

2. The method for generating virtual human assets based on 3D supervision according to claim 1, characterized in that: The semantic matrix parameters are fused with the preset cross attention layer parameters through the attention mechanism to obtain semantic enhancement features, including: Performing dimension alignment processing on the semantic matrix parameters to obtain standardized matrix parameters; Performing a first linear transformation and a second linear transformation on the standardized matrix parameters respectively to obtain key matrix parameters and value matrix parameters in a preset cross attention layer; Using the feature vector in the preset cross-modal interaction layer as the query vector of the preset cross-attention layer, and performing similarity calculation on the key matrix parameters according to the query vector to obtain a semantic association weight; The value matrix parameters are subjected to feature weighted fusion according to the semantic association weights to obtain semantic enhancement features.

3. The method for generating virtual human assets based on 3D supervision according to claim 1, characterized in that: The determining of the viewing angle parameters of the target virtual human according to the preset camera extrinsic parameters includes: Normalizing the rotation matrix and translation vector in the preset camera extrinsic parameters to obtain a normalized rotation matrix and a normalized translation vector; Performing feature splicing on the normalized rotation matrix and the normalized translation vector to obtain normalized parameter data; Performing dimension conversion on the normalized parameter data to obtain a high-dimensional viewing angle feature vector, and using the high-dimensional viewing angle feature vector as the encoding viewing angle feature of the target virtual person; Performing a splicing operation on the coded view feature and the preset multi-view image coding feature to obtain a fusion feature parameter; Perform feature correlation analysis on the fused feature parameters to obtain viewing angle parameters.

4. The method for generating virtual human assets based on 3D supervision according to claim 1, wherein: Generating a multi-view picture video corresponding to the two-dimensional image data according to the semantic enhancement feature, the viewing angle parameter, and the image feature noise matrix includes: Performing noise reduction processing on the image feature noise matrix to obtain a filtered image feature matrix; Analyzing key semantic information of the semantic enhancement feature; Using the viewing angle parameter to adjust the viewing angle of the filtered image feature matrix to obtain a target viewing angle feature matrix; The key semantic information is subjected to feature fusion with the target perspective feature matrix to obtain a multi-perspective picture video corresponding to the two-dimensional image data.

5. The method for generating virtual human assets based on 3D supervision according to claim 1, characterized in that: The extracting depth information from the multi-view image and video to obtain a depth information feature map includes: Performing frame sequence analysis on the multi-view picture video to obtain single-frame image data; Extracting depth information from the single-frame image data using a preset depth extraction model to obtain an initial depth map; Noise filtering is performed on the pixel distribution features in the initial depth map to obtain a depth information feature map.

6. The method for generating virtual human assets based on 3D supervision according to claim 1, characterized in that: Generating a two-dimensional diffusion result of the two-dimensional image data according to the multi-view image video and the supervision information includes: Extracting features from the multi-view images and videos to obtain multi-view feature vectors; Using the supervision information to constrain the multi-view feature vector to obtain a target constraint feature; A two-dimensional diffusion result of the two-dimensional image data is generated according to the target constraint feature.

7. The method for generating virtual human assets based on 3D supervision according to claim 1, characterized in that: The geometrically reconstructing the two-dimensional diffusion result to obtain a virtual human asset corresponding to the two-dimensional image data includes: Performing three-dimensional coordinate mapping on the two-dimensional diffusion result to obtain a spatial depth point cloud set; Performing surface modeling on the spatial depth point cloud set to obtain Gaussian sphere surface elements; Multi-view fusion rendering is performed on the Gaussian sphere surface element to obtain a virtual human asset corresponding to the two-dimensional image data.

8. A virtual human asset generation device based on 3D supervision, characterized in that: include: An image feature noise matrix extraction module is used to obtain two-dimensional image data of a target virtual person, perform feature extraction on the two-dimensional image data to obtain an image feature matrix, and add preset structured noise to the image feature matrix to obtain an image feature noise matrix; A semantic enhancement feature fusion module is used to perform a feature encoding operation on the two-dimensional image data to obtain semantic matrix parameters, and fuse the semantic matrix parameters with preset cross-attention layer parameters through an attention mechanism to obtain semantic enhancement features; a multi-view image and video generation module, configured to determine the view parameters of the target virtual human according to preset camera extrinsics, and generate a multi-view image and video corresponding to the two-dimensional image data according to the semantic enhancement features, the view parameters, and the image feature noise matrix; a supervisory information analysis module, configured to extract depth information from the multi-view image or video to obtain a depth information feature map, and use the depth information feature map and the viewing angle parameter as supervisory information for the two-dimensional image data; A virtual human asset reconstruction module is used to generate a two-dimensional diffusion result of the two-dimensional image data based on the multi-view image video and the supervision information, and geometrically reconstruct the two-dimensional diffusion result to obtain the virtual human assets corresponding to the two-dimensional image data.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the method for generating virtual human assets based on 3D supervision according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the method for generating virtual human assets based on 3D supervision according to any one of claims 1 to 7 are implemented.

Images