Data processing method and device based on artificial intelligence, electronic equipment, computer readable storage medium and computer program product

By obtaining unit plane texture parameters and weight parameters of the object mesh model and combining image semantic loss updates, the problem of low generation quality of the three-dimensional mesh model is solved, and higher accuracy modeling and local detail expression are achieved, which is suitable for the rapid application of digital assets.

CN120525708APending Publication Date: 2025-08-22TENCENT TECHNOLOGY (SHENZHEN) CO LTD +1
View PDF 0 Cites 2 Cited by

Patent Information

Application Number
CN202410199365.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-02-22
Publication Date
2025-08-22

AI Technical Summary

Technical Problem

When generating a three-dimensional grid model in the prior art, there is a low quality of grid model asset generation, which makes it difficult to directly realize digital assetization, and the generated three-dimensional digital human model is difficult to inherit the properties of the source input grid model and express local details.

Method used

By obtaining the first texture parameters of the unit plane of the object mesh model, deformation processing is performed based on the weight parameters, and weight parameters are updated in combination with image semantic loss, local fine-grained control of the deformation of the mesh model and improving modeling accuracy.

Benefits of technology

The modeling accuracy of the three-dimensional grid model is improved, and the generated digital human model can better inherit the properties of the source input grid model and have local details, which is suitable for rapid editing and expansion applications of digital assets.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN120525708A_ABST
    Figure CN120525708A_ABST
Patent Text Reader

Abstract

The invention provides a data processing method and device based on artificial intelligence, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: acquiring an object grid model and a first texture parameter corresponding to the object grid model, and performing deformation processing on the object grid model based on a first weight parameter to obtain a first deformed object grid model; performing rendering processing based on the first deformation object grid model and the first texture parameter to obtain a first rendered object image, and performing image generation processing based on the object description text to obtain a generated object image conforming to the object description text; determining image semantic loss based on the first rendered object image and the generated object image, and updating the first weight parameter based on the image semantic loss to obtain a second weight parameter; and performing deformation processing on the object grid model based on the second weight parameter to obtain a second deformation object grid model. According to the invention, the modeling accuracy of the second deformation object grid model is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to artificial intelligence technology, and in particular to an artificial intelligence-based data processing method, device, electronic device, computer-readable storage medium, and computer program product. Background Art

[0002] Artificial Intelligence (AI) is a comprehensive field of computer science. By studying the design principles and implementation methods of various intelligent machines, AI enables them to possess the capabilities of perception, reasoning, and decision-making. AI technology is an interdisciplinary discipline encompassing a wide range of fields, including natural language processing and machine learning / deep learning. With technological advancements, AI will be applied in even more areas and play an increasingly important role.

[0003] Relevant technologies mainly focus on using deep learning algorithm models to achieve good rendering effects, but high-precision texture maps often mask the generation quality of mesh model assets, resulting in low generation quality of mesh model assets. Mesh denoising and topology correction are still required in the later stage. Summary of the Invention

[0004] The embodiments of the present application provide an artificial intelligence-based data processing method, device, electronic device, computer-readable storage medium, and computer program product, which can improve the modeling accuracy of the second deformable object mesh model.

[0005] The technical solution of the embodiment of the present application is implemented as follows:

[0006] The present invention provides an artificial intelligence-based data processing method, including:

[0007] Obtain an object mesh model including a plurality of unit planes, and obtain a first texture parameter corresponding to the object mesh model;

[0008] deforming the object mesh model based on a first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model;

[0009] Performing rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a first rendered object image, and performing image generation processing based on the object description text to obtain a generated object image that conforms to the object description text;

[0010] Determining an image semantic loss based on the first rendered object image and the generated object image, and updating a first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain a second weight parameter corresponding to each of the unit planes;

[0011] The object mesh model is deformed based on a second weight parameter corresponding to each of the unit planes to obtain a second deformed object mesh model.

[0012] The present invention provides an artificial intelligence-based data processing device, comprising:

[0013] A model module, configured to obtain an object mesh model comprising a plurality of unit planes, and obtain a first texture parameter corresponding to the object mesh model;

[0014] a parameter module, configured to deform the object mesh model based on a first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model;

[0015] a rendering module configured to perform rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a first rendered object image, and to perform image generation processing based on the object description text to obtain a generated object image that conforms to the object description text;

[0016] an updating module, configured to determine an image semantic loss based on the first rendered object image and the generated object image, and update a first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain a second weight parameter corresponding to each of the unit planes;

[0017] The deformation module is used to deform the object mesh model based on the second weight parameter corresponding to each unit plane to obtain a second deformed object mesh model.

[0018] An embodiment of the present application provides an electronic device, including:

[0019] a memory for storing computer-executable instructions;

[0020] The processor is used to implement the artificial intelligence-based data processing method provided in the embodiment of the present application when executing the computer-executable instructions stored in the memory.

[0021] An embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions for implementing the artificial intelligence-based data processing method provided in an embodiment of the present application when executed by a processor.

[0022] An embodiment of the present application provides a computer program product, including computer-executable instructions. When the computer-executable instructions are executed by a processor, the artificial intelligence-based data processing method provided in the embodiment of the present application is implemented.

[0023] The embodiments of the present application have the following beneficial effects:

[0024] An object mesh model including a plurality of unit planes is obtained, and a first texture parameter corresponding to the object mesh model and a first texture parameter corresponding to the object mesh model are obtained, and the object mesh model is deformed based on a first weight parameter corresponding to each unit plane to obtain a first deformed object mesh model. The weight parameter involved in the embodiment of the present application is based on a unit plane, that is, different unit planes correspond to different weight parameters, so that the deformation of the mesh model can be controlled locally and finely; rendering is performed based on the first deformed object mesh model and the first texture parameter to obtain a first rendered object image, and image generation is performed based on the object description text to obtain a generated object image that conforms to the object description text; based on the first rendering The object image and the generated object image are used to determine the image semantic loss, where the image obtained based on semantics and the rendered image constitute the image semantic loss, and the first weight parameter is updated based on the image semantic loss to obtain the second weight parameter corresponding to each unit plane, that is, the unit plane weight parameter can be constrained by using the image semantic loss, and the updated unit plane weight parameter can generate a rendered image that is closer to the text semantics; the object mesh model is deformed based on the second weight parameter corresponding to each unit plane to obtain a second deformed object mesh model, and finally the updated second weight parameter can be used to control the object mesh model to perform a more precise and accurate deformation in units of unit planes, thereby improving the modeling accuracy of the mesh model. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1A-1C This is a schematic diagram of the modeling effect of the artificial intelligence-based data processing method provided by related technologies

[0026] Figure 2 This is a schematic diagram of the structure of the artificial intelligence-based data processing system provided in an embodiment of the present application;

[0027] Figure 3 is a structural diagram of an electronic device provided in an embodiment of the present application;

[0028] Figures 4A-4D This is a flow chart of an artificial intelligence-based data processing method provided in an embodiment of the present application;

[0029] Figure 5 This is a schematic diagram of the framework of the data processing method based on artificial intelligence provided in the embodiment of the present application;

[0030] Figures 6A-6C This is a schematic diagram of the rendering effect of the artificial intelligence-based data processing method provided in an embodiment of the present application;

[0031] Figure 7A-7B This is a schematic diagram of facial expression changes in the artificial intelligence-based data processing method provided in an embodiment of the present application;

[0032] Figures 8A-8E This is a text editing schematic diagram of the artificial intelligence-based data processing method provided in an embodiment of the present application;

[0033] Figure 9 is a labeled schematic diagram of the data processing method based on artificial intelligence provided in an embodiment of the present application;

[0034] Figures 10A-10B This is a schematic diagram of local effect optimization of the artificial intelligence-based data processing method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solutions and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limiting this application. All other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0036] In the following description, reference is made to “some embodiments”, which describes a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by those skilled in the art to which this application pertains. The terms used herein are for the purpose of describing the embodiments of this application only and are not intended to limit this application.

[0039] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0040] 1) Mesh Deformation: refers to changing the size, shape, details and other three-dimensional features of the overall mesh by changing the position of the vertices of the three-dimensional mesh.

[0041] 2) Attribute Inheritance: This means that the new 3D portrait generated using mesh deformation technology inherits the complete properties of the source mesh model, such as consistent mesh topology and animation binding.

[0042] 3) Diffusion Models: An unsupervised generative algorithm model that mainly learns the reverse diffusion process to construct the required data samples from noise and understands the characteristics of the dataset in the latent space.

[0043] 4) Deformation Mapping refers to a function that acts on the position change of vertices in a 3D mesh. The first-order derivative of the deformation mapping (Jacobi deformation field) represents the rotation and scaling of each unit plane, thereby guiding the overall deformation of the 3D mesh model.

[0044] 5) Differentiable Rendering: Rendering parameters can be optimized through gradient feedback of the rendering results, thereby guiding the mesh to transform into a 3D portrait that matches the text description.

[0045] 6) Parametric portrait model: refers to the parametric processing of a large number of facial 3D scanning results to represent the facial model, including parameters such as shape, expression, and posture.

[0046] Related technologies extend the generation tasks of generative AI technology from two-dimensional images to three-dimensional models. For example, cultural graph models such as diffusion models are applied to three-dimensional model generation tasks through large-scale three-dimensional data training or multi-view image sampling. The task of generating three-dimensional digital portraits based on text descriptions emphasizes converting user-input description text into a three-dimensional digital portrait that matches the description text. This involves technologies that integrate three-dimensional representation-related technologies such as mesh topology, neural rendering, and tri-plane with deep learning algorithms (such as differentiable rendering, multi-view image loss function design, and the use of parameterized portrait models as three-dimensional prior knowledge).

[0047] See also Figure 1A ,Relevant technologies mainly focus on using deep learning algorithm models to show good rendering effects, but high-precision texture maps often mask the generation quality of mesh model assets, resulting in low generation quality of mesh model assets. Mesh denoising and topology correction are still required in the later stage, which creates certain obstacles to the rapid application of generative artificial intelligence technology in three-dimensional scene products, that is, the quality of the mesh model is not high and it is difficult to directly realize digital assetization.

[0048] See also Figure 1BIn the mainstream technical solutions of Wensheng 3D digital humans in related technologies, the mainstream method is portrait generation based on 3D implicit expression. Although implicit 3D expression can support portrait generation with high rendering quality, the extracted mesh model loses the characteristics of the input source mesh model (cannot inherit the properties of the source input mesh model), such as complete topology change, vertex semantic loss, etc., and the generated mesh model has a lot of meaningless vertex offset noise problems, which makes the generated 3D digital humans often only able to be used for static display or video generation related applications, or to visually inspire designers' inspiration, but it is difficult to truly serve as a digital asset for designers to quickly edit and expand applications.

[0049] To address the shortcomings of implicit 3D representation, related technologies have proposed algorithmic models that optimize mesh vertex offsets directly on 3D mesh models. These solutions often use parameterized portrait models as strong prior constraints, or optimize both the mesh vertex offsets and the parameterized portrait model as training parameters, allowing the newly generated 3D digital portrait to fully inherit the characteristics of the parameterized portrait model. However, these solutions still suffer from the problem of excessive mesh model noise. Furthermore, these solutions' strong reliance on parameterized portrait models also imposes a disguised constraint on diverse portrait generation. Their mesh vertex offsets struggle to exceed the prior constraints of the parameterized portrait, making it difficult to express cartoon characters whose features differ significantly from those of real-life portraits.

[0050] See also Figure 1C For mesh deformation optimization of vertex offset, related technologies propose that a more globally smooth deformation can be achieved by optimizing its first-order derivative (Jacobi deformation field), reducing mesh noise while supporting the complete inheritance of various 3D properties of the source input mesh model. For mesh deformation solutions based on deep learning and differentiable rendering, although they directly control the 3D mesh model, they strongly rely on global overall deformation and manual adjustment of regularization hyperparameters of the deformation energy equation, which often makes the model surface too smooth and lacks local detail features (local features are not fully expressed, for example). Figure 1C The pointed ears of the goblins are not fully expressed), and for the 3D portrait generation task, it is easy to produce average faces or common faces.

[0051] The embodiments of the present application provide an artificial intelligence-based data processing method, a data processing method, an apparatus, an electronic device, a computer-readable storage medium, and a computer program product, which can improve the modeling accuracy of the second deformable object mesh model.

[0052] The artificial intelligence-based data processing method provided in the embodiments of the present application can be implemented by the terminal / server alone; it can also be implemented by the terminal and the server in collaboration, for example, the terminal alone undertakes the artificial intelligence-based data processing method below, or the terminal sends an artificial intelligence-based data processing request (carrying object description text) to the server, and the server executes the artificial intelligence-based data processing method according to the received artificial intelligence-based data processing request.

[0053] The electronic devices for executing the artificial intelligence-based data processing methods provided in the embodiments of the present application can be various types of terminal devices or servers, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services; the terminal can be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited to these. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application.

[0054] Taking servers as an example, it can be a server cluster deployed in the cloud, opening artificial intelligence cloud services (AI as a Service, AIaaS) to users. The AIaaS platform will split several common AI services and provide independent or packaged services in the cloud. This service model is similar to an AI theme mall. All users can access and use one or more artificial intelligence services provided by the AIaaS platform through the application programming interface.

[0055] See also Figure 2 , Figure 2 This is a schematic diagram of the architecture of the artificial intelligence-based data processing system provided in an embodiment of the present application. The terminal 400 is connected to the server 200 via the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0056] The terminal 400 can be used to obtain a data processing request based on artificial intelligence. For example, the user inputs a data processing request based on artificial intelligence through the input interface of the terminal 400, and the terminal 400 sends the data processing request based on artificial intelligence to the server 200. The object description text can be directly collected by the terminal 400 and carried in the data processing request. The server 200 obtains an object mesh model including a plurality of unit planes and obtains a first texture parameter corresponding to the object mesh model; obtains the first texture parameter corresponding to the object mesh model, and deforms the object mesh model based on the first weight parameter corresponding to each unit plane to obtain a first deformed object mesh model; and deforms the object mesh model based on the first deformed object mesh model. The server 200 performs rendering processing on the object mesh model and the first texture parameter to obtain a first rendered object image, and performs image generation processing based on the object description text to obtain a generated object image that conforms to the object description text; determines the image semantic loss based on the first rendered object image and the generated object image, and updates the first weight parameter based on the image semantic loss to obtain a second weight parameter corresponding to each unit plane; deforms the object mesh model based on the second weight parameter corresponding to each unit plane to obtain a second deformed object mesh model, and the server 200 sends the second deformed object mesh model to the terminal 400, so that the user of the terminal 400 can use the second deformed object mesh model as a digital asset of a certain character object.

[0057] In some embodiments, a data processing plug-in may be embedded in a client running in a terminal to implement a data processing method locally on the client. For example, the terminal 400 obtains an object mesh model including multiple unit planes and obtains a first texture parameter corresponding to the object mesh model; obtains the first texture parameter corresponding to the object mesh model and deforms the object mesh model based on a first weight parameter corresponding to each unit plane to obtain a first deformed object mesh model; renders the object mesh model based on the first deformed object mesh model and the first texture parameter to obtain a first rendered object image, and performs image generation based on the object description text to obtain a generated object image that conforms to the object description text; determines image semantic loss based on the first rendered object image and the generated object image, and updates the first weight parameter based on the image semantic loss to obtain a second weight parameter corresponding to each unit plane; deforms the object mesh model based on the second weight parameter corresponding to each unit plane to obtain a second deformed object mesh model, so that the user of the terminal 400 can use the second deformed object mesh model as a digital asset of a certain character object.

[0058] The embodiments of the present application can be applied to various scenarios, including but not limited to artificial intelligence scenarios. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, pre-trained model technology, operation / interaction systems, mechatronics, etc. Among them, the pre-trained model is also called a large model or a basic model. After fine-tuning, it can be widely used in downstream tasks in various major directions of artificial intelligence. Artificial intelligence software technology mainly includes several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0059] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, digital twins, virtual humans, robots, artificial intelligence generated content (AIGC), conversational interaction, smart medical care, smart customer service, game AI, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0060] See also Figure 3 , Figure 3 is a structural diagram of an electronic device for data processing provided in an embodiment of the present application, Figure 3 The server with a database deployed as shown includes: at least one processor 210, a memory 250, at least one network interface 220 and a user interface 230. The various components in the terminal 400 are coupled together via a bus system 240. It is understood that the bus system 240 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 240 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 240 is not described in detail. Figure 3 Various buses are labeled as bus system 240 .

[0061] The processor 210 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0062] The user interface 230 includes one or more output devices 231 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 230 also includes one or more input devices 232, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0063] The memory 250 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 250 may optionally include one or more storage devices that are physically remote from the processor 210.

[0064] The memory 250 includes volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 250 described in the embodiments of the present application is intended to include any suitable type of memory.

[0065] In some embodiments, the memory 250 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplified below.

[0066] Operating system 251, including system programs for processing various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, and driver layer, which are used to implement various basic services and process hardware-based tasks;

[0067] A network communication module 252 for reaching other electronic devices via one or more (wired or wireless) network interfaces 220 , exemplary network interfaces 220 including Bluetooth, WiFi, and USB;

[0068] a presentation module 253 for enabling presentation of information via one or more output devices 231 (e.g., a display screen, a speaker, etc.) associated with the user interface 230 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0069] The input processing module 254 is configured to detect one or more user inputs or interactions from one of the one or more input devices 232 and to translate the detected inputs or interactions.

[0070] In some embodiments, the artificial intelligence-based data processing device provided in the embodiments of the present application can be implemented in software. Figure 3 The AI-based data processing device 255 stored in the memory 250 is shown. This device can be software in the form of a program or plug-in, and includes the following software modules: an acquisition module 2551, an encoding module 2552, a denoising module 2553, a loss module 2554, and an update module 2555. These modules are logical and can be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0071] As mentioned above, the data processing method based on artificial intelligence provided by the embodiment of the present application can be implemented by various types of electronic devices. Figure 4A , Figure 4A This is a flow chart of the data processing method based on artificial intelligence provided by the embodiment of the present application, combined with Figure 4A Steps 101 to 105 are shown for explanation.

[0072] In step 101, an object mesh model including a plurality of unit planes is obtained, and first texture parameters corresponding to the object mesh model are obtained.

[0073] As an example, the unit plane can be a triangle or a quadrilateral. Figure 5 The object mesh model here is a source mesh model, which can be understood as a mesh model template with various customized properties, such as head shape and size. The object mesh model here can be a mesh model of a human object, an animal object, or a mesh model of a non-living object, such as a prop, table, or chair.

[0074] For example, see Figure 5 , perform UV unfolding on the object mesh model to obtain the first texture parameters of the object mesh model. UV unfolding is an important step in 3D modeling, which involves mapping the 3D model surface to a 2D texture to facilitate the placement and display of the map.

[0075] In step 102, the object mesh model is deformed based on a first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model.

[0076] In some embodiments, see Figure 4B In step 102, the object mesh model is deformed based on the first weight parameter corresponding to each unit plane to obtain a first deformed object mesh model, which can be obtained by Figure 4B Steps 1021 to 1023 are implemented as shown.

[0077] In step 1021, the deformation difference of the object mesh model is determined based on the first weight parameter corresponding to each of the unit planes and the deformation function variable.

[0078] In some embodiments, determining the deformation difference of the object mesh model based on the first weight parameter and deformation function variable corresponding to each unit plane in step 1021 can be achieved by the following technical solution: obtaining the first-order derivative of the deformation function variable corresponding to each unit plane; for each unit plane, determining the first deformation error between the first-order derivative of the unit plane and the first weight parameter of the unit plane; and fusing the first deformation errors of multiple unit planes to obtain the deformation difference of the object mesh model. Through embodiments of the present application, it is possible to constrain the first deformed mesh model to have a topological structure similar to that defined by the source mesh model template.

[0079] As an example, the optimization of the deformation function variables can be defined by the following least squares form of formula (1):

[0080]

[0081] Among them, f i ∈F represents each unit plane of the object mesh model, Φ represents the overall deformation map (deformation function variable), For each unit plane f i The first-order derivative (Jacobi) corresponding to the overall deformation map, w i J i is the weighted Jacobian matrix, |f i | represents the area of ​​a unit plane.

[0082] In step 1022 , a solution result of the deformation function variable is obtained as a first deformation function, wherein the solution result makes the deformation difference converge to a minimum value.

[0083] As an example, here, the deformation function variable that makes the deformation difference converge to the minimum value is solved by Poisson solution as the first deformation function.

[0084] In step 1023, the object mesh model is deformed based on the first deformation function to obtain a first deformed object mesh model.

[0085] As an example, the first deformation function may be used to map the vertex position of each unit plane in the object mesh model, thereby achieving deformation of the object mesh model and obtaining a first deformed object mesh model.

[0086] In step 103, rendering processing is performed based on the first deformable object mesh model and the first texture parameters to obtain a first rendered object image, and image generation processing is performed based on the object description text to obtain a generated object image that conforms to the object description text.

[0087] In some embodiments, rendering processing is performed based on the first deformable object mesh model and the first texture parameters to obtain a first rendered object image in step 103. This can be achieved by the following technical solutions: performing normal rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a normal rendered image; performing texture rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a texture rendered image; and combining the texture rendered image and the normal rendered image to form the first rendered object image. This embodiment of the present application can achieve a two-dimensional rendered image. Subsequently, loss is resolved and updated based on the two-dimensional rendered images, which helps improve the modeling accuracy of the second deformable mesh model.

[0088] For example, see Figure 5 In the differentiable rendering process, under a fixed camera perspective, normal rendering processing and texture rendering processing are performed based on the first deformable object mesh model and the first texture parameters respectively, and a normal rendering image and a texture rendering image can be obtained.

[0089] In some embodiments, see Figure 4C In step 103, image generation processing is performed based on the object description text to obtain a generated object image that conforms to the object description text. Figure 4C Step 1031 or step 1032 shown is implemented.

[0090] In step 1031, the object description text is forward propagated in the generative network of the adversarial generative network to obtain a generated object image that conforms to the object description text.

[0091] As an example, a generative adversarial network (GAN) is a deep learning architecture consisting of two competing neural networks: a generator and a discriminator. The core idea behind GANs is to pit these two networks against each other: the generator attempts to generate realistic images, while the discriminator strives to distinguish between real images and generated images (generated images of objects that match the textual representation of the object). This competition drives the generator network to continuously optimize its ability to generate images that are increasingly close to real images. Real images, in this context, refer to images that semantically match the textual representation of the object description.

[0092] In step 1032, the object description text and the noise image are forward propagated in a diffusion model to obtain a generated object image that conforms to the object description text.

[0093] In some embodiments, the diffusion model includes N cascaded denoising networks, where N satisfies 2 ≤ N. Propagating the object description text and the noisy image forward in the diffusion model in step 1032 to obtain a generated object image that conforms to the object description text can be achieved through the following technical solution: Denoise the input of the n-th denoising network through the n-th denoising network in the N cascaded denoising networks, and transmit the n-th denoising result output by the n-th denoising network to the (n + 1)-th denoising network to continue denoising, obtaining the (n + 1)-th denoising result corresponding to the (n + 1)-th denoising network; Take the N-th denoising result output by the N-th denoising network as the generated object image that conforms to the object description text; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n = 1, the input of the n-th denoising network is the noisy image and the object description text. When 2 ≤ n < N, the input of the n-th denoising network is the (n - 1)-th denoising result output by the (n - 1)-th denoising network and the object description text. Through the embodiments of the present application, the denoising ability can be improved through an iterative manner, so as to improve the generation effect of the generated object image.

[0094] As an example, there are N denoising networks cascaded in the diffusion model, which is equivalent to performing N denoising processes. The predicted denoising results obtained in step 103 are all obtained through denoising processes. Each time, denoising is performed based on the predicted denoising result obtained from the previous denoising, and then input into the next denoising network for denoising. n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n = 1, the input of the n-th denoising network is the noisy image (or the latent space noise encoding of the noisy image) and the object description text (or the text encoding of the object description text). When 2 ≤ n < N, the input of the n-th denoising network is the (n - 1)-th denoising result output by the (n - 1)-th denoising network and the object description text (or the text encoding of the object description text).

[0095] As an example, taking N = 3 for illustration, denoise the latent space noise encoding and the object description text (or the text encoding of the object description text) through the first denoising network to obtain the first denoising result. Denoise the first denoising result and the object description text (or the text encoding of the object description text) through the second denoising network to obtain the second denoising result. Denoise the second denoising result and the object description text (or the text encoding of the object description text) through the third denoising network to obtain the third denoising result. The denoising process performed by each denoising network is equivalent to a denoising process for one time step. Finally, the denoising result output by the last denoising network is the generated object image.

[0096] In some embodiments, each denoising network includes M cascaded sampling networks, and each sampling network includes a residual layer and an attention layer; denoising the input of the n-th denoising network in the N cascaded denoising networks can be achieved through the following technical solution: extracting residual features from the input of the m-th sampling network in the n-th denoising network through the residual layer of the m-th sampling network in the n-th denoising network to obtain residual features; performing attention processing on the residual features and the object description text through the attention layer of the m-th sampling network in the n-th denoising network to obtain attention features, and using the attention features as the m-th sampling result of the m-th sampling network; transmitting the m-th sampling result of the m-th sampling network to the (m + 1)-th sampling network to continue the sampling process to obtain the (m + 1)-th sampling result of the (m + 1)-th sampling network; using the M-th sampling result output by the M-th sampling network in the n-th denoising network as the n-th denoising result; where m is an integer variable starting from 1, and the value range of m is 1 ≤ m ≤ M - 1. When m = 1, the input of the m-th sampling network is the (n - 1)-th denoising result. When 2 ≤ m < M, the input of the m-th sampling network is the (m - 1)-th sampling result output by the (m - 1)-th sampling network. Within each denoising network, the denoising ability inside the denoising network can be improved through the sampling iteration method, so as to improve the generation effect of the generated object image.

[0097] As an example, the n-th denoising network includes M cascaded sampling networks (including a downsampling network and an upsampling network), and the value of M satisfies 2 ≤ M; denoising the input of the n-th denoising network in the N cascaded denoising networks can be achieved through the following technical solution: performing downsampling on the (n - 1)-th denoising result and the object description text (or the text encoding of the object description text) through multiple cascaded downsampling networks to obtain the downsampling result of the n-th denoising network; performing upsampling on the downsampling result of the n-th denoising network through multiple cascaded upsampling networks to obtain the upsampling result of the n-th denoising network as the n-th denoising result of the corresponding n-th denoising network. By performing downsampling and upsampling in each denoising process, more detailed information can be retained during the denoising process.

[0098] Continuing with the above example, taking the 2nd denoising network as an example for illustration, the denoising network can include 3 downsampling networks and 3 upsampling networks. Performing downsampling on the 1st denoising result and the object description text (or the text encoding of the object description text) through 3 cascaded downsampling networks to obtain the downsampling result of the 2nd denoising network; performing upsampling on the downsampling result of the 2nd denoising network through 3 cascaded upsampling networks to obtain the upsampling result of the 2nd denoising network as the 2nd denoising result of the corresponding 2nd denoising network.

[0099] Continuing with the above example, the input of the first downsampling network is downsampled through the first downsampling network to obtain a downsampling result corresponding to the first downsampling network. The downsampling result corresponding to the first downsampling network is transmitted to the second downsampling network for further downsampling processing to obtain a second downsampling result corresponding to the second downsampling network. The input of the second downsampling network is downsampled through the second downsampling network to obtain a downsampling result corresponding to the second downsampling network. The downsampling result corresponding to the second downsampling network is transmitted to the third downsampling network for further downsampling processing to obtain a third downsampling result corresponding to the third downsampling network. The third downsampling result output by the third downsampling network is used as the second denoising result. Here, the input of each downsampling network includes object description text (or text encoding of the object description text). The processing process of the upsampling network is the same as that of the downsampling network.

[0100] In some embodiments, the above-mentioned attention processing of the residual features and the object description text through the attention layer of the mth sampling network in the nth denoising network can be achieved through the following technical solutions: performing mapping processing on the residual features based on the query matrix to obtain the attention query matrix; performing mapping processing on the text encoding of the object description text based on the key matrix to obtain the attention key matrix; performing mapping processing on the text encoding of the object description text based on the value matrix to obtain the attention value matrix; multiplying the attention query matrix with the transposed matrix of the attention key matrix to obtain the multiplication result, and obtaining the ratio between the multiplication result and the dimension of the attention key matrix; performing maximum likelihood processing on the ratio, and multiplying the maximum likelihood result with the attention value matrix to obtain the attention feature.

[0101] As an example, the mth sampling network includes a residual layer and an attention layer; the input of the mth sampling network is sampled and processed by the mth sampling network in the M cascaded sampling networks to obtain the mth sampling result corresponding to the mth sampling network, which can be achieved by the following technical solution: the m-1th sampling result corresponding to the m-1th sampling network is subjected to residual feature extraction processing by the residual layer to obtain the mth residual feature corresponding to the mth sampling network; the mth residual feature and the object description text (or the text encoding of the object description text) are subjected to attention processing by the attention layer to obtain the mth sampling result (attention feature) corresponding to the mth sampling network. Through the embodiment of the present application, more effective information can be retained through the residual layer, and the spatial dimension and the temporal dimension can be modeled based on the text encoding through the attention layer, thereby improving the denoising effect.

[0102] As an example, each sampling network includes two layers, namely the residual layer and the attention layer. The input of the residual layer is x, and the output of the residual layer is f(x)+x, where f(x) represents the feature extraction process in the residual layer. The input of the attention layer is the output of the residual layer, and the input of the residual layer is the output of the previous cascaded sampling network (that is, the output of the attention layer included in the previous cascaded sampling network). If the residual layer belongs to the first sampling network, the input of the residual layer is the output of the previous denoising network.

[0103] As an example, the residual feature extraction process is performed on the first sampling result corresponding to the first sampling network through the residual layer to obtain the second residual feature corresponding to the second sampling network; the attention process is performed on the second residual feature and the object description text (or the text encoding of the object description text) through the attention layer to obtain the second sampling result corresponding to the second sampling network.

[0104] As an example, the conditional signal is introduced through the cross-attention mechanism. This cross-attention mechanism can also integrate the conditional information of the object description text into the denoising process. Specifically, the embedded text encoding of the object description text is obtained through the embedding encoding process. Then, this embedded text encoding is merged by the U-Net network including the cross-attention layer, see formula (2) to formula (4):

[0105]

[0106]

[0107]

[0108] in, represents the output of the residual layer (residual features), is the value matrix, is the query matrix, is a key matrix, is a projection matrix with learnable parameters, τ θ (y) is the text encoding.

[0109] In step 104, an image semantic loss is determined based on the first rendered object image and the generated object image, and a first weight parameter corresponding to each of the unit planes is updated based on the image semantic loss to obtain a second weight parameter corresponding to each of the unit planes.

[0110] In some embodiments, the image generation process is implemented by a diffusion model, and the diffusion model includes N cascaded denoising networks, where the value of N satisfies 2≤N, see Figure 4D In step 104, the image semantic loss is determined based on the first rendered object image and the generated object image, which can be obtained by Figure 4DSteps 1041 to 1043 are implemented as shown.

[0111] In step 1041 , an image code corresponding to the first rendering object image is obtained.

[0112] For example, see Figure 5 The first rendered object image includes a first rendered image obtained by normal rendering and a second rendered image obtained by texture rendering. The first rendered image is encoded to obtain a first image code, the second rendered image is encoded to obtain a second image code, and the first image code and the second image code are spliced ​​to obtain an image code corresponding to the first rendered object image.

[0113] In step 1042 , for each of the denoising networks, an image semantic loss corresponding to the denoising network is determined based on the image encoding of the first rendered object image and the generated object image.

[0114] In some embodiments, determining the image semantic loss corresponding to the denoising network based on the image encoding of the first rendered object image and the generated object image in step 1042 can be achieved by the following technical solution: obtaining a first partial derivative of the input of the denoising network with respect to the image encoding, and a second partial derivative of the image encoding with respect to a deformation function variable; obtaining a first difference between the output of the denoising network and the noise image; and multiplying the first difference, the first partial derivative, and the second partial derivative to obtain the image semantic loss corresponding to the denoising network. This embodiment of the present application can constrain the semantic loss between the rendered image and the generated image, thereby improving the modeling accuracy of the three-dimensional mesh model.

[0115] As an example, the image semantic loss is a semantic loss function used to calculate the semantic difference between the first rendered object image and the diffusion model. The image semantic loss of a denoising network is shown in formula (5):

[0116]

[0117] Among them, a time step t corresponds to the denoising process of a denoising network, L(t) is the image semantic loss of a denoising network, ∈ θ is the denoising function from the diffusion model parameter θ (the output of the denoising network), y is the object text description, and x t is the noise at time t (the input of the denoising network), x is the image code, Φ is the deformation function variable, and ∈ is the noise image.

[0118] In step 1043, the image semantic losses of the multiple denoising networks are fused to obtain the image semantic loss.

[0119] As an example, the fusion process can be seen in formula (6):

[0120]

[0121] Among them, w(t) is the weight function about time t (about the denoising network), L(t) is the image semantic loss of a denoising network, is the fused image semantic loss.

[0122] In some embodiments, before updating the first weight parameter corresponding to each of the unit planes based on the image semantic loss and obtaining the second weight parameter corresponding to each of the unit planes, the mesh model loss is determined based on the first deformable object mesh model and the object mesh model; in step 104, updating the first weight parameter corresponding to each of the unit planes based on the image semantic loss and obtaining the second weight parameter corresponding to each of the unit planes can be achieved through the following technical solution: updating the first weight parameter corresponding to each of the unit planes based on the image semantic loss and the mesh model loss and obtaining the second weight parameter corresponding to each of the unit planes.

[0123] As an example, in addition to updating the first weight parameter based on image semantic loss, the first weight parameter can also be updated based on image semantic loss and mesh model loss. Therefore, before updating, the mesh model loss can be determined based on the first deformable object mesh model and the object mesh model. The mesh model loss can constrain the subsequently generated second deformable object mesh to fully inherit the various custom properties of the input object mesh model, thereby seamlessly connecting with the current three-dimensional model design workflow.

[0124] As an example, the overall loss function is shown in formula (7):

[0125]

[0126] in, is the image semantic loss, L lmk is the facial feature point regularization loss function (feature point loss), L op The mesh model loss is composed of the object mask image regularization loss (contour loss), the facial landmark regularization loss, and the object mask image regularization loss. λ1, λ2, and λ3 are the weights corresponding to each loss function. In the experimental setup, λ1 = 1, λ2 = 200, and λ3 = 250 are used as hyperparameters to help output better 3D objects.

[0127] In some embodiments, the above-mentioned determination of mesh model loss based on the first deformable object mesh model and the object mesh model can be achieved through the following technical solutions: projecting the first deformable object mesh model to obtain a first object mask, and obtaining the first object feature points of the first deformable object mesh model; projecting the object mesh model to obtain a second object mask, and obtaining the second object feature points of the object mesh model; determining feature point loss based on the difference between the first object feature points and the second object feature points; determining contour loss based on the difference between the first object mask and the second object mask; fusing the feature point loss and the contour loss to obtain the mesh model loss. Through the embodiments of the present application, the subsequently generated second deformable object mesh can be constrained to fully inherit the various custom properties of the input object mesh model, thereby seamlessly connecting with the current three-dimensional model design workflow.

[0128] As an example, feature point loss, whose full name is facial feature point regularization loss function, is used to measure the difference between the facial feature points of the object mesh model and the deformed mesh model, see formula (8):

[0129]

[0130] Among them, see Figure 5 , k i The facial landmarks defined for the object mesh model, k i ′ is the facial feature point of the deformed mesh model (the first deformed object mesh model), L lmk is the feature point loss.

[0131] As an example, the contour loss, whose full name is the object mask image regularization loss function, is used to constrain the overall contour of the object's head, see formula (9):

[0132]

[0133] Where W and H are the width and height of the mask image, see Figure 5 , O is the mask of the object mesh model (second object mask), O′ is the mask of the first deformed object mesh model (first object mask), L op is the contour loss.

[0134] In step 105, the object mesh model is deformed based on the second weight parameter corresponding to each of the unit planes to obtain a second deformed object mesh model.

[0135] In some embodiments, the first texture parameters are updated based on the image semantic loss to obtain second texture parameters corresponding to the object mesh model; and rendering is performed based on the second deformable object mesh model and the second texture parameters to obtain a second rendered object image. Embodiments of the present application allow the texture parameters to be updated as well, thereby optimizing the texture parameters while completing mesh model building, facilitating the subsequent direct generation of a textured three-dimensional object model as a digital asset.

[0136] As an example, only the first weight parameter may be updated based on the image semantic loss, or the first weight parameter and the first texture parameter may be updated based on the image semantic loss, or the first weight parameter and the first texture parameter may be updated based on the image semantic loss and the mesh model loss.

[0137] As an example, rendering processing corresponding to one perspective can be performed based on the second deformable object mesh model and the second texture parameters to obtain a second rendered object image under one perspective, and a three-dimensional object model can be generated based on the second rendered objects under multiple perspectives.

[0138] Through the embodiment of the present application, an object mesh model including a plurality of unit planes is obtained, and a first texture parameter corresponding to the object mesh model and a first texture parameter corresponding to the object mesh model are obtained, and the object mesh model is deformed based on a first weight parameter corresponding to each unit plane to obtain a first deformed object mesh model. The weight parameters involved in the embodiment of the present application are based on unit planes, that is, different unit planes correspond to different weight parameters, so that the deformation of the mesh model can be controlled locally and finely; rendering processing is performed based on the first deformed object mesh model and the first texture parameter to obtain a first rendered object image, and image generation processing is performed based on the object description text to obtain a generated object image that conforms to the object description text; based on First, render the object image and generate the object image, determine the image semantic loss, where the image obtained based on semantics and the rendered image constitute the image semantic loss, and update the first weight parameter based on the image semantic loss to obtain the second weight parameter corresponding to each unit plane, that is, the unit plane weight parameter can be constrained by using the image semantic loss. The updated unit plane weight parameter can generate a rendered image that is closer to the text semantics; based on the second weight parameter corresponding to each unit plane, the object mesh model is deformed to obtain a second deformed object mesh model. Finally, the updated second weight parameter can control the object mesh model to perform more precise and accurate deformation in units of unit planes, thereby improving the modeling accuracy of the mesh model.

[0139] The following describes an exemplary application of the embodiments of the present application in a practical application scenario.

[0140] The terminal can be used to obtain an artificial intelligence-based data processing request. For example, a user inputs an artificial intelligence-based data processing request through the terminal's input interface, and the terminal sends the artificial intelligence-based data processing request to a server. The object description text can be directly collected by the terminal and carried in the data processing request. The server obtains an object mesh model including multiple triangular faces; obtains a first texture parameter corresponding to the object mesh model, and deforms the object mesh model based on a first weight parameter corresponding to each triangular face to obtain a first deformed object mesh model; renders based on the first deformed object mesh model and the first texture parameter to obtain a first rendered object image, and performs image generation based on the object description text to obtain a generated object image that conforms to the object description text; determines image semantic loss based on the first rendered object image and the generated object image, and updates the first weight parameter based on the image semantic loss to obtain a second weight parameter corresponding to each triangular face; deforms the object mesh model based on the second weight parameter corresponding to each triangular face to obtain a second deformed object mesh model, and the server sends the second deformed object mesh model to the terminal, so that the user of the terminal can use the second deformed object mesh model as a digital asset of a certain character object.

[0141] The embodiment of the present application proposes an optimization based on the technical solution of mesh deformation based on the Jacobi deformation field, and gives a trainable weight parameter to the deformation field acting on each triangle, thereby providing a finer-grained mesh deformation control, which can be called a weighted Jacobi deformation field. The introduction of weight parameters not only ensures the diversity of portrait generation, but also can fully highlight the local features of the face described in the text. The three-dimensional mesh model obtained by the embodiment of the present application can not only achieve high-quality rendering of visual content, but the three-dimensional mesh model itself is also of high quality, and fully inherits the various custom properties of the input mesh model, which can be seamlessly connected with the current three-dimensional model design workflow.

[0142] This embodiment of the application leverages image-based diffusion modeling and differentiable rendering to extend the multi-perspective Vincent image task to the Vincent 3D portrait task, forming a complete technical framework for 3D portrait generation based on weighted Jacobi mesh deformation. The 3D portrait generated by this embodiment of the application can be directly applied in parallel to the 3D rendering software commonly used by 3D designers. This reduces the 3D modeling threshold and repetitive processes through the Vincent 3D portrait technique, significantly enhancing the ability of 3D algorithm models to expand the diverse and high-quality annotated 3D portrait dataset.

[0143] See also Figure 5 , Figure 5The Wensheng 3D portrait framework provided by an embodiment of the present application is shown. First, the input mesh model template is UV unfolded to obtain an RGB color map. At the same time, the mesh model template is Poisson solved based on the weighted Jacobian matrix of each triangle to obtain an optimized deformation function. The mesh model template is then deformed based on the optimized deformation function to obtain a deformed mesh model. Rendering is performed based on the RGB color map and the deformed mesh model to obtain a rendered image (normal rendering image and texture rendering image). The deformed mesh model is projected to obtain a facial mask and facial feature points. A generated image is obtained by inputting text into a diffusion model. A semantic loss is constructed based on the rendered image and the generated image. A mesh model loss is constructed based on the facial mask and the facial feature points. The RGB color map and the weighted Jacobian matrix are updated based on the semantic loss and the mesh model loss. The updated weighted Jacobian matrix can be used to control the deformation to obtain a mesh model that conforms to the semantics of the text.

[0144] The embodiment of the present application controls mesh model deformation by optimizing the weighted Jacobian matrix of each triangle, thereby achieving the generation of a three-dimensional portrait that conforms to the text description. During the differentiable rendering process, the normal image and texture image rendered from a fixed camera perspective are placed in a diffusion model with the text description to calculate the loss function, and gradient backpropagation is used to optimize the RGB color map and weighted Jacobian matrix. Furthermore, information obtained from the facial mask and facial feature points is used to calculate the regularization term of the weighted Jacobian matrix, thereby ensuring the rationality of the facial features of the portrait mesh model and their consistency with the text description.

[0145] The embodiment of this application mainly optimizes three parameters, namely the weighted Jacobian matrix w of each triangle i , J i ∈R 3x3 (corresponding to the first weight parameter or the second weight parameter in the previous text) and the RGB color texture map (the first texture parameter or the second texture parameter).

[0146] For the weighted Jacobian matrix, the weighted Jacobian matrix after gradient optimization can help restore the topology structure defined by the source mesh model template. Therefore, the optimization of mesh deformation can be defined by the following least squares formula (10):

[0147]

[0148] Among them, f i ∈F represents each triangle face of the mesh model, Φ represents the overall deformation mapping (deformation function variable), For each triangle f i The first-order derivative (Jacobi) corresponding to the overall deformation map, w i J iis the weighted Jacobian matrix (corresponding to the first weight parameter or the second weight parameter in the previous text), |f i | represents the area of ​​a triangle.

[0149] For the RGB color texture map, the gradient feedback calculated by the loss function is continuously optimized. The UV coordinates of the RGB color texture map come from the input mesh model template. The texture map resolution set in the current experiment is 2048x2048.

[0150] The final generated 3D portrait contains a high-quality mesh model that conforms to the text semantics and a high-definition texture map. It is a 3D asset with reasonable mesh topology and clear coloring that can be edited and optimized by designers.

[0151] The loss function design involved in the portrait generation framework provided in the embodiment of this application is specifically as follows:

[0152] 1) Semantic loss function, used to calculate the semantic difference between the rendered image and the pre-trained diffusion model, see formula (11):

[0153]

[0154] in, is the semantic loss, ∈θ is the denoising function from the diffusion model parameter θ, w(t) is the weight function about time t, y is the text description, x t is the noise at time t, x is the data encoded from the normal rendered image and the texture rendered image, Φ is the global deformation map, and ∈ is the noise image.

[0155] 2) Facial feature point regularization loss function, which is used to measure the difference between the facial feature points of the source mesh model and the deformed mesh model, see formula (12):

[0156]

[0157] Among them, k i The facial feature points defined by the input source mesh model, k′ i are the facial feature points of the deformed mesh model.

[0158] 3) Object mask image regularization loss function, which is used to constrain the overall contour of the object's head, see formula (13):

[0159]

[0160] Where W and H are the width and height of the mask image, O is the source mesh model mask, and O′ is the deformed mesh model mask.

[0161] The overall loss function is shown in formula (14):

[0162]

[0163] in, is the semantic loss function, L lmk is the regularization loss function of facial feature points, L op is the object mask image regularization loss function, and λ1, λ2, and λ3 are the weights corresponding to the respective loss functions. In the experimental setting, λ1 = 1, λ2 = 200, and λ3 = 250 are used as hyperparameters to help output better 3D portraits.

[0164] See also Figures 6A-6C The object mesh model optimization solution provided by the embodiment of the present application can be implemented in the current mature 3D model design and rendering process, and is adapted to the workflow of 3D designers. That is, there is no need to pinch the face from scratch. The high-quality mesh portrait model with reasonable facial shape and features can be directly edited and optimized. It often only takes 3-5 minutes of design time instead of starting modeling from scratch. For example, Figures 6A-6C The high-quality mesh portrait model shown can be added with hair and accessories in 3D model design software.

[0165] See also Figure 7A-7B The present embodiment of the application does not lose the properties of the input mesh portrait model (the original object mesh model provided by the present embodiment of the application), allowing 3D designers to reuse a set of customized input mesh portrait model designs in multiple portrait generation tasks, accelerating the rapid application of 3D assets. Character rigging animations can be perfectly preserved and reused after being generated based on the input mesh portrait model. If the input mesh portrait model comes from a portrait parameterized model, these portrait parameters can also be perfectly preserved, allowing for rapid control of the expression and shape changes of the generated new portrait.

[0166] The embodiment of the present application also supports the ability to edit 3D materials by text input, and uses the rich semantics of text descriptions to help designers quickly reproduce design ideas, thereby reducing repeated iterations and rollbacks. Figure 8A-8B The embodiment of the present application supports the ability to edit mesh models and texture maps by text input, and can quickly change local features of the generated model, such as clothing color, nose length, etc. Figure 8C-8D By changing the text description, you can control different high-quality mesh portrait models, for example, changing the style of the nose or changing the face shape, see Figure 8E By changing the text description, you can control different map styles, for example, changing the color of the headscarf and clothing.

[0167] See also Figure 9In terms of three-dimensional data sets, since the embodiment of the present application can completely retain the various three-dimensional features of the source input mesh model, the diversified three-dimensional portraits generated by the diversified text description of the embodiment of the present application can quickly expand high-quality two-dimensional / three-dimensional facial annotation information, such as facial feature points, facial language segmentation, etc., and only need to be annotated once on the source input mesh model (the original object mesh model provided by the embodiment of the present application), and the annotation information is also complete and accurate on the newly generated diversified three-dimensional portraits.

[0168] See also Figures 10A-10B , Figures 10A-10B The comparison of the effect of using the weighted Jacobian matrix and not using the trainable weighting parameters is shown ( Figure 10A as well as Figure 10B The effect shown on the left is the effect of not using the weighted Jacobian matrix. Figure 10A as well as Figure 10B The effect shown on the right is the effect of using the weighted Jacobian matrix). It can be clearly observed that after using such adaptive and trainable weight parameters, the local features of the generated three-dimensional portrait can be amplified and generated in units of triangles in accordance with the text semantic description, thereby improving the problems of "common face" and "average face". For example, Figure 10A The mesh model on the middle right has a more prominent nose and ears than the one on the left. Figure 10B The mesh model on the center right has a more prominent chin than the one on the left.

[0169] The image generation algorithm model involved in the embodiment of the present application, in addition to adopting the diffusion model, can also use a generative adversarial network. The embodiment of the present application focuses on generating high-quality mesh models while fully inheriting the designer's predefined three-dimensional attributes (three-dimensional facial feature point annotation, three-dimensional facial semantic segmentation, etc.). It will help the subsequent process to generate better three-dimensional assets. The textureless solid-color three-dimensional digital portraits generated in the embodiment of the present application (the second deformable network model provided in the embodiment of the present application) can be collected, and other technical solutions specializing in the generation of three-dimensional model texture maps or allowing designers to make color adjustments can also be used to quickly iterate and produce a large number of three-dimensional portrait materials required for three-dimensional design products.

[0170] The following further describes an exemplary structure of the artificial intelligence-based data processing device 255 provided in the embodiment of the present application as a software module. In some embodiments, for example, Figure 2As shown, the software modules stored in the artificial intelligence-based data processing device 255 of the memory 250 may include: a model module 2551, which is used to obtain an object mesh model including multiple unit planes and obtain a first texture parameter corresponding to the object mesh model; a parameter module 2552, which is used to deform the object mesh model based on a first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model; a rendering module 2553, which is used to perform rendering processing based on the first deformed object mesh model and the first texture parameter to obtain a first rendered object image, and perform image generation processing based on the object description text to obtain a generated object image that conforms to the object description text; an update module 2554, which is used to determine the image semantic loss based on the first rendered object image and the generated object image, and update the first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain a second weight parameter corresponding to each of the unit planes; and a deformation module 2555, which is used to deform the object mesh model based on the second weight parameter corresponding to each of the unit planes to obtain a second deformed object mesh model.

[0171] In some embodiments, the parameter module 2552 is further used to: determine the deformation difference of the object mesh model based on the first weight parameter corresponding to each of the unit planes and the deformation function variable; obtain the solution result of the deformation function variable as the first deformation function, wherein the solution result makes the deformation difference converge to a minimum value; and deform the object mesh model based on the first deformation function to obtain a first deformed object mesh model.

[0172] In some embodiments, the parameter module 2552 is further used to: obtain the first-order derivative of the deformation function variable corresponding to each of the unit planes; for each of the unit planes, determine the first deformation error between the first-order derivative of the unit plane and the first weight parameter of the unit plane; and fuse the first deformation errors of multiple unit planes to obtain the deformation difference of the object mesh model.

[0173] In some embodiments, the rendering module 2553 is further used to: perform normal rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a normal rendering image; perform texture rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a texture rendering image; and combine the texture rendering image and the normal rendering image to form the first rendering object image.

[0174] In some embodiments, the rendering module 2553 is further configured to perform any one of the following processes: performing forward propagation of the object description text in the generation network of the adversarial generation network to obtain a generated object image that conforms to the object description text; performing forward propagation of the object description text and the noise image in the diffusion model to obtain a generated object image that conforms to the object description text.

[0175] In some embodiments, the diffusion model includes N cascaded denoising networks, where the value of N satisfies 2 ≤ N; the rendering module 2553 is further configured to: denoise the input of the n-th denoising network through the n-th denoising network in the N cascaded denoising networks, and transmit the n-th denoising result output by the n-th denoising network to the (n + 1)-th denoising network to continue denoising to obtain the (n + 1)-th denoising result corresponding to the (n + 1)-th denoising network; using the N-th denoising result output by the N-th denoising network as the generated object image that conforms to the object description text; where n is an integer variable starting from 1 and increasing, and the value range of n is 1 ≤ n < N. When n = 1, the input of the n-th denoising network is the noise image and the object description text. When 2 ≤ n < N, the input of the n-th denoising network is the (n - 1)-th denoising result output by the (n - 1)-th denoising network and the object description text.

[0176] In some embodiments, each denoising network includes M cascaded sampling networks, and each sampling network includes a residual layer and an attention layer; the rendering module 2553 is further configured to: extract residual features from the input of the m-th sampling network through the residual layer of the m-th sampling network in the n-th denoising network to obtain residual features; perform attention processing on the residual features and the object description text through the attention layer of the m-th sampling network in the n-th denoising network to obtain attention features, and use the attention features as the m-th sampling result of the m-th sampling network; transmit the m-th sampling result of the m-th sampling network to the (m + 1)-th sampling network to continue sampling to obtain the (m + 1)-th sampling result of the (m + 1)-th sampling network; using the M-th sampling result output by the M-th sampling network in the n-th denoising network as the n-th denoising result; where m is an integer variable starting from 1 and increasing, and the value range of m is 1 ≤ m ≤ M - 1. When m = 1, the input of the m-th sampling network is the (n - 1)-th denoising result. When 2 ≤ m < M, the input of the m-th sampling network is the (m - 1)-th sampling result output by the (m - 1)-th sampling network.

[0177] In some embodiments, the rendering module 2553 is further used to: perform query matrix-based mapping processing on the residual features to obtain an attention query matrix; perform key matrix-based mapping processing on the text encoding of the object description text to obtain an attention key matrix; perform value matrix-based mapping processing on the text encoding of the object description text to obtain an attention value matrix; multiply the attention query matrix with the transposed matrix of the attention key matrix to obtain a multiplication result, and obtain the ratio between the multiplication result and the dimension of the attention key matrix; perform maximum likelihood processing on the ratio, and multiply the maximum likelihood result with the attention value matrix to obtain the attention feature.

[0178] In some embodiments, the image generation process is implemented through a diffusion model, and the diffusion model includes N cascaded denoising networks, where the value of N satisfies 2≤N. The update module 2554 is further used to: obtain the image encoding corresponding to the first rendered object image; for each of the denoising networks, determine the image semantic loss corresponding to the denoising network based on the image encoding of the first rendered object image and the generated object image; and fuse the image semantic losses of multiple denoising networks to obtain the image semantic loss.

[0179] In some embodiments, the update module 2554 is further used to: obtain a first partial derivative of the input of the denoising network with respect to the image encoding, and a second partial derivative of the image encoding with respect to the deformation function variable; obtain a first difference between the output of the denoising network and the noisy image; and perform multiplication processing on the first difference, the first partial derivative, and the second partial derivative to obtain the image semantic loss corresponding to the denoising network.

[0180] In some embodiments, before updating the first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain the second weight parameter corresponding to each of the unit planes, the update module 2554 is further used to: determine the mesh model loss based on the first deformable object mesh model and the object mesh model; and update the first weight parameter corresponding to each of the unit planes based on the image semantic loss and the mesh model loss to obtain the second weight parameter corresponding to each of the unit planes.

[0181] In some embodiments, the update module 2554 is further used to: perform projection processing on the first deformable object mesh model to obtain a first object mask, and obtain the first object feature point of the first deformable object mesh model; perform projection processing on the object mesh model to obtain a second object mask, and obtain the second object feature point of the object mesh model; determine the feature point loss based on the difference between the first object feature point and the second object feature point; determine the contour loss based on the difference between the first object mask and the second object mask; and fuse the feature point loss and the contour loss to obtain the mesh model loss.

[0182] In some embodiments, the update module 2554 is further used to: update the first texture parameters based on the image semantic loss to obtain second texture parameters corresponding to the object mesh model; perform rendering processing based on the second deformable object mesh model and the second texture parameters to obtain a second rendered object image.

[0183] An embodiment of the present application provides a computer program product comprising computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the artificial intelligence-based data processing method described above in the embodiment of the present application.

[0184] The embodiment of the present application provides a computer-readable storage medium storing computer-executable instructions, wherein the computer-executable instructions are stored. When the computer-executable instructions are executed by a processor, the processor will execute the artificial intelligence-based data processing method provided by the embodiment of the present application, for example, Figures 4A-4D The data processing method based on artificial intelligence is shown.

[0185] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or various devices including one or any combination of the above memories.

[0186] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0187] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0188] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0189] In summary, through the embodiment of the present application, an object mesh model including multiple unit planes is obtained, and a first texture parameter corresponding to the object mesh model is obtained, and the object mesh model includes multiple unit planes; the first texture parameter corresponding to the object mesh model is obtained, and the object mesh model is deformed based on the first weight parameter corresponding to each unit plane to obtain a first deformed object mesh model. The weight parameter involved in the embodiment of the present application is based on the unit plane, that is, different unit planes correspond to different weight parameters, so that the deformation of the mesh model can be controlled locally and finely; rendering processing is performed based on the first deformed object mesh model and the first texture parameter to obtain a first rendered object image, and image generation processing is performed based on the object description text to obtain an image that conforms to the object description text. The object image is generated based on the first rendered object image and the generated object image, and the image semantic loss is determined, where the image obtained based on semantics and the rendered image constitute the image semantic loss, and the first weight parameter is updated based on the image semantic loss to obtain the second weight parameter corresponding to each unit plane, that is, the unit plane weight parameter can be constrained by using the image semantic loss, and the updated unit plane weight parameter can generate a rendered image that is closer to the text semantics; the object mesh model is deformed based on the second weight parameter corresponding to each unit plane to obtain a second deformed object mesh model, and finally the object mesh model can be controlled by the updated second weight parameter to perform more precise and accurate deformation in units of unit planes, thereby improving the modeling accuracy of the mesh model.

[0190] The above description is merely an embodiment of the present application and is not intended to limit the scope of protection of the present application. Any modifications, equivalent replacements, and improvements made within the spirit and scope of the present application are included in the scope of protection of the present application.

Claims

1. A data processing method based on artificial intelligence, characterized in that: The method comprises: Obtain an object mesh model including a plurality of unit planes, and obtain a first texture parameter corresponding to the object mesh model; deforming the object mesh model based on a first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model; Performing rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a first rendered object image, and performing image generation processing based on the object description text to obtain a generated object image that conforms to the object description text; Determining an image semantic loss based on the first rendered object image and the generated object image, and updating a first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain a second weight parameter corresponding to each of the unit planes; The object mesh model is deformed based on a second weight parameter corresponding to each of the unit planes to obtain a second deformed object mesh model.

2. The method according to claim 1, characterized in that The deforming the object mesh model based on the first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model includes: determining a deformation difference of the object mesh model based on a first weight parameter corresponding to each of the unit planes and a deformation function variable; Obtaining a solution result of a deformation function variable as a first deformation function, wherein the solution result makes the deformation difference converge to a minimum value; The object mesh model is deformed based on the first deformation function to obtain a first deformed object mesh model.

3. The method according to claim 2, characterized in that The determining of the deformation difference of the object mesh model based on the first weight parameter corresponding to each of the unit planes and the deformation function variable includes: Obtaining the first-order derivative of the deformation function variable corresponding to each of the unit planes; For each of the unit planes, determining a first deformation error between a first-order derivative of the unit plane and a first weight parameter of the unit plane; The first deformation errors of the plurality of unit planes are fused to obtain deformation differences of the object mesh model.

4. The method according to claim 1, wherein The performing rendering processing based on the first deformable object mesh model and the first texture parameter to obtain a first rendered object image includes: performing normal rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a normal rendered image; Performing texture rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a texture rendered image; The texture rendering image and the normal rendering image are combined into the first rendering object image.

5. The method according to claim 1, wherein The performing of image generation processing based on the object description text to obtain a generated object image that conforms to the object description text includes: Forward propagating the object description text in the generative network of the adversarial generative network to obtain a generated object image that conforms to the object description text; or The object description text and the noise image are forward propagated in a diffusion model to obtain a generated object image that conforms to the object description text.

6. The method according to claim 5, characterized in that The diffusion model includes N cascaded denoising networks, where the value of N satisfies 2≤N; Forward propagating the object description text and the noisy image in the diffusion model to obtain a generated object image that conforms to the object description text, including: Denosing the input of the n-th denoising network through the n-th denoising network in N cascaded denoising networks, and transmitting the n-th denoising result output by the n-th denoising network to the (n + 1)-th denoising network to continue denoising, obtaining the (n + 1)-th denoising result corresponding to the (n + 1)-th denoising network; Taking the N-th denoising result output by the N-th denoising network as the generated object image that conforms to the object description text; where n is an integer variable starting from 1 and increasing, the value range of n is 1 ≤ n < N. When n = 1, the input of the n-th denoising network is the noisy image and the object description text. When 2 ≤ n < N, the input of the n-th denoising network is the (n - 1)-th denoising result output by the (n - 1)-th denoising network and the object description text.

7. The method according to claim 6, characterized in that Each of the denoising networks includes M cascaded sampling networks, and each of the sampling networks includes a residual layer and an attention layer; Denosing the input of the n-th denoising network through the n-th denoising network in N cascaded denoising networks, including: Performing residual feature extraction processing on the input of the m-th sampling network in the n-th denoising network through the residual layer of the m-th sampling network in the n-th denoising network to obtain residual features; Performing attention processing on the residual features and the object description text through the attention layer of the m-th sampling network in the n-th denoising network to obtain attention features, and taking the attention features as the m-th sampling result of the m-th sampling network; Transmitting the m-th sampling result of the m-th sampling network to the (m + 1)-th sampling network to continue sampling processing, obtaining the (m + 1)-th sampling result of the (m + 1)-th sampling network; Taking the M-th sampling result output by the M-th sampling network in the n-th denoising network as the n-th denoising result; where m is an integer variable starting from 1 and increasing, the value range of m is 1 ≤ m ≤ M - 1. When m = 1, the input of the m-th sampling network is the (n - 1)-th denoising result. When 2 ≤ m < M, the input of the m-th sampling network is the (m - 1)-th sampling result output by the (m - 1)-th sampling network.

8. The method according to claim 7, characterized in that Performing attention processing on the residual features and the object description text through the attention layer of the m-th sampling network in the n-th denoising network to obtain attention features, including: Performing mapping processing on the residual features based on a query matrix to obtain an attention query matrix; Performing mapping processing on the text encoding of the object description text based on a key matrix to obtain an attention key matrix; Performing mapping processing on the text encoding of the object description text based on a value matrix to obtain an attention value matrix; Multiplying the attention query matrix by the transposed matrix of the attention key matrix to obtain a multiplication result, and obtaining the ratio between the multiplication result and the dimension of the attention key matrix; The ratio is subjected to maximum likelihood processing, and the maximum likelihood result is multiplied by the attention value matrix to obtain the attention feature.

9. The method according to claim 1, characterized in that The image generation process is implemented by a diffusion model, and the diffusion model includes N cascaded denoising networks, where the value of N satisfies 2≤N. The image semantic loss is determined based on the first rendered object image and the generated object image, including: Obtaining an image code corresponding to the first rendering object image; For each of the denoising networks, determining an image semantic loss corresponding to the denoising network based on the image encoding of the first rendered object image and the generated object image; The image semantic loss of the multiple denoising networks is fused to obtain the image semantic loss.

10. The method according to claim 9, characterized in that The determining, based on the image encoding of the first rendered object image and the generated object image, an image semantic loss corresponding to the denoising network, comprises: Obtaining a first partial derivative of an input of the denoising network with respect to the image code, and a second partial derivative of the image code with respect to a deformation function variable; Obtaining a first difference between an output of the denoising network and a noisy image; The first difference, the first partial derivative, and the second partial derivative are multiplied to obtain an image semantic loss corresponding to the denoising network.

11. The method according to claim 1, characterized in that Before updating the first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain the second weight parameter corresponding to each of the unit planes, the method further includes: Determining a mesh model loss based on the first deformable object mesh model and the object mesh model; The updating of the first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain the second weight parameter corresponding to each of the unit planes includes: Based on the image semantic loss and the grid model loss, a first weight parameter corresponding to each of the unit planes is updated to obtain a second weight parameter corresponding to each of the unit planes.

12. The method according to claim 11, characterized in that The determining of mesh model loss based on the first deformable object mesh model and the object mesh model includes: Performing projection processing on the first deformable object mesh model to obtain a first object mask, and acquiring first object feature points of the first deformable object mesh model; Performing projection processing on the object mesh model to obtain a second object mask, and acquiring second object feature points of the object mesh model; determining a feature point loss based on a difference between the first object feature point and the second object feature point; determining a contour loss based on a difference between the first object mask and the second object mask; The feature point loss and the contour loss are fused to obtain the mesh model loss.

13. The method according to claim 1, wherein The method further comprises: updating the first texture parameter based on the image semantic loss to obtain a second texture parameter corresponding to the object mesh model; Rendering is performed based on the second deformable object mesh model and the second texture parameters to obtain a second rendered object image.

14. A data processing device based on artificial intelligence, characterized in that: The device comprises: A model module, configured to obtain an object mesh model comprising a plurality of unit planes, and obtain a first texture parameter corresponding to the object mesh model; a parameter module, configured to deform the object mesh model based on a first weight parameter corresponding to each of the unit planes to obtain a first deformed object mesh model; a rendering module configured to perform rendering processing based on the first deformable object mesh model and the first texture parameters to obtain a first rendered object image, and to perform image generation processing based on the object description text to obtain a generated object image that conforms to the object description text; an updating module, configured to determine an image semantic loss based on the first rendered object image and the generated object image, and update a first weight parameter corresponding to each of the unit planes based on the image semantic loss to obtain a second weight parameter corresponding to each of the unit planes; The deformation module is used to deform the object mesh model based on the second weight parameter corresponding to each unit plane to obtain a second deformed object mesh model.

15. An electronic device, characterized in that: The electronic device comprises: a memory for storing computer-executable instructions; A processor, configured to implement the artificial intelligence-based data processing method according to any one of claims 1 to 13 when executing computer-executable instructions stored in the memory.

16. A computer-readable storage medium storing computer-executable instructions, characterized in that: When the executable instructions are executed by the processor, the artificial intelligence-based data processing method according to any one of claims 1 to 13 is implemented.

17. A computer program product comprising computer executable instructions, characterized in that When the computer executable instructions are executed by a processor, the artificial intelligence-based data processing method according to any one of claims 1 to 13 is implemented.

Citation Information

Cited By

  • Medical image registration method and system based on language large model prompt

    CN120747182A

  • Medical image registration method and system based on language large model prompt

    CN120747182B