Image generation method and device, electronic equipment, computer readable storage medium and computer program product

Abstract

The invention provides an image generation method and device, electronic equipment, a computer readable storage medium and a computer program product. The method comprises the following steps: projecting a first vertex of a first three-dimensional model of a first object to a first texture map to obtain a first texture coordinate; projecting a second vertex of a second three-dimensional model of a second object to the second texture map to obtain a second texture coordinate; determining a second texture coordinate corresponding to each first texture coordinate; generating a dyed second texture map based on the second texture coordinate and the color data of the first texture map; performing editing processing on the dyed second texture image based on the first description text to obtain a dyed third texture image; and performing rendering processing based on the dyed third texture map and the second three-dimensional model to obtain an image of a third object. According to the invention, the image generation efficiency and accuracy can be improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of image processing, and in particular to an image generation method and device, electronic equipment, computer readable storage medium and computer program product. BACKGROUND

[0002] In related technologies, a three-dimensional model with texture is collected, a BlendShape binding is manually added to the three-dimensional model, and a digital human image is generated by using a rendering engine for rendering processing. The related technologies have low execution efficiency and low accuracy. SUMMARY

[0003] The embodiments of the present application provide an image generation method and device, electronic equipment, computer readable storage medium and computer program product, which can improve the image generation efficiency and accuracy.

[0004] The technical solutions of the embodiments of the present application are as follows:

[0005] The embodiments of the present application provide an image generation method, comprising:

[0006] projecting a plurality of first vertices of a first three-dimensional model of a first object to a first texture map of the first three-dimensional model to obtain a first texture coordinate corresponding to each of the first vertices;

[0007] projecting a plurality of second vertices of a second three-dimensional model of a second object to a second texture map of the second three-dimensional model to obtain a second texture coordinate of each of the second vertices;

[0008] determining a second texture coordinate corresponding to each of the first texture coordinates, wherein the correspondence between the first texture coordinates and the second texture coordinates is the same as the correspondence between the first vertices and the second vertices in a deformation, and the deformation is a deformation from the first three-dimensional model to the second three-dimensional model;

[0009] generating a dyed second texture map based on the second texture coordinate corresponding to each of the first texture coordinates and color data of the first texture map;

[0010] editing the dyed second texture map based on a first description text to obtain a dyed third texture map;

[0011] performing rendering processing based on the dyed third texture map and the second three-dimensional model to obtain an image of a third object.

[0012] The embodiments of the present application provide an image generation device, comprising:

[0013] The projection module is configured to project a plurality of first vertices of a first three-dimensional model of a first object to a first texture map of the first three-dimensional model to obtain a first texture coordinate corresponding to each of the first vertices;

[0014] The projection module is further configured to project a plurality of second vertices of a second three-dimensional model of a second object to a second texture map of the second three-dimensional model to obtain a second texture coordinate of each of the second vertices;

[0015] The determination module is configured to determine a second texture coordinate corresponding to each of the first texture coordinates, wherein a correspondence relationship between the first texture coordinate and the second texture coordinate is the same as a correspondence relationship between the first vertex and the second vertex in a deformation, and the deformation is a deformation from the first three-dimensional model to the second three-dimensional model;

[0016] The generation module is configured to generate a dyed second texture map based on the second texture coordinate corresponding to each of the first texture coordinates and color data of the first texture map;

[0017] The editing module is configured to perform editing processing on the dyed second texture map based on a first description text to obtain a dyed third texture map;

[0018] The rendering module is configured to perform rendering processing based on the dyed third texture map and the second three-dimensional model to obtain an image of a third object.

[0019] An electronic device is provided in an embodiment of the present application, and the electronic device comprises:

[0020] A memory is configured to store computer executable instructions or computer programs;

[0021] A processor is configured to execute the computer executable instructions or computer programs stored in the memory to implement the image generation method provided in the embodiments of the present application.

[0022] A computer readable storage medium is provided in an embodiment of the present application, and the computer readable storage medium stores computer programs or computer executable instructions, and is configured to be executed by a processor to implement the image generation method provided in the embodiments of the present application.

[0023] A computer program product is provided in an embodiment of the present application, and the computer program product comprises computer programs or computer executable instructions, and the computer programs or computer executable instructions are executed by a processor to implement the image generation method provided in the embodiments of the present application.

[0024] The embodiments of the present application have the following beneficial effects:

[0025] First, the vertices of the first three-dimensional model of the first object are projected onto the first texture map to obtain first texture coordinates, and the vertices of the second three-dimensional model of the second object are projected onto the second texture map to obtain second texture coordinates, and the correspondence between the texture coordinates in the deformation of the two models is determined, so that a dyed second texture map can be generated based on the second texture coordinates corresponding to the first texture coordinates and the color data of the first texture map, so as to automatically migrate the color data of the first texture map to the second texture map, thereby realizing the transmission of the color information of the texture map; then, the dyed second texture map is edited based on the first description text to obtain a dyed third texture map, and the details of the texture map are adjusted according to the content of the description text, thereby enhancing the visual expressiveness of the texture map; finally, the image of the third object is obtained based on the dyed third texture map and the second three-dimensional model. This can ensure that the generated image meets specific requirements, and no artificial dyeing is required in this process, which can improve the generation efficiency of the object image. BRIEF DESCRIPTION OF THE DRAWINGS

[0026] Figure 1 1 is a schematic diagram of the architecture of an image generation system 100 provided in an embodiment of the present application;

[0027] Figure 2 is a structural diagram of an electronic device 500 provided in an embodiment of the present application;

[0028] Figure 3 Schematic diagram of the process of generating an image according to an embodiment of the present invention;

[0029] Figure 4 Schematic diagram of the process of generating an image according to an embodiment of the present invention;

[0030] Figure 5 This is a schematic diagram of the process of the image generation method provided in an embodiment of the present application;

[0031] Figure 6 This is a schematic diagram of key points of a 3D model and a 2D texture map provided in an embodiment of the present application;

[0032] Figure 7 is a schematic diagram of a texture map of a 3D model provided in an embodiment of the present application;

[0033] Figure 8 The batch texture map and the texture map with enhanced decoration provided in the embodiments of the present application are:

[0034] Figure 9 Schematic diagram of the model training structure of the discriminator provided in an embodiment of the present application;

[0035] Figure 10 Schematic diagram of the model training structure of the text editing model provided in the embodiment of the present application;

[0036] Figure 11 is a schematic diagram of automatically migrating and performing texture enhancement on a texture map provided by an embodiment of the present application;

[0037] Figure 12 is a schematic diagram of automatically converting a texture map to a target model provided by an embodiment of the present application;

[0038] Figure 13 is a schematic diagram of performing decorative texture enhancement on a texture map provided by an embodiment of the present application. DETAILED DESCRIPTION

[0039] In order to make the purposes, technical solutions and advantages of the present application clearer, the present application will be described in further detail below with reference to the drawings, and the described embodiments should not be regarded as limiting the present application, and all other embodiments obtained by those of ordinary skill in the art without making creative efforts fall within the scope of protection of the present application.

[0040] In the following description, "some embodiments" are related to a subset of all possible embodiments, but it can be understood that "some embodiments" can be the same subset or different subsets of all possible embodiments, and can be combined with each other without conflict.

[0041] It can be understood that in the embodiments of the present application, data related to user information and the like are involved, and when the embodiments of the present application are applied to specific products or technologies, user permission or consent needs to be obtained, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards.

[0042] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined target, and can be implemented entirely or partially by using software, hardware (such as a processing circuit or a memory) or a combination thereof. Similarly, one processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be a part of an overall module or unit that includes the functions of the module or unit.

[0043] In the following description, the term "first\second\..." involved is only to distinguish similar objects, and does not represent a specific order of the objects. It can be understood that "first\second\..." can be interchanged with a specific order or sequence as allowed, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0044] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used in the description herein is for describing the embodiments of the application only and is not intended to be limiting of the application.

[0045] Before the embodiments of the application are further described in detail, the terms and phrases involved in the embodiments of the application are explained, and the terms and phrases involved in the embodiments of the application are applicable to the following explanations.

[0046] 1) 3D Morphable Model (3DMM): 3DMM is a mathematical model used for three-dimensional face representation and generation in the field of computer vision and image science, widely used in face modeling tasks.

[0047] 2) UV (Texture Coordinate System): In a three-dimensional model, the UV coordinate system is used to map a two-dimensional texture image to the surface of a three-dimensional model. Through UV coordinates, each vertex of a three-dimensional model can be associated with a pixel (texture) of a two-dimensional image, so that the texture details can be displayed on the model.

[0048] 3) BlendShape: BlendShape is a technique used in computer graphics to control and animate three-dimensional models. By interpolating a model (usually a static model) with multiple target shapes, the model can smoothly transition to the target shape, achieving dynamic deformation.

[0049] 4) PCA-based 3D model: PCA-based 3D model is a mathematical model that uses PCA method for data conversion and dimensionality reduction. It mainly finds the most important features (i.e. principal components) in the data and projects the data onto these principal components, thereby achieving dimensionality reduction.

[0050] 5) Camera extrinsic parameters: Camera extrinsic parameters are used to describe the position and orientation of the camera relative to the world coordinate.

[0051] 6) Camera intrinsic parameters: Camera intrinsic parameters include focal length and other performance parameters of the camera device itself.

[0052] 7) Key coordinates: The 68 key points in the image are two-dimensional and three-dimensional, providing training data for the algorithm that regresses the 68 key points of the head from the image.

[0053] In the related art, by collecting three-dimensional models with textures, manually adding BlendShape binding to the three-dimensional models, and using a rendering engine to render and process new three-dimensional models, the execution efficiency and accuracy of the related art are low.

[0054] Based on this, the embodiment of the present application provides an image generation method, device, electronic equipment, computer readable storage medium and computer program product, which can improve the image generation efficiency and accuracy. The electronic equipment provided by the embodiment of the present application is described below. The electronic equipment provided by the embodiment of the present application can be implemented as a server, or as a terminal, or can be implemented by a server and a terminal in cooperation. The image generation method provided by the embodiment of the present application is taken as an example for description.

[0055] For example, referring to Figure 1 , Figure 1 is the architecture diagram of the image generation system 100 provided by the embodiment of the present application, which is used to support an image generation application, such as Figure 1 As shown in the figure, the image generation system 100 includes a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 through the network 300. The network 300 can be a local area network or a wide area network, or a combination of the two.

[0056] In some embodiments, the user inputs the first three-dimensional model of the first object, the second three-dimensional model of the second object, and the first description text through the terminal 400, and transmits the first three-dimensional model, the second three-dimensional model, and the first description text to the server 200 through the network 300; then, the server 200 determines the first texture coordinates of each vertex of the first three-dimensional model and the second texture coordinates of each vertex of the second three-dimensional model; then, the server 200 determines the second texture coordinates corresponding to the first texture coordinates of each vertex; then, based on the second texture coordinates corresponding to the first texture coordinates and the color data of the first texture map of the first three-dimensional model, a dyed second texture map is generated; then, the server 200 edits the dyed second texture map according to the first description text to obtain a dyed third texture map, and performs rendering processing according to the dyed third texture map and the second three-dimensional model to obtain an image of the third object; finally, the server 200 transmits the image of the third object to the terminal 400 through the network 300, and displays it on the terminal 400.

[0057] The image generation method provided by the embodiment of the present application can be applied to various scenes that need to generate object images, for example, it can be applied to the following scenes:

[0058] 1) Game development scenario, taking a role-playing game as an example, assuming that the first three-dimensional model is an angry expression and the second three-dimensional model is a neutral expression, projecting the key vertices (such as the vertices of the eyes and the mouth) of the first three-dimensional model onto the corresponding texture map to obtain the texture coordinates; then, determining the corresponding positions of the vertices in the second three-dimensional model through a morphing algorithm (such as an energy-based morphing model); subsequently, coloring the texture map of the second three-dimensional model using the color data of the texture map of the first three-dimensional model, so that the character in the neutral expression presents an angry expression; then, adjusting the colored texture map according to the text description requirements of the game designer, for example, adding some red shadows to the expression; finally, rendering the processed texture map through a rendering engine to obtain an object image, thereby generating different object images required by the character in the game.

[0059] 2) Movie post-production scenario, taking movie production as an example, assuming that there is a high-resolution three-dimensional model texture map of an actor in a specific expression and a three-dimensional model of a character in the movie that needs to be replaced, mapping the texture coordinates corresponding to the vertices of the three-dimensional model of the actor to the three-dimensional model of the character in the movie through mapping technology; then, ensuring that the correspondence of the texture coordinates between the two models remains consistent through a matching algorithm, coloring the texture map of the three-dimensional model using the color data of the texture map of the actor in the specific expression, to generate a colored texture map of the three-dimensional model; subsequently, adjusting the colored texture map of the three-dimensional model according to the specific scene requirements of the character in the movie, to obtain an adjusted texture map, for example, adding costume textures; finally, applying the adjusted texture map to the three-dimensional model of the character in the movie and performing rendering processing to generate an object image, to replace the original character and bring a more realistic visual experience to the audience.

[0060] 3) Virtual reality scenario, taking a virtual reality (VR) experience as an example, assuming that there is a three-dimensional scan model of a user and a corresponding texture map, as well as a default three-dimensional model in the VR environment; then, determining the texture coordinates corresponding to each vertex in the two three-dimensional models through mapping technology, and determining the correspondence of the texture coordinates in the two models; subsequently, generating a colored texture map of the default three-dimensional model in the VR environment according to the correspondence and the color data of the corresponding texture map of the three-dimensional scan model of the user; then, performing special effect processing on the colored texture map of the default three-dimensional model in the VR environment according to the individualized requirements of the user, for example, adding tattoos or changing hair color; finally, applying the processed texture map to the three-dimensional model of the VR environment and generating an object image through a rendering engine, to make the VR experience more immersive.

[0061] The structure of the electronic device provided in the embodiments of the present application will be further described below. Taking the electronic device as a terminal as an example, referring to Figure 2 , Figure 2FIG. 1 is a schematic diagram of an example of an electronic device 500, Figure 2 The electronic device 500 shown includes at least one processor 510, memory 550, at least one network interface 520, and a user interface 530. The various components of the electronic device 500 are coupled together by a bus system 540, which is configured to facilitate the communication of information between each of the components. The bus system 540 can be implemented with a series of buses and wires, which are used to directly or indirectly couple the various components together. Figure 2 For the sake of clarity, the various buses are shown as a single bus system 540 in

[0062] The processor 510 can be an integrated circuit chip located on a bus, with the capability to process computer program instructions. In this example, the processor 510 can be a general-purpose processor, a Digital Signal Processor (DSP), or other programmable logic device, discrete gate or transistor logic, discrete hardware components, or any combination thereof designed to process computer program instructions. The general-purpose processor can be a microprocessor or any conventional processor, etc.

[0063] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532 that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0064] The memory 550 can be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, and the like. The memory 550 optionally includes one or more storage devices physically located in proximity to the processor 510.

[0065] The memory 550 includes volatile memory or non-volatile memory, or both. Non-volatile memory can be read only memory (ROM), volatile memory can be random access memory (RAM). The memory 550 described herein is intended to include any suitable type of memory.

[0066] In some embodiments, the memory 550 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or a subset or superset thereof, which are described below.

[0067] The operating system 551 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks;

[0068] The network communication module 552 is configured to reach other computing devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 including Bluetooth, wireless fidelity (WiFi), universal serial bus (USB), and the like;

[0069] The presentation module 553 is configured to enable presentation of information via one or more output devices 531 (e.g., a display screen, a speaker, and the like) associated with the user interface 530 (e.g., a user interface for operating peripheral devices and displaying content and information);

[0070] The input processing module 554 is configured to detect and interpret one or more user inputs or interactions from one or more input devices 532.

[0071] In some embodiments, the apparatus provided by the present application can be implemented in software, Figure 2 An image generation apparatus 555 stored in the memory 550 is shown, which can be in the form of programs and plug-ins and the like, including the following software modules: a projection module 5551, a determination module 5552, a generation module 5553, an editing module 5554, a rendering module 5555, and a training module 5556. These modules are logical, and thus can be combined or further split according to the implemented functions. It should be noted that, in order to facilitate expression, all the above modules are shown at one time, but should not be regarded as excluding the implementation that only includes the projection module 5551, the determination module 5552, the generation module 5553, the editing module 5554, and the rendering module 5555 in the image generation apparatus 555. The functions of each module will be described below. Figure 2

[0072] The image generation method provided by the present application will be described in detail below in combination with the exemplary application and implementation of the terminal provided by the present application.

[0073] Referring to Figure 3 , Figure 3 is a flowchart of the image generation method provided by the present application, which will be described in combination with the steps shown in Figure 3 .

[0074] It should be noted that, Figure 3 ​The method shown can be executed by various forms of computer programs run by the terminal, and is not limited to the client, for example, can also be the operating system, software module, script, applet and the like described above, therefore, the following examples of the client should not be regarded as a limitation of the embodiments of the present application. In addition, for the convenience of description, the terminal and the client run by the terminal are not specifically distinguished in the following.

[0075] In step 101, a plurality of first vertices of a first three-dimensional model of a first object are projected to a first texture map of the first three-dimensional model, to obtain a first texture coordinate corresponding to each of the first vertices.

[0076] As an example, assuming that the first three-dimensional model (taking a three-dimensional head model as an example) is a simple cube shape representing the basic structure of the head, and the corresponding texture map is a two-dimensional image containing facial details, the three-dimensional head model contains a plurality of vertices, assuming that a vertex is located at the center position of the eye, denoted as vertex A, the corresponding position of vertex A on the corresponding texture map is determined, to obtain the first texture coordinate of vertex A in the texture coordinate system; the above process is repeated for each vertex in the head model to obtain the first texture coordinate corresponding to each vertex.

[0077] In step 102, a plurality of second vertices of a second three-dimensional model of a second object are projected to a second texture map of the second three-dimensional model, to obtain a second texture coordinate of each of the second vertices.

[0078] It should be noted that the implementation of step 102 is similar to the implementation of step 101 described above, and the implementation of step 101 can be referred to for details, which will not be repeated here.

[0079] In some embodiments, before step 102 is executed, the following processing can also be performed: based on the respective weights corresponding to the basic three-dimensional models of each local part, the basic three-dimensional models of the local parts are fused, and the fusion result is superimposed with the average three-dimensional model to obtain the second three-dimensional model, wherein the weights correspond to the second object. In this way, by adjusting the weights, a three-dimensional model conforming to the characteristics of a specific individual can be created, realizing the individualization of the three-dimensional model; through the automatic fusion process, the workload of manual modeling can be reduced, and the modeling efficiency can be improved; and in the fusion process, important features of the model can be preserved through weight adjustment, avoiding the loss of details when merging the model; in addition, the weight-based model fusion method can be better extended to different numbers of basic three-dimensional models and different weight strategies to adapt to more complex modeling requirements, having high adaptability and scalability.

[0080] As an example, assume there is a standard virtual character head model, the model is segmented into several local regions, such as left eye, right eye, nose, mouth, etc.; each local region has a base 3D model, these base 3D models are pre-designed according to different features, such as different eye sizes, nose shapes, etc.; then, for each local base 3D model, a set of weights is defined, the weights can be user input based parameters or preset parameters, for example, the user can choose a large nose or a small nose, if the user chooses a large nose feature, the nose local base 3D model will have a higher weight, and the weights of other regions remain default; then, the local base 3D models are fused, for example, for the nose region, since the user chooses a large nose, the nose base 3D model has a higher weight and contributes more to the fusion result, for the eye region, the user has no special choice, so the eye base 3D model weight remains the default value, and the influence on the fusion result is relatively small, through weight distribution, each local base 3D model will affect the final head model according to its weight, the application of weights may cause some features to be emphasized, while other features remain unchanged or are weakened; then, the fused head model is superimposed with an average 3D model to obtain a second 3D model, the average 3D model is a standard, neutral head shape, providing a basis for the fused model, here, during the superimposition process, each vertex position of the fused model is combined with the corresponding vertex position of the average model.

[0081] In step 103, a second texture coordinate corresponding to each of the first texture coordinates is determined.

[0082] Here, the correspondence between the first texture coordinates and the second texture coordinates is the same as the correspondence between the first vertex and the second vertex in the deformation, the deformation being the deformation of the first 3D model to the second 3D model.

[0083] In some embodiments, the step 103 described above can be implemented by: generating a plurality of first polygonal faces and a plurality of first edges based on the first texture coordinates of the plurality of first vertices, and generating a plurality of second polygonal faces and a plurality of second edges based on the second texture coordinates of the plurality of second vertices; determining a matching degree between each of the first texture coordinates and the corresponding second texture coordinates to be solved based on each of the first texture coordinates and the corresponding second texture coordinates to be solved; determining a first difference of a deformation gradient between each of the first polygonal faces and the corresponding second polygonal faces to be solved based on each of the first texture coordinates and the corresponding second texture coordinates to be solved; determining a second difference between each of the first edges and the corresponding second edges to be solved based on each of the first texture coordinates and the corresponding second texture coordinates to be solved; and fusing the matching degree between each of the first texture coordinates and the corresponding second texture coordinates to be solved, the first difference of the deformation gradient between each of the first polygonal faces and the corresponding second polygonal faces to be solved, and the second difference between each of the first edges and the corresponding second edges to be solved to obtain the energy function of the deformation. Each of the corresponding second texture coordinates to be solved of each of the first texture coordinates is solved to obtain the corresponding second texture coordinates of each of the first texture coordinates when the energy function of the deformation converges to a minimum value. In this way, by optimizing the correspondence of the texture coordinates, the texture can be more accurately mapped to the deformed model, thereby improving the realism of the rendering effect; the optimized texture coordinates and deformation can adapt to different animations and expressions to improve the applicability of the model; and by optimizing the texture mapping of the three-dimensional model during deformation, the consistency and realism of the appearance of the model under complex deformation can be ensured.

[0084] It should be noted that the energy function described above is a smooth energy function, mainly refers to an energy function used in the fields of optimization problems, machine learning, and computer vision, etc., has the properties of continuity and smoothness, can balance the deformation gradient consistency of the triangular face composed of the key points, the accurate matching of the key points of the texture map, and the smoothness of the texture map edge, and by minimizing the energy function, the corresponding second texture coordinates of each first texture coordinate can be determined.

[0085] As an example, for the first three-dimensional model, a series of polygonal faces and edges are generated according to the texture coordinates of the vertices, constituting the geometric structure of the initial model, and for the second three-dimensional model, polygonal faces and edges are generated according to the texture coordinates of the vertices; then, the corresponding texture coordinates of each texture coordinate in the first three-dimensional model in the second three-dimensional model are found, which can be determined by comparing similar texture features (such as color patterns, patterns, etc.) in the two models; then, the deformation gradient difference between the corresponding polygonal faces in the first three-dimensional model and the second three-dimensional model is compared, which reflects the bending degree or stretching degree of the polygonal face, to determine the change of the texture in the model deformation process; then, the difference between the corresponding edges in the first three-dimensional model and the second three-dimensional model is compared, where the difference can be the length change or direction change of the edge, to determine the change of the model edge in the deformation process; finally, the matching degree, the first difference of the deformation gradient and the second difference of the edge are combined to obtain an energy function, and the texture coordinates in the second three-dimensional model are adjusted by an optimization algorithm to minimize the energy function, thereby obtaining the second texture coordinates, where the energy function represents the total cost of texture mapping, and the optimization process finds the best texture coordinate position.

[0086] In step 104, a dyed second texture map is generated based on the corresponding second texture coordinates of each of the first texture coordinates and the color data of the first texture map.

[0087] As an example, assuming that the texture coordinates of the eye region in the first three-dimensional model are mapped to a specific position on the texture map, with specific color and details; in the second three-dimensional model, the texture coordinates of the eyes have been updated to reflect the new position information of the eyes; when generating the second texture map, the color data of the eye region is extracted from the corresponding texture map of the first three-dimensional model according to the new texture coordinates of the eyes in the second three-dimensional model, and is applied to the corresponding position on the corresponding texture map (i.e. the second texture map) of the second three-dimensional model; finally, the second texture map contains updated color information, obtaining the dyed second texture map, thereby mapping the color information of the texture map of the first three-dimensional model to the texture map on the second three-dimensional model, to ensure the consistency and authenticity of the texture in the deformation process, making the animation expression more vivid and realistic.

[0088] In step 105, the dyed second texture map is edited based on the first description text to obtain a dyed third texture map.

[0089] As an example, assuming that the dyed second texture map contains basic color and pattern information, and it is desired to edit the dyed second texture map, for example, to add a tattoo or change the color of facial features, etc.; then, according to the first description text, it is determined that the content that needs to be edited in the second texture map is, for example, to add a tattoo; after the editing is completed, the edited texture map is saved to obtain a dyed third texture map, where the dyed third texture map contains changes to the texture map during the editing process. By editing the texture map to enhance its texture features, a three-dimensional model with more personalization and artisticity can be created.

[0090] In some embodiments, referring to Figure 4 , Figure 4 is a flowchart of an image generation method provided by an embodiment of the present application, as shown in Figure 4 , before step 105 shown in Figure 3 is executed, steps 107 to 108 shown in Figure 4 may also be executed to implement, which will be described in combination with the steps shown in Figure 4 .

[0091] In step 107, the initialized first network is first trained based on a first loss for constraining texture quality, to obtain the first network after the first training.

[0092] In some embodiments, the step 107 described above can also be implemented in the following manner: the following processing is performed by the initialized first network: editing processing is performed on a dyed fourth texture map of a fourth object based on a second description text, to obtain a dyed fifth texture map; random damage processing is performed on the fourth texture map, to obtain a damaged fourth texture map, and texture quality evaluation is performed on the fifth texture map based on the damaged fourth texture map, to obtain a first quality score of the fifth texture map; a first loss negatively correlated with the first quality score is determined, and parameter updating is performed on the initialized first network based on the first loss, to obtain the first network after the first training. In this way, by performing damage processing and texture quality evaluation on the texture, the first network can learn to process more complex and diverse inputs, thereby enhancing the robustness of the first network to noise and abnormalities; in addition, by introducing a loss function and iteratively updating the first network parameters based on the loss function, the fidelity and quality of the texture generated by the first network can be gradually improved, to optimize the performance of the first network itself, thereby generating higher quality texture images.

[0093] As an example, assuming that the structure of the first network is a convolutional neural network that can understand and generate image textures, first, the dyed fourth texture map of the fourth object (for example, the head texture map of the user) and the corresponding second description text (for example, "the pupil color of the eyes is red") are input into the initialized first network. The initialized first network parses the second description text to determine the texture content to be edited, and edits the dyed fourth texture map of the fourth object according to the texture content to be edited, adjusts its color, stroke and details, adjusts the pupil color of the eyes to red, and obtains a dyed fifth texture map. Then, random noise or damage such as scratches, color blocks, blurring, etc. is added to the fourth texture map to generate a damaged fourth texture map. Then, the fifth texture map is compared with the damaged fourth texture map to evaluate the texture quality of the fifth texture map, thereby obtaining a first quality score of the fifth texture map. Then, a first loss negatively correlated with the first quality score is calculated, wherein the first loss reflects the difference between the texture map generated by the first network and the expectation. Finally, based on the calculated first loss, the parameters of the initialized first network are updated using gradient descent or other optimization algorithms. Through multiple iterations, the network gradually learns how to generate better stylized texture maps according to the text description until the desired effect is achieved, thereby obtaining the first network after the first training.

[0094] This illustrates the backpropagation. The training sample data is input into the input layer of the neural network, passes through the hidden layer, and finally reaches the output layer and outputs the result, which is the forward propagation process of the neural network. Since there is an error between the output result of the neural network and the actual result, the difference between the generated texture map and the expectation is calculated, and the difference is propagated from the output layer to the hidden layer until it propagates to the input layer. In the process of backpropagation, the values of the network parameters are adjusted according to the difference. The above process is iterated continuously until convergence.

[0095] In some embodiments, the texture quality evaluation is performed by a trained second network, and before step 107, the following processing can also be performed: performing damage processing on the first region in the ninth texture map to obtain a tenth texture map; performing texture quality evaluation on the tenth texture map based on the ninth texture map by the initialized second network to obtain a third quality score of the tenth texture map; mapping a first ratio between the area of the damaged region of the ninth texture map and the overall area of the ninth texture map to a cosine value interval to obtain a second ratio; determining a third loss based on the difference between the third quality score and the second ratio; updating the initialized second network based on the third loss to obtain the trained second network, wherein the trained second network is used to perform the texture quality evaluation. In this way, by performing random damage processing on the image region of the texture map, the training sample required for training the second network is constructed, so that the texture quality evaluation result of the second network is more accurate; mapping the ratio between the area of the damaged region and the overall area to the cosine interval to obtain the second ratio can make the second network more flexible to handle different sizes of damage, and calculating the loss based on the difference between the quality score and the second ratio can make the second network learn and improve the output result, thereby improving the accuracy of the texture quality evaluation result of the second network.

[0096] As an example, assuming that the second network is a multi-layer texture discriminator network constructed based on a convolutional neural network (CNN), first, the original sample image (ninth texture map) is input into the initialized second network, the initialized second network performs random damage processing on the texture region in the picture, for example, adding more noise, scratches or blur, etc., to obtain a tenth texture map; then, the initialized second network is used to perform texture quality evaluation on the damaged tenth texture map to obtain the quality score corresponding to the tenth texture map; subsequently, the ratio between the damaged area of the ninth texture map and the overall area of the ninth texture map is calculated and mapped to the cosine value interval to obtain the second ratio as the reference true value; then, according to the difference between the quality score and the second ratio, the corresponding loss is determined; finally, based on the loss, the parameters of the second network are updated, and through multiple iterations, until the loss reaches a preset value or the number of iterations reaches a preset value, so that the quality score of the second network is more accurate.

[0097] In step 108, the first network is second trained based on the first loss and a second loss for constraining the reconstruction quality to obtain a second trained first network.

[0098] Here, the first network after the second training is used to be called to perform editing processing on the dyed second texture map based on the first description text, to obtain a dyed third texture map.

[0099] In some embodiments, the step 108 described above can be implemented by performing the following processing through the first network after the first training: performing editing processing on a dyed sixth texture map of a fifth object based on a third description text to obtain a dyed seventh texture map; performing random destruction processing on the sixth texture map to obtain a destroyed sixth texture map, and performing texture quality evaluation on the dyed seventh texture map based on the destroyed sixth texture map to obtain a second quality score of the dyed seventh texture map; determining a first loss negatively correlated with the second quality score, and determining a second loss based on a pixel difference between the dyed seventh texture map and an eighth texture map, wherein the eighth texture map is a texture map matched with the third description text and the dyed sixth texture map; and performing parameter updating on the first network after the first training based on a fusion loss of the first loss and the second loss to obtain the first network after the second training. In this way, by customizing the texture through the text description, the first network after the training can be made to generate a satisfactory texture image according to the text description, thereby improving the satisfaction of the user; and by training through the fusion loss of the first loss and the second loss, the first network can generate a more high-quality and more natural texture editing result; in addition, by comprehensively considering the first loss of the texture quality and the second loss of the pixel difference, the first network can reduce the deviation from the target texture map while maintaining the texture style, thereby improving the overall performance of the first network.

[0100] As an example, assuming that the structure of the first network is a convolutional neural network that can understand and generate image textures, first, the dyed sixth texture map of the fifth object (for example, the head texture map of the player) and the corresponding third description text (for example, "ears add earrings") are input into the first network that has been first trained, the first network that has been first trained parses the third description text to determine the texture content to be edited, edits the sixth texture map according to the texture content to be edited, adds earrings to the ears, and obtains a dyed seventh texture map; then, random noise or damage is added to the sixth texture map to generate a damaged sixth texture map; subsequently, the seventh texture map is subjected to texture quality assessment to obtain a second quality score of the seventh texture map; then, a first loss that is negatively correlated with the second quality score is calculated, the first loss reflects the difference between the texture map generated by the network and the expectation, and a second loss that is used to describe the texture map reconstruction quality is determined according to the pixel difference between the dyed seventh texture map and the eighth texture map; finally, the first loss and the second loss are fused by weighting to obtain the total loss of the network, and the parameters of the first network that has been first trained are updated by back propagation through gradient descent or other optimization algorithms according to the total loss, the network gradually learns how to better edit the texture map according to the user description through multiple iterations, until the desired effect is achieved, thereby obtaining the first network that has been second trained.

[0101] In this way, the first loss is used to constrain the texture quality, which can avoid generating unnatural textures, the first network is first trained based on the first loss, so that the first network can generate textures with higher quality and more in line with expectations; the second loss is used to constrain the reconstruction quality, which can make the first network generate more realistic textures, and the first network is second trained based on the first loss and the second loss, which can make the first network find a balance point between the generation quality and the reconstruction quality of the texture, so that the first network generates textures with high visual quality.

[0102] In step 106, rendering processing is performed based on the dyed third texture map and the second three-dimensional model to obtain an image of the third object.

[0103] As an example, taking a game scene as an example, assuming that some unique tattoos and facial features are added to the dyed third texture map, and the dyed third texture map is applied to the head model of the virtual character, here, the head model contains the geometric shape and vertex information of the character; then, through the rendering engine, the dyed third texture map is mapped to the head model of the virtual character, and appropriate lighting and other light and shadow effects are added, so that the skin, hair and tattoos of the character have realistic light and color, and a higher-realism object image can be obtained, thereby bringing a more lively and immersive visual experience to the user.

[0104] Below, an exemplary application of the embodiments of the present application in an actual application scenario will be described. The exemplary application describes the specific implementation process of the image generation method in the generation of a digital human face scene.

[0105] In the related art, by collecting three-dimensional models with textures, manually adding BlendShape bindings to the three-dimensional models, and using a rendering engine to render and process to obtain generated object images.

[0106] Since a large number of three-dimensional head models containing texture images need to be collected in the related art, and the types of collected texture images are single, it is difficult to generate rich and complex decorative textures; in addition, the texture migration step of the texture image in the related art is redundant and complex, therefore, referring to Figure 5 , Figure 5 is a process schematic diagram of the image generation method provided by the embodiments of the present application, as shown in Figure 5 the texture adaptation technology is used to universalize the textures of different three-dimensional head models, so as to automatically synthesize any object image, wherein the object image contains an image, camera parameters, two-dimensional / three-dimensional key point coordinates, BlendShape animation data, light and shadow, etc. Below, the processing flow of the image generation method of the present application will be described in detail. Figure 5

[0107] Firstly, a plurality of 3D Morphable Models (3DMM) are collected, which are self-equipped with BlendShape panels, based on the existing 100 PCA basic three-dimensional models, by using the formula of 3DMM, the weights of different models of PCA are modified, and a new BlendShape binding model of any face shape can be obtained.

[0108] Next, the deformation migration of the two-dimensional texture feature surface is constructed. Referring to Figure 6 , Figure 6 is a key point schematic diagram of a 3D model and a 2D texture map provided by the embodiments of the present application, as shown in Figure 6 68 key points (vertices) of the 3D model A are projected into the corresponding two-dimensional texture map, and 68 two-dimensional texture coordinate points of the model A are obtained, denoted as A i , wherein i = 1, …, 68, referring to Figure 7 , Figure 7 is a texture map schematic diagram of a 3D model provided by the embodiments of the present application; similarly, 68 key points (vertices) of the 3D model B are projected into the corresponding two-dimensional texture map, and 68 two-dimensional texture coordinate points of the model B are obtained, denoted as B i , wherein i = 1, …, 68. Next, three points are randomly selected from the 68 A i points to form a triangular surface, and TriangleA j ​where j = 1,..., 50116; similarly, TriangleB is generated j where j = 1,..., 50116. Then, TriangleA is randomly selected j with two-dimensional texture vertices p1, p2, p3, and TriangleB j with corresponding vertices q1, q2, q3, P = [p2-p1, p3-p1] and Q = [q2-q1, q3-q1] are obtained by mapping, and the deformation matrix B of the triangle can be obtained by formula (1) j A j Then, other two-dimensional texture coordinate points of model B are included in TriangleB j The coordinates of other texture coordinate points remain unchanged, i.e., the deformation matrix B of TriangleB j remains unchanged, i.e., B j A j Finally, to ensure the smoothness of all triangular meshes and the accurate change of the 68 key points, a smoothing energy function is set, such as formula (2) to formula (5), by ensuring that the energy of all key points of TriangleB j is minimum, Newton method, gradient descent or a special mesh deformation solver can be used to obtain the solution of the key points of two-dimensional texture map B and the key points of two-dimensional texture map A.

[0109] It should be noted that the triangular faces of the above texture can be any polygonal face, which is not specifically limited here.

[0110] B j A J = Q x P -1 (1)

[0111]

[0112] E smooth = ∑ (j,k) ‖(q j -q k )-(p j -p k )‖ 2 (4)

[0113] E = aE gradient + bE uv_lmd + gE smooth (5)

[0114] where Q is the mapping matrix of the vertices of the triangular patch in Model B, P is the mapping matrix of the vertices of the triangular patch in Model A; B0A0 represents the change of the vertices of the triangular patch TriangleB0 relative to the vertices of TriangleA0 (deformation matrix) in the initial state, B j A j is the change of the vertices of the triangular patch TriangleB j relative to the vertices of TriangleA j during the deformation process; A i represents the key points in the two-dimensional texture space of Model A, q j and q k represent the positions of the adjacent vertices in the two-dimensional texture space, p j and p k represent the positions of the adjacent vertices before deformation, and α, β and γ are weights used to balance the deformation gradient consistency of the triangular patch formed by the key points. gradient , the accurate matching of the two-dimensional texture key points, uv_lmd , the smoothing of the UV adjacent edges, smooth and (j, k) are the adjacent edge key points.

[0115] It should be noted that E gradient is used to measure the gradient consistency during the deformation of the triangular patch, and the square of the difference between B j A j and B0A0 is calculated and summed to reflect the degree of gradient change of the triangular patch during the deformation process; E uv_lmd is used to accurately match the key points in the UV space, and by minimizing the square of the difference between the key points, the accurate correspondence of the key points in the UV space can be ensured; E smooth is used to maintain the smoothness between the adjacent edges in the UV space, which can ensure that the deformed vertices maintain the adjacent relationship and smooth change to maintain the overall smoothness of the network.

[0116] As an example, assuming there is a UV map of a 3D model (Model B) including 68 key points that need to be accurately matched, first set an initial triangular mesh (Model A) including the vertices that need to be optimized; then, according to the optimization requirements and mesh characteristics, select appropriate weight coefficients α, β and γ; then, for each vertex, calculate E gradient , E uv_lmd and E smooth ; then, use an optimization algorithm (such as Newton's method, gradient descent or mesh deformation solver, etc.) to find the minimum value of the energy function, and adjust the position of the vertices through multiple iterations until the energy function reaches the minimum value, thereby obtaining the mapping relationship between the vertices of the two models.

[0117] Subsequently, texture graph augmentation is performed based on the text editing model. First, referring to Figure 8 , Figure 8 is a batch texture graph and a texture graph with enhanced decoration provided by the embodiment of the present application, as shown in Figure 8 2000 portrait texture images and 200 different texture graphs (such as holiday decorations, accessories, tattoos, etc.) are collected, and 1 million reasonable portrait texture graphs (1024*1024) with decorations, texture region coordinates and corresponding text descriptions are generated through permutation and combination and different regions and a rendering engine. Next, referring to Figure 9 , Figure 9 is a model training structure diagram of the discriminator provided by the embodiment of the present application, as shown in Figure 9 a multi-layer texture discriminator network is constructed through CNN, random occlusion editing is performed on the texture coordinate region according to the proportion of the area, the ratio of the edited area to the texture area is taken as the label data, the label data range is 0 to 1, the label data is mapped to -1 to 1, and a model for discriminating the reasonableness of the texture is trained based on the texture data. Next, referring to Figure 10 , Figure 10 is a model training structure diagram of the text editing model provided by the embodiment of the present application, as shown in Figure 10 the text editing model is fine-tuned to enable the model to generate reasonable decoration texture, in the model training process, the pre-trained diffusion model is frozen, a bypass Lora branch is constructed, only the parameters of Lora are fine-tuned, and the fine-tuning mainly includes two stages: first stage fine-tuning: only through the text description and the avatar texture graph, the model is constrained by the texture discrimination loss of the texture discriminator to generate reasonable decoration texture graph according to the text description, and 100 iterations are performed; second stage fine-tuning: the model is trained through the text description, the avatar texture graph and the avatar texture graph with decoration, wherein the avatar texture graph with decoration is taken as the true value, and the model parameters are adjusted through the weighted fusion of the reconstruction loss and the texture discrimination loss in the whole training process, so as to obtain the final text editing model.

[0118] It should be noted that the discriminator network can be a Transformer, a diffusion model structure, etc., which is not specifically limited here.

[0119] Finally, the texture graph is automatically migrated based on the deformation, and the texture is automatically enhanced according to the texture description text. First, referring to Figure 11 , Figure 11 is a schematic diagram of automatically migrating and enhancing the texture graph provided by the embodiment of the present application, as shown in Figure 11 the avatar texture graph is obtained, and deformation migration is performed, so as to topologically map the avatar texture graph to the target model, referring to Figure 12 , Figure 12is a schematic diagram provided by an embodiment of the present application for automatically converting a texture map to a target model. Then, referring to Figure 13 , Figure 13 is a schematic diagram provided by an embodiment of the present application for performing decorative texture enhancement on a texture map, as shown in Figure 13 , the texture map is processed to perform decorative texture enhancement, to obtain a texture map with device texture enhancement; finally, the texture map with decorative texture enhancement and the three-dimensional head model with mixed shape animation are controlled by mixed shape animation, under the control of different lights, cameras and environments, to generate an arbitrary object image using a rendering engine, and the mixed shape animation, camera parameters, key point coordinates and light data of each three-dimensional head model are saved for subsequent downstream model tasks, such as a three-dimensional key point detection model, a digital human model driven in a single frame / sequence, etc.

[0120] In some embodiments, by automatically obtaining standard synthetic digital human data, it can be applied to downstream digital human related model pre-training. Since the entire data is generated by a rendering engine, all rendering engine parameters can be synthesized according to actual needs to obtain the desired digital human data set.

[0121] The following continues to illustrate an exemplary structure of the image generation apparatus 555 provided by an embodiment of the present application as a software module. In some embodiments, as shown in Figure 2 , the software module stored in the image generation apparatus 555 in the memory 550 can include a projection module 5551, a determination module 5552, a generation module 5553, an editing module 5554 and a rendering module 5555.

[0122] The projection module 5551 is configured to project a plurality of first vertices of a first three-dimensional model of a first object to a first texture map of the first three-dimensional model to obtain a first texture coordinate corresponding to each of the first vertices; the projection module 5551 is further configured to project a plurality of second vertices of a second three-dimensional model of a second object to a second texture map of the second three-dimensional model to obtain a second texture coordinate of each of the second vertices; the determination module 5552 is configured to determine a second texture coordinate corresponding to each of the first texture coordinates, wherein a correspondence relationship between the first texture coordinate and the second texture coordinate is the same as a correspondence relationship between the first vertex and the second vertex in a deformation, and the deformation is a deformation from the first three-dimensional model to the second three-dimensional model; the generation module 5553 is configured to generate a dyed second texture map based on the second texture coordinate corresponding to each of the first texture coordinates and color data of the first texture map; the editing module 5554 is configured to perform editing processing on the dyed second texture map based on a first description text to obtain a dyed third texture map; and the rendering module 5555 is configured to perform rendering processing based on the dyed third texture map and the second three-dimensional model to obtain an image of a third object.

[0123] In some embodiments, the projection module 5551 is further configured to perform fusion processing on the base three-dimensional models of the local parts based on respective weights corresponding to the base three-dimensional models of the local parts, and perform superimposition processing on a fusion result and the average three-dimensional model to obtain the second three-dimensional model, wherein the weights correspond to the second object.

[0124] In some embodiments, the determining module 5552 is further configured to generate a plurality of first polygonal faces and a plurality of first edges based on the first texture coordinates of the plurality of first vertices, and generate a plurality of second polygonal faces and a plurality of second edges based on the second texture coordinates of the plurality of second vertices; determine a matching degree between each of the first texture coordinates and a corresponding second texture coordinate to be solved based on each of the first texture coordinates and the corresponding second texture coordinate to be solved; determine a first difference in deformation gradient between each of the first polygonal faces and a corresponding second polygonal face to be solved based on each of the first texture coordinates and the corresponding second texture coordinate to be solved; determine a second difference between each of the first edges and a corresponding second edge to be solved based on each of the first texture coordinates and the corresponding second texture coordinate to be solved; and fuse the matching degree between each of the first texture coordinates and the corresponding second texture coordinate to be solved, the first difference in deformation gradient between each of the first polygonal faces and the corresponding second polygonal face to be solved, and the second difference between each of the first edges and the corresponding second edge to be solved to obtain the energy function of the deformation; and solve the corresponding second texture coordinate to be solved for each of the first texture coordinates to obtain the second texture coordinate corresponding to each of the first texture coordinates when the energy function of the deformation converges to a minimum value.

[0125] In some embodiments, the image generation apparatus 555 further includes a training module 5556.

[0126] The training module 5556 is configured to perform first training on the initialized first network based on a first loss used to constrain texture quality to obtain a first network subjected to the first training, and perform second training on the first network subjected to the first training based on the first loss and a second loss used to constrain reconstruction quality to obtain a first network subjected to the second training, wherein the first network subjected to the second training is used to be invoked to perform editing processing on the dyed second texture map based on the first description text to obtain a dyed third texture map.

[0127] In some embodiments, the training module 5556 is further configured to perform the following processing by the initialized first network: perform editing processing on a fourth texture map based on a second description text to obtain a dyed fifth texture map; perform random damage processing on the fourth texture map to obtain a damaged fourth texture map, and perform texture quality evaluation on the fifth texture map based on the damaged fourth texture map to obtain a first quality score of the fifth texture map; determine a first loss negatively correlated with the first quality score, and perform parameter updating on the initialized first network based on the first loss to obtain the first network subjected to the first training.

[0128] In some embodiments, the training module 5556 is further configured to perform the following processing by using the first network after the first training: performing editing processing on a sixth texture map based on a third description text to obtain a dyed seventh texture map; performing random damage processing on the sixth texture map to obtain a damaged sixth texture map, and performing texture quality evaluation on the dyed seventh texture map based on the damaged sixth texture map to obtain a second quality score of the dyed seventh texture map; determining a first loss negatively correlated with the second quality score, and determining a second loss based on a pixel difference between the dyed seventh texture map and an eighth texture map, where the eighth texture map is a texture map matched with the third description text and the dyed sixth texture map; and performing parameter updating on the first network after the first training based on a fusion loss of the first loss and the second loss to obtain a second network after the second training.

[0129] In some embodiments, the training module 5556 is further configured to perform damage processing on a first region in a ninth texture map to obtain a tenth texture map, perform texture quality evaluation on the tenth texture map based on the ninth texture map by using an initialized second network to obtain a third quality score of the tenth texture map, map a first ratio between a damaged region area of the ninth texture map and an overall area of the ninth texture map to a cosine value interval to obtain a second ratio, determine a third loss based on a difference between the third quality score and the second ratio, and update the initialized second network based on the third loss to obtain a trained second network, where the trained second network is configured to perform the texture quality evaluation.

[0130] It should be noted that the description of the device embodiments of the present application is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments, and thus is not described herein. For technical details of the image generation device not described in the embodiments of the present application, they can be understood according to the description of any figure. Figure 3 , or Figure 4 any figure.

[0131] The embodiments of the present application provide a computer program product, which includes a computer program or computer executable instructions stored in a computer readable storage medium. The processor of the electronic device reads the computer executable instructions from the computer readable storage medium, and the processor executes the computer executable instructions to enable the electronic device to perform the image generation method described above in the embodiments of the present application.

[0132] The embodiment of the present application provides a computer-readable storage medium in which computer-executable instructions or computer programs are stored. When the computer-executable instructions or computer programs are executed by a processor, the processor will execute the image generation method provided by the embodiment of the present application, for example, Figure 3 ,or Figure 4 The image generation method is shown.

[0133] In some embodiments, the computer-readable storage medium may be a ferroelectric random access memory (FRAM), ROM, programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), flash memory, magnetic surface memory, optical disk, or compact disc read-only memory (CD-ROM); or various devices including one or any combination of the above memories.

[0134] In some embodiments, computer-executable instructions may be in the form of a program, software, software module, script, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine, or other unit suitable for use in a computing environment.

[0135] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinating files (e.g., files storing one or more modules, subroutines, or code portions).

[0136] By way of example, computer-executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed across multiple sites and interconnected by a communication network.

[0137] To sum up, by the embodiment of the present application, first, the vertices of the first three-dimensional model of the first object are projected to the first texture map to obtain the first texture coordinates, and the vertices of the second three-dimensional model of the second object are projected to the second texture map to obtain the second texture coordinates, and the correspondence relationship of the texture coordinates in the deformation of the two models is determined, so that the dyed second texture map can be generated based on the second texture coordinates corresponding to the first texture coordinates and the color data of the first texture map, the color data of the first texture map is automatically migrated to the second texture map, and the color information of the texture map is realized. The transmission; then, the dyed second texture map is edited based on the first description text to obtain a dyed third texture map, and the details of the texture map are adjusted according to the content of the description text, so as to enhance the visual expressiveness of the texture map; finally, the dyed third texture map and the second three-dimensional model are rendered to obtain an image of a third object, which can ensure that the generated image meets the specific requirements, and in this process, manual dyeing is not required, and the generation efficiency of the object image can be improved.

[0138] The above is only an embodiment of the present application, and is not used to limit the protection scope of the present application. Any modification, equivalent replacement and improvement made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. An image generation method, characterized in that: The method comprises: Projecting a plurality of first vertices of a first three-dimensional model of a first object onto a first texture map of the first three-dimensional model to obtain a first texture coordinate corresponding to each first vertex; projecting a plurality of second vertices of the second three-dimensional model of the second object onto a second texture map of the second three-dimensional model to obtain a second texture coordinate of each of the second vertices; determining a second texture coordinate corresponding to each of the first texture coordinates, wherein a correspondence between the first texture coordinates and the second texture coordinates is the same as a correspondence between the first vertex and the second vertex in a deformation from the first three-dimensional model to the second three-dimensional model; generating a dyed second texture map based on a second texture coordinate corresponding to each of the first texture coordinates and color data of the first texture map; Editing the dyed second texture map based on the first description text to obtain a dyed third texture map; Rendering is performed based on the dyed third texture map and the second three-dimensional model to obtain an image of the third object.

2. The method according to claim 1, characterized in that The method further comprises: Performing a first training on the initialized first network based on a first loss for constraining texture quality to obtain a first network that has undergone the first training; Performing second training on the first network after the first training based on the first loss and a second loss for constraining reconstruction quality, to obtain the first network after the second training; The first network that has undergone the second training is used to be called to edit the dyed second texture image based on the first description text to obtain the dyed third texture image.

3. The method according to claim 2, characterized in that The first training of the initialized first network based on the first loss for constraining texture quality to obtain the first trained first network includes: The following processing is performed through the initialized first network: editing the dyed fourth texture map of the fourth object based on the second description text to obtain a dyed fifth texture map; performing random destruction processing on the fourth texture map to obtain a destroyed fourth texture map, and performing texture quality evaluation on the fifth texture map based on the destroyed fourth texture map to obtain a first quality score of the fifth texture map; A first loss negatively correlated with the first quality score is determined, and parameters of the initialized first network are updated based on the first loss to obtain a first network that has undergone the first training.

4. The method according to claim 2, characterized in that The performing second training on the first network after the first training based on the first loss and a second loss for constraining reconstruction quality to obtain the first network after the second training includes: The following processing is performed by the first network that has undergone the first training: editing the dyed sixth texture map of the fifth object based on the third description text to obtain a dyed seventh texture map; performing random destruction processing on the sixth texture map to obtain a destroyed sixth texture map, and performing texture quality evaluation on the dyed seventh texture map based on the destroyed sixth texture map to obtain a second quality score of the dyed seventh texture map; determining a first loss that is negatively correlated with the second quality score, and determining a second loss based on pixel differences between the stained seventh texture map and an eighth texture map, wherein the eighth texture map is a texture map that matches the third description text and the stained sixth texture map; Parameters of the first network that has undergone the first training are updated based on a fusion loss of the first loss and the second loss to obtain the first network that has undergone the second training.

5. The method according to claim 3 or 4, characterized in that The method further comprises: Destroy the first area in the ninth texture map to obtain a tenth texture map; performing, by means of the initialized second network, a texture quality assessment on the tenth texture map based on the ninth texture map, to obtain a third quality score for the tenth texture map; Mapping a first ratio between the area of ​​the damaged region of the ninth texture image and the entire area of ​​the ninth texture image to a cosine value interval to obtain a second ratio; determining a third loss based on a difference between the third quality score and the second ratio; The initialized second network is updated based on the third loss to obtain a trained second network, wherein the trained second network is used to perform the texture quality assessment.

6. The method according to claim 1, characterized in that Before projecting the plurality of second vertices of the second three-dimensional model of the second object onto the second texture map of the second three-dimensional model to obtain a second texture coordinate of each of the second vertices, the method further includes: Based on the weights corresponding to the basic three-dimensional models of each local part, the basic three-dimensional models of the multiple local parts are fused, and the fusion result is superimposed with the average three-dimensional model to obtain the second three-dimensional model, wherein the weights correspond to the second object.

7. The method according to claim 1, characterized in that The determining of a second texture coordinate corresponding to each of the first texture coordinates comprises: generating a plurality of first polygonal faces and a plurality of first edges based on the first texture coordinates of the plurality of first vertices, and generating a plurality of second polygonal faces and a plurality of second edges based on the second texture coordinates of the second vertices; Determining, based on each of the first texture coordinates and the corresponding second texture coordinate to be solved for each of the first texture coordinates, a degree of matching between each of the first texture coordinates and the corresponding second texture coordinate to be solved, a first difference in deformation gradient between each of the first polygonal faces and the corresponding second polygonal face, and a second difference between each of the first edges and the corresponding second edges; fusing the degree of match between each of the first texture coordinates and the corresponding second texture coordinate to be solved, the first difference in deformation gradient between each of the first polygonal faces and the corresponding second polygonal face, and the second difference between each of the first edges and the corresponding second edges to obtain an energy function of the deformation; The second texture coordinate corresponding to each first texture coordinate to be solved is solved based on the energy function of the deformation to obtain the second texture coordinate corresponding to each first texture coordinate when the energy function of the deformation converges to a minimum value.

8. An image generating device, characterized in that: The device comprises: a projection module, configured to project a plurality of first vertices of a first three-dimensional model of a first object onto a first texture map of the first three-dimensional model to obtain a first texture coordinate corresponding to each first vertex; The projection module is further configured to project a plurality of second vertices of the second three-dimensional model of the second object onto a second texture map of the second three-dimensional model to obtain a second texture coordinate of each second vertex; a determining module configured to determine a second texture coordinate corresponding to each of the first texture coordinates, wherein a correspondence between the first texture coordinates and the second texture coordinates is the same as a correspondence between the first vertex and the second vertex in a deformation from the first three-dimensional model to the second three-dimensional model; a generating module, configured to generate a dyed second texture map based on a second texture coordinate corresponding to each of the first texture coordinates and color data of the first texture map; An editing module, configured to edit the dyed second texture map based on the first description text to obtain a dyed third texture map; A rendering module is used to perform rendering processing based on the dyed third texture map and the second three-dimensional model to obtain an image of a third object.

9. An electronic device, characterized in that: include: a memory for storing computer executable instructions or computer programs; The processor is configured to implement the image generation method according to any one of claims 1 to 7 when executing the computer-executable instructions or computer program stored in the memory.

10. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the image generation method according to any one of claims 1 to 7 is implemented.

11. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the image generation method according to any one of claims 1 to 7 is implemented.

Images