Training methods for texture prediction models, texture prediction methods, devices and equipment

Abstract

This application provides a training method, apparatus, and device for a texture prediction model, belonging to the field of computer graphics technology. The method includes: acquiring at least two training samples, each training sample including geometric information of a 3D model and a first texture image, the first texture image including at least two first texture fragments; inputting the geometric information and the first texture image into a texture prediction model, and outputting a second texture image from the texture prediction model, the second texture image including at least two second texture fragments, each of the at least two second texture fragments corresponding to a second region on the 3D model, wherein the second region corresponding to at least one second texture fragment includes at least two sub-regions, each sub-region including at least a portion of the first region corresponding to a first texture fragment; iteratively training the texture prediction model based on the difference between the second texture image and the first texture image. This method ensures the quality of the predicted texture image.

Description

Technical Field

[0001] This application relates to the field of computer graphics technology, and in particular to a training method, texture prediction method, apparatus and device for a texture prediction model. Background Technology

[0002] With the development of computer technology, the application of 3D models is becoming increasingly widespread, and users have higher and higher requirements for the visual effects of 3D models. In order to improve the visual effects of 3D models, texture images can be used to render 3D models, so that the 3D models have a sense of color and relief.

[0003] In related technologies, two-dimensional texture images are typically generated manually by art engineers. A three-dimensional model can be divided into hundreds or thousands of regions, and the texture image includes numerous texture fragments, each indicating the texture of a region on the 3D model. When rendering a 3D model based on a texture image, each region is rendered separately based on each texture fragment. Due to the large number of texture fragments and the numerous rendering iterations, the computational load is enormous, leading to errors such as texture ghosting and texture blurring during the rendering process. Summary of the Invention

[0004] This application provides a method for training a texture prediction model, a texture prediction method, an apparatus, and a device. The technical solutions provided by this application can be as follows.

[0005] According to one aspect of the embodiments of this application, a method for training a texture prediction model is provided, the method comprising:

[0006] At least two training samples are obtained, each training sample including geometric information of a 3D model and a first texture image, the first texture image including at least two first texture fragments, the at least two first texture fragments being used to determine the texture of a first region on the 3D model;

[0007] The geometric information and the first texture image are input into the texture prediction model, and the texture prediction model outputs a second texture image. The second texture image includes at least two second texture fragments, and the at least two second texture fragments correspond to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment.

[0008] The texture prediction model is trained iteratively based on the difference between the second texture image and the first texture image.

[0009] According to another aspect of the embodiments of this application, a texture prediction method is provided, the method comprising:

[0010] Obtain the geometric information of the three-dimensional model and a first texture image, wherein the first texture image includes at least two first texture fragments, and the at least two first texture fragments are respectively used to determine the texture of a first region on the three-dimensional model;

[0011] The geometric information and the first texture image are input into the texture prediction model, and the texture prediction model outputs a second texture image. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment.

[0012] According to another aspect of the embodiments of this application, a training apparatus for a texture prediction model is provided, the apparatus comprising:

[0013] The acquisition module is used to acquire at least two training samples, each training sample including geometric information of a 3D model and a first texture image, the first texture image including at least two first texture fragments, the at least two first texture fragments being used to determine the texture of a first region on the 3D model;

[0014] The input / output module is used to input the geometric information and the first texture image into the texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, and the at least two second texture fragments respectively correspond to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment.

[0015] The training module is used to iteratively train the texture prediction model based on the difference between the second texture image and the first texture image.

[0016] According to another aspect of the embodiments of this application, a texture prediction apparatus is provided, the apparatus comprising:

[0017] The acquisition module is used to acquire the geometric information of the three-dimensional model and a first texture image, wherein the first texture image includes at least two first texture fragments, and the at least two first texture fragments are respectively used to determine the texture of a first region on the three-dimensional model;

[0018] The input / output module is used to input the geometric information and the first texture image into the texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment.

[0019] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store a computer program, the computer program being loaded and executed by the processor to implement the training method or texture prediction method of the texture prediction model in the embodiments of this application.

[0020] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the training method or texture prediction method of the texture prediction model in the embodiments of this application.

[0021] On the other hand, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, a processor of a computer device reading the computer program from the computer-readable storage medium, the processor executing the computer program, causing the computer device to execute the training method or texture prediction method of the texture prediction model in the embodiments of this application.

[0022] The technical solutions provided in this application have at least the following beneficial effects:

[0023] On the one hand, based on the geometric information of the 3D model and the first texture image, a texture prediction model generates a second texture image for the 3D model. At least one texture fragment in the second texture image corresponds to a second region comprising at least two sub-regions. Since each sub-region includes at least a portion of the first region corresponding to a first texture fragment in the first texture image, the second texture image aggregates textures from some sub-regions into larger second texture fragments. This increases the size of the region on the 3D model corresponding to the second texture fragment, while reducing the number of texture fragments in the second texture image. Because each texture fragment is rendered once, the number of rendering operations is reduced when rendering the 3D model based on the second texture image, thus reducing computation and rendering errors, thereby improving rendering efficiency and quality. On the other hand, the texture prediction model is iteratively trained based on the differences between the first and second texture images. This allows the texture prediction model to not only re-divide the texture image but also ensure that the predicted texture image is as similar as possible to the original texture image of the 3D model, guaranteeing the quality of the predicted texture image. Attached Figure Description

[0024] Figure 1 This is a schematic diagram of a training method for a texture prediction model provided in one embodiment of this application;

[0025] Figure 2 This is a schematic diagram of a computer system provided in one embodiment of this application;

[0026] Figure 3 This is a flowchart of a training method for a texture prediction model provided in one embodiment of this application;

[0027] Figure 4 This is a schematic diagram of a texture prediction model provided in one embodiment of this application;

[0028] Figure 5 This is a schematic diagram of a residual block provided in one embodiment of this application;

[0029] Figure 6 This is a flowchart of a training method for a texture prediction model provided in one embodiment of this application;

[0030] Figure 7 This is a schematic diagram illustrating the comparison of performance parameters provided in one embodiment of this application;

[0031] Figure 8 This is a flowchart of a texture prediction method provided in one embodiment of this application;

[0032] Figure 9 This is a rendering flowchart of a three-dimensional model provided in one embodiment of this application;

[0033] Figure 10 This is a block diagram of a training apparatus for a texture prediction model provided in one embodiment of this application;

[0034] Figure 11 This is a block diagram of a texture prediction apparatus provided in one embodiment of this application;

[0035] Figure 12 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0036] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0037] Before introducing the technical solutions of this application, some terms involved in this application will be explained. The following related explanations are optional solutions and can be arbitrarily combined with the technical solutions of the embodiments of this application, all of which fall within the protection scope of the embodiments of this application. The embodiments of this application include at least some of the following contents.

[0038] A 3D model is a polygonal representation of an object. 3D models can be used to represent real-world entities or virtual objects. A 3D model can be a model with 3D data constructed in a virtual 3D space.

[0039] Vertex: The smallest unit of geometry, a single point in three-dimensional space, used to define the corners and intersections of a mesh. Vertices connect to form edges and faces, collectively constituting the surface and shape of a three-dimensional model.

[0040] Rendering a 3D model refers to the process of converting a 3D model into a 2D image. The rendering process simulates how light interacts with the 3D model in a virtual 3D space, thus giving the rendered 2D image (hereinafter referred to as the rendered image) visual representations such as depth, lighting, and shadow effects. Rendering of a 3D model can be performed on a vertex-by-vertex basis.

[0041] Texture image: A two-dimensional image used to describe the texture of a 3D model. Each pixel in the texture image can be mapped to different locations on the 3D model, thus providing information such as the color, texture, bumps, and lighting of the 3D model—in other words, providing the texture of the 3D model. During the rendering process, mapping the texture image onto the surface of the 3D model allows the 3D model to present a more realistic effect. A texture image consists of multiple texture fragments, with each fragment indicating the texture of a region on the 3D model.

[0042] 3DCNN (3D Convolutional Neural Networks) model: A neural network model that can perform three-dimensional convolution operations.

[0043] In related technologies, texture images are generally generated manually by art engineers. Since 3D models can be divided into hundreds or thousands of small regions, the texture image of a 3D model includes multiple texture fragments. Each texture fragment corresponds to a region on the 3D model. When rendering a 3D model based on a texture image, for each texture fragment, the corresponding region on the 3D model is determined, and the texture from that fragment is rendered to that region. Therefore, if there are many texture fragments in the texture image, and each texture fragment undergoes a separate rendering process, the number of rendering operations is obviously high, the computational load is large, and various rendering errors are easily caused.

[0044] Please refer to Figure 1 The diagram illustrates a training method for a texture prediction model provided in one embodiment of this application. The method includes at least one of the following steps:

[0045] 1. Obtain training samples: Obtain at least two training samples for training the texture prediction model. The training samples include the geometric information 101 of the 3D model and the first texture image 102. The first texture image 102 is also the original texture image used to render the 3D model. If it is generated by the manual drawing of the art engineer, it contains a lot of texture fragments, which leads to a large amount of calculation during rendering and some rendering errors.

[0046] 2. Predicting the Texture Image: Due to the large number of texture fragments in the first texture image 101, it is necessary to re-segment the texture fragments so that textures in similar regions are clustered into a larger texture fragment. Specifically, the geometric information 101 of the 3D model and the first texture image 102 are input into the texture prediction model 103 to obtain the second texture image 104. The texture prediction model 103 is used to re-segment the texture fragments based on the similarity of the textures in the first texture image 101, thereby obtaining a new texture image (i.e., the second texture image). This reduces the number of texture fragments in the second texture image 104, and the area on the 3D model indicated by a single texture fragment becomes larger. Therefore, when rendering the 3D model based on the second texture image 104, the number of rendering iterations can be reduced, thereby reducing the computational load and minimizing errors during the rendering process.

[0047] 3. Training the model: Based on the difference between the first texture image 102 and the second texture image 104, the texture prediction model 103 is trained iteratively so that the texture prediction model 103 can predict the second texture image 104 that is as similar as possible to the first texture image 101 and reduces texture fragmentation.

[0048] In this embodiment, the original texture image of the 3D model is processed to group textures in similar regions into large texture fragments. Correspondingly, the region corresponding to the texture fragment also becomes larger. At the same time, the number of texture fragments included in the texture image is reduced. As a result, when rendering the 3D model based on the texture image, the number of rendering times is reduced and the amount of computation is reduced, thereby improving rendering efficiency and rendering effect.

[0049] Please refer to Figure 2 The diagram illustrates a computer system provided in one embodiment of this application. The computer system includes at least one of the following: a terminal device 10 and a server 20.

[0050] Terminal device 10 can be an electronic device such as a mobile phone, tablet computer, multimedia playback device, PC (Personal Computer), wearable device, in-vehicle terminal device, VR (Virtual Reality) device, AR (Augmented Reality) device, MR (Mixed Reality) device, etc. Terminal device 10 can have a client running the target application. The target application can be an application that requires the use of a 3D model, or it can be an application used to construct a 3D model; this application embodiment does not limit the specific implementation form of the target application. For example, it can be an application that requires downloading and installation, a small program that does not require installation, a web application, etc. For example, the target application can include at least one of the following: a game application, a video application, a game engine application, or a modeling application. Texture images are loaded into the target application to render the 3D model.

[0051] The game application can be any of the following: Massive Multiplayer Online Role-Playing Game (MMORPG), casual game, party game, sandbox game, tower defense game, action-adventure game, multiplayer online battle arena (MOBA) game, first-person shooter (FPS) game, multiplayer shooting survival game, third-person shooter (TPS) game, strategy game (SLG), simulation management game, survival building game, real-time strategy game, etc. This application embodiment does not limit this.

[0052] In this embodiment, server 20 provides background services for the target application. For example, server 20 generates texture images of a 3D model. A texture prediction model is embedded in server 20, which obtains the texture image of the 3D model. Server 20 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Terminal device 10 can communicate with server 20 via a network, such as a wireless or wired network.

[0053] In some embodiments, server 20 provides terminal 10 with geometric information and texture images of a 3D model. Terminal 10 can render the 3D model based on the geometric information and texture images to give the surface of the 3D model a sense of color and texture. Alternatively, terminal 10 may have a texture prediction model embedded in it. Terminal 10 obtains a new texture image of the 3D model through this texture prediction model and then renders the 3D model based on the geometric information and the new texture image.

[0054] The technical solutions provided in this application can be applied to any scenario that requires rendering 3D models, such as game rendering, 3D model construction and other scenarios that require real-time rendering of 3D models, as well as scenarios that can render 3D models offline, such as images, animations, videos, and movies, and other scenarios such as rendering repair (e.g., 3D model rendering failure requires re-rendering), rendering updates, and game development.

[0055] Please refer to Figure 3 This document illustrates a flowchart of a training method for a texture prediction model provided in one embodiment of this application. The execution entity for each step of this method can be a computer device, such as... Figure 2 The terminal device 10 in the computer system shown can be a client, such as a server 20. Taking the server as the execution subject of each step of the method as an example, the method may include at least one of the following steps 310 to 340.

[0056] Step 310: Obtain at least two training samples, each training sample including geometric information of the 3D model and a first texture image, the first texture image including at least two first texture fragments, the at least two first texture fragments being used to determine the texture of a first region on the 3D model.

[0057] In this embodiment, the geometric information of the 3D model is used to map the texture image onto the 3D model. The 3D model is composed of points, lines, and faces. The geometric information of the 3D model includes the vertex coordinates of at least two vertices of the 3D model, face information, and the texture coordinates of each of the at least two vertices. Vertex coordinates are used to represent the position of the vertex in the 3D model. Face information is used to represent the relationship between each face and vertex in the 3D model, and the vertices in each face information are the at least three vertices that constitute a face. For example, if a face is a triangle, then there are three vertices that constitute the face, and the face information includes the indices of the three vertices to indicate the vertices. The texture coordinates of the vertex are used to indicate the coordinates of the vertex on the texture image, and are used to map the texture at the position indicated by the texture coordinates of the vertex on the texture image to that vertex on the 3D model.

[0058] In this embodiment, the surface of the 3D model is composed of triangles or polygons, and the texture image is a two-dimensional planar image. This two-dimensional plane is called the UV plane, which is obtained by unfolding the surface of the 3D model into a two-dimensional plane. The coordinates of the vertices on the two-dimensional plane are called UV coordinates (texture coordinates), and the two-dimensional image formed by unfolding the 3D model on this two-dimensional plane is called the texture image. U and V represent two directions on the two-dimensional image, respectively. Each vertex in the 3D model has a 3D coordinate (X, Y, Z) in 3D space, and also a corresponding coordinate (U, V) on the UV plane, which is the texture coordinate. Since the texture image is obtained by unfolding the 3D model on a two-dimensional plane, the shape of the area occupied by each texture fragment on the texture image is the same as the shape of the corresponding area on the 3D model, and the size can be the same or scaled proportionally.

[0059] In this model, a first texture fragment on the first texture image corresponds to a first region on the 3D model. Each first texture fragment is used to indicate the texture on the corresponding first region. The surface of the 3D model is composed of triangles or polygons, and each first region is a surface region on the 3D model composed of multiple faces. Accordingly, the information of each first region includes the information of its multiple faces, and the information of each face is indicated by the geometric information of the 3D model.

[0060] In this embodiment, at least two training samples correspond to at least two 3D models, and each training sample includes the geometric information of a 3D model and a first texture image. The set of at least two 3D models can be represented as... N represents the total number of 3D models, which is also the total number of training samples, M i Let V represent the i-th 3D model, where i and N are integers greater than 1. For each 3D model, extract its geometric information. Optionally, for the i-th 3D model, V i Let F represent the set of vertices of the i-th 3D model.i Let T represent the set of faces of the i-th 3D model. i Let represent the set of texture coordinates for the i-th 3D model. These sets can be represented as follows:

[0061]

[0062] Where, n i v represents the number of vertices in the i-th 3D model. i,j Let m represent the j-th vertex of the i-th 3D model. i f represents the number of faces in the i-th 3D model. i,k p represents the k-th face of the i-th 3D model. i t represents the number of texture coordinates for the i-th 3D model. i,l This represents the I-th texture coordinate of the i-th 3D model.

[0063] In some embodiments, the first texture image is a normalized texture image. Specifically, the original texture image of the 3D model is normalized to obtain the first texture image of the 3D model; the normalization process maps the pixel values ​​of the pixels in the original texture image to a third numerical range.

[0064] The original texture image can be represented as H×W×3, where H and W represent the height and width of the original texture image, respectively, and 3 represents the three color channels RGB (RED, GREEN, BLUE). For example, the first texture image is obtained using the following formula:

[0065]

[0066] Among them, I i I represents the original texture image of the i-th 3D model. i ′ represents the normalized texture image of the i-th 3D model, which is also the first texture image of the i-th 3D model. The formula means that for each pixel in the original texture image, the quotient of the pixel value of the pixel and 255.0 is used as the normalized pixel value. Since the range of pixel values ​​is [0, 255.0], the range of the third value is also [0, 1].

[0067] In this embodiment, by normalizing the original texture image of the 3D model, the pixel values ​​of the original texture image are limited to a certain range. Then, after the normalized texture image is input into the texture prediction model, the stability of the numerical calculation of the texture prediction model can be improved, and numerical problems caused by large numerical gradients, such as gradient explosion or gradient vanishing, can be reduced. This helps the texture prediction model learn faster and thus improves training efficiency.

[0068] In some embodiments, to enable the trained texture prediction model to generalize across different data, the acquired training samples are divided into a training set and a test set. Each training set and test set includes at least two training samples. Optionally, the training set is represented as D. train The test set is represented as D. test If the training set and test set account for 80% and 20% of the total training sample data, respectively, then for example, the training set and test set are shown below:

[0069]

[0070] Where N represents the total number of training samples, which is also the total number of 3D models.

[0071] In some embodiments, after training a texture prediction model based on a training set, the texture prediction model is tested based on a test set. If the test results meet the preset requirements, the texture prediction model is obtained. If the test results do not meet the preset requirements, the texture prediction model continues to be trained.

[0072] In this embodiment of the application, by dividing a test set, the performance of the texture prediction model on unseen data is evaluated based on the test set to measure the generalization ability of the texture prediction model, that is, to measure the predictive ability of the texture prediction model on new data, thereby ensuring the reliability and practicality of the texture prediction model.

[0073] In some embodiments, to increase the diversity of training samples, data augmentation processing is performed on the 3D model to obtain a processed 3D model. The data augmentation processing includes at least one of cropping, rotating, and flipping. Based on the geometric information of the processed 3D model and the first texture image, expanded training samples are generated.

[0074] Since there are many 3D models, data augmentation can be performed on a subset of them. These subsets can be individually cropped, rotated, and flipped to perform multiple types of data augmentation on a single 3D model, thus expanding the training sample. Alternatively, data augmentation can be performed on all 3D models corresponding to at least two training samples.

[0075] The process of cropping the 3D model involves both the 3D model itself and its texture image. Cropping the 3D model itself refers to trimming the 3D model in 3D space to remove certain areas, thereby simulating different perspectives and positions of objects in the real world and increasing the diversity of the 3D model. Cropping the texture image involves adjusting the texture mapping based on the cropping result of the 3D model, removing corresponding texture portions from the texture image. This is because after the 3D model itself is cropped, some texture areas may no longer correspond to any part of the 3D model and therefore need to be cropped to maintain the consistency between the texture image and the 3D model, thus ensuring the correctness of the texture mapping.

[0076] Among them, clipping can be random clipping, which randomly selects a region from the 3D model for clipping, thus obtaining multiple different clipping results. This can help the texture prediction model adapt to different clipping results of the 3D model and improve the robustness of the texture prediction model to random changes in the 3D model.

[0077] Rotation of the 3D model can be either a rotation of the 3D model itself or a rotation of its texture image. Rotating the 3D model changes its orientation in 3D space, altering its geometry and allowing it to present different perspectives from different angles. Rotating the texture image changes its orientation without altering the geometry of the 3D model; the shape of the 3D model remains unchanged, but the surface texture changes with the rotation angle. Furthermore, both the 3D model and the texture image can be rotated simultaneously for more comprehensive data augmentation. This ensures that the texture prediction model has good generalization ability in terms of both geometry and texture features.

[0078] In this embodiment of the application, data augmentation processing is performed on the training samples to expand the training samples. Since the training samples include both the original training samples and the data augmented training samples, the data diversity of the training samples can be increased, thereby improving the generalization ability of the texture prediction model.

[0079] Step 320: Input geometric information and the first texture image into the texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on the three-dimensional model. The region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to a first texture fragment.

[0080] In this context, a second texture fragment on the second texture image corresponds to a second region on the 3D model. Each second texture fragment is used to indicate the texture on the corresponding second region. The surface of the 3D model is composed of triangles or polygons, and each second region is a surface region on the 3D model composed of multiple faces. Correspondingly, the information of each first region includes the information of its multiple faces, and the information of each face is indicated by the geometric information of the 3D model.

[0081] In this embodiment of the application, the second region corresponding to the second texture fragment includes at least two sub-regions, which can be indirectly adjacent or adjacent to each other. For example, the at least two sub-regions include three sub-regions A, B, and C, where A and B are adjacent, and B and C are adjacent; or, A is adjacent to B and C respectively, and B is adjacent to C.

[0082] Each sub-region includes at least a portion of the first region corresponding to a first texture fragment. That is, each sub-region may include the entire first region corresponding to a first texture fragment, or it may include only a portion of the first region corresponding to a first texture fragment. Since the textures on each first texture fragment belong to the same texture type, but the pixels in different regions under the same texture type can be different, such as wood texture, the first region corresponding to a first texture fragment can be divided into one or at least two sub-regions. The sub-regions of the second texture fragment may only include a portion of the first region corresponding to the first texture fragment.

[0083] In some embodiments, the textures of at least two sub-regions on the first texture image satisfy a preset similarity, which can be set and changed as needed. The similarity of textures between sub-regions can be determined as follows: feature vectors of the texture of each sub-region are extracted based on a neural network, and then the texture similarity is obtained by calculating the Euclidean distance or cosine similarity between the feature vectors. It should be noted that, since at least two sub-regions are indirectly adjacent or pairwise adjacent, for each sub-region, only the texture similarity between that sub-region and its adjacent sub-regions is determined.

[0084] In this process, a first texture fragment in a first texture image is re-divided on a second texture image, and a second region corresponding to at least one second texture fragment in the second texture image comprises at least two sub-regions. Optionally, since the region corresponding to the second texture fragment comprises at least two sub-regions, and since each sub-region comprises at least a portion of the first region corresponding to the first texture fragment—meaning the textures between these at least two sub-regions are only similar, not identical—a new texture needs to be generated for the second texture fragment. Accordingly, the texture of the second texture fragment is generated based on the textures of these at least two sub-regions. Optionally, the texture prediction model generates the texture of the second texture fragment based on the textures of these at least two sub-regions, i.e., the texture is generated using artificial intelligence. Alternatively, the average pixel value of the textures of the at least two sub-regions on the first texture fragment is used as the pixel value of the texture of the second texture fragment.

[0085] In this embodiment, geometric information and a first texture image are input into a texture prediction model, which then outputs a second texture image. The texture prediction model is used to obtain the structural features of the 3D model in 3D space based on the geometric information; that is, the geometric information provides the texture prediction model with the structural features of the 3D model in 3D space. Based on the geometric information and the first texture image, the texture features at various structural locations of the 3D model are obtained. The texture prediction model then fuses the structural and texture features of the 3D model to output the second texture image.

[0086] In some embodiments, please refer to Figure 4 The texture prediction model includes a 3D convolutional network 41 and an output network 42. The process of inputting geometric information and a first texture image into the texture prediction model and outputting a second texture image by the texture prediction model includes at least one of the following steps:

[0087] Step 321: Input the geometric information and the first texture image into the three-dimensional convolutional network, and output the first fused feature of the geometric information and the first texture image from the three-dimensional convolutional network.

[0088] In some embodiments, please refer to Figure 4 The three-dimensional convolutional network 41 includes a three-dimensional convolutional layer 411 and a normalization layer 412, and step 321 above includes at least one of the following steps:

[0089] 1. Input the geometric information and the first texture image into the three-dimensional convolutional layer. The three-dimensional convolutional layer performs convolution processing on the geometric information and the first texture image to obtain the second fused feature.

[0090] In some embodiments, to adapt to the input format of the deep learning model, the geometric information and the first texture image are converted into tensor form. This process includes tensor transformations of the test set and the training set. For example, the input tensor X of the training set... trainand output tensor Y train and the input tensor X of the test set test and output tensor Y test They are shown below:

[0091]

[0092] Vertex coordinates are represented as (X, Y, Z) coordinates and can be stored in a P×3 matrix, resulting in a tensor of vertex coordinates. N represents the number of vertices, and each row of the matrix represents a vertex coordinate. Texture coordinates are used to map the 2D texture image onto the surface of the 3D model. Each vertex has a texture coordinate (U, V), representing the vertex's position on the texture image. Texture coordinates can be stored in a P×2 matrix, resulting in a tensor of texture coordinates, with each row of the matrix representing a texture coordinate. Concatenating these two matrices along a feature dimension yields a P×5 matrix, which is the concatenated tensor of vertex and texture coordinates. Each row of this matrix contains information about a vertex (X, Y, Z, U, V).

[0093] Converting the first texture image into a tensor can be achieved as follows: Feature extraction is performed on the first texture image. This feature extraction can be implemented using a neural network, and the output of the neural network is a feature tensor. The dimensions of this tensor can be represented as [C, H, W], where C represents the number of color channels in the first texture image, and H and W represent the height and width of the first texture image, respectively. Then, the feature tensor of the first texture image is fused with the concatenated tensor described above. This maps the feature tensor of the first texture image to each vertex, that is, the corresponding feature vector is extracted from the feature tensor based on the texture coordinates of each vertex. The feature tensor of the first texture image is in the form [P, C]. Concatenating this feature tensor with the concatenated tensor of the vertex coordinates and texture coordinates yields a tensor of [P, 5+C].

[0094] Face information is represented by the vertex indices that make up each face. For example, a triangular face is defined by three vertex indices (i, j, k), which can be stored in a Q×3 matrix, resulting in a tensor of face information. Each row in the tensor represents a face, and Q represents the number of faces. The face information tensor is converted to the form [Q, 1, 3]. The 1 in the tensor indicates that there is only one element in that dimension, which serves as a placeholder dimension. This allows the face information tensor to be concatenated with the tensor [P, 5+C] of the vertex coordinates, texture coordinates, and the first texture image along a new dimension, forming a single tensor [P, 5+C+3] containing all the information, which is the input tensor of the 3D convolutional layer.

[0095] For example, a 3D convolutional layer performs convolution processing using the following formula:

[0096]

[0097] Where D represents the depth of the input data for the 3D convolutional layer, H represents the height of the input data for the 3D convolutional layer, and W represents the width of the input data for the 3D convolutional layer. d,h,w X represents the convolution kernel in a 3D convolutional layer. i+d-1,j+h-1,k+w-1 Y represents the value at depth i+d-1, width j+h-1, and height k+w-1 in the input data. i,j,k This represents the value at depth i, width j+h-1, and height h in the output data of a 3D convolutional layer. D, H, and W are all integers greater than 1.

[0098] 2. Input the second fusion feature into the normalization layer, and the normalization layer performs the second mapping process on the second fusion feature to obtain the third fusion feature. The second mapping process is used to map the value of the second fusion feature to the second numerical range.

[0099] In some embodiments, the second mapping process can be batch normalization. The batch in batch normalization refers to a subset of training samples used in each iteration of the training process. This subset of training samples is organized into a batch, and the second mapping process is performed on this batch of training samples based on the second fusion features. For example, the second mapping process is implemented using the following formula:

[0100]

[0101] Among them, Y i,j,k This represents the input to the normalization layer, i.e., the second fused feature. ∈ represents a small constant to prevent the denominator from being zero. μ B This represents the mean of the second fusion feature of all training samples in the batch to which the current training sample belongs. This represents the variance of the second fusion feature of all training samples in the batch to which the current training sample belongs. This represents the output of the normalization layer, also known as the third fusion feature.

[0102] 3. The third fusion feature is processed by an activation function to obtain the first fusion feature.

[0103] In this embodiment, the activation function is a function used by the 3D convolutional layer to introduce non-linearity. The activation function determines whether the output of a node in the 3D convolutional layer is activated. The activation function in this embodiment can be a Sigmoid function, a Tanh function, or a ReLU function, etc.

[0104] In this embodiment, the activation function is described using the ReLU function as an example. For instance, the activation function is shown in the following formula:

[0105] f(x) = max(0,x)

[0106] Here, x represents the third fusion feature, and f(x) represents the first fusion feature. This activation function sets negative values ​​in the input data to 0, while keeping positive values ​​unchanged. This activation function is simple to compute and differentiable in the positive interval, which helps alleviate the gradient vanishing problem.

[0107] In some embodiments, the 3D convolutional network includes at least two sub-convolutional layers. Each sub-convolutional layer includes a 3D convolutional layer, a normalization layer, and an activation function. That is, each sub-convolutional layer performs convolution processing on the input data through the 3D convolutional layer, performs a second mapping processing on the convolution result, and performs non-linear processing on the mapping result through the activation function to obtain the output data of the sub-convolutional layer. The input data of the first sub-convolutional layer is the concatenated tensor of vertex coordinates, texture coordinates, surface information, and the first texture image. The output data of the last sub-convolutional layer is the output data of the 3D convolutional network.

[0108] In this embodiment, the three-dimensional convolutional network includes a three-dimensional convolutional layer and a normalization layer. The three-dimensional convolutional layer obtains the fusion features of geometric information and the first texture image, while the normalization layer maps the numerical values ​​of the features output by the three-dimensional convolutional layer to a certain numerical range. This keeps the output of the three-dimensional convolutional network within a reasonable range, which helps the gradient to propagate in the deep network, helps to alleviate the gradient vanishing problem, and thus reduces the number of iterations during training, allowing the model to converge faster, thereby improving training efficiency.

[0109] In this embodiment, a three-dimensional convolutional network is introduced. Since a three-dimensional convolutional network is more suitable for processing three-dimensional data than a two-dimensional convolutional network, it can better capture the structural and texture features of the three-dimensional model in three-dimensional space, obtain rich feature information of the three-dimensional model, and then improve the quality of the predicted texture image based on the feature information.

[0110] Step 322: Input the first fused feature into the output network, and output the second texture image from the output network.

[0111] In some embodiments, please refer to Figure 4 The output network 42 includes an initial convolutional layer 421 and at least two residual blocks 422, and step 322 above includes at least one of the following steps:

[0112] 1. Input the first fused feature into the initial convolutional layer, and the initial convolutional layer outputs the local features of the 3D model.

[0113] In this embodiment, the initial convolutional layer can be a two-dimensional convolutional layer or a three-dimensional convolutional layer. Here, a two-dimensional convolutional layer is used as an example. The initial convolutional layer may include one or at least two two-dimensional convolutional layers, each including one or at least two convolutional kernels. The convolutional kernels perform convolution operations on the first fused features to obtain the local features of the three-dimensional model.

[0114] Local features include local texture features, local grayscale variation features, local texture direction features, local structural features, and edge features. The local features output by the initial convolutional layer can be a multidimensional array or tensor. For example, the dimension of a local feature is H1×W1×D1, which is a three-dimensional matrix, where D1 represents the number of channels, and the number of channels is also the number of dimensions of the feature. Each two-dimensional slice of H1×W1 represents a specific feature channel, and the entire H1×W1×D1 represents the set of all feature channels.

[0115] For example, the local features obtained by the initial convolutional layer can be represented by the following formula:

[0116] Z i =Conv(f i )

[0117] Where Conv represents the convolution operation, F i Z represents the input data of the initial convolutional layer, i.e., the first fused feature. i This represents the output data of the initial convolutional layer, i.e., local features.

[0118] 2. For the first residual block, input the local features into the first residual block, and obtain the first residual feature from the local features. Then, superimpose the first residual feature and the local features to obtain the first superimposed feature.

[0119] For example, the first superimposed feature can be obtained as shown in the following formula:

[0120] H i =f(Z) i )+Z i

[0121] Where i is 1, then Z i This represents the input of the first residual block, i.e., the local features, f(Z). i ) represents the first residual feature, H i This represents the first superimposed feature.

[0122] 3. For the nth residual block, input the (n-1)th superposition feature into the nth residual block. The nth residual block obtains the nth residual feature based on the (n-1)th superposition feature. Superimpose the nth residual feature and the (n-1)th superposition feature to obtain the nth superposition feature, where n is an integer greater than 1.

[0123] In some embodiments, please refer to Figure 5 At least one residual block 422 includes a convolutional layer 4221, an attention layer 4222, and a normalization layer 4223 connected in sequence. The process of inputting the (n-1)th superimposed feature into the nth residual block and obtaining the nth residual feature from the nth residual block based on the (n-1)th superimposed feature includes at least one of the following steps:

[0124] (1) Input the (n-1)th superimposed feature into the convolutional layer in the nth residual block, and perform convolution processing on the (n-1)th superimposed feature by the convolutional layer in the nth residual block to obtain the first feature.

[0125] The convolutional layers in the residual blocks can be two-dimensional or three-dimensional convolutional layers.

[0126] (2) Input the first feature into the attention layer in the nth residual block, and perform feature extraction processing on the first feature by the attention layer in the nth residual block to obtain the second feature. The feature extraction processing is used to extract the texture features and structural features of the three-dimensional model in the first feature.

[0127] The attention layer employs an attention mechanism where the softmax function converts the attention score of each input data point into a probability value. These probabilities sum to 1, forming a probability distribution. This indicates that the attention layer integrates the relative importance of each input data point. The softmax function applies an exponential function to each attention score and normalizes it, ensuring all attention scores are between 0 and 1 and their sum is 1. This normalization ensures the attention layer balances the contributions of different inputs. Through the softmax function, the attention layer learns which inputs are important and which are not in a specific task. Specifically, the attention layer extracts feature representations of the 3D model based on the first feature, which includes both important and unimportant features. Then, the attention layer performs feature enhancement through its attention mechanism, enhancing important features such as texture and structural features while suppressing other unimportant features. This results in more prominent texture and structural features, thus increasing their influence on texture fragment segmentation of the first texture image.

[0128] In this embodiment, the attention layer extracts texture and structural features of the 3D model for texture image segmentation and texture image generation; that is, the attention layer has the capability of texture image segmentation and texture image generation. In some embodiments, the attention layer includes a feature extraction unit, a segmentation unit, and an output unit. The feature extraction unit extracts texture and structural features of the 3D model from a first feature. The segmentation unit segments a first texture fragment on the first texture image based on the texture and structural features to obtain segmentation features. The segmentation features indicate the regions corresponding to each second texture fragment on the second texture image, i.e., the area contours of each second texture fragment are obtained. The output unit obtains a second feature based on the segmentation features, texture features, and structural features. The second feature includes the texture features of each second texture fragment on the second texture image; that is, the output unit generates textures for each second texture fragment, and the second feature can be a feature map. Optionally, the output unit performs convolution operations on the segmentation features, texture features, and structural features through at least one convolutional layer to obtain the second feature.

[0129] For example, the feature extraction unit extracts features using the following formula:

[0130]

[0131] Among them, A i The output of the feature extraction unit is represented by Q, which includes texture features and structural features; i This represents the product of the first feature and the query vector. V represents the transpose of the product of the first feature and the key vector. i This represents the product of the first feature and the value vector. The query vector, key vector, and value vector are represented as Q, K, and V, respectively, all of which are neural network parameters in the attention mechanism. k K represents the dimension, and softmax represents the normalization function.

[0132] The texture features and structural features include the texture features and structural features of each sub-region on the 3D model. These texture features and structural features are used to divide the first texture fragment on the first texture image. Both texture features and structural features are feature vectors. Optionally, a first similarity between each sub-region is calculated based on the texture features of each sub-region, and a second similarity between each sub-region is calculated based on the structural features of each sub-region. Either similarity can be Euclidean distance or cosine similarity. The first similarity and the second similarity are weighted and summed to obtain the texture similarity between each sub-region. Then, sub-regions whose similarity satisfies the preset similarity are grouped into a second region corresponding to a second texture fragment.

[0133] In the embodiments of this application, each residual block may include a convolutional layer, an attention layer, a normalization layer, and an activation function; alternatively, some residual blocks may only include a convolutional layer, a normalization layer, and an activation function without an attention layer, such as only the last few residual blocks including an attention layer.

[0134] In some embodiments, the attention layer is a pre-trained neural network, and its parameters are not adjusted during the training of the texture prediction model. Optionally, the attention layer is trained based on at least two training samples. Each training sample of the attention layer includes the geometric information of the 3D model and a sample texture image. During training, the geometric information and the sample texture image are input into the attention layer, and the attention layer outputs a predicted texture image; the attention layer is iteratively trained based on the difference between the sample texture image and the predicted texture image.

[0135] (3) Input the second feature into the normalization layer in the nth residual block, and perform the first mapping process on the second feature by the normalization layer in the nth residual block to obtain the third feature. The first mapping process is used to map the value of the second feature to the first value range.

[0136] In some embodiments, the first mapping process is similar to the second mapping process described above, and will not be repeated here.

[0137] In other embodiments, the first mapping process is shown in the following formula:

[0138] F=γE+β

[0139] Where F represents the third feature, E represents the first feature, γ represents the scaling factor, and β represents the offset factor. The two factors are used to adjust the value of the first feature.

[0140] (4) The third feature is processed by the activation function to obtain the nth residual feature.

[0141] In this embodiment, the activation function may be a Sigmoid function, a Tanh function, or a ReLU function. In this embodiment, the ReLU function will be used as an example for explanation.

[0142] In this embodiment, the residual block includes an attention layer, meaning that a residual connection mechanism and an attention mechanism are introduced into the output network. The residual connection mechanism solves the gradient vanishing problem in deep neural networks. Furthermore, the introduction of the attention mechanism enables the texture prediction model to more effectively focus on important texture and structural features. This combination of mechanisms not only improves the expressive power of the texture prediction model, but also enhances its ability to capture and learn complex patterns and relationships in the data. This allows the texture prediction model to more accurately identify, understand, and predict features and trends in the data, thereby enhancing the detail and quality of the generated texture image.

[0143] 4. The superimposed features output by the last residual block are determined as the second texture image.

[0144] In some embodiments, if the first texture image is a normalized texture image, then the second texture image is subjected to inverse normalization, that is, the pixel of each pixel in the second texture image is multiplied by 255.0 to restore the value of the pixel in the second texture image to [0, 255.0].

[0145] In this embodiment, the texture prediction model predicts texture images through a three-dimensional convolutional network and an output network, which can be expressed by the following formula:

[0146] F i =3D-CNN(X) i )

[0147] I″ i =RESNET(F i )

[0148] Where 3D-CNN represents a three-dimensional convolutional network, RESNET represents the output network, and X... i F represents the geometric information and first texture image of the i-th 3D model, which is also the input data of the 3D convolutional network. i The first fusion feature representing geometric information and the first texture image, which is also the output data of the 3D convolutional network, is simultaneously the input data of the output network; I″ i This represents the output data of the output network, which is also the second texture image.

[0149] In this embodiment, the output network includes an initial convolutional layer and at least two residual blocks, which introduces a residual connection mechanism. The residual connection mechanism solves the gradient vanishing problem in deep neural networks. This mechanism allows gradients to propagate smoothly even in very deep neural networks, thereby enabling the weights of the neural network to be updated effectively. This allows the neural network to learn effective feature representations faster and improves the model training speed.

[0150] In this embodiment, a three-dimensional convolutional network and a residual network are combined to form an end-to-end deep learning framework, which achieves seamless fusion of geometric information and texture images. This enables the deep learning framework to automatically handle the entire process from geometric feature and texture feature extraction to texture image generation, thereby improving efficiency.

[0151] In other embodiments, the output network includes an attention layer but excludes the initial convolutional layer and residual blocks. The attention layer is used to extract texture and structural features of the 3D model based on the first fusion features. The texture and structural features are used to segment the first texture fragments on the first texture image to obtain the second texture image.

[0152] In this embodiment, by introducing an attention layer into the output network, important texture and structural features are extracted from the rich features of the 3D model extracted by the 3D convolutional network. This makes the extracted texture and structural features more accurate. Based on these important features, the attention layer performs texture image segmentation and texture image generation, thereby improving the accuracy of texture image segmentation and the quality of the generated texture image.

[0153] Step 330: Based on the difference between the second texture image and the first texture image, iteratively train the texture prediction model.

[0154] In this embodiment, the parameters of the texture prediction model are adjusted with the goal of minimizing the difference between the second texture image and the first texture image. By adjusting the parameters of the texture prediction model with the goal of minimizing the difference between the two texture images, the second texture image generated by the texture prediction model gradually becomes closer to the first texture image, resulting in a high-quality second texture image.

[0155] Optionally, the difference between the second texture image and the first texture image includes pixel differences and differences in the feature space. The process described above, which aims to minimize the difference between the second texture image and the first texture image, of adjusting the parameters of the texture prediction model includes at least one of the following implementation methods.

[0156] The first implementation method includes at least one of the following steps.

[0157] 1. Based on the second texture image and the first texture image, determine the first loss function value, which is used to indicate the difference between the pixels of the second texture image and the pixels of the first texture image.

[0158] In some embodiments, the first loss function value is the mean squared loss function value, which can be obtained by the following formula:

[0159]

[0160] Among them, L MSE This represents the first loss function value, where G represents the number of pixels in the first and second texture images. The two images have the same number of pixels. I″ i I represents the pixel of the i-th pixel in the second texture image. i′ represents the pixel of the i-th pixel in the first texture image, |·| 2 It represents the square of the absolute value.

[0161] 2. After iteratively adjusting the parameters of the texture prediction model based on the first loss function value at least twice, and when the first iteration stopping condition is met, determine the second loss function value based on the second texture image and the first texture image obtained in the last iteration process. The second loss function value is used to indicate the difference between the second texture image and the first texture image obtained in the last iteration process at the feature level of the deep neural network.

[0162] In the embodiments of this application, one or at least two training samples are used in each iteration process. If at least two training samples are used, the mean between the first loss function values ​​of the at least two training samples is determined, and the parameters of the texture prediction model are adjusted based on the mean.

[0163] The first iteration stopping condition includes at least one of the following: the number of iterations reaches a first threshold, and the first loss function value is less than or equal to a second threshold. The first iteration stopping condition is the criterion for determining when to terminate the updating of model parameters based on the first loss function value. The first threshold is the maximum number of iterations. Optionally, the first threshold for the number of iterations is preset by those skilled in the art. The second threshold represents the minimum acceptable error of the first loss function value. Optionally, the second threshold is preset by those skilled in the art, and this embodiment does not limit this.

[0164] In this process, features of the first texture image and the second texture image can be extracted using a pre-trained deep neural network (such as a convolutional neural network). The extracted features of the first texture image and the second texture image are used to obtain the second loss function value. For example, the Euclidean distance between the two features can be calculated and used as the second loss function value.

[0165] The features extracted by the deep neural network include the texture and structural information of the texture image. By comparing the distance between two texture images in the feature space, rather than comparing pixel differences, the second loss function value focuses more on the perceptual quality of the texture image, which is consistent with the human eye's perception of image quality. This makes the reconstructed second texture image visually more similar to the first texture image. Then, the parameters of the texture prediction model are adjusted based on the second loss function value, so that the second texture image output by the texture prediction model is as visually similar to the first texture image as possible, and closer to the first texture image.

[0166] For example, the value of the second loss function is obtained by the following formula:

[0167]

[0168] Among them, L Perceptual This represents the value of the second loss function, and φ represents the feature extractor of the deep neural network, which can be a convolutional network. φ(I″) i ) represents the features extracted from the second texture image by the deep neural network, φ(I i ′) represents the features extracted from the first texture image by a deep neural network, |·| 2 The L2 (Euclidean distance) distance is represented by S, which represents the number of training samples used in this iteration. In other words, the second loss function value is the total second loss function value of multiple training samples.

[0169] 3. If the second loss function value satisfies the second iteration stopping condition, the texture prediction model is obtained; if the second loss function value does not satisfy the second iteration stopping condition, the parameters of the texture prediction model are adjusted iteratively based on the second loss function value.

[0170] The second iteration condition refers to the second loss function value being less than or equal to the third threshold. The third threshold represents the minimum acceptable error of the second loss function value. Optionally, the third threshold is preset by those skilled in the art, and this embodiment does not limit this.

[0171] In other embodiments, if the second loss function value does not meet the second iteration stopping condition, the parameters of the texture prediction model can be iteratively adjusted based on the first and second loss function values. For example, a total loss function value between the first and second loss function values ​​can be determined, and the parameters of the texture prediction model can be iteratively adjusted based on this total loss function value. Optionally, the total loss function value is the mean of the two loss function values.

[0172] Adjusting the parameters of a texture prediction model using only pixel-based loss might result in a smoother predicted texture image, leading to a loss of detail. In this embodiment, after training the texture prediction model to a certain extent using pixel-based loss, introducing a visual perception loss enhances the detail and texture features of the predicted texture image, making it closer to the texture features of the first texture image. Furthermore, adjusting the parameters of the texture prediction model using a second loss function value can be used for style transfer tasks, applying the style of one texture image to another. This helps generate a second texture image with the style of the first texture image during texture image reconstruction. In other words, adjusting the parameters of the texture prediction model based on the second loss function value improves the quality of the reconstructed texture image, resulting in better local sharpness and structural integrity.

[0173] The second implementation method includes at least one of the following steps.

[0174] 1. Based on the second texture image and the first texture image, determine the first loss function value, which is used to indicate the difference between the pixels of the second texture image and the pixels of the first texture image.

[0175] In this step, the process of determining the first loss function value is the same as the process of determining the first loss function value in the first implementation method described above, and will not be repeated here.

[0176] 2. Based on the second texture image and the first texture image, determine the second loss function value. The second loss function value is used to indicate the difference between the second texture image and the first texture image at the feature level of the deep neural network.

[0177] In this step, the process of determining the value of the second loss function is the same as that of determining the value of the second loss function in the first implementation method described above, and will not be repeated here.

[0178] 3. Determine the total loss function value based on the first loss function value and the second loss function value.

[0179] In this embodiment, the total loss function value can be determined by the following implementation method (1) or (2): (1) Determine the mean of the first loss function value and the second loss function value, and use this mean as the total loss function value. (2) Perform a weighted summation of the first loss function value and the second loss function value, and use the resulting value as the total loss function value. The weights of the first loss function value and the second loss function value may need to be set. Optionally, the weights of the first loss function value and the second loss function value may be preset by a person skilled in the art.

[0180] 4. Based on the total loss function value, iteratively adjust the parameters of the texture prediction model.

[0181] In the embodiments of this application, one or at least two training samples are used in each iteration process. If at least two training samples are used in each iteration process, the mean of the total loss function of the at least two training samples is determined, and the parameters of the texture prediction model are adjusted based on the mean.

[0182] Specifically, based on the total loss function value, the parameters of the texture prediction model are iteratively adjusted until the third iteration stopping condition is met, thus obtaining the texture prediction model.

[0183] The third iteration stopping condition includes at least one of the following: the number of iterations reaches a fourth threshold, and the total loss function value is less than or equal to a fifth threshold. The third iteration stopping condition is the criterion for determining when to terminate the update of model parameters based on the total loss function value. The fourth threshold is the maximum number of iterations. Optionally, the fourth threshold for the number of iterations is preset by those skilled in the art. The fifth threshold represents the minimum acceptable error of the total loss function value. Optionally, the fifth threshold is preset by those skilled in the art, and this embodiment does not limit this.

[0184] In this embodiment, the parameters of the texture prediction model are adjusted by combining pixel loss (first loss function value) and perceptual loss (second loss function value). Since the two losses focus on pixel-level differences and high-level visual feature differences respectively, this combination makes the second texture image generated by the texture prediction model not only numerically closer to the first texture image, but also visually more natural and realistic, thereby improving the quality of the second texture image.

[0185] In some embodiments, after performing steps 310-330 above, a texture prediction model is obtained. In other embodiments, please refer to... Figure 6 In addition, at least one of the following steps 340-370 is performed.

[0186] Step 340: Render the 3D model based on the first texture image to obtain a first performance parameter, the first performance parameter including at least one of a first rendering time and a first rendering completeness; Render the 3D model based on the second texture image to obtain a second performance parameter, the second performance parameter including at least one of a second rendering time and a second rendering completeness.

[0187] In this embodiment, both the first and second texture images are rendered onto the 3D model based on its geometric information. In some embodiments, a computer device has an application that supports texture rendering installed and running, through which the 3D model is rendered. This application provides a rendering engine. For example, it may provide at least one of Unreal Engine 4 (UE4), Unity3D, RAGE Engine, or Frostbite Engine.

[0188] Here, any rendering time refers to the time spent rendering the texture image onto the 3D model. Any rendering completeness refers to the ratio of the area of ​​the rendered texture on the 3D model to the total area of ​​the 3D model; the larger the ratio, the greater the rendering completeness. Alternatively, rendering completeness can also be the difference between the texture rendered on the 3D model and the texture in the texture image; the smaller the difference, the greater the rendering completeness. For example, rendering completeness can be the negative of the difference between the pixel value of a pixel on the 3D model and the pixel value of a pixel on the texture image; this difference can be the mean squared error.

[0189] Step 350: Compare the first performance parameter and the second performance parameter to obtain the comparison result.

[0190] Specifically, if the first performance parameter includes a first rendering duration and the second performance parameter includes a second rendering duration, then the two rendering durations are compared. If the first performance parameter includes a first rendering completeness and the second performance parameter includes a second rendering completeness, then the two rendering completenesses are compared. If the first performance parameter includes both the first rendering duration and the first rendering completeness, and the second performance parameter includes both the second rendering duration and the second rendering completeness, then the two rendering durations and the two rendering completenesses are compared separately.

[0191] Step 360: If the comparison result indicates that the second performance parameter is better than the first performance parameter, the texture prediction model is determined as the neural network model for predicting texture images.

[0192] Specifically, if the first performance parameter includes a first rendering duration, the second performance parameter includes a second rendering duration, and the first rendering duration is longer than the second rendering duration, then the comparison result indicates that the second performance parameter is superior to the first performance parameter. If the first performance parameter includes a first rendering completeness, the second performance parameter includes a second rendering completeness, and the first rendering completeness is less than the second rendering completeness, then the comparison result indicates that the second performance parameter is superior to the first performance parameter. If the first performance parameter includes both the first rendering duration and the first rendering completeness, the second performance parameter includes both the second rendering duration and the second rendering completeness, and the first rendering duration is longer than the second rendering duration and the first rendering completeness is less than the second rendering completeness, then the comparison result indicates that the second performance parameter is superior to the first performance parameter.

[0193] Step 370: If the comparison result indicates that the first performance parameter is better than the second performance parameter, continue training the texture prediction model.

[0194] Specifically, if the first performance parameter includes a first rendering duration, the second performance parameter includes a second rendering duration, and the first rendering duration is less than the second rendering duration, then the comparison result indicates that the first performance parameter is superior to the second performance parameter. If the first performance parameter includes a first rendering completeness, the second performance parameter includes a second rendering completeness, and the first rendering completeness is greater than the second rendering completeness, then the comparison result indicates that the first performance parameter is superior to the second performance parameter. If the first performance parameter includes both the first rendering duration and the first rendering completeness, the second performance parameter includes both the second rendering duration and the second rendering completeness, and the first rendering duration is less than the second rendering duration and the first rendering completeness is greater than the second rendering completeness, then the comparison result indicates that the first performance parameter is superior to the second performance parameter.

[0195] Please refer to Figure 7 This paper uses the rendering of a 3D model in a single frame of a game application as an example to illustrate the comparison of performance parameters. First, a first texture image of the 3D model is obtained, and a second texture image is obtained through a texture prediction model. The performance parameters of rendering the 3D model based on the first and second texture images are recorded respectively, and the performance parameters are compared to see if there is any optimization. If the second performance parameter is better than the first performance parameter, it means that rendering the 3D model using the second texture image does not increase performance consumption excessively, indicating excellent performance, and thus a texture prediction model is obtained.

[0196] It should be noted that steps 360 and 370 are optional steps, and their numbers are for illustrative purposes only and are not intended to restrict the execution order of the two steps.

[0197] In this embodiment, after obtaining the trained texture prediction model, the second texture image predicted by the texture prediction model and the original first texture image are used to render a 3D model respectively. Then, the rendering performance parameters are compared. Only when the second performance parameter is better than the first performance parameter is the final texture prediction model obtained. Otherwise, the texture prediction model is trained again. In this way, the performance of the texture prediction model is evaluated based on the actual rendering performance of the texture image predicted by the texture prediction model, and the predicted texture model is adjusted based on the rendering performance. Thus, when rendering a 3D model based on the texture image predicted by the trained texture prediction model, the rendering performance can be effectively improved, which also ensures the practicality of the texture prediction model.

[0198] On the one hand, based on the geometric information of the 3D model and the first texture image, a texture prediction model generates a second texture image for the 3D model. At least one second texture fragment in the second texture image corresponds to a second region comprising at least two sub-regions. Since each sub-region includes at least a portion of the first region corresponding to a first texture fragment in the first texture image, the second texture image aggregates textures from some sub-regions into larger second texture fragments. This increases the size of the region on the 3D model corresponding to the second texture fragment, while reducing the number of texture fragments included in the second texture image. Because each texture fragment is rendered once, the number of rendering operations is reduced when rendering the 3D model based on the second texture image, thus reducing computation and rendering errors, thereby improving rendering efficiency and quality. On the other hand, the texture prediction model is iteratively trained based on the differences between the first and second texture images. This allows the texture prediction model to not only re-divide the texture image but also ensure that the predicted texture image is as similar as possible to the original texture image of the 3D model, guaranteeing the quality of the predicted texture image.

[0199] Please refer to Figure 8 This document illustrates a flowchart of a texture prediction method provided in one embodiment of this application. The execution entity for each step of this method can be a computer device. The texture prediction model used in this embodiment is either the texture prediction model obtained through steps 310-330 above, or the texture prediction model obtained through steps 310-370 above. The method may include at least one of the following steps 810-820.

[0200] Step 810: Obtain the geometric information of the three-dimensional model and the first texture image. The first texture image includes at least two first texture fragments, which are used to determine the texture of a first region on the three-dimensional model.

[0201] In this embodiment, the geometric information of the three-dimensional model and the first texture image are similar to those described in step 310, and will not be repeated here.

[0202] In some embodiments, the first texture image is a normalized texture image. Specifically, the original texture image of the 3D model is normalized to obtain the first texture image of the 3D model; the normalization process maps the pixel values ​​of the pixels in the original texture image to a third numerical range.

[0203] The specific process of performing normalization on the original texture image is the same as that in step 310, and will not be repeated here.

[0204] Step 820: Input geometric information and the first texture image into the texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to a first texture fragment.

[0205] In some embodiments, the texture prediction model includes a three-dimensional convolutional network and an output network; the process of inputting geometric information and a first texture image into the texture prediction model and outputting a second texture image by the texture prediction model includes at least one of the following steps: inputting geometric information and the first texture image into the three-dimensional convolutional network and outputting a first fusion feature of the geometric information and the first texture image by the three-dimensional convolutional network; inputting the first fusion feature into the output network and outputting the second texture image by the output network.

[0206] In some embodiments, the output network includes an attention layer for extracting texture and structural features of a 3D model based on a first fusion feature. The texture and structural features are used to segment a first texture fragment on a first texture image to obtain a second texture image.

[0207] In some embodiments, the output network includes an initial convolutional layer and at least two residual blocks; the process of inputting the first fusion feature into the output network and outputting the second texture image by the output network includes at least one of the following steps: inputting the first fusion feature into the initial convolutional layer, and outputting local features of the 3D model by the initial convolutional layer; for the first residual block, inputting the local features into the first residual block, obtaining the first residual feature from the first residual block based on the local features, and superimposing the first residual feature and the local features to obtain the first superimposed feature; for the nth residual block, inputting the (n-1)th superimposed feature into the nth residual block, obtaining the nth residual feature from the nth residual block based on the (n-1)th superimposed feature, and superimposing the nth residual feature and the (n-1)th superimposed feature to obtain the nth superimposed feature, where n is an integer greater than 1; and determining the superimposed feature output by the last residual block as the second texture image.

[0208] In some embodiments, a residual block includes a convolutional layer, an attention layer, and a normalization layer connected in sequence. The process of inputting the aforementioned n-1 superimposed features into the nth residual block and obtaining the nth residual feature from the nth residual block based on the n-1 superimposed features includes at least one of the following steps: inputting the n-1 superimposed features into the convolutional layer in the nth residual block, performing convolution processing on the n-1 superimposed features by the convolutional layer in the nth residual block to obtain a first feature; inputting the first feature into the attention layer in the nth residual block, performing feature extraction processing on the first feature by the attention layer in the nth residual block to obtain a second feature, wherein the feature extraction processing is used to extract the texture features and structural features of the 3D model in the first feature; inputting the second feature into the normalization layer in the nth residual block, performing a first mapping processing on the second feature by the normalization layer in the nth residual block to obtain a third feature, wherein the first mapping processing is used to map the value of the second feature to a first numerical range; and processing the third feature through an activation function to obtain the nth residual feature.

[0209] In some embodiments, the three-dimensional convolutional network includes a three-dimensional convolutional layer and a normalization layer; the process of inputting geometric information and a first texture image into the three-dimensional convolutional network and outputting a first fusion feature of the geometric information and the first texture image by the three-dimensional convolutional network includes at least one of the following steps: inputting geometric information and the first texture image into the three-dimensional convolutional layer, performing convolution processing on the geometric information and the first texture image by the three-dimensional convolutional layer to obtain a second fusion feature; inputting the second fusion feature into the normalization layer, performing a second mapping processing on the second fusion feature by the normalization layer to obtain a third fusion feature, wherein the second mapping processing is used to map the value of the second fusion feature to a second numerical range; and processing the third fusion feature through an activation function to obtain the first fusion feature.

[0210] For details not covered in step 820, please refer to step 320 above, which will not be repeated here.

[0211] Please refer to Figure 9 , Figure 9 This is a flowchart illustrating the rendering process of a 3D model provided in an embodiment of this application. In the rendering scene of the 3D model, data preparation is first performed, namely, acquiring the geometric information and a first texture image of the 3D model. Then, the geometric information and the first texture image are input into a texture prediction model, which outputs a second texture image. This texture prediction model is a neural network model combining a 3D convolutional network and a residual network. Finally, based on the second texture image and the geometric information, the 3D model is rendered to obtain the rendered 3D model.

[0212] The solution provided in this application embodiment, on the one hand, is to generate a second texture image for the 3D model based on the geometric information of the 3D model and the first texture image, by a texture prediction model. The second texture image contains at least one second texture fragment. Since each sub-region includes at least a portion of the region corresponding to a first texture fragment in the first texture image, that is, the second texture image is a large second texture fragment formed by aggregating the textures of some sub-regions. In this way, the region on the 3D model corresponding to the second texture fragment also becomes larger. At the same time, the number of texture fragments included in the second texture image is reduced. Since one texture fragment is rendered once, the number of rendering times can be reduced when rendering the 3D model based on the second texture image, thereby reducing the amount of computation and reducing the occurrence of rendering errors. Therefore, rendering efficiency and rendering effect can be improved.

[0213] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0214] Please refer to Figure 10 This diagram illustrates a block diagram of a training apparatus for a texture prediction model according to an embodiment of this application. The apparatus has the function of implementing the training method for the aforementioned texture prediction model; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be installed within a computer device. Figure 10 As shown, the device 1000 may include: an acquisition module 1010, an input / output module 1020, and a training module 1030.

[0215] The acquisition module 1010 is used to acquire at least two training samples, each training sample including geometric information of a 3D model and a first texture image, the first texture image including at least two first texture fragments, the at least two first texture fragments being used to determine the texture of a first region on the 3D model respectively;

[0216] The input / output module 1020 is used to input geometric information and a first texture image into a texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on a three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to a first texture fragment.

[0217] Training module 1030 is used to iteratively train the texture prediction model based on the difference between the second texture image and the first texture image.

[0218] In some embodiments, the texture prediction model includes a three-dimensional convolutional network and an output network;

[0219] Input / output module 1020 includes:

[0220] The first input-output submodule is used to input geometric information and a first texture image into a three-dimensional convolutional network, and the three-dimensional convolutional network outputs a first fusion feature of geometric information and the first texture image.

[0221] The second input-output submodule is used to input the first fused feature into the output network and output the second texture image from the output network.

[0222] In some embodiments, the output network includes an attention layer for extracting texture and structural features of a 3D model based on a first fusion feature. The texture and structural features are used to segment a first texture fragment on a first texture image to obtain a second texture image.

[0223] In some embodiments, the output network includes an initial convolutional layer and at least two residual blocks; the second input / output submodule is used for:

[0224] The first fused feature is input into the initial convolutional layer, and the initial convolutional layer outputs the local features of the 3D model;

[0225] For the first residual block, the local features are input into the first residual block, and the first residual block obtains the first residual feature based on the local features. The first residual feature and the local features are superimposed to obtain the first superimposed feature.

[0226] For the nth residual block, the (n-1)th superposition feature is input into the nth residual block. The nth residual block obtains the nth residual feature based on the (n-1)th superposition feature. The nth residual feature and the (n-1)th superposition feature are superimposed to obtain the nth superposition feature, where n is an integer greater than 1.

[0227] The superimposed features output from the last residual block are used to determine the second texture image.

[0228] In some embodiments, the residual block includes a convolutional layer, an attention layer, and a normalization layer connected in sequence; the second input / output submodule is used for:

[0229] The (n-1)th superimposed feature is input into the convolutional layer in the nth residual block, and the convolutional layer in the nth residual block performs convolution processing on the (n-1)th superimposed feature to obtain the first feature;

[0230] The first feature is input into the attention layer in the nth residual block, and the attention layer in the nth residual block performs feature extraction processing on the first feature to obtain the second feature. The feature extraction processing is used to extract the texture features and structural features of the 3D model in the first feature.

[0231] The second feature is input into the normalization layer in the nth residual block, and the normalization layer in the nth residual block performs the first mapping process on the second feature to obtain the third feature. The first mapping process is used to map the value of the second feature to a first value range.

[0232] The third feature is processed by an activation function to obtain the nth residual feature.

[0233] In some embodiments, the three-dimensional convolutional network includes a three-dimensional convolutional layer and a normalization layer; a first input / output submodule is configured to:

[0234] The geometric information and the first texture image are input into the three-dimensional convolutional layer, and the three-dimensional convolutional layer performs convolution processing on the geometric information and the first texture image to obtain the second fused feature.

[0235] The second fusion feature is input into the normalization layer, and the normalization layer performs a second mapping process on the second fusion feature to obtain the third fusion feature. The second mapping process is used to map the value of the second fusion feature to a second numerical range.

[0236] The first fusion feature is obtained by processing the third fusion feature through an activation function.

[0237] In some embodiments, the training module 1030 is used for:

[0238] The parameters of the texture prediction model are adjusted with the goal of minimizing the difference between the second texture image and the first texture image.

[0239] In some embodiments, the training module 1030 is used for:

[0240] Based on the second texture image and the first texture image, a first loss function value is determined, which is used to indicate the difference between the pixels of the second texture image and the pixels of the first texture image;

[0241] After iteratively adjusting the parameters of the texture prediction model based on the first loss function value at least twice, and when the first iteration stopping condition is met, a second loss function value is determined based on the second texture image and the first texture image obtained in the last iteration process. The second loss function value is used to indicate the difference between the second texture image and the first texture image obtained in the last iteration process at the feature level of the deep neural network.

[0242] The texture prediction model is obtained when the value of the second loss function satisfies the second iteration stopping condition;

[0243] If the second loss function value does not meet the second iteration stopping condition, the parameters of the texture prediction model are adjusted iteratively based on the second loss function value.

[0244] In some embodiments, the training module 1030 is used for:

[0245] Based on the second texture image and the first texture image, a first loss function value is determined, which is used to indicate the difference between the pixels of the second texture image and the pixels of the first texture image;

[0246] Based on the second texture image and the first texture image, a second loss function value is determined. The second loss function value is used to indicate the difference between the second texture image and the first texture image at the feature level of the deep neural network.

[0247] The total loss function value is determined based on the first loss function value and the second loss function value.

[0248] The parameters of the texture prediction model are iteratively adjusted based on the total loss function value.

[0249] In some embodiments, the acquisition module 1010 is used for:

[0250] Normalization is performed on the original texture image of the 3D model to obtain the first texture image of the 3D model. The normalization process is used to map the pixel values ​​of the pixels in the original texture image to a third numerical range.

[0251] In some embodiments, the apparatus further includes:

[0252] The processing module is used to perform data augmentation on the 3D model to obtain the processed 3D model. The data augmentation includes at least one of cropping, rotating and flipping.

[0253] The generation module is used to generate augmented training samples based on the geometric information of the processed 3D model and the first texture image.

[0254] In some embodiments, the apparatus further includes:

[0255] A rendering module is used to render a 3D model based on a first texture image and obtain a first performance parameter, the first performance parameter including at least one of a first rendering duration and a first rendering completeness;

[0256] The rendering module is also used to render a 3D model based on a second texture image to obtain a second performance parameter, which includes at least one of a second rendering duration and a second rendering completeness.

[0257] The comparison module is used to compare the first performance parameter and the second performance parameter to obtain the comparison result.

[0258] The determination module is used to determine the texture prediction model as a neural network model for predicting texture images if the comparison result indicates that the second performance parameter is better than the first performance parameter.

[0259] Continue training module 1030, which is used to continue training the texture prediction model if the comparison result indicates that the first performance parameter is better than the second performance parameter.

[0260] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0261] Please refer to Figure 11 This diagram illustrates a block diagram of a texture prediction apparatus according to an embodiment of this application. The apparatus has the function of implementing the texture prediction method described above; this function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be installed within a computer device. For example... Figure 11 As shown, the device 1100 may include: an acquisition module 1110 and an input / output module 1120.

[0262] The acquisition module 1110 is used to acquire the geometric information and first texture image of the three-dimensional model, and at least two first texture fragments are used to determine the texture of a first region on the three-dimensional model.

[0263] The input / output module 1120 is used to input geometric information and a first texture image into a texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on a three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to a first texture fragment.

[0264] In some embodiments, the texture prediction model includes a three-dimensional convolutional network and an output network;

[0265] Input / output module 1120 includes:

[0266] The first input-output submodule is used to input geometric information and a first texture image into a three-dimensional convolutional network, and the three-dimensional convolutional network outputs a first fusion feature of geometric information and the first texture image.

[0267] The second input-output submodule is used to input the first fused feature into the output network and output the second texture image from the output network.

[0268] In some embodiments, the output network includes an attention layer for extracting texture and structural features of a 3D model based on a first fusion feature. The texture and structural features are used to segment a first texture fragment on a first texture image to obtain a second texture image.

[0269] In some embodiments, the output network includes an initial convolutional layer and at least two residual blocks; the second input / output submodule is used for:

[0270] The first fused feature is input into the initial convolutional layer, and the initial convolutional layer outputs the local features of the 3D model;

[0271] For the first residual block, the local features are input into the first residual block, and the first residual block obtains the first residual feature based on the local features. The first residual feature and the local features are superimposed to obtain the first superimposed feature.

[0272] For the nth residual block, the (n-1)th superposition feature is input into the nth residual block. The nth residual block obtains the nth residual feature based on the (n-1)th superposition feature. The nth residual feature and the (n-1)th superposition feature are superimposed to obtain the nth superposition feature, where n is an integer greater than 1.

[0273] The superimposed features output from the last residual block are used to determine the second texture image.

[0274] In some embodiments, the residual block includes a convolutional layer, an attention layer, and a normalization layer connected in sequence; the second input / output submodule is used for:

[0275] The (n-1)th superimposed feature is input into the convolutional layer in the nth residual block, and the convolutional layer in the nth residual block performs convolution processing on the (n-1)th superimposed feature to obtain the first feature;

[0276] The first feature is input into the attention layer in the nth residual block, and the attention layer in the nth residual block performs feature extraction processing on the first feature to obtain the second feature. The feature extraction processing is used to extract the texture features and structural features of the 3D model in the first feature.

[0277] The second feature is input into the normalization layer in the nth residual block, and the normalization layer in the nth residual block performs the first mapping process on the second feature to obtain the third feature. The first mapping process is used to map the value of the second feature to a first value range.

[0278] The third feature is processed by an activation function to obtain the nth residual feature.

[0279] In some embodiments, the three-dimensional convolutional network includes a three-dimensional convolutional layer and a normalization layer; a first input / output submodule is configured to:

[0280] The geometric information and the first texture image are input into the three-dimensional convolutional layer, and the three-dimensional convolutional layer performs convolution processing on the geometric information and the first texture image to obtain the second fused feature.

[0281] The second fusion feature is input into the normalization layer, and the normalization layer performs a second mapping process on the second fusion feature to obtain the third fusion feature. The second mapping process is used to map the value of the second fusion feature to a second numerical range.

[0282] The first fusion feature is obtained by processing the third fusion feature through an activation function.

[0283] In some embodiments, the acquisition module 1110 is used for:

[0284] Normalization is performed on the original texture image of the 3D model to obtain the first texture image of the 3D model. The normalization process is used to map the pixel values ​​of the pixels in the original texture image to a third numerical range.

[0285] It should be noted that the apparatus provided in the above embodiments is only illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments belong to the same concept, and the specific implementation process can be found in the method embodiments, which will not be repeated here.

[0286] Please refer to Figure 12 This diagram illustrates a structural block diagram of a computer device 1200 provided in one embodiment of this application. The computer device 1200 can be a computer device in an implementation environment, used to implement the training method or texture prediction method of the texture prediction model provided in the above embodiments. Specifically:

[0287] Typically, computer device 1200 includes a processor 1210 and a memory 1220.

[0288] Processor 1210 may include one or more processing cores, such as a quad-core processor or a twelfth-core processor. Processor 1210 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1210 may also include a main processor and a coprocessor. The main processor, also known as the central processing unit, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1210 may include a GPU, which is responsible for executing the method steps provided in this application. In some embodiments, processor 1210 may also include an AI processor, which is used to handle computational operations related to machine learning.

[0289] The memory 1220 may include one or more computer-readable storage media, which may be non-transitory. The memory 1220 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1220 are used to store a computer program configured to be executed by one or more processors (such as GPUs) to implement the above-described texture prediction model training method or texture prediction method.

[0290] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on the computer device 1200 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0291] This application also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement a training method or a texture prediction method for any of the above-described implementations of the texture prediction model.

[0292] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform a training method or a texture prediction method for the texture prediction model of any of the above implementations.

[0293] In some embodiments, the computer program product involved in the present application can be deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network can form a blockchain system.

[0294] It should be noted that the data collection and processing in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0295] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0296] All the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here. The above are only optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for a texture prediction model, characterized in that, The method includes: At least two training samples are obtained, each training sample including geometric information of a 3D model and a first texture image, the first texture image including at least two first texture fragments, the at least two first texture fragments being used to determine the texture of a first region on the 3D model; The geometric information and the first texture image are input into the texture prediction model, and the texture prediction model outputs a second texture image. The second texture image includes at least two second texture fragments, and the at least two second texture fragments correspond to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment. The texture prediction model is trained iteratively based on the difference between the second texture image and the first texture image.

2. The method according to claim 1, characterized in that, The texture prediction model includes a three-dimensional convolutional network and an output network; The step of inputting the geometric information and the first texture image into the texture prediction model, and outputting the second texture image by the texture prediction model, includes: The geometric information and the first texture image are input into the three-dimensional convolutional network, and the three-dimensional convolutional network outputs the first fusion feature of the geometric information and the first texture image; The first fusion feature is input into the output network, and the output network outputs the second texture image.

3. The method according to claim 2, characterized in that, The output network includes an attention layer, which is used to extract texture features and structural features of the 3D model based on the first fusion feature. The texture features and the structural features are used to segment the first texture fragment on the first texture image to obtain the second texture image.

4. The method according to claim 3, characterized in that, The output network includes an initial convolutional layer and at least two residual blocks; The step of inputting the first fused feature into the output network and outputting the second texture image by the output network includes: The first fused feature is input into the initial convolutional layer, and the initial convolutional layer outputs the local features of the 3D model; For the first residual block, the local features are input into the first residual block, and the first residual block obtains the first residual feature based on the local features. The first residual feature and the local features are then superimposed to obtain the first superimposed feature. For the nth residual block, the (n-1)th superposition feature is input into the nth residual block, and the nth residual block obtains the nth residual feature based on the (n-1)th superposition feature. The nth residual feature and the (n-1)th superposition feature are superimposed to obtain the nth superposition feature, where n is an integer greater than 1. The superimposed features output by the last residual block are determined as the second texture image.

5. The method according to claim 4, characterized in that, The residual block comprises a convolutional layer, an attention layer, and a normalization layer connected in sequence. The step of inputting the (n-1)th superimposed feature into the nth residual block, and obtaining the nth residual feature from the nth residual block based on the (n-1)th superimposed feature, includes: The (n-1)th superimposed feature is input into the convolutional layer of the nth residual block, and the convolutional layer of the nth residual block performs convolution processing on the (n-1)th superimposed feature to obtain the first feature; The first feature is input into the attention layer in the nth residual block, and the attention layer in the nth residual block performs feature extraction processing on the first feature to obtain the second feature. The feature extraction processing is used to extract the texture features and structural features of the three-dimensional model in the first feature. The second feature is input into the normalization layer in the nth residual block, and the normalization layer in the nth residual block performs a first mapping process on the second feature to obtain a third feature. The first mapping process is used to map the value of the second feature to a first value range. The third feature is processed by an activation function to obtain the nth residual feature.

6. The method according to any one of claims 2 to 5, characterized in that, The three-dimensional convolutional network includes three-dimensional convolutional layers and normalization layers; The step of inputting the geometric information and the first texture image into the three-dimensional convolutional network, and outputting a first fused feature of the geometric information and the first texture image by the three-dimensional convolutional network, includes: The geometric information and the first texture image are input into the three-dimensional convolutional layer, and the three-dimensional convolutional layer performs convolution processing on the geometric information and the first texture image to obtain the second fusion feature; The second fusion feature is input into the normalization layer, and the normalization layer performs a second mapping process on the second fusion feature to obtain a third fusion feature. The second mapping process is used to map the value of the second fusion feature to a second numerical range. The first fusion feature is obtained by processing the third fusion feature through an activation function.

7. The method according to any one of claims 1 to 6, characterized in that, The step of iteratively training the texture prediction model based on the difference between the second texture image and the first texture image includes: The parameters of the texture prediction model are adjusted with the goal of minimizing the difference between the second texture image and the first texture image.

8. The method according to claim 7, characterized in that, The step of adjusting the parameters of the texture prediction model with the goal of minimizing the difference between the second texture image and the first texture image includes: Based on the second texture image and the first texture image, a first loss function value is determined, which is used to indicate the difference between the pixels of the second texture image and the pixels of the first texture image; After iteratively adjusting the parameters of the texture prediction model based on the first loss function value at least twice, and upon reaching the first iteration stopping condition, a second loss function value is determined based on the second texture image obtained in the last iteration process and the first texture image. The second loss function value is used to indicate the difference between the second texture image obtained in the last iteration process and the first texture image at the deep neural network feature level. The texture prediction model is obtained when the value of the second loss function satisfies the second iteration stopping condition; If the second loss function value does not meet the second iteration stopping condition, the parameters of the texture prediction model are iteratively adjusted based on the second loss function value.

9. The method according to claim 7, characterized in that, The step of adjusting the parameters of the texture prediction model with the goal of minimizing the difference between the second texture image and the first texture image includes: Based on the second texture image and the first texture image, a first loss function value is determined, which is used to indicate the difference between the pixels of the second texture image and the pixels of the first texture image; Based on the second texture image and the first texture image, a second loss function value is determined, which is used to indicate the difference between the second texture image and the first texture image at the feature level of the deep neural network. Based on the first loss function value and the second loss function value, determine the total loss function value; Based on the total loss function value, the parameters of the texture prediction model are iteratively adjusted.

10. The method according to any one of claims 1 to 9, characterized in that, The acquisition of at least two training samples includes: Normalization processing is performed on the original texture image of the three-dimensional model to obtain the first texture image of the three-dimensional model. The normalization processing is used to map the pixel values ​​of the pixels in the original texture image to a third numerical range.

11. The method according to any one of claims 1 to 10, characterized in that, The method further includes: Perform data augmentation processing on the three-dimensional model to obtain a processed three-dimensional model, wherein the data augmentation processing includes at least one of cropping, rotating and flipping; Based on the geometric information of the processed 3D model and the first texture image, the expanded training samples are generated.

12. A texture prediction method, characterized in that, The method includes: Obtain the geometric information of the three-dimensional model and a first texture image, wherein the first texture image includes at least two first texture fragments, and the at least two first texture fragments are respectively used to determine the texture of a first region on the three-dimensional model; The geometric information and the first texture image are input into the texture prediction model, and the texture prediction model outputs a second texture image. The second texture image includes at least two second texture fragments, each of which corresponds to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment.

13. The method according to claim 12, characterized in that, The texture prediction model includes a three-dimensional convolutional network and an output network; The step of inputting the geometric information and the first texture image into the texture prediction model, and outputting the second texture image by the texture prediction model, includes: The geometric information and the first texture image are input into the three-dimensional convolutional network, and the three-dimensional convolutional network outputs the first fusion feature of the geometric information and the first texture image; The first fusion feature is input into the output network, and the output network outputs the second texture image.

14. The method according to claim 13, characterized in that, The output network includes an attention layer, which is used to extract texture features and structural features of the 3D model based on the first fusion feature. The texture features and the structural features are used to segment the first texture fragment on the first texture image to obtain the second texture image.

15. The method according to claim 14, characterized in that, The output network includes an initial convolutional layer and at least two residual blocks; The step of inputting the first fused feature into the output network and outputting the second texture image by the output network includes: The first fused feature is input into the initial convolutional layer, and the initial convolutional layer outputs the local features of the 3D model; For the first residual block, the local features are input into the first residual block, and the first residual block obtains the first residual feature based on the local features. The first residual feature and the local features are then superimposed to obtain the first superimposed feature. For the nth residual block, the (n-1)th superposition feature is input into the nth residual block, and the nth residual block obtains the nth residual feature based on the (n-1)th superposition feature. The nth residual feature and the (n-1)th superposition feature are superimposed to obtain the nth superposition feature, where n is an integer greater than 1. The superimposed features output by the last residual block are determined as the second texture image.

16. The method according to claim 15, characterized in that, The residual block comprises a convolutional layer, an attention layer, and a normalization layer connected in sequence. The step of inputting the (n-1)th superimposed feature into the nth residual block, and obtaining the nth residual feature from the nth residual block based on the (n-1)th superimposed feature, includes: The (n-1)th superimposed feature is input into the convolutional layer of the nth residual block, and the convolutional layer of the nth residual block performs convolution processing on the (n-1)th superimposed feature to obtain the first feature; The first feature is input into the attention layer in the nth residual block, and the attention layer in the nth residual block performs feature extraction processing on the first feature to obtain the second feature. The feature extraction processing is used to extract the texture features and structural features of the three-dimensional model in the first feature. The second feature is input into the normalization layer in the nth residual block, and the normalization layer in the nth residual block performs a first mapping process on the second feature to obtain a third feature. The first mapping process is used to map the value of the second feature to a first value range. The third feature is processed by an activation function to obtain the nth residual feature.

17. The method according to any one of claims 13 to 16, characterized in that, The three-dimensional convolutional network includes three-dimensional convolutional layers and normalization layers; The step of inputting the geometric information and the first texture image into the three-dimensional convolutional network, and outputting a first fused feature of the geometric information and the first texture image by the three-dimensional convolutional network, includes: The geometric information and the first texture image are input into the three-dimensional convolutional layer, and the three-dimensional convolutional layer performs convolution processing on the geometric information and the first texture image to obtain the second fusion feature; The second fusion feature is input into the normalization layer, and the normalization layer performs a second mapping process on the second fusion feature to obtain a third fusion feature. The second mapping process is used to map the value of the second fusion feature to a second numerical range. The first fusion feature is obtained by processing the third fusion feature through an activation function.

18. A training device for a texture prediction model, characterized in that, The device includes: The acquisition module is used to acquire at least two training samples, each training sample including geometric information of a 3D model and a first texture image, the first texture image including at least two first texture fragments, the at least two first texture fragments being used to determine the texture of a first region on the 3D model; The input / output module is used to input the geometric information and the first texture image into the texture prediction model, and output a second texture image from the texture prediction model. The second texture image includes at least two second texture fragments, and the at least two second texture fragments respectively correspond to a second region on the three-dimensional model. The second region corresponding to at least one second texture fragment includes at least two sub-regions, and each sub-region includes at least a portion of the first region corresponding to the first texture fragment. The training module is used to iteratively train the texture prediction model based on the difference between the second texture image and the first texture image.

19. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store a computer program, the computer program being loaded by the processor and executed as a training method for the texture prediction model according to any one of claims 1 to 11 or a texture prediction method according to any one of claims 12 to 17.

20. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for executing the training method of the texture prediction model according to any one of claims 1 to 11 or the texture prediction method according to any one of claims 12 to 17.

Images