Three-dimensional object texture enhancement method based on coordinate mapping and super-resolution

Abstract

The embodiment of the invention discloses a three-dimensional object texture enhancement method based on coordinate mapping and super-resolution. A specific embodiment of the method comprises the steps of obtaining a scene image data sequence corresponding to a target scene; constructing a three-dimensional scene grid model; generating a scene texture expansion view and image data of each model; the following training steps are executed: obtaining a high-resolution texture expansion graph; generating image data of each rendering model; generating a pixel loss value and a perception loss value; generating an adversarial loss value; generating an image loss value; in response to determining that the image loss value does not meet the preset loss condition, adjusting the parameters of the initial neural network, and executing the training step again; in response to determining that the image loss value meets the loss condition, determining a generative network in the initial neural network as a generative model; and generating a target three-dimensional scene grid model. According to the embodiment, enough training data can be provided for training of the generative network, and the precision and accuracy of the generated three-dimensional grid model can be improved.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the field of computer technology, and more particularly to a method for enhancing the texture of three-dimensional objects based on coordinate mapping and super-resolution. Background Art

[0002] With the rapid progress of digital media and graphics technology, obtaining high-quality three-dimensional models has become increasingly crucial. As a core technology in 3D modeling, UV texture mapping mainly serves to accurately map 2D textures onto the surface of 3D models, thereby significantly enhancing the visual effect. Therefore, in multiple fields, especially virtual reality (VR), augmented reality (AR), and game development, the demand for creating high-resolution UV texture maps is on the rise. These application fields have extremely high requirements for realistic visual effects, and high-quality texture details are the key to achieving this goal.

[0003] In the fields of VR and AR, the clarity and details of images are crucial for enhancing the user's immersion. Textures play a core role in the rendering process of three-dimensional models, endowing the models with a more realistic and delicate appearance. By accurately mapping textures onto the model surface, the color and detail features of objects can be simulated, making the models more realistic when rendered. Therefore, super-resolution processing of low-resolution textures can not only improve the rendering quality of three-dimensional scenes but also enable users to enjoy a more refined and realistic visual experience during the interaction process. This is of great significance for enhancing the attractiveness of VR / AR applications and improving their market competitiveness.

[0004] In recent years, with the development of deep learning technology, significant progress has been made in image super-resolution reconstruction technology. From early convolutional neural network architectures such as SRCNN (Super Resolution Convolutional Network) to later deep networks with residual structures such as EDSR (Enhanced Deep Residual Networks) and RCAN (Residual Channel Attention Network), and then to methods such as Real-ESRGAN that introduce adversarial learning, image super-resolution technology has continuously broken through in terms of reconstruction quality and efficiency. However, these methods are mainly designed for ordinary 2D images, only considering the planar information of the images and lacking the understanding and constraints of three-dimensional geometric structures. When applied to the texture mapping scenario of 3D models, due to the failure to fully utilize geometric information, ideal reconstruction effects are often not achieved.

[0005] In recent years, some researchers have attempted to improve traditional image super-resolution methods for quality optimization of UV textures to achieve better model rendering effects.

[0006] Existing super-resolution methods usually perform super-resolution on a single ordinary RGB image and cannot effectively utilize the geometric information of 3D models, resulting in limited improvement of texture maps. Some researchers have introduced normal information during the training and inference processes of super-resolution neural networks to generate higher-quality texture maps, thereby enhancing the rendering effect of 3D scenes.

[0007] However, this method often requires high-quality textures as training data. In practical applications, it is very difficult to obtain high-resolution texture images as training data. High-quality texture annotation of real scenes requires a large amount of manpower and material resources, while synthetic data often lacks authenticity. This limits supervised learning, makes it difficult to establish model evaluation metrics, and also affects the generalization performance of the model.

[0008] The method proposed in this paper uses multi-view RGB images with pose information generated during 3D reconstruction as training data. After applying the super-resolution texture to the 3D model through differentiable rendering to obtain multi-view images, these images are compared with the images of the training data to calculate the loss to optimize the super-resolution model.

[0009] Currently, when performing super-resolution reconstruction on UV texture maps, the commonly used technical solution is as follows: The UV texture map is input into a neural network model. The neural network model used usually is based on a convolutional neural network and is divided into two parts; The first part starts with 1 convolutional layer, fills in multiple ResBlock modules in the middle, and ends with 1 convolutional layer. The main function of this part is to perform feature extraction to obtain a low-resolution feature map corresponding to the UV texture map; The second part consists of an upsampling module and a convolutional layer. The upsampling module merges feature maps of multiple channels through sub-pixel convolution to increase the resolution of the feature map, and then outputs a high-resolution texture map through a convolutional layer. The basic structure of the normal map is the same as that of the texture map, and the difference is that the corresponding pixels store normal information of the corresponding spatial geometry instead of color information, thereby being able to introduce geometric information into the super-resolution process to a certain extent.

[0010] However, when performing super-resolution reconstruction on UV texture maps in the above manner, the following technical problems often exist:

[0011] First, the loss used for model training is obtained by comparing the generated texture map with the original high-resolution texture map. To a certain extent, it reflects the quality of the rendering result, but the comparison only in the texture space may not fully represent the rendering effect.

[0012] Second, model training requires a 3D model with high-quality UV textures as training data. In practical applications, the reconstructed scene often lacks high-quality textures itself, resulting in difficulty in texture optimization of the reconstruction results of actual objects and a lack of generalization for such objects. Summary of the Invention

[0013] This disclosure is in part for introducing concepts in a concise form, which will be described in detail in the following Detailed Description section. This disclosure is not intended to identify the key features or essential features of the claimed technical solution, nor is it intended to limit the scope of the claimed technical solution.

[0014] Some embodiments of this disclosure propose a 3D object texture enhancement method based on coordinate mapping and super-resolution to solve one or more of the technical problems mentioned in the above Background section.

[0015] In a first aspect, some embodiments of this disclosure provide a 3D object texture enhancement method based on coordinate mapping and super-resolution. The method includes: obtaining a sequence of scene image data corresponding to a target scene, where each scene image data in the sequence of scene image data corresponds to color information, depth information, and camera parameter information, and the camera parameter information includes camera intrinsics, camera extrinsics, and an intrinsic matrix; constructing a 3D scene grid model based on the sequence of scene image data; generating a scene texture unwrapping and each model image data based on the 3D scene grid model, where each model image data in the each model image data corresponds to camera pose information; based on the 3D scene grid model, the scene texture unwrapping, and the each model image data, performing the following training steps: inputting the scene texture unwrapping into a generation network in an initial neural network to obtain a high-resolution texture unwrapping, where the initial neural network includes a generation network and a discriminator; generating each rendered model image data based on the high-resolution texture unwrapping and the 3D scene grid model; generating a pixel loss value and a perceptual loss value based on the each model image data and the each rendered model image data; generating an adversarial loss value based on the discriminator in the initial neural network, the each model image data, and the each rendered model image data; generating an image loss value based on the pixel loss value, the perceptual loss value, and the adversarial loss value; in response to determining that the image loss value does not meet a preset loss condition, adjusting the parameters of the initial neural network, using the adjusted initial neural network as the initial neural network, and performing the above training steps again; in response to determining that the image loss value meets the loss condition, determining the generation network in the initial neural network as a generation model; generating a target 3D scene grid model based on the generation model, the 3D scene grid model, and the scene texture unwrapping.

[0016] The present invention proposes a UV texture super-resolution method based on differentiable rendering technology. This method utilizes differentiable rendering technology, allowing the use of multi-view RGB images with pose information to supervise the training of the super-resolution network and introducing geometric information to a certain extent. For this special two-dimensional image of UV texture, we have made corresponding adjustments and optimizations to the traditional super-resolution method. In addition, the present invention also proposes a training data acquisition method to support the training of the above super-resolution network. This method combines traditional 3D reconstruction technology and UV texture unwrapping tools to create 3D models, rough textures, multi-view RGB images, and their corresponding camera poses. Through this method, we can provide the high-quality data required for the training of the super-resolution network, thereby improving the performance and accuracy of the network. Brief Description of the Drawings

[0017] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages, and aspects of the various embodiments of the present disclosure will become more apparent. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the elements and elements are not necessarily drawn to scale.

[0018] Figure 1 is a flowchart of some embodiments of a 3D object texture enhancement method based on coordinate mapping and super-resolution according to the present disclosure;

[0019] Figure 2 is a schematic structural diagram of the generation network in the initial neural network;

[0020] Figure 3 is a schematic structural diagram of the discriminator in the initial neural network. Detailed Description of the Embodiments

[0021] The embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0022] It should also be noted that, for the sake of convenience of description, only the parts related to the relevant invention are shown in the drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0023] It should be noted that the concepts such as "first", "second", etc. mentioned in this disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependent relationships.

[0024] It should be noted that the modifications of "one" and "multiple" mentioned in this disclosure are illustrative rather than restrictive. Those skilled in the art should understand that unless otherwise clearly specified in the context, it should be understood as "one or more".

[0025] The names of the messages or information exchanged between multiple devices in the embodiments of this disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0026] The present disclosure will be described in detail below with reference to the accompanying drawings and in conjunction with embodiments.

[0027] Figure 1 Flow 100 of some embodiments of a three-dimensional object texture enhancement method based on coordinate mapping and super-resolution according to the present disclosure is shown. The three-dimensional object texture enhancement method based on coordinate mapping and super-resolution includes the following steps:

[0028] Step 101, obtaining a sequence of scene image data corresponding to a target scene.

[0029] In some embodiments, an execution subject (such as a computing device) of the three-dimensional object texture enhancement method based on coordinate mapping and super-resolution can obtain a sequence of scene image data corresponding to a target scene. Among them, the above target scene can be a real scene. Each scene image data in the above sequence of scene image data can be RGBD image data corresponding to the above target scene. Each scene image data in the above sequence of scene image data can be image data of different angles corresponding to the above target scene. Each scene image data in the above sequence of scene image data corresponds to color information, depth information, and camera parameter information. The above color information can represent the RGB values corresponding to each pixel point in the scene image data. The above depth information can represent the depth values corresponding to each pixel point in the scene image data. The above camera parameter information can be the parameters of the camera used when shooting the scene image data. The above camera parameter information can include camera internal parameters, camera external parameters, and internal parameter matrices. The above camera internal parameters can include focal lengths and principal points. The above focal lengths can include the focal length in the horizontal coordinate direction and the focal length in the vertical coordinate direction. The above principal points can include the horizontal coordinate and the vertical coordinate. The above camera external parameters can include a rotation matrix and a translation vector. In practice, a sequence of scene image data can be obtained from a preset scene image database. Among them, the above scene image database can be a database for storing scene image data.

[0030] Step 102: Construct a three-dimensional scene grid model based on the sequence of scene image data.

[0031] In some embodiments, a three-dimensional scene grid model can be constructed based on the above sequence of scene image data. Among them, the above three-dimensional scene grid model can be a three-dimensional model constructed based on the above sequence of scene image data. The above three-dimensional scene grid model includes each model vertex. The above model vertex can be a vertex in the three-dimensional model. Each model vertex in the above model vertices corresponds to vertex color information. The above vertex color information can be the color information corresponding to the vertex in the three-dimensional model. The above color information can be data used to represent color. In practice, the above sequence of scene image data can be reconstructed through a model reconstruction algorithm to obtain a three-dimensional scene grid model. Among them, the above model reconstruction algorithm can be the BundleFusion algorithm.

[0032] In the process of adopting technical solutions to solve the above technical problems, the following problems often arise:

[0033] Directly generating a high-resolution texture map based on the UV texture image without modeling the three-dimensional scene, resulting in only being able to refer to the UV texture image when generating the high-resolution texture map, lacking the geometric structure constraint of the three-dimensional scene, easily leading to the inconsistency between the generated high-resolution texture map and the geometric structure of the three-dimensional scene, resulting in the need to consume computing resources to regenerate the high-resolution texture map, and thus causing waste of computing resources when generating the high-resolution texture map.

[0034] Facing the above technical problems, the following solution is decided to be adopted:

[0035] In some optional implementation manners of some embodiments, a three-dimensional scene grid model can be constructed based on the above sequence of scene image data through the following steps:

[0036] The first step: For each adjacent two scene image data in the above sequence of scene image data, perform the following steps:

[0037] The first sub-step: Determine the scene image data that meets the preset image frame condition in the above two scene image data as the first target scene image data. Among them, the above preset image frame condition can be that the ranking of the scene image data in the above sequence of scene image data is more forward.

[0038] The second sub-step: Determine the scene image data that does not meet the above preset image frame condition in the above two scene image data as the second target scene image data.

[0039] The third sub-step is to perform feature point extraction processing on the above-mentioned first target scene image data and the above-mentioned second target scene image data to obtain each first feature point and each second feature point. Among them, each first feature point in the above-mentioned first feature points can be a feature point corresponding to the first target scene image data. Each first feature point in the above-mentioned first feature points corresponds to a feature point coordinate. The above-mentioned feature point coordinate can be the coordinate of the feature point in the corresponding image coordinate system. The above-mentioned image coordinate system can be a coordinate system constructed with the first pixel point in the lower left corner of the image as the coordinate origin, the horizontal right direction as the horizontal coordinate axis, and the vertical upward direction as the vertical coordinate axis. Each second feature point in the above-mentioned second feature points can be a feature point corresponding to the second target scene image data. Each second feature point in the above-mentioned second feature points corresponds to a feature point coordinate. In practice, a feature point extraction algorithm can be used to perform feature point extraction processing on the above-mentioned first target scene image data and the above-mentioned second target scene image data to obtain each first feature point and each second feature point. Among them, the above-mentioned feature point extraction algorithm can be the SIFT algorithm (Scale-Invariant Feature Transform).

[0040] The fourth sub-step is to perform feature point matching processing on the above-mentioned first feature points and the above-mentioned second feature points to obtain each feature point pair. Among them, each feature point pair in the above-mentioned feature point pairs can be a data pair composed of two mutually matching feature points. Each feature point pair in the above-mentioned feature point pairs can include a first feature point and a second feature point. In practice, a feature point matching algorithm can be used to perform preliminary matching on the above-mentioned first feature points and the above-mentioned second feature points to obtain each feature point pair. Among them, the above-mentioned feature point matching algorithm can be the k-Nearest Neighbor (KNN) classification algorithm.

[0041] The fifth sub-step is to perform data screening processing on the above-mentioned feature point pairs to obtain each screened feature point pair. Among them, each screened feature point pair in the above-mentioned screened feature point pairs can be each feature point pair retained after screening. In practice, an outlier detection algorithm can be used to screen the above-mentioned feature point pairs to obtain each screened feature point pair after screening as each screened feature point pair. Among them, the above-mentioned outlier detection algorithm can be the RANSAC algorithm (Random Sample Consensus).

[0042] Sixth sub-step: Generate each target screening feature point pair based on the above-mentioned screening feature point pairs. Each target screening feature point pair in the above-mentioned target screening feature point pairs can be a selected screening feature point pair. In practice, a preset number of screening feature point pairs can be randomly selected from the above-mentioned screening feature point pairs as each target screening feature point pair. The above-mentioned preset number of feature point pairs can be a preset value. Here, there is no limitation on the specific setting of the above-mentioned preset number of feature point pairs.

[0043] Seventh sub-step: For each target screening feature point pair in the above-mentioned target screening feature point pairs, perform the following steps:

[0044] Sub-step 1: Generate a first feature point matrix and a second feature point matrix corresponding to the above-mentioned target screening feature point pair based on the above-mentioned target screening feature point pair. The above-mentioned first feature point matrix can be the matrix corresponding to the above-mentioned first feature point. The above-mentioned second feature point matrix can be the matrix corresponding to the above-mentioned second feature point. In practice, the first feature point included in the above-mentioned target screening feature point pair can be determined as the first target feature point. Then, the feature point coordinates corresponding to the above-mentioned first target feature point can be determined as the first feature point coordinates. Then, a preset value can be added to the above-mentioned first feature point coordinates to obtain the first feature point coordinates added with the above-mentioned preset value as the first feature point matrix. The above-mentioned preset value can be 1. In practice, the method of obtaining the second feature point matrix can refer to the specific implementation manner of obtaining the above-mentioned first feature point matrix, which will not be elaborated here.

[0045] Sub-step 2: Determine the transpose of the above-mentioned second feature point matrix as the feature point transpose matrix.

[0046] Sub-step 3: Construct non-linear information based on the above-mentioned first feature point matrix, a preset fundamental matrix, and the above-mentioned feature point transpose matrix. The above-mentioned non-linear information can be an equation composed of the above-mentioned first feature point matrix, a preset fundamental matrix, and the above-mentioned feature point transpose matrix. In practice, an equation can be constructed as non-linear information by making the product of the above-mentioned first feature point matrix, a preset fundamental matrix, and the above-mentioned feature point transpose matrix equal to 0. The above-mentioned fundamental matrix can be an unknown third-order matrix.

[0047] Eighth sub-step: Generate a target matrix based on the constructed non-linear information. The above-mentioned target matrix can be the fundamental matrix solved from the above-mentioned non-linear information. In practice, for the constructed non-linear information, the value of the fundamental matrix in the above-mentioned non-linear information can be solved through a polynomial solving algorithm. Then, the solved fundamental matrix can be determined as the target matrix. The above-mentioned polynomial solving algorithm can be the least squares method.

[0048] The ninth sub-step is to generate an essential matrix based on the above target matrix and the intrinsic matrices corresponding to the above two scene image data. In practice, the product of the above intrinsic matrix, the above target matrix, and the transpose of the above intrinsic matrix can be determined as the essential matrix.

[0049] The tenth sub-step is to perform matrix decomposition on the above essential matrix to obtain a first decomposition matrix and a second decomposition matrix. In practice, the above essential matrix can be decomposed through matrix decomposition technology to obtain a first decomposition matrix and a second decomposition matrix. Among them, the above matrix decomposition technology can be the singular value decomposition technology. The above first decomposition matrix can be the left singular matrix obtained by performing singular value decomposition on the above essential matrix. The above second decomposition matrix can be the right singular matrix obtained by performing singular value decomposition on the above essential matrix.

[0050] The eleventh sub-step is to determine the inter-frame rotation matrix as the product of the above first decomposition matrix, a preset matrix, and the above second decomposition matrix. Among them, the above preset matrix can be a third-order matrix with the element in the first row and second column being -1, the element in the second row and first column being 1, the element in the third row and third column being 1, and the rest of the elements being 0.

[0051] The twelfth sub-step is to generate an inter-frame translation vector based on the above first decomposition matrix. In practice, the third column in the above first decomposition matrix can be determined as the inter-frame translation vector.

[0052] The thirteenth sub-step is to combine the above inter-frame rotation matrix and the above inter-frame translation vector to obtain inter-frame pose information. In practice, the above inter-frame rotation matrix and the above inter-frame translation vector can be combined into inter-frame pose information.

[0053] Step 2: Based on the above scene image data sequence and the obtained inter-frame pose information, generate the global pose information for each frame. Among them, each global pose information in the above global pose information can be the pose obtained after the corresponding inter-frame pose information is transformed into the world coordinate system. In practice, first, the scene image data in the above scene image data sequence can be grouped to obtain each scene image data group. Among them, the number of scene image data corresponding to each scene image data group in the above scene image data groups is the same. There is the same scene image data between every two scene image data groups in the above scene image data groups. Then, for each scene image data group in the above scene image data groups, the pose optimization technology can be used to locally optimize the inter-frame pose information corresponding to the above scene image data group to obtain the optimized inter-frame pose information for each frame as the target inter-frame pose information. Among them, the above pose optimization technology can be the bundle adjustment technology. Then, the obtained target inter-frame pose information for each frame can be transformed into the world coordinate system through the pose transformation technology to obtain the transformed target inter-frame pose information for each frame as the global pose information for each frame. Among them, the above pose transformation technology can be the SLAM technology (Simultaneous Localization And Mapping).

[0054] Step 3: Based on the above scene image data sequence, the generated global pose information for each frame, and the preset three-dimensional space, generate a three-dimensional scene grid model. Among them, the above three-dimensional space can be a virtual three-dimensional space. In practice, the above scene image data sequence, the above global pose information, and the above three-dimensional space can be input into the model reconstruction function to obtain the distance field. Among them, the above model reconstruction function can be the truncated signed distance function. Then, the model extraction algorithm can be used to extract the model from the above distance field to obtain the three-dimensional scene grid model. The above model extraction algorithm can be the Marching Cubes algorithm.

[0055] The above technical solution and its related content in combination with steps 101 to 105 are an inventive point of an embodiment of the present disclosure, which solves the problem of "computing resource waste". The factors leading to computing resource waste are often as follows: directly generating a high-resolution texture map based on the UV texture image without modeling the three-dimensional scene, resulting in only being able to refer to the UV texture image when generating the high-resolution texture map, lacking the geometric structure constraint of the three-dimensional scene, and easily causing the generated high-resolution texture map to be inconsistent with the geometric structure of the three-dimensional scene, resulting in the need to consume computing resources to regenerate the high-resolution texture map, and further leading to computing resource waste when generating the high-resolution texture map. If the above factors are solved, computing resource waste can be reduced. To achieve this effect, the present disclosure first performs the following steps for each adjacent pair of scene image data in the above scene image data sequence: First, determine the scene image data that meets the preset image frame condition among the two scene image data as the first target scene image data. Second, determine the scene image data that does not meet the above preset image frame condition among the two scene image data as the second target scene image data. Then, perform feature point extraction processing on the above first target scene image data and the above second target scene image data to obtain each first feature point and each second feature point. Thus, each feature point corresponding to each scene image data in the two scene image data can be obtained. Then, perform feature point matching processing on the above each first feature point and the above each second feature point to obtain each feature point pair. Thus, the above each first feature point and the above each second feature point can be matched to obtain each feature point pair. Then, perform data screening processing on the above each feature point pair to obtain each screened feature point pair. Thus, each feature point pair with a higher matching degree can be screened out from the above each feature point pair. Then, based on the above each screened feature point pair, generate each target screened feature point pair. Then, for each target screened feature point pair in the above each target screened feature point pair, perform the following steps: First, generate a first feature point matrix and a second feature point matrix corresponding to the above target screened feature point pair based on the above target screened feature point pair. Thus, a first feature point matrix and a second feature point matrix can be obtained. Second, determine the transpose of the above second feature point matrix as the feature point transpose matrix. Then, construct non-linear information based on the above first feature point matrix, a preset fundamental matrix, and the above feature point transpose matrix. Thus, non-linear information can be constructed. Then, generate a target matrix based on the constructed non-linear information. Thus, the target matrix can be obtained by solving the above non-linear information. Then, generate an essential matrix based on the above target matrix and the internal parameter matrix corresponding to the two scene image data. Thus, an essential matrix can be obtained. Then, perform matrix decomposition on the above essential matrix to obtain a first decomposition matrix and a second decomposition matrix.Then, determine the product of the above first decomposition matrix, the preset matrix, and the above second decomposition matrix as the inter-frame rotation matrix. Then, based on the above first decomposition matrix, generate an inter-frame translation vector. Thus, the inter-frame translation vector can be obtained. Then, combine the above inter-frame rotation matrix and the above inter-frame translation vector to obtain inter-frame pose information. Thus, the pose between the above two scene image data can be obtained. Then, based on the above two scene image data and the obtained respective inter-frame pose information, generate respective global pose information. Thus, the obtained respective inter-frame pose information can be optimized to obtain respective global pose information. Finally, based on the above scene image data sequence, the generated respective global pose information, and a preset three-dimensional space, generate a three-dimensional scene grid model. Thus, through the above scene image data sequence, the above respective global pose information, and the above three-dimensional space, a three-dimensional model corresponding to the above scene image data sequence can be obtained. Also, because a three-dimensional model corresponding to the scene can be first constructed according to each scene image data corresponding to the scene, and then the generation network can be trained by the loss value between the RGB images of each angle in the three-dimensional model corresponding to the scene and the RGB images of each angle corresponding to the three-dimensional model applied with a high-resolution texture map, and finally the trained generation network is used to generate a high-resolution texture map, the probability of the situation where the generated high-resolution texture map is inconsistent with the geometric structure of the three-dimensional model can be reduced, and further the probability of the situation where the high-resolution texture map is regenerated by consuming computing resources can be reduced, and further the waste of computing resources can be reduced.

[0056] Step 103: Based on the three-dimensional scene grid model, generate a scene texture unfolding map and each model image data.

[0057] In some embodiments, a scene texture unfolding map and each model image data may be generated based on the above three-dimensional scene grid model. Among them, the above scene texture unfolding map may be an image obtained by performing UV unfolding on the above three-dimensional scene grid model. Each model image data in the above each model image data may be the image data of the above three-dimensional scene grid model. The above each model image data corresponds to different angles of the above three-dimensional scene grid model.

[0058] In some alternative implementation manners of some embodiments, the following steps may be used to generate a scene texture unfolding map and each model image data based on the above three-dimensional scene grid model:

[0059] First step, perform parameterization processing on the above three-dimensional scene grid model to obtain a scene texture unfolding map. In practice, the above three-dimensional scene grid model may be UV unfolded through a UV unfolding library function to obtain a scene texture unfolding map. Among them, the above UV unfolding library function may be the xatlas library.

[0060] As an example, the first step of the xatlas library can be to preprocess the input 3D scene mesh model. The main goal of this step is to clean the above-mentioned 3D scene mesh model and segment the above-mentioned 3D scene mesh model into sub-meshes (called charts) suitable for parameterization. These sub-meshes are usually regions with simple topology, such as planes or cylindrical surfaces. Each sub-mesh is a local region suitable for individual parameterization. Through this segmentation, the xatlas library can decompose the complex global parameterization problem into multiple simple local problems.

[0061] Secondly, local parameterization can be performed on each sub-mesh to map each sub-mesh to a two-dimensional parameter domain (UV space). This process usually uses conformal mapping or area-preserving mapping methods to ensure that the three-dimensional to two-dimensional mapping preserves geometric properties as much as possible. The boundaries of each sub-mesh are unfolded into a continuous two-dimensional shape in the above UV space and do not overlap with other sub-meshes.

[0062] Then, after local parameterization is completed, the local UV coordinates of all sub-meshes can be merged into the global UV space and global optimization can be performed. The goal of this stage is to adjust the UV layout to reduce distortion, overlap, and waste of texture space. The xatlas library ensures that they are closely arranged and do not overlap in the UV space by translating, rotating, and scaling the positions and sizes of the charts. At the same time, the xatlas library optimizes the UV coordinates to make the three-dimensional to two-dimensional mapping as conformal or area-preserving as possible, thereby reducing distortion during texture mapping. Finally, the xatlas library generates a globally optimized UV layout, and the UV coordinates of all charts are adjusted to the optimal positions.

[0063] Then, the UV coordinates can be normalized to the range [0,1], that is, for each face of the triangular mesh, it is determined by three points. Each point only stores color information and spatial coordinate information before parameterization. After parameterization, UV coordinate information will be added to the relevant data of the point. The UV coordinates correspond to a point on the two-dimensional image. At this time, the UV unwrapping can be considered as a blank image.

[0064] Finally, the UV texture in the UV unwrapping can be filled according to the vertex color information in the above-mentioned 3D scene mesh model to obtain a rough UV texture map corresponding to the model as the scene texture unwrapping.

[0065] Step 2: For each camera pose information among the preset camera pose information, based on the above three-dimensional scene grid model, obtain the respective image data corresponding to the above camera pose information as respective model image data. Among them, each camera pose information among the above camera pose information can be information for characterizing the angle of the above three-dimensional scene grid model. The above camera pose information may include a rotation matrix and a translation vector. In practice, for each camera pose information among the above camera pose information, first, the above three-dimensional scene grid model can be read through a model processing library. Then, through the above model processing library, the respective RGB images corresponding to the angle of the above three-dimensional scene grid model corresponding to the above camera pose information can be determined as the respective model image data. Among them, the above model processing library can be the Pytorch3D library.

[0066] Step 104: Based on the three-dimensional scene grid model, the scene texture unwrapping map, and the respective model image data, perform the following training steps:

[0067] Step 1041: Input the scene texture unwrapping map into the generation network in the initial neural network to obtain a high-resolution texture unwrapping map.

[0068] In some embodiments, the above scene texture unwrapping map can be input into the generation network in the initial neural network to obtain a high-resolution texture unwrapping map. Among them, the above high-resolution texture unwrapping map can be a UV texture map with higher accuracy corresponding to the above scene texture unwrapping map.

[0069] The above initial neural network can include a generation network and a discriminator. The above generation network can be a neural network model that takes the scene texture unwrapping map as input and outputs a high-resolution texture unwrapping map. The above generation network can include a convolutional layer, an activation function, respective residual convolutional blocks, an upsampling layer, and a convolutional layer. Each of the respective residual convolutional blocks can include a convolutional layer, a batch normalization layer, an activation function, a convolutional layer, and a batch normalization layer.

[0070] As Figure 2 shown, Figure 2 is a schematic structural diagram of the generation network in the initial neural network.

[0071] The discriminator described above can be a neural network model that takes an image as input and outputs a discriminant value corresponding to the image. The discriminant value described above can be a numerical value between [0, 1]. The discriminant value can represent the probability that the image input to the discriminator is model image data. Among them, 0 can represent that the image input to the discriminator is rendered model image data. 1 can represent that the image input to the discriminator is model image data. The discriminator can include a convolutional layer, an activation function, each feature extraction module, and a classifier. Each feature extraction module among the above-mentioned feature extraction modules can include a convolutional layer, a batch normalization layer, and an activation function.

[0072] As Figure 3 shown, Figure 3 is a schematic structural diagram of the discriminator in the initial neural network.

[0073] Step 1042, generate each piece of rendered model image data based on the high-resolution texture unwrapping map and the three-dimensional scene mesh model.

[0074] In some embodiments, each piece of rendered model image data can be generated based on the high-resolution texture unwrapping map and the three-dimensional scene mesh model described above.

[0075] In some optional implementation manners of some embodiments, each piece of rendered model image data can be generated based on the high-resolution texture unwrapping map and the three-dimensional scene mesh model described above through the following steps:

[0076] First step, apply the high-resolution texture unwrapping map to the three-dimensional scene mesh model to obtain a high-resolution three-dimensional mesh model. Among them, the high-resolution three-dimensional mesh model can be a three-dimensional scene mesh model whose surface is replaced with the high-resolution texture unwrapping map. In practice, first, the three-dimensional scene mesh model can be read through a differentiable rendering framework. Then, the high-resolution texture unwrapping map can be applied to the three-dimensional scene mesh model through a UV texture application class to obtain a high-resolution three-dimensional mesh model. The differentiable rendering framework described above can be Pytorch3D. The UV texture application class described above can be the TexturesUV class in the Pytorch3D framework.

[0077] Step 2: Based on the above high-resolution three-dimensional grid model, generate respective rendered model image data corresponding to the respective model image data. Among them, each of the respective rendered model image data can be the image data corresponding to the above high-resolution three-dimensional grid model. The respective rendered model image data correspond to different angles of the above high-resolution three-dimensional grid model. The respective rendered model image data are in one-to-one correspondence with the respective model image data. Each of the respective rendered model image data in the respective rendered model image data corresponds to camera pose information. In practice, first, the above high-resolution three-dimensional grid model can be read through the above differentiable rendering framework. Then, for each of the respective camera pose information in the above respective camera pose information, the respective RGB images corresponding to the above high-resolution three-dimensional grid model at the angle of the above camera pose information can be determined as the respective rendered model image data. Finally, the respective rendered model image data corresponding to the above respective camera pose information are obtained.

[0078] Step 1043: Generate a pixel loss value and a perceptual loss value based on the respective model image data and the respective rendered model image data.

[0079] In some embodiments, a pixel loss value and a perceptual loss value can be generated based on the respective model image data and the respective rendered model image data. Among them, the above pixel loss value can be the loss value between the respective pixel points in the above respective model image data and the above respective rendered model image data. The above perceptual loss value can be the loss value between the respective feature maps corresponding to the above respective model image data and the respective feature maps corresponding to the above respective rendered model image data.

[0080] In some optional implementation manners of some embodiments, the following steps can be used to generate a pixel loss value and a perceptual loss value based on the respective model image data and the respective rendered model image data:

[0081] First step: For each of the respective camera pose information in the above respective camera pose information, perform the following steps:

[0082] First sub-step: Determine the respective model image data corresponding to the above camera pose information in the above respective model image data as the respective model image data to be processed. In practice, the respective model image data with the same camera pose information as the above camera pose information in the above respective model image data can be determined as the respective model image data to be processed.

[0083] Second sub-step: Determine each rendered model image data corresponding to the camera pose information in each of the above-mentioned rendered model image data as each to-be-processed rendered model image data. In practice, each rendered model image data with the same camera pose information as the above-mentioned camera pose information in each of the above-mentioned rendered model image data can be determined as each to-be-processed rendered model image data.

[0084] Third sub-step: Determine the image pixel loss value between each of the above-mentioned to-be-processed model image data and each of the above-mentioned to-be-processed rendered model image data through the following formula:

[0085]

[0086] Wherein, PixelLoss can be the above-mentioned image pixel loss value. The above-mentioned image pixel loss value can be the loss value between the to-be-processed model image data and the corresponding to-be-processed rendered model image data. N can be the number of each of the above-mentioned to-be-processed model image data. n can represent the nth to-be-processed model image data, and at the same time can represent the nth to-be-processed rendered model image data. H can be the image height. i can be each pixel point corresponding to when the image height is i. W can be the image width. j can be each pixel point corresponding to when the image width is j. C can be the number of color channels. The above-mentioned number of color channels can be the quantity of color channels. k can be the kth color channel. can represent the nth to-be-processed rendered model image data in each of the above-mentioned to-be-processed rendered model image data. can be the pixel value corresponding to the pixel point with image height i, image width j, and color channel k in the above-mentioned nth to-be-processed rendered model image data. can represent the nth to-be-processed model image data in each of the above-mentioned to-be-processed model image data. can be the pixel value corresponding to the pixel point with image height i, image width j, and color channel k in the above-mentioned nth to-be-processed model image data.

[0087] Fourth sub-step: perform feature extraction processing on each of the above model image data to obtain respective model feature maps corresponding to each of the above model image data. Among them, each model feature map in each of the above model feature maps can be the feature map corresponding to the model image data. In practice, each of the above model image data can be input into a pre-trained feature extraction model to obtain respective model feature maps. The above feature extraction model can be a neural network model that takes each of the model image data or each of the rendered model image data as input and outputs respective feature maps corresponding to each of the model image data or each of the rendered model image data. The above feature extraction model can be a pre-trained VGG19 network. The above pre-training can be to fine-tune the VGG19 network using a labeled three-dimensional model image dataset and a cross-entropy loss function. Each three-dimensional model image data in the above three-dimensional model image dataset can be an image corresponding to a three-dimensional model.

[0088] Fifth sub-step: perform feature extraction processing on each of the above rendered model image data to obtain respective rendered model feature maps corresponding to each of the above rendered model image data. Among them, each rendered model feature map in each of the above rendered model feature maps can be the feature map corresponding to the rendered model image data. In practice, each of the above rendered model image data can be input into the above feature extraction model to obtain respective rendered model feature maps.

[0089] Sixth sub-step: generate an image perception loss value based on each of the above model feature maps and each of the above rendered model feature maps. Among them, the above image perception loss value can be the loss value between each of the above model feature maps and each of the above rendered model feature maps. In practice, for each model feature map in each of the above model feature maps, first, the rendered model feature map corresponding to the above model feature map in each of the above rendered model feature maps can be determined as the target rendered model feature map. Then, the root mean square error between the above model feature map and the above target rendered model feature map can be determined as the feature image loss value. Finally, the sum of the determined respective feature image loss values can be determined as the image perception loss value.

[0090] Second step: generate a pixel loss value based on the generated respective image pixel loss values. In practice, the sum of the above respective image pixel loss values can be determined as the pixel loss value.

[0091] Third step: generate a perception loss value based on the generated respective image perception loss values. In practice, the sum of the above respective image perception loss values can be determined as the perception loss value.

[0092] Step 1044: generate an adversarial loss value based on the discriminator in the initial neural network, each of the model image data, and each of the rendered model image data.

[0093] In some embodiments, an adversarial loss value may be generated based on the discriminator in the initial neural network, the respective model image data, and the respective rendered model image data.

[0094] In some alternative implementation manners of some embodiments, the adversarial loss value may be generated based on the discriminator in the initial neural network, the respective model image data, and the respective rendered model image data through the following steps:

[0095] First step, input the respective model image data into the discriminator in the initial neural network to obtain respective model prediction values corresponding to the respective model image data. Each of the respective model prediction values may be a numerical value within the interval [0, 1].

[0096] Second step, for each of the respective model prediction values, perform the following steps:

[0097] First sub-step, determine the logarithm of the model prediction value as the model prediction logarithm. The logarithm may be a logarithm with base 2.

[0098] Second sub-step, determine the negative of the model prediction logarithm as the model loss value.

[0099] Third step, input the respective rendered model image data into the discriminator in the initial neural network to obtain respective rendered model prediction values corresponding to the respective rendered model image data. Each of the respective rendered model prediction values may be a numerical value within the interval [0, 1].

[0100] Fourth step, for each of the respective rendered model prediction values, perform the following steps:

[0101] First sub-step, determine the difference between a preset true value and the rendered model prediction value as the model prediction difference. The true value may be a preset numerical value. For example, the true value may be 1.

[0102] Second sub-step, determine the logarithm of the model prediction difference as the model logarithmic data. The logarithm may be a logarithm with base 2.

[0103] Third sub-step, determine the negative of the model logarithmic data as the rendered model loss value.

[0104] Fifth step, generate an adversarial loss value based on the determined respective model loss values and the determined respective rendered model loss values. In practice, the sum of the respective model loss values and the respective rendered model loss values may be determined as the adversarial loss value.

[0105] Step 1045: Generate an image loss value based on the pixel loss value, the perceptual loss value, and the adversarial loss value.

[0106] In some embodiments, an image loss value may be generated based on the above-mentioned pixel loss value, the above-mentioned perceptual loss value, and the above-mentioned adversarial loss value. In practice, the sum of the above-mentioned pixel loss value, the above-mentioned perceptual loss value, and the above-mentioned adversarial loss value may be determined as the image loss value.

[0107] Step 1046: In response to determining that the image loss value does not meet the preset loss condition, adjust the parameters of the initial neural network, use the adjusted initial neural network as the initial neural network, and execute the above training steps again.

[0108] In some embodiments, in response to determining that the above-mentioned image loss value does not meet the preset loss condition, the parameters of the initial neural network may be adjusted, and the adjusted initial neural network may be used as the initial neural network to execute the above training steps again. Among them, the above-mentioned loss condition may be that the above-mentioned image loss value is less than a preset loss threshold. The above-mentioned loss threshold may be a preset value. Here, the specific setting of the above-mentioned loss threshold is not limited. In practice, the backpropagation algorithm (Back Propagation Algorithm, BP algorithm) and the gradient descent method (such as the mini-batch gradient descent algorithm) may be used to adjust the parameters of the above-mentioned initial neural network.

[0109] Step 1047: In response to determining that the image loss value meets the loss condition, determine the generation network in the initial neural network as the generation model.

[0110] In some embodiments, in response to determining that the above-mentioned image loss value meets the above-mentioned loss condition, the generation network in the initial neural network may be determined as the generation model.

[0111] Step 105: Generate a target three-dimensional scene grid model based on the generation model, the three-dimensional scene grid model, and the scene texture unfolding diagram.

[0112] In some embodiments, a target three-dimensional scene mesh model may be generated based on the above-mentioned generation model, the above-mentioned three-dimensional scene mesh model, and the above-mentioned unfolded scene texture map. Among them, the above-mentioned target three-dimensional scene mesh model may be a three-dimensional scene mesh model with a higher resolution of surface texture. In practice, first, the above-mentioned unfolded scene texture map may be input into the above-mentioned generation model to obtain a high-resolution UV texture map. Then, the above-mentioned high-resolution UV texture map may be applied to the above-mentioned three-dimensional scene mesh model to obtain the target three-dimensional scene mesh model. The manner of applying the above-mentioned high-resolution UV texture map to the above-mentioned three-dimensional scene mesh model may refer to the specific implementation manner of step 1042, which will not be elaborated here. In practice, first, the above-mentioned three-dimensional scene mesh model may be read through the above-mentioned differentiable rendering framework. Then, the above-mentioned high-resolution UV texture map may be applied to the above-mentioned three-dimensional scene mesh model through the above-mentioned UV texture application class to obtain the target three-dimensional scene mesh model.

[0113] The above description is only some preferred embodiments of the present disclosure and an explanation of the applied technical principles. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, and should also cover other technical solutions formed by any combination of the above-mentioned technical features or their equivalent features without departing from the above-mentioned inventive concept. For example, the technical solutions formed by mutually replacing the above-mentioned features with the technical features (but not limited to) having similar functions disclosed in the embodiments of the present disclosure.

Claims

1. A three-dimensional object texture enhancement method based on coordinate mapping and super-resolution, comprising: Acquire a scene image data sequence corresponding to the target scene, wherein each scene image data in the scene image data sequence corresponds to color information, depth information and camera parameter information, and the camera parameter information includes camera intrinsic parameters, camera extrinsic parameters and an intrinsic parameter matrix; Based on the scene image data sequence, construct a three-dimensional scene grid model; Based on the three-dimensional scene mesh model, a scene texture unfolding map and various model image data are generated, wherein each model image data in the various model image data corresponds to camera pose information; Based on the three-dimensional scene mesh model, the scene texture expansion map and the respective model image data, the following training steps are performed: Inputting the scene texture expansion map into a generation network in an initial neural network to obtain a high-resolution texture expansion map, wherein the initial neural network includes a generation network and a discriminator; Generate each rendering model image data based on the high-resolution texture expansion map and the three-dimensional scene mesh model; Generate pixel loss values ​​and perceptual loss values ​​based on the respective model image data and the respective rendered model image data; Generate an adversarial loss value based on the discriminator in the initial neural network, the respective model image data, and the respective rendered model image data; Generate an image loss value based on the pixel loss value, the perceptual loss value and the adversarial loss value; In response to determining that the image loss value does not satisfy a preset loss condition, adjusting parameters of the initial neural network, using the adjusted initial neural network as the initial neural network, and performing the training step again; In response to determining that the image loss value satisfies the loss condition, determining a generative network in the initial neural network as a generative model; A target three-dimensional scene mesh model is generated based on the generation model, the three-dimensional scene mesh model and the scene texture unfolding map.

2. The method according to claim 1, wherein: The step of generating a scene texture expansion map and various model image data based on the three-dimensional scene mesh model comprises: Performing parameterization processing on the three-dimensional scene mesh model to obtain a scene texture expansion map; For each piece of camera pose information in each preset piece of camera pose information, based on the three-dimensional scene mesh model, each piece of image data corresponding to the camera pose information is acquired as each piece of model image data.

3. The method according to claim 1, wherein: The generating of each rendering model image data based on the high-resolution texture expansion map and the three-dimensional scene mesh model comprises: Applying the high-resolution texture expansion map to the three-dimensional scene mesh model to obtain a high-resolution three-dimensional mesh model; Based on the high-resolution three-dimensional mesh model, each rendering model image data corresponding to the each model image data is generated.

4. The method according to claim 2, wherein: Each of the rendering model image data corresponds to camera pose information; And generating a pixel loss value and a perception loss value based on each model image data and each rendered model image data comprises: For each piece of camera pose information in the individual pieces of camera pose information, perform the following steps: Determine each model image data corresponding to the camera pose information among the each model image data as each model image data to be processed; Determine each rendering model image data corresponding to the camera pose information among the respective rendering model image data as each rendering model image data to be processed; The image pixel loss value between each of the to-be-processed model image data and each of the to-be-processed rendered model image data is determined by the following formula: Among them, PixelLoss is the image pixel loss value, N is the number of each model image data to be processed, n represents the nth model image data to be processed, and also represents the nth rendered model image data to be processed, H is the image height, i is the corresponding pixel points when the image height is i, W is the image width, j is the corresponding pixel points when the image width is j, C is the number of color channels, k is the kth color channel, representing the nth to-be-processed rendering model image data among the to-be-processed rendering model image data, is the pixel value corresponding to the pixel point with image height i, image width j, and color channel k in the nth to-be-processed rendering model image data, representing the nth model image data to be processed among the model image data to be processed, is the pixel value corresponding to the pixel point with image height i, image width j, and color channel k in the nth model image data to be processed; Performing feature extraction processing on each model image data to obtain each model feature graph corresponding to each model image data; Performing feature extraction processing on each of the rendering model image data to obtain each of the rendering model feature graphs corresponding to each of the rendering model image data; Generate an image perception loss value based on each of the model feature maps and each of the rendering model feature maps; Generate a pixel loss value based on the generated pixel loss values ​​of each image; Based on the generated perceptual loss values ​​for each image, a perceptual loss value is generated.

5. The method according to claim 1, wherein: The generating of the adversarial loss value based on the discriminator in the initial neural network, the respective model image data and the respective rendered model image data comprises: Inputting each model image data into a discriminator in an initial neural network to obtain each model prediction value corresponding to each model image data; For each of the model prediction values, perform the following steps: determining the logarithm of the model prediction value as the model prediction logarithm; Determine the negative of the model prediction logarithm as the model loss value; Inputting each of the rendering model image data into a discriminator in an initial neural network to obtain each of the rendering model prediction values ​​corresponding to each of the rendering model image data; For each of the rendering model prediction values, perform the following steps: Determine the difference between the preset true value and the predicted value of the rendering model as the model prediction difference; Determining the logarithm of the model prediction difference as model logarithmic data; Determine the negative of the model logarithmic data as a rendering model loss value; An adversarial loss value is generated based on the determined individual model loss values ​​and the determined individual rendering model loss values.

Images