Method and system for improving texture resolution consistency of three-dimensional model based on arbitrary-scale super-division

Abstract

The invention discloses a three-dimensional model texture resolution consistency improving method and system based on any-scale super-division, and the method comprises the steps: dividing a three-dimensional geometric module in a grid format into a plurality of triangular surface clusters, and generating an initial texture image set; a scale factor is introduced to represent the texture resolution of the triangular surface cluster, and then a super-resolution factor of the triangular surface cluster is calculated based on the scale factor; according to the super-resolution factor, adaptively improving the resolution of a texture block corresponding to each triangular surface cluster by using a super-resolution network of any scale based on reference information; and after each texture image block completes any-scale super-resolution processing, packaging the texture image blocks after super-resolution to generate a new texture image set, and generating a final three-dimensional texture model based on the new texture image set. According to the method, the texture resolution can be improved, the consistency of the texture resolutions of the adjacent triangular surface clusters can be ensured, and the three-dimensional model with the consistent texture resolution and the vivid visual effect can be finally generated.

Description

Technical Field

[0001] The present invention belongs to the technical field of three-dimensional real scene model reconstruction, and in particular relates to a method and system for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution. Background Art

[0002] Realistic 3D models are widely used in smart cities, virtual / augmented reality, urban planning, etc. Image-based 3D reconstruction has attracted widespread attention from researchers due to its convenient image data acquisition and low cost. The image-based 3D reconstruction process can be roughly divided into 3D model geometric reconstruction and texture mapping. Geometric reconstruction usually uses sparse matching, dense matching and other technologies to reconstruct the 3D surface model of the target from multi-view images. This model is usually called a "white model" - only geometric shape, no texture information. Texture mapping is to give color and texture to the "white model" based on camera internal and external parameters and multi-view images, so that the 3D model has rich details and more realistic. Specifically, the "white model" is composed of a series of triangular faces, and texture mapping needs to give texture information to each triangular face based on multi-view image information. According to the way the triangular face texture is generated, the texture mapping method can be divided into a fusion-based method and a mosaic-based method. The fusion-based method performs weighted fusion of all images corresponding to the triangular face to generate the color of the triangular face; the mosaic-based method selects the best viewing angle image for the triangular face from all candidate images. Compared with the fusion-based method, the mosaic-based method has less deformation when resampling the multi-view image to the texture space, and the generated texture has better realism and fine details. In order to improve the local consistency of the 3D texture model, the mosaic-based method usually merges the adjacent triangles with the same best viewing angle image into a triangle cluster, which is used as the basic unit to obtain texture tile information from the multi-view image.

[0003] When the shooting angles, distances and camera parameters of multi-view images are different, there are significant differences in the texture resolution of adjacent triangle clusters on the 3D texture model that collect texture information from different multi-view images. This problem is very common and greatly reduces the visual quality of the texture model. Although many traditional texture mapping methods have been proposed in recent years to generate texture models, most of these methods focus on camera pose optimization, viewpoint selection, color correction, etc. The few works that focus on improving the texture resolution of 3D models can improve the resolution and restore the detail information by performing fixed-scale super-resolution on the texture atlas, but there is still a problem of inconsistent resolution on the texture model. Summary of the invention

[0004] Considering that the existing technical solutions cannot solve the problem of inconsistent clarity on three-dimensional texture models well, the present invention innovatively proposes an arbitrary scale super-resolution network with multi-view information enhancement, which integrates multi-view image information to improve the resolution consistency of the texture model, and finally can enhance the resolution of adjacent triangular face clusters on the three-dimensional model to the same level, thereby improving the visualization effect of the three-dimensional model. Although many arbitrary scale image super-resolution methods have been proposed, these methods are designed for two-dimensional images rather than three-dimensional texture models. In the present invention, the three-dimensional model in a grid format is divided into several triangular face clusters, and the texture tile corresponding to each triangular face cluster is used as a processing primitive instead of the entire texture atlas. Since the resolutions of different triangular face clusters are inconsistent and in arbitrary proportions, the scale factor of each triangular face cluster is adaptively calculated, and the factor determines the final resolution of the corresponding texture tile. That is, for different triangular face clusters, their super-resolution factors are different. In addition, considering the ill-posedness of single-image super-resolution and the requirement of the present invention that the content of the super-resolution result must meet multi-view consistency, a cross-view feature aggregation module is designed to integrate multi-view image information to improve the performance of the super-resolution network. Thanks to the adaptive calculation of super-resolution factors and the fusion of multi-view image information, the texture resolution of the 3D model can be improved and the resolutions of adjacent triangle clusters can be made consistent.

[0005] The technical solution of the present invention is: a method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution, comprising the following steps:

[0006] Step 1, determine the best viewing angle image corresponding to each triangle, and generate a triangle cluster and an initial texture atlas consisting of texture blocks;

[0007] Step 2, calculating the super-resolution factor of the triangular face cluster based on the area of ​​the triangular face cluster and the texture block;

[0008] Step 3: Use a super-resolution network to improve the resolution of the texture tiles corresponding to the triangular face clusters;

[0009] The object to be super-resolved is the smallest enclosing rectangular area of ​​the texture block in its optimal view image. The super-resolution network generates super-resolution blocks based on the super-resolution factor. The reference image information is introduced to achieve higher quality super-reconstruction. The reference image is a multi-view image that is spatially adjacent to the optimal view image.

[0010] Step 4: After each texture tile has completed the super-resolution processing at any scale, the super-resolution texture tiles are packaged to generate a new texture atlas, and a three-dimensional model with consistent texture resolution and realistic visual effects is generated based on the updated texture atlas.

[0011] Furthermore, in step 1, to generate a triangular face cluster, first determine each triangular face f i The corresponding best viewing angle image is recorded as image label l i Specifically, the energy equation E(l) is constructed to select the best viewing angle image, where the data term E data Used to select a better visible viewing angle image for the triangle, the smoothing term E smooth Used to minimize the visibility of visual seams; the energy equation E(l) is expressed as:

[0012] E(l)=E data +E smooth

[0013]

[0014] Among them, l represents the image label set of all triangular faces, Faces represents the set of all triangular faces, Edge represents the edge formed by two adjacent triangular faces, Edges represents the set of all edges, and l j Represents the triangle face f j The best viewing angle image, j is to distinguish it from i, j≠i, φ(f i ,l i ) represents the triangular face f i In the image The projection on represents the Sobel operator, and [·] represents the Iverson bracket. The above energy equation is solved by the graph cut method to obtain the labels of the best viewing angle images of all triangles, and the adjacent triangles with the same label are aggregated into triangle clusters. Where C is the number of triangle clusters; then, the texture tiles corresponding to the triangle clusters are obtained from the multi-view images, packaged to generate a texture atlas, and all texture tiles are recorded as

[0015] Furthermore, the specific implementation of step 2 is as follows:

[0016] Step 2.1, generate the scale factor of the triangular face cluster: scale factor r i Defined as a triangular face cluster c i The area of ​​the triangle cluster and the texture tile t i The ratio of the areas is as follows:

[0017]

[0018] in and Respectively represent c i and t i The area of ​​​​the ,∈ is a very small non-zero constant; r iThe larger the value, the clearer the corresponding texture on the 3D model;

[0019] Step 2.2, generating super-resolution factors of triangular face clusters: Since the texture resolutions of different triangular face clusters are different, the super-resolution factor of each triangular face cluster is adaptively calculated based on the scale factor defined in formula (2); before calculating the super-resolution factor, a reference scale factor r is set ref ; Then, the super-resolution factor s of each triangle cluster is i It is obtained by the following formula:

[0020]

[0021] Furthermore, the reference scale factor r ref is the maximum value r of the scale factor of all triangle clusters max or greater than r max The value of .

[0022] Furthermore, the specific implementation of step 3 is as follows:

[0023] Step 3.1, feature extraction: The super-resolution network EDSR-b is used as a feature extractor to generate texture patches t LR The feature map F LR and reference image feature map J represents the number of reference images;

[0024] Among them, t LR is the texture tile t i Image at its best viewing angle The minimum enclosing rectangular area in ;

[0025] Step 3.2, cross-view feature aggregation: For the feature map F LR For each pixel point p in the image, we first project it to the reference image perspective according to the multi-view geometric relationship, and the projection point is recorded as Also called a query point. Create a small window for the center Use the cross attention mechanism to enhance the image features of point p and generate the final enhanced feature map To integrate the reference information in multi-view images into the low-resolution image feature map;

[0026] Step 3.3, information integration within the view: use the arbitrary scale super-resolution method CiaoSR to integrate the information within the view to obtain the implicit code corresponding to the query point coordinates;

[0027] Step 3.4, super-resolution image reconstruction: For each pixel coordinate in the super-resolution image, the coordinates of the query point and its implicit encoding are combined to predict its color value through the implicit image function, and then the color prediction values ​​of all pixel coordinates are added pixel by pixel to the bicubic interpolation result map of the low-resolution image to generate a super-resolution image; then, the effective texture area in the super-resolution image is obtained through mask processing, and then the texture block with improved resolution is obtained;

[0028] Step 3.5, using the loss function: Based on the training sample set, train the super-resolution network of any scale until the entire network converges to the optimal accuracy.

[0029] Furthermore, in step 3.2, the cross attention mechanism is used to enhance the image features of point p. The calculation formula is as follows:

[0030]

[0031] in, represents the feature of point p after the cross-attention mechanism aggregates the reference information, F inter is the corresponding feature map; (m,n) indicates the area contained in the window The coordinate position inside, σ represents the Softmax function; Q, and They represent the query, key, and value in the attention mechanism, respectively, and are defined as follows:

[0032]

[0033] in, is the low-resolution image feature of pixel p, is the reference image feature at coordinate (m,n), dis m,n The projection point The relative distance between the pixel (m, n) and its neighborhood window, cell is the grid decoding used to specify the shape of the query area; φ k and φ v are the key network and value network implemented by multi-layer perceptron, both of which are used for feature mapping; according to formula (4), the feature generation of all reference images of aggregated pixel p In order to generate the final enhanced feature map Further integration of F inter and F LR First, the two feature maps are concatenated in the channel dimension, and then they are processed by two convolutional layers Conv and two nonlinear activation functions ReLU. The specific process of the feature fusion module is as follows:

[0034]

[0035] Furthermore, the specific implementation of step 3.3 is as follows;

[0036] Assume x is the super-resolution image t SR The pixel coordinates in x q For low-resolution images LR The corresponding query point coordinates are in x q A small local area centered Perform intra-view attention to obtain implicit encoding Formulated as:

[0037]

[0038] Among them, the query, key and value of the intra-view attention mechanism are defined as:

[0039]

[0040] In the formula (m * ,n * ) is the closest query point x q Pixels, and Indicated in (m * ,n * ) and (m,n) positions after the enhancement of low-scoring features, φ k ′ and φ v ′ are key network and value network respectively, Represents the enhanced low-score feature map The non-local features captured in .

[0041] Furthermore, in step 3.4, the super-resolution image t SR The calculation of is as follows:

[0042]

[0043] in, is the bicubic interpolation result image, f θ (z x ,x) is the implicit image function, implemented by multi-layer perceptron, z x Represents the implicit code corresponding to each pixel coordinate x in the super-resolution image.

[0044] Furthermore, in step 3.5, the loss function is norm, which constrains the difference between the super-resolution result image and the high-resolution image at the pixel level.

[0045] The present invention also provides a system for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as described in the above technical solution.

[0046] Compared with the prior art, the present invention has the following three advantages:

[0047] 1) The problem of improving the consistency of 3D texture model resolution is transformed into an arbitrary-scale texture super-resolution task, and a method is proposed to solve it.

[0048] 2) A super-resolution factor is introduced to achieve adaptive improvement of the resolution of triangle clusters, so that the resolution consistency of the texture model can be improved while improving the texture resolution.

[0049] 3) A cross-view feature aggregation module is designed in the arbitrary-scale super-resolution network to better integrate multi-view image information, thereby improving the fidelity of the super-resolution result image and the consistency of content between perspectives. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1 This is a framework diagram for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution proposed by the present invention;

[0051] Figure 2 The arbitrary-scale super-resolution network for multi-view image information enhancement proposed by the present invention;

[0052] Figure 3 Schematic diagram of experimental results of the arbitrary scale super-resolution network in the present invention on test data, from left to right are the effects of different super-resolution scales;

[0053] Figure 4 This is a schematic diagram of the experimental results of the method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution in the present invention on a test data set, from left to right are the results of texture mapping using the original low-resolution texture atlas, the results of texture mapping after super-resolution of the texture atlas using the default reference scale factor, and the results of texture mapping after super-resolution of the texture atlas using a larger scale factor. DETAILED DESCRIPTION

[0054] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the present invention is further described in detail below in conjunction with the accompanying drawings. It should be understood that the implementation examples described herein are only used to illustrate and explain the present invention, and are not used to limit the present invention.

[0055] The present invention provides a Figure 1The method for improving the consistency of texture resolution of 3D models based on arbitrary scale super-resolution is shown in the figure. Given a 3D geometric model M (with vertices and triangles ), a series of multi-view images and its camera parameters (V, F, N are the number of vertices, the number of triangles and the number of multi-view images respectively), the present invention aims to generate a three-dimensional model with consistent texture resolution. When generating the three-dimensional model, it is necessary to solve the problem of inconsistent texture resolution of the three-dimensional model caused by inconsistent resolution of multi-view images, so as to generate a three-dimensional model with better visual effect. In order to achieve this purpose, the present invention proposes a method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution. The method proposed in the present invention can adaptively improve the texture resolution of each triangular face cluster, eliminate the visual seams caused by inconsistent texture resolution, and improve the visual effect of the three-dimensional model. The technical solution adopted by the present invention includes the following steps:

[0056] Step 1, generating triangular face clusters and texture atlases. The present invention uses the mosaic-based texture mapping method mvs-texturing to generate triangular face clusters and initial texture atlases. To generate triangular face clusters, it is first necessary to determine the texture of each triangular face f i The corresponding best viewing angle image is recorded as image label l i Specifically, the energy equation E(k) is constructed to select the best viewing angle image, where the data term E data Used to select a better visible viewing angle image for the triangle, the smoothing term E smooth Used to minimize the visibility of visual seams. The energy equation E(l) is expressed as:

[0057]

[0058] Among them, l represents the image label set of all triangular faces, Faces represents the set of all triangular faces, Edge represents the edge formed by two adjacent triangular faces, Edges represents the set of all edges, and l j Represents the triangle face f j The best viewing angle image, j is to distinguish it from i, j≠i, φ(f i ,l i ) represents the triangular face f i In the image The projection on Denotes the Sobel operator, and [·] denotes the Iverson bracket. Solving the above energy equation using the graph cut method can obtain the labels of the best viewing angle images of all triangles. Adjacent triangles with the same label are aggregated into triangle clusters. Where C is the number of triangle clusters. Then, the texture tiles corresponding to the triangle clusters are obtained from the multi-view images and packaged to generate a texture atlas. All texture tiles are recorded as

[0059] Step 2: Calculate the super-resolution factor of the triangular face cluster. To unify the resolution of the triangular face cluster, the present invention introduces a scale factor to represent the texture resolution of the triangular face cluster. Then, the super-resolution factor of the triangular face cluster is calculated based on the scale factor. Specifically,

[0060] Step 2.1, generate the scale factor of the triangular face cluster. The scale factor r i Defined as a triangular face cluster c i The area of ​​the triangle cluster and the texture tile t i The ratio of the areas is as follows:

[0061]

[0062] in and Respectively represent c i and t i The area of ​​​​the ,∈ is a very small constant. i The larger the value, the clearer the corresponding texture on the 3D model.

[0063] Step 2.2, generating super-resolution factors of triangular face clusters. Since the texture resolutions of different triangular face clusters are different, the present invention adaptively calculates the super-resolution factor of each triangular face cluster based on the scale factor defined in formula (2). Before calculating the super-resolution factor, a reference scale factor r needs to be set. ref In the present invention, r ref The default value is the maximum value r of all triangle cluster scale factors. max .Right now Then, the super-resolution factor s of each triangle cluster is i It can be obtained by the following formula:

[0064]

[0065] It is worth noting that the reference scale factor r ref It can also be set to be greater than r max This will further increase the overall resolution of the texture model.

[0066] Step 3, arbitrary scale super-resolution based on reference information. After obtaining the super-resolution factor of each triangular face cluster, a super-resolution network is needed to improve the resolution of the texture tile corresponding to the triangular face cluster. Considering that the arbitrary scale super-resolution network can process arbitrary scale factors through a single model, the present invention adopts an arbitrary scale super-resolution network to achieve resolution improvement of texture tiles. In addition, the effective area of ​​the texture tile is usually irregular. In order to better utilize two-dimensional convolution and integrate neighborhood information to achieve super-resolution, the object to be super-resolved in the present invention is the texture tile t i Image at its best viewing angle The minimum enclosing rectangular area in is called t i ′. For simplicity, in the super-resolution network, the block t ′ ′ is called t LR The super-resolution network will be based on the super-resolution factor s i To generate super-resolution patches t SR Considering the ill-posedness of single-image super-resolution and the requirement of the present invention that the super-resolution result content must meet the multi-view consistency requirement, reference image information is introduced to achieve higher quality super-resolution reconstruction. Is the image with the optimal viewing angle In the spatially adjacent multi-view images, J represents the number of reference images. The multi-view geometric relationship used to integrate the reference information is known in the texture mapping task, which can quickly match the corresponding points between the optimal view image and the reference image.

[0067] The arbitrary scale super-resolution network based on reference information proposed in the present invention is as follows Figure 2 As shown, it includes the following sub-steps:

[0068] Step 3.1, feature extraction. The classic and lightweight super-resolution network EDSR-b is used as a feature extractor for images to generate local patches t LR The feature map F LR and reference image feature map

[0069] Step 3.2: Cross-view feature aggregation. For the feature map F LR For each pixel point p in the image, we first project it to the reference image perspective according to the multi-view geometric relationship, and the projection point is recorded as Also called query point. Since the camera pose and geometric model may not be accurate, There is a deviation from the actual projection position of p in the reference image perspective, so Create a small window for the center To aggregate the information of the reference image. Specifically, the cross attention mechanism is used to enhance the image features of point p, as shown below:

[0070]

[0071] in, represents the feature of point p after the cross-attention mechanism aggregates the reference information, F inter is the corresponding feature map. (m,n) indicates the number of pixels contained in the window. The coordinate position inside, σ represents the Softmax function. Q, and They represent the query, key, and value in the attention mechanism, respectively, and are defined as follows:

[0072]

[0073] in, is the low-resolution image feature of pixel p, is the reference image feature at coordinate (m,n), It is based on the pixel point p from the low-score feature map F LR Get, According to the coordinates (m,n) from the reference image feature map Get, dis m,n The projection point The relative distance between the pixel (m,n) and its neighborhood window, that is, cell is a grid decoding used to specify the shape of the query area. Cell is based on the super-resolution ratio s i Calculated, specifically [2 / s i ,2 / s i ].φ k and φ v are the key network and value network implemented by multi-layer perceptron, both of which are used for feature mapping. According to equation (4), the feature generation of all reference images of aggregated pixel p is In order to generate the final enhanced feature map Further integration of F inter and F LR First, the two feature maps are concatenated in the channel dimension, and then they are processed by two convolutional layers Conv and two nonlinear activation functions ReLU. The specific process of the feature fusion module is as follows:

[0074]

[0075] Through the above method, the reference information in the multi-view image can be integrated into the low-resolution image feature map.

[0076] Step 3.3, information integration within the view. The present invention uses the arbitrary scale super-resolution method CiaoSR to integrate the information within the view. CiaoSR is a super-resolution method based on implicit neural expression. This type of method will use the implicit image function f according to the coordinates of the query point and its implicit encoding θ (Realized by multi-layer perceptron) predicts the color value of the coordinate point. In order to obtain information-rich implicit coding, CiaoSR uses an attention mechanism to jointly consider the spatial distance and feature similarity between coordinate points when integrating the neighborhood information of the query point. Specifically, assuming that x is the super-resolution result image t SR The pixel coordinates in x q For low-resolution images LR The corresponding query point coordinates in x q A small local area centered Perform intra-view attention to obtain implicit encoding It can be formulated as:

[0077]

[0078] Among them, the query, key and value of the intra-view attention mechanism are defined as:

[0079]

[0080] In the formula (m * ,n * ) is the closest query point x q Pixels, and Indicated in (m * ,n * ) and (m,n) positions, respectively, according to the coordinates (m * ,n * ) and coordinates (m,n) from the enhanced low-resolution feature map Get, φ k ′ and φ v ′ are key network and value network respectively, Represents the enhanced low-score feature map The non-local features captured in are generated by the non-local module.

[0081] Step 3.4: Super-resolution image reconstruction. For each pixel coordinate x in the super-resolution image, its implicit encoding is obtained. Then, through the implicit image function f θ (z x,x) to predict its color value (the implicit image function is usually implemented by a multi-layer perceptron). Then the color prediction values ​​of all pixel coordinates are added pixel by pixel to the bicubic interpolation result map t of the low-resolution image. LR↑ Then, the super-resolution image t is generated sR , as shown below:

[0082]

[0083] Then, the effective texture area in the super-resolution image is obtained through mask processing, and then the texture block with improved resolution is obtained.

[0084] Step 3.5, use the loss function. Based on the training sample set, train the super-resolution network of any scale. The loss function is The norm constrains the difference between the super-resolution result image and the high-resolution image at the pixel level. The loss function is used to train the arbitrary scale super-resolution network proposed in the present invention until the entire network converges to the optimal accuracy.

[0085] Step 4: Texture atlas update and 3D texture model generation. After each texture tile has completed the super-resolution processing at any scale, the super-resolution texture tile is Pack and generate a new texture atlas. Based on the updated texture atlas, a 3D model with consistent texture resolution and realistic visual effects can be generated.

[0086] The present invention has carried out experiments on image arbitrary scale super-resolution and texture resolution consistency improvement on the test data of the multi-view geometry dataset DTU. The results are shown in the following examples: Figure 3 and Figure 4 shown. Figure 3 These are the result diagrams of low-resolution images at different super-resolution scales, from which we can see that the present invention can stably improve the resolution of images, and can correctly reproduce the details in low-resolution images even in the super-resolution result diagram of a larger scale of 4.0×. Figure 4 The result of texture mapping the same 3D geometric model using texture atlases of different resolutions. Through the three local close-ups of the 3D model, we can see that the present invention can improve the resolution of the model by using the default reference scale factor to super-resolve the texture atlas and then perform texture mapping, and effectively alleviate the resolution inconsistency problem existing in the texture mapping result image using the original low-resolution texture atlas. Furthermore, the present invention uses a scale factor larger than the default reference scale factor to super-resolve the texture atlas and then perform texture mapping. At this time, the resolution of the 3D model is further improved and still maintains good model resolution consistency.

[0087] On the other hand, an embodiment of the present invention also provides a system for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution, including a processor and a memory, the memory being used to store program instructions, and the processor being used to call the stored instructions in the memory to execute the method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as described in the above technical solution.

[0088] It should be understood that the above description of the preferred embodiment is relatively detailed and cannot be regarded as limiting the scope of patent protection of the present invention. Under the enlightenment of the present invention, ordinary technicians in this field can also make substitutions or modifications without departing from the scope of protection of the claims of the present invention, which all fall within the scope of protection of the present invention. The scope of protection requested for the present invention shall be based on the attached claims.

Claims

1. A method for improving the consistency of texture resolution of a 3D model based on arbitrary scale super-resolution, characterized by: The steps include: Step 1, determine the best viewing angle image corresponding to each triangle, and generate a triangle cluster and an initial texture atlas consisting of texture blocks; Step 2, calculating the super-resolution factor of the triangular face cluster based on the area of ​​the triangular face cluster and the texture block; Step 3: Use a super-resolution network to improve the resolution of the texture tiles corresponding to the triangular face clusters; The object to be super-resolved is the smallest enclosing rectangular area of ​​the texture block in its optimal view image. The super-resolution network generates super-resolution blocks based on the super-resolution factor. It also introduces reference image information to achieve higher quality super-reconstruction. The reference image is a multi-view image that is spatially adjacent to the optimal view image. Step 4: After each texture tile has completed the super-resolution processing at any scale, the super-resolution texture tiles are packaged to generate a new texture atlas, and a three-dimensional model with consistent texture resolution and realistic visual effects is generated based on the updated texture atlas.

2. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 1, characterized in that: In step 1, to generate a triangular face cluster, first determine each triangular face f i The corresponding best viewing angle image is recorded as image label l i Specifically, the energy equation E(l) is constructed to select the best viewing angle image, where the data term E data Used to select a better visible viewing angle image for the triangle, the smoothing term E smooth Used to minimize the visibility of visual seams; the energy equation E(l) is expressed as: E(l)=E data +E smooth Among them, l represents the image label set of all triangular faces, Faces represents the set of all triangular faces, Edge represents the edge formed by two adjacent triangular faces, Edges represents the set of all edges, and l j Represents the triangle face f j The best viewing angle image, j is to distinguish it from i, j≠i, φ(f i , l i ) represents the triangular face f i In the image The projection on represents the Sobel operator, and [·] represents the Iverson bracket. The above energy equation is solved by the graph cut method to obtain the labels of the best viewing angle images of all triangles, and the adjacent triangles with the same label are aggregated into triangle clusters. Where C is the number of triangle clusters; then, the texture tiles corresponding to the triangle clusters are obtained from the multi-view images, packaged to generate a texture atlas, and all texture tiles are recorded as 3. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 1, characterized in that: The specific implementation of step 2 is as follows: Step 2.1, generate the scale factor of the triangular face cluster: scale factor r i Defined as a triangular face cluster c i The area of ​​the triangle cluster and the texture tile t i The ratio of the areas is as follows: in and Respectively represent c i and t i The area of ​​​​the ,∈ is a constant; r i The larger the value, the clearer the corresponding texture on the 3D model; Step 2.2, generating super-resolution factors of triangular face clusters: Since the texture resolutions of different triangular face clusters are different, the super-resolution factor of each triangular face cluster is adaptively calculated based on the scale factor defined in formula (2); before calculating the super-resolution factor, a reference scale factor r is set ref ; Then, the super-resolution factor s of each triangle cluster is i Obtained by the following formula:

4. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 3, characterized in that: Reference scale factor r ref is the maximum value r of the scale factor of all triangle clusters max or greater than r max The value of .

5. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 1, characterized in that: The specific implementation of step 3 is as follows: Step 3.1, feature extraction: The super-resolution network EDSR-b is used as a feature extractor to generate texture patches t LR The feature map F LR and reference image feature map J represents the number of reference images; Among them, t LR is the texture tile t i Image at its best viewing angle The minimum enclosing rectangular area in ; Step 3.2, cross-view feature aggregation: For the feature map F LR For each pixel point p in the image, we first project it to the reference image perspective according to the multi-view geometric relationship, and the projection point is recorded as Also called a query point. Create a small window for the center Use the cross attention mechanism to enhance the image features of point p and generate the final enhanced feature map To integrate the reference information in multi-view images into the low-resolution image feature map; Step 3.3, information integration within the view: use the arbitrary scale super-resolution method CiaoSR to integrate the information within the view to obtain the implicit code corresponding to the query point coordinates; Step 3.4, super-resolution image reconstruction: For each pixel coordinate in the super-resolution image, the coordinates of the query point and its implicit encoding are combined to predict its color value through the implicit image function, and then the color prediction values ​​of all pixel coordinates are added pixel by pixel to the bicubic interpolation result map of the low-resolution image to generate a super-resolution image; then, the effective texture area in the super-resolution image is obtained through mask processing, and then the texture block with improved resolution is obtained; Step 3.5, using the loss function: Based on the training sample set, train the super-resolution network of any scale until the entire network converges to the optimal accuracy.

6. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 5, characterized in that: In step 3.2, the cross attention mechanism is used to enhance the image features of point p. The calculation formula is as follows: in, represents the feature of point p after the cross-attention mechanism aggregates the reference information, F inter is the corresponding feature map; (m, n) represents the feature map contained in the window The coordinate position inside, σ represents the Softmax function; Q, and They represent the query, key, and value in the attention mechanism, respectively, and are defined as follows: in, is the low-resolution image feature of pixel p, is the reference image feature at coordinate (m, n), dis m，n The projection point The relative distance between the pixel (m, n) and its neighborhood window, cell is the grid decoding used to specify the shape of the query area; φ k and φ v are the key network and value network implemented by multi-layer perceptron, both of which are used for feature mapping; according to formula (4), the feature generation of all reference images of aggregated pixel p In order to generate the final enhanced feature map Further integration of F inter and F LR First, the two feature maps are concatenated in the channel dimension, and then they are processed by two convolutional layers Conv and two nonlinear activation functions ReLU. The specific process of the feature fusion module is as follows:

7. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 5, characterized in that: The specific implementation of step 3.3 is as follows; Assume x is the super-resolution image t SR The pixel coordinates in x q For low-resolution images LR The corresponding query point coordinates are in x q A small local area centered Perform intra-view attention to obtain implicit encoding Formulated as: Among them, the query, key and value of the intra-view attention mechanism are defined as: In the formula (m * , n * ) is the closest query point x q Pixels, and Indicated in (m * , n * ) and (m, n) positions after the enhancement of low-scoring features, φ k ′ and φ v ′ are key network and value network respectively, Represents the enhanced low-score feature map The non-local features captured in .

8. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 5, characterized in that: In step 3.4, the super-resolution image t SR The calculation of is as follows: in, is the bicubic interpolation result image, f θ (z x , x) is the implicit image function, implemented by multi-layer perceptron, z x Represents the implicit code corresponding to each pixel coordinate x in the super-resolution image.

9. The method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as claimed in claim 5, characterized in that: In step 3.5, the loss function is norm, which constrains the difference between the super-resolution result image and the high-resolution image at the pixel level.

10. A 3D model texture resolution consistency improvement system based on arbitrary scale super-resolution, characterized by: It includes a processor and a memory, the memory is used to store program instructions, and the processor is used to call the stored instructions in the memory to execute the method for improving the consistency of texture resolution of a three-dimensional model based on arbitrary scale super-resolution as described in any one of claims 1 to 9.

Images