Training methods for image restoration models, image restoration methods, devices and equipment

Abstract

This application provides a training method for an image correction model, an image correction method, an apparatus, and a device, belonging to the field of image processing technology. The method adds noise features to the viewpoint features corresponding to a second viewpoint image after texture defect correction. Then, combining the viewpoint features corresponding to a first viewpoint image with texture defects and reference features, it predicts the added noise features. Based on the difference between the predicted noise features and the actual added noise features, the image correction model is trained, enabling it to accurately predict noise features. After training the image correction model using this method, for any viewpoint image with texture defects, the image correction model can predict the noise corresponding to the texture defects in that viewpoint image and correct the viewpoint image by removing the noise, without requiring manual correction, thus improving correction efficiency. Furthermore, this method enhances the image correction model's ability to correct texture defects, thereby improving correction quality.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to a training method for an image correction model, an image correction method, an apparatus, and a device. Background Art

[0002] Image retouching is an important task in image processing. Defects such as blurred or incorrect textures in an image can reduce its quality. Therefore, image retouching can be performed to correct these defects and obtain a high-quality image.

[0003] In related technologies, in order to obtain high-quality images, image correction usually requires manual correction based on experience, which makes image correction inefficient. Summary of the Invention

[0004] This application provides a training method for an image correction model, an image correction method, an apparatus, and a device, which improves image correction efficiency and quality. The technical solutions provided by this application are as follows.

[0005] According to one aspect of the embodiments of this application, a method for training an image correction model is provided, the method comprising:

[0006] Multiple sets of first training samples are obtained. Each set of first training samples includes a first view image, a second view image, and a reference image of the three-dimensional model. The texture of the three-dimensional model in the first view image has defects. The second view image is the first view image after the defects are corrected. The reference image is a view image showing the appearance of the three-dimensional model.

[0007] The first viewpoint image, the second viewpoint image, and the reference image are input into an image correction model. The image correction model processes the first viewpoint image, the second viewpoint image, and the reference image respectively to obtain a first viewpoint feature, a second viewpoint feature, and a reference feature. A first noise feature is added to the second viewpoint feature to obtain a third viewpoint feature. Based on the first viewpoint feature, the third viewpoint feature, and the reference feature, a prediction is made to obtain a second noise feature that represents the noise feature in the third viewpoint feature. The first viewpoint feature, the second viewpoint image, and the reference feature are used to describe the first viewpoint image, the second viewpoint image, and the reference image, respectively.

[0008] Based on the difference information between the first noise feature and the second noise feature of each of the multiple sets of first training samples, the image correction model is iteratively trained. The image correction model is used to predict the noise in the viewpoint map and correct the viewpoint map by removing the noise.

[0009] According to another aspect of the embodiments of this application, an image correction method is provided, the method comprising:

[0010] Obtain a noise map, a view of the 3D model, and a reference map. The texture of the 3D model in the view of the model has defects, and the reference map is a view of the model that provides the appearance features of the 3D model.

[0011] The noise map, the viewpoint map, and the reference map are input into an image correction model. The image correction model processes the noise map, the viewpoint map, and the reference map respectively to obtain noise features, viewpoint features, and reference features. Based on the noise features, the viewpoint features, and the reference features, prediction is performed to obtain predicted noise features that represent noise features other than viewpoint features in the noise features. The predicted noise features are removed from the noise features to obtain corrected viewpoint features. Based on the corrected viewpoint features, the corrected viewpoint map is obtained.

[0012] The noise feature, the viewpoint feature, and the reference feature are used to describe the noise map, the viewpoint map, and the reference map, respectively.

[0013] According to another aspect of the embodiments of this application, a training apparatus for an image correction model is provided, the apparatus comprising:

[0014] The acquisition module is used to acquire multiple sets of first training samples. Each set of first training samples includes a first view image, a second view image, and a reference image of the three-dimensional model. The texture of the three-dimensional model in the first view image has defects. The second view image is the first view image after the defects are corrected. The reference image is a view image showing the appearance of the three-dimensional model.

[0015] The input / output module is used to input the first viewpoint image, the second viewpoint image, and the reference image into an image correction model, and process the first viewpoint image, the second viewpoint image, and the reference image through the image correction model to obtain a first viewpoint feature, a second viewpoint feature, and a reference feature. A first noise feature is added to the second viewpoint feature to obtain a third viewpoint feature. Based on the first viewpoint feature, the third viewpoint feature, and the reference feature, a prediction is made to obtain a second noise feature that represents the noise feature in the third viewpoint feature. The first viewpoint feature, the second viewpoint image, and the reference feature are used to describe the first viewpoint image, the second viewpoint image, and the reference image, respectively.

[0016] The training module is used to iteratively train the image correction model based on the difference information between the first noise features and the second noise features of each of the multiple sets of first training samples. The image correction model is used to predict noise in the viewpoint map and correct the viewpoint map by removing noise.

[0017] In some embodiments, the input / output module is configured to:

[0018] By splicing the first viewpoint features and the third viewpoint features, a spliced ​​feature is obtained;

[0019] The second noise feature is obtained by prediction based on the splicing features and the reference features.

[0020] In some embodiments, the input / output module is configured to:

[0021] The convolution module of the image correction model performs convolution processing on the stitched features to obtain processed stitched features. The convolution processing is used to extract features from the stitched features and limit the dimensions of the stitched features to a preset range.

[0022] The attention module of the image correction model processes the processed spliced ​​features and the reference features to obtain cross-attention features. The query parameter in the attention module is the processed spliced ​​features, and the key parameter and value parameter in the attention module are both the reference features.

[0023] The image correction model's denoising module fuses the processed spliced ​​features and the cross-attention features, and prediction is performed based on the fused features to obtain the second noise feature.

[0024] In some embodiments, the acquisition module is further configured to:

[0025] The image correction model is obtained based on the pre-trained model. The pre-trained model is trained based on multiple sets of second training samples. The model parameters of the convolution module corresponding to the second viewpoint features, the model parameters of the attention module, and the model parameters of the denoising module in the image correction model are the model parameters of the convolution module, attention module, and denoising module in the pre-trained model, respectively. The second training samples include sample images and sample images with added noise.

[0026] In some embodiments, the input / output module is configured to:

[0027] The first encoding module of the image correction model processes the first view map and the second view map respectively to obtain the first view feature and the second view feature. The first view feature is the representation of the first view map in the latent space, and the second view feature is the representation of the second view map in the latent space.

[0028] The reference image is processed by the second encoding module of the image correction model to obtain the reference features, which are the representation of the reference image in the embedding space.

[0029] According to another aspect of the embodiments of this application, an image correction apparatus is provided, the apparatus comprising:

[0030] The acquisition module is used to acquire a noise map, a view map of the 3D model, and a reference map. The texture of the 3D model in the view map has defects, and the reference map is a view map that provides the appearance features of the 3D model.

[0031] The input / output module is used to input the noise map, the viewpoint map, and the reference map into an image correction model, process the noise map, the viewpoint map, and the reference map respectively through the image correction model to obtain noise features, viewpoint features, and reference features, predict based on the noise features, the viewpoint features, and the reference features to obtain predicted noise features that represent noise features other than viewpoint features in the noise features, remove the predicted noise features from the noise features to obtain corrected viewpoint features, and obtain the corrected viewpoint map based on the corrected viewpoint features.

[0032] The noise feature, the viewpoint feature, and the reference feature are used to describe the noise map, the viewpoint map, and the reference map, respectively.

[0033] In some embodiments, the input / output module is configured to:

[0034] By splicing the noise features and the viewpoint features, a spliced ​​feature is obtained;

[0035] Based on the splicing features and the reference features, prediction is performed to obtain the predicted noise features.

[0036] In some embodiments, the input / output module is configured to:

[0037] The convolution module of the image correction model performs convolution processing on the stitched features to obtain processed stitched features. The convolution processing is used to extract features from the stitched features and limit the dimensions of the stitched features to a preset range.

[0038] The attention module of the image correction model processes the processed spliced ​​features and the reference features to obtain cross-attention features. The query parameter in the attention module is the processed spliced ​​features, and the key parameter and value parameter in the attention module are both the reference features.

[0039] The image correction model's denoising module fuses the processed splicing features and the cross-attention features, and prediction is performed based on the fused features to obtain the predicted noise features.

[0040] In some embodiments, the input / output module is configured to:

[0041] The noise map and the view map are processed by the first encoding module of the image correction model to obtain the noise feature and the view feature. The noise feature is the representation of the noise map in the latent space, and the view feature is the representation of the view map in the latent space.

[0042] The reference image is processed by the second encoding module of the image correction model to obtain the reference features, which are the representation of the reference image in the embedding space.

[0043] In some embodiments, the input / output module is configured to:

[0044] The image correction model's decoding module processes the corrected viewpoint features to obtain the corrected viewpoint image.

[0045] In some embodiments, the input / output module is configured to:

[0046] Based on the noise features, the viewpoint features, and the reference features, multiple first-iteration calculations are performed until the first-iteration stopping condition is met, thereby obtaining the corrected viewpoint features; the multiple first-iteration calculations include:

[0047] In the first iteration calculation, a prediction is made based on the noise feature, the view feature, and the reference feature to obtain the first predicted noise feature. The first predicted noise feature is used to represent the noise features other than the view feature in the noise features. The first predicted noise feature is removed from the noise features to obtain the first corrected view feature.

[0048] In the first iteration of the Mth calculation, prediction is performed based on the (M-1)th corrected viewpoint feature, the viewpoint feature, and the reference feature to obtain the Mth predicted noise feature. The Mth predicted noise feature is used to represent the noise features other than the viewpoint feature in the (M-1)th corrected viewpoint feature. The Mth predicted noise feature is removed from the (M-1)th corrected viewpoint feature to obtain the Mth corrected viewpoint feature, where M is an integer greater than 1.

[0049] In some embodiments, the apparatus further includes:

[0050] The first correction module is used to correct multiple viewpoint images of the three-dimensional model through the image correction model to obtain multiple corrected viewpoint images. The multiple viewpoint images are rendered based on the texture map of the three-dimensional model, and the texture map includes textures at multiple locations on the three-dimensional model.

[0051] The second correction module is used to correct the texture map based on the plurality of view maps and the plurality of corrected view maps.

[0052] In some embodiments, the second correction module is configured to:

[0053] Based on the multiple viewpoints and the multiple corrected viewpoints, multiple second-iteration calculations are performed until the second-iteration stopping condition is met; the multiple second-iteration calculations include:

[0054] In the first second iteration calculation, the difference information between each view map and its corresponding corrected view map is determined, and the texture map is corrected based on the difference information between the multiple view maps and the multiple corrected view maps;

[0055] In the Nth second iteration calculation, based on the corrected texture map from the (N-1)th second iteration calculation and the multiple perspectives corresponding to the multiple perspective maps, the 3D model is rendered to obtain the Nth multiple perspective maps. The difference information between each perspective map in the Nth iteration and its corresponding corrected perspective map is determined. Based on the difference information between the multiple perspective maps in the Nth iteration and the multiple corrected perspective maps, the corrected texture map from the (N-1)th second iteration calculation is corrected, where N is an integer greater than 1.

[0056] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store a computer program, the computer program being loaded and executed by the processor to implement the training method or image correction method of the image correction model in the embodiments of this application.

[0057] On the other hand, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, the computer program being loaded and executed by a processor to implement the training method or image correction method of the image correction model in the embodiments of this application.

[0058] On the other hand, a computer program product is provided, the computer program product including a computer program stored in a computer-readable storage medium, a processor of a computer device reading the computer program from the computer-readable storage medium, the processor executing the computer program, causing the computer device to perform the training method or image correction method of the image correction model described in any of the above implementations.

[0059] This application provides a training method for an image correction model. During training, features of each image are first extracted using the image correction model. Then, noise features are added to the viewpoint features corresponding to the second viewpoint image after texture defect correction. The added noise features are then predicted by combining the viewpoint features corresponding to the first viewpoint image with texture defects and reference features. The image correction model is trained based on the difference between the predicted noise features and the actual added noise features, enabling the image correction model to accurately predict noise features. After training the image correction model using this method, for any viewpoint image with texture defects, the image correction model can predict the noise corresponding to the texture defects in that viewpoint image and correct the viewpoint image by removing the noise, without requiring manual correction, thus improving correction efficiency. Furthermore, when predicting noise features, first-view features and reference features are combined. Since the first-view features provide prior information about the first-view image, the predicted noise features are noise features that, after removal, can make the view image close to the first-view image but correct texture defects. Since the reference features provide appearance control information for the view image, the predicted noise features are noise features that, after removal, can make the view image match the overall appearance of the 3D model. In other words, this method makes the noise features predicted by the image correction model accurate, which can improve the image correction model's ability to correct texture defects and thus improve the correction quality. Attached Figure Description

[0060] Figure 1 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0061] Figure 2 This is a schematic diagram of the implementation environment of a solution provided in one embodiment of this application;

[0062] Figure 3 This is a flowchart of a training method for an image correction model provided in one embodiment of this application;

[0063] Figure 4 This is a flowchart of a training method for an image correction model provided in one embodiment of this application;

[0064] Figure 5 This is a schematic diagram illustrating the principle of a pre-trained model provided in one embodiment of this application;

[0065] Figure 6 This is a flowchart of a training method for an image correction model provided in one embodiment of this application;

[0066] Figure 7 This is a flowchart of an image correction method provided in one embodiment of this application;

[0067] Figure 8This is a flowchart of an image correction method provided in one embodiment of this application;

[0068] Figure 9 This is a flowchart of an image correction method provided in one embodiment of this application;

[0069] Figure 10 This is a flowchart of texture map correction provided in one embodiment of this application;

[0070] Figure 11 This is a comparison image of the texture map before and after correction according to an embodiment of this application;

[0071] Figure 12 This is a flowchart of an image correction method provided in one embodiment of this application;

[0072] Figure 13 This is a flowchart of texture map correction provided in one embodiment of this application;

[0073] Figure 14 This is a block diagram of a training apparatus for an image correction model provided in one embodiment of this application;

[0074] Figure 15 This is a block diagram of an image correction apparatus provided in one embodiment of this application;

[0075] Figure 16 This is a structural block diagram of a computer device provided in one embodiment of this application. Detailed Implementation

[0076] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0077] Please refer to Figure 1 The diagram illustrates a training method for an image correction model provided in one embodiment of this application. The method includes at least one of the following steps:

[0078] 1. Obtain training samples: Obtain multiple sets of training samples for training the image correction model. Each set of training samples includes a first-view image 101, a second-view image 102, and a reference image 103 of the 3D model. The first-view image 101 is a view image with texture defects, the second-view image 102 is a first-view image after the defects are corrected, and the reference image 103 is a view image used to show the appearance of the 3D model.

[0079] 2. Predicting Noise Features: Due to texture defects in the first-view image 101, an image correction model is used to correct these defects to obtain a high-quality view image. Specifically, the first-view image 101, the second-view image 102, and the reference image 103 are processed separately to obtain their respective first-view feature 104, second-view feature 105, and reference feature 106. Then, a first noise feature 107 is added to the second-view feature 105 to obtain a third noise feature 108. This third noise feature 108 is then combined with the first-view feature 104 and the reference feature 106 for prediction, resulting in a second noise feature 109 representing the added noise feature.

[0080] 3. Training the model: Based on the difference between the predicted second noise feature 109 and the actual added first noise feature 107, the image correction model is iteratively trained so that the image correction model can predict the noise features accurately.

[0081] In this embodiment of the application, noise features are predicted by training an image correction model. Subsequently, after any viewpoint image with texture defects is given, the noise features in it are predicted by the image correction model. These noise features correspond to the parts with texture defects in the viewpoint image. By removing the noise features, the viewpoint image with texture defects corrected can be obtained.

[0082] Please refer to Figure 2 The diagram illustrates a computer system provided in one embodiment of this application. The computer system includes at least one of the following: a terminal device 10 and a server 20.

[0083] Terminal device 10 can be an electronic device such as a mobile phone, tablet computer, multimedia playback device, PC (Personal Computer), wearable device, in-vehicle terminal device, VR (Virtual Reality) device, AR (Augmented Reality) device, MR (Mixed Reality) device, etc. Terminal device 10 can have a client running the target application. The target application can be an application that requires the use of a 3D model, or it can be an application used to construct a 3D model; this application embodiment does not limit the specific implementation form of the target application. For example, it can be an application that requires downloading and installation, a small program that does not require installation, a web application, etc. For example, the target application can include at least one of the following: a game application, a video application, a game engine application, or a modeling application. Texture images are loaded into the target application to render the 3D model.

[0084] In this embodiment, server 20 is used to provide background services for the target application. For example, server 20 is used to correct the viewpoint of a 3D model. Server 20 has an image correction model embedded in it, and server 20 corrects the viewpoint of the 3D model using the image correction model. In other embodiments, server 20 provides a viewpoint of a 3D model to terminal 10, terminal 10 has an image correction model embedded in it, and terminal 10 corrects the viewpoint provided by server 20 using the image correction model.

[0085] Server 20 can be a single server or a server cluster consisting of multiple servers. Terminal device 10 can communicate with server 20 via a network, such as a wireless or wired network.

[0086] Please refer to Figure 3 , Figure 3 This is a flowchart of a training method for an image correction model provided in one embodiment of this application. The execution entity for each step of this method can be a computer device, such as a terminal or a server. The method is described using a server as an example, and may include at least one of the following steps 301 to 303.

[0087] 301. Obtain multiple sets of first training samples. Each set of first training samples includes a first-view image, a second-view image, and a reference image of the 3D model. The texture of the 3D model in the first-view image has defects. The second-view image is the first-view image after the defects are corrected. The reference image is a view showing the appearance of the 3D model.

[0088] In this embodiment, the three-dimensional model is composed of points, lines, and surfaces. The three-dimensional model can be used to represent real-world entities or virtual objects. The three-dimensional model can be a model with a three-dimensional structure constructed in a virtual three-dimensional space.

[0089] In some embodiments, the first-view image is an image of the 3D model from a certain perspective, which can be a camera view. From this perspective, the 3D model is rendered based on its texture map to obtain the first-view image. A rendering engine is used to render the textures from the 3D model's texture map onto the 3D model, resulting in the view image. The resolution of the view image can be set as needed, such as 512*512.

[0090] In the embodiments of this application, the first perspective view can be a full-body perspective view of the three-dimensional model, or it can be a partial perspective view of the three-dimensional model, such as a face perspective view or a leg perspective view of the three-dimensional model.

[0091] A 3D model's texture map includes textures at multiple locations on the 3D model. The surface of a 3D model is composed of triangles or polygons, and the texture map is a two-dimensional planar image obtained by unfolding the surface of the 3D model into a two-dimensional plane. The coordinates of vertices on the texture map are called texture coordinates. Each vertex in the 3D model has a corresponding texture coordinate on the texture map. Texture coordinates indicate the coordinates of a vertex in the 3D model on the texture map. During texture rendering, the texture at the location indicated by the texture coordinates of a vertex in the texture map is mapped to that vertex in the 3D model to render the texture onto the 3D model.

[0092] Texture in a 3D model refers to a 2D image mapped onto the surface of a 3D object. It is used to enhance the visual effect of the object, making it look more realistic and detailed. Texture typically includes attributes such as color, smoothness, and reflectivity, and plays a crucial role in the rendering process of 3D models.

[0093] In the embodiments of this application, the defects in the texture include at least one of texture blurring, texture errors, and insufficient clarity of texture details. Among them, texture blurring can refer to texture edges that are not sharp or color distortion, while texture errors can refer to geometric distortion of texture lines.

[0094] In this embodiment of the application, the reference figure is a perspective view used to show the appearance of a 3D model. Further, the reference figure is a perspective view that provides as comprehensive a range of texture and structural features as possible of the 3D model's appearance, such as a front view of the 3D model.

[0095] In this embodiment, multiple 3D models can be acquired first, and then multiple sets of first training samples can be obtained through the 3D models. Since viewpoint images of the 3D models can be obtained from different perspectives, optionally, multiple sets of first training samples can be obtained based on multiple 3D models, and one or more sets of first training samples can be obtained based on each 3D model. Multiple sets of first training samples corresponding to the same 3D model may include first viewpoint images obtained from different perspectives and the same reference image. For example, at least some of the first training samples may include multiple first viewpoint images that are respectively the front view, top view, side view, or other viewpoint images of the same 3D model, etc., without specific limitations here.

[0096] 302. Input the first-view image, the second-view image, and the reference image into the image correction model. Process the first-view image, the second-view image, and the reference image through the image correction model to obtain the first-view feature, the second-view feature, and the reference feature. Add the first noise feature to the second-view feature to obtain the third-view feature. Make predictions based on the first-view feature, the third-view feature, and the reference feature to obtain the second noise feature used to represent the noise feature in the third-view feature. The first-view feature, the second-view feature, and the reference feature are used to describe the first-view image, the second-view image, and the reference image, respectively.

[0097] In the embodiments of this application, the first viewpoint feature, the second viewpoint feature, the third viewpoint feature, the reference feature, the first noise feature, and the second noise feature can be vectors or matrices. In this embodiment, these features are used as matrices for illustration.

[0098] Optionally, the first noise feature is a feature representation of random noise, such as a feature representation of random Gaussian noise. The first noise feature and the second viewpoint feature have the same dimension, making it easier to add the first noise feature to the second viewpoint feature through summation. Correspondingly, the first noise feature and the second noise feature have the same dimension, making it easier to determine the difference information between them.

[0099] In this embodiment of the application, a first noise feature is added to the second view feature, so that the second view image corresponding to the second view feature produces texture defects. The first noise feature is also the feature corresponding to the texture defect part in the second view image. Therefore, the image correction model is trained to predict the noise in the view image, that is, to predict the feature corresponding to the texture defect part in the view image. Thus, the texture defects in the view image can be corrected by removing noise.

[0100] In this embodiment, the first view is a view with texture defects, the second view is the first view after the defects are corrected, and the reference image is a view showing the appearance of the 3D model. Based on these three, noise features in the first view are predicted. In this way, the prediction of noise features has the prior of the first view, so as to generate a view that is close to the first view but of higher quality (i.e., the defect is corrected). The reference image is used as a condition constraint on the appearance of the 3D model, so that the generated view is not only of high quality in local appearance, but also matches the overall appearance of the 3D model, so that the corrected view is of higher quality.

[0101] 303. Based on the difference information between the first noise features and the second noise features of each of the multiple sets of first training samples, the image correction model is iteratively trained. The image correction model is used to predict the noise in the view map and correct the view map by removing the noise.

[0102] In this embodiment, with the goal of minimizing the difference between the first noise feature and the second noise feature, the parameters of the image correction model are iteratively adjusted so that the second noise feature predicted by the image correction model gradually approaches the actual added first noise feature.

[0103] In this process, by training the image correction model to predict noise features, and then given any viewpoint feature with texture defects, the image correction model predicts the noise features in the viewpoint feature. By removing the noise features from the viewpoint feature, the corrected viewpoint feature can be obtained, and thus the corrected viewpoint map can be obtained.

[0104] This application provides a training method for an image correction model. During training, features of each image are first extracted using the image correction model. Then, noise features are added to the viewpoint features corresponding to the second viewpoint image after texture defect correction. The added noise features are then predicted by combining the viewpoint features corresponding to the first viewpoint image with texture defects and reference features. The image correction model is trained based on the difference between the predicted noise features and the actual added noise features, enabling the image correction model to accurately predict noise features. After training the image correction model in this way, for any viewpoint image with texture defects, the image correction model can predict the noise corresponding to the texture defects in that viewpoint image and correct the viewpoint image by removing the noise, without requiring manual correction, thus improving correction efficiency. Furthermore, when predicting noise features, first-view features and reference features are combined. Since the first-view features provide prior information about the first-view image, the predicted noise features are noise features that, after removal, can make the view image close to the first-view image but correct texture defects. Since the reference features provide appearance control information for the view image, the predicted noise features are noise features that, after removal, can make the view image match the overall appearance of the 3D model. In other words, this method makes the noise features predicted by the image correction model accurate, which can improve the image correction model's ability to correct texture defects and thus improve the correction quality.

[0105] See Figure 4 , Figure 4 This is a flowchart of a training method for an image correction model provided in one embodiment of this application. The execution entity for each step of this method can be a computer device, such as a terminal or a server. The method is described using a server as an example, and may include at least one of the following steps 401 to 407.

[0106] 401. Obtain multiple sets of first training samples. Each set of first training samples includes a first-view image, a second-view image, and a reference image of the 3D model. The texture of the 3D model in the first-view image has defects. The second-view image is the first-view image after the defects are corrected. The reference image is a view showing the appearance of the 3D model.

[0107] In the embodiments of this application, step 401 is the same as step 301, and will not be described again here.

[0108] 402. Input the first view image, the second view image, and the reference image into the image correction model. Process the first view image and the second view image through the first encoding module of the image correction model to obtain the first view feature and the second view feature. The first view feature is the representation of the first view image in the latent space, and the second view feature is the representation of the second view image in the latent space.

[0109] In this embodiment, the first viewpoint feature is a matrix, representing a low-dimensional representation of the first viewpoint graph in the latent space. The first viewpoint feature is obtained by compressing and reducing the dimensionality of the first viewpoint graph using a first encoding module. Similarly, the second viewpoint feature is a matrix, representing a low-dimensional representation of the second viewpoint graph in the latent space. The second viewpoint feature is obtained by compressing and reducing the dimensionality of the second viewpoint graph using a first encoding module.

[0110] For example, the dimensions of the first-view image and the second-view image are 4*512*512. After processing by the first encoding module, the first-view features and second-view features with dimensions of 4*64*64 are obtained.

[0111] The first encoding module is an encoder, which can be an encoder from a Variational Autoencoder (VAE). Accordingly, the model parameters in the first encoding module are pre-trained, so the first and second viewpoint images can be processed directly using the first encoding module without further training of the first encoding module, thereby improving the training efficiency of the image correction model.

[0112] 403. The reference image is processed by the second encoding module of the image correction model to obtain reference features, which are the representation of the reference image in the embedding space.

[0113] In the embodiments of this application, the reference feature is a matrix, and the reference image is converted into a matrix form so that the reference image can participate in the calculation in the image correction model.

[0114] The second encoding module is an encoder, which can be the image encoder in a Contrastive Language-Image Pre-training (CLIP) model. Accordingly, the model parameters in the second encoding module are pre-trained, so the reference image can be processed directly using the second encoding module without further training of the second encoding module, thereby improving the training efficiency of the image correction model.

[0115] It should be noted that the sequence numbers of steps 402 and 403 above are only for ease of explanation and are not intended to restrict the execution order of the two. For example, step 402 can be executed before step 403, after step 403, or simultaneously. In this embodiment, the simultaneous execution of the two is taken as an example to improve efficiency.

[0116] In this embodiment, steps 402 and 403 above are used to process the first-view image, the second-view image, and the reference image using an image correction model to obtain first-view features, second-view features, and reference features, respectively. In this embodiment, the first and second-view images are mapped to a latent space using a first encoding module. Since the latent space typically has a lower dimension than the original image space, it helps reduce the number of model parameters, thereby reducing computational complexity and improving training efficiency. Furthermore, the low-dimensionality of the latent space helps the model capture the essential features of the data, thus improving generalization ability. The second encoding module maps the reference image to the embedding space, enabling feature extraction from the reference image. This facilitates the image correction model's understanding of the reference image, improving the efficiency and accuracy of the image correction model in predicting noise features based on the reference image.

[0117] 404. By adding the first noise feature to the second viewpoint feature through the image correction model, the third viewpoint feature is obtained.

[0118] In some embodiments, the first noise feature and the second view feature have the same dimension, such as 4*64*64. Then, the first noise feature can be added to the second view feature by summing to obtain a third view feature with the same dimension as the second view feature. The third view feature is also the second view feature carrying the first noise feature.

[0119] In some embodiments, the image correction model includes a noise addition module, which adds a first noise feature to the second viewpoint feature.

[0120] 405. By stitching together the first-view features and the third-view features using an image correction model, the stitched features are obtained.

[0121] In this embodiment, since the first-view feature and the third-view feature have the same dimension, the first-view feature and the third-view feature can be concatenated on a certain sub-dimension to obtain a concatenated feature. For example, if the dimensions of the two view features are 4*64*64 respectively, the two view features can be concatenated on the channel sub-dimension to obtain a concatenated feature with a dimension of 8*64*64.

[0122] 406. Using an image correction model, predictions are made based on splicing features and reference features to obtain the second noise feature.

[0123] In some embodiments, the image correction model includes a convolution module, a cross-attention module, and a denoising module. Accordingly, the process described above, which uses the image correction model to predict based on stitched features and reference features to obtain a second noise feature, includes the following steps: The stitched features are convolved using the convolution module of the image correction model to obtain processed stitched features; the convolution process is used for feature extraction from the stitched features and to limit the dimension of the stitched features to a preset range; the processed stitched features and reference features are processed using the attention module of the image correction model to obtain cross-attention features; the query parameter in the attention module is the processed stitched features, and the key and value parameters in the attention module are both reference features; the processed stitched features and cross-attention features are fused using the denoising module of the image correction model, and prediction is performed based on the fused features to obtain the second noise feature.

[0124] The convolutional module comprises one or more convolutional layers. The number of channels in the first convolutional layer is the same as the channel dimension of the concatenated features, facilitating its processing. For example, if the concatenated features are 8*64*64, the first convolutional layer will have 8 channels. The convolutional module extracts preliminary features from the concatenated features, which are then used for deeper processing. Correspondingly, the first convolutional layer increases the dimension of the concatenated features, for example, from 8 dimensions to 320 dimensions, to provide richer reference information for feature extraction in subsequent downsampling stages.

[0125] In this approach, the cross-attention mechanism extracts the most important features from the input data. By using the processed concatenated features as query parameters and the reference features as key and value parameters, the model combines the reference features with the viewpoint features in the latent space. This allows the model to consider additional conditional information during noise prediction, resulting in more accurate predictions. Furthermore, using the reference features as key and value parameters enhances the control over the reference map, strengthening the correlation between the denoised viewpoint map and the reference map, thereby improving the quality of the corrected viewpoint map.

[0126] The fusion method for the processed concatenated features and cross-attention features can be either superposition or concatenation, without specific limitations. Optionally, the processed concatenated features and cross-attention features have the same dimension, thus facilitating their fusion. The denoising module can perform predictions using convolutional networks or other neural networks, without specific limitations.

[0127] It should be noted that the convolutional module, attention module, and denoising module are all modules within the prediction unit of the image correction model, such as the U-net (U-shaped network). The U-net network is a convolutional neural network used for image segmentation, capable of pixel-level processing of input image features with good accuracy and robustness. The U-net network can process image features through its various processing modules; in this embodiment, the U-net network includes a convolutional module, an attention module, and a denoising module, etc., and uses these modules to predict noise features.

[0128] In this embodiment, the spliced ​​features are first processed by a convolutional module to facilitate subsequent processing of the spliced ​​features by the attention module. By combining the reference features and the processed spliced ​​features through the attention module, additional conditional information is considered during the noise prediction process. Furthermore, the spliced ​​features and attention cross-features are combined for prediction, resulting in more accurate predictions. This enhances the correlation between the denoised viewpoint image and the reference image, thereby improving the correction quality.

[0129] In some embodiments, an initial image correction model is obtained before acquiring multiple sets of first training samples. Optionally, the image correction model is an untrained model. Alternatively, the image correction model is obtained based on a pre-trained model, which is trained on multiple sets of second training samples. The model parameters of the convolutional module corresponding to the second viewpoint features, the model parameters of the attention module, and the model parameters of the denoising module in the image correction model are the model parameters of the convolutional module, attention module, and denoising module in the pre-trained model, respectively. The second training samples include sample images and sample images with added noise.

[0130] The pre-trained model is a generalized image generation model that also generates images based on denoising principles. For example, a pre-trained model might be a stable diffusion model. When generating images using the pre-trained model, random noise is processed based on text prompts to predict a noise feature. This predicted noise feature is then removed from the random noise to obtain the predicted image. Correspondingly, during training, noise is added to sample images. Based on these sample images and the noisy sample images, the pre-trained model is trained to predict noise features, thus enabling it to generate images by removing noise from random noise.

[0131] In this embodiment, an initial image correction model is obtained based on a pre-trained model. Training the model on the basis of the pre-trained model reduces the number of training samples, decreases the number of iterations, and improves training efficiency. Furthermore, since the training samples of the pre-trained model consist of sample images and noisy sample images (meaning the pre-trained model generates images based on denoising principles), and the training samples are not limited to the viewpoints of 3D models, using the model parameters of the pre-trained model also ensures the generalization of the image correction model. This means that the image correction module achieves controllable image correction functionality while retaining the generalization ability across large datasets.

[0132] It's important to note that the convolutional modules of a pre-trained model are generally used to process features from a single viewpoint, not to handle higher-dimensional stitched features. Therefore, the number of channels in the pre-trained model's convolutional modules does not correspond to the dimension of the stitched features; it only corresponds to the dimension of a single viewpoint feature. Consequently, the model parameters in the pre-trained model's convolutional modules can only provide parameters corresponding to the second viewpoint feature. For the model parameters corresponding to the first viewpoint feature, the number of channels needs to be increased in the image restoration model's convolutional modules, and the model parameters for each newly added channel are trained from scratch, meaning the initial value of the model parameters for each newly added channel is zero. For example, if the pre-trained model's convolutional modules have 4 channels, while the image restoration model's convolutional modules have 8 channels, then the image restoration model's convolutional modules need to add 4 more channels to the 4 channels in the pre-trained model's convolutional modules.

[0133] In some embodiments, the pre-trained model is a diffusion model. The principle of the pre-trained model is as follows: Figure 5 As shown, random noise X3 is denoised multiple times to obtain the original image X0. Each denoising step involves passing the image through a UNet network, which includes convolutional modules, attention modules, and denoising modules. During training, the original image X0 can be input and different levels of noise can be added to it to obtain noisy images. Figure Xt, as shown in Formula 1 below. The added noise is predicted using a pre-trained model, as shown in Formula 2 below. The difference information (error) between the predicted noise and the actual added noise is calculated, and then the pre-trained model is trained based on this difference information to achieve correct denoising capabilities. The pre-trained model can predict various types of noise during inference, and after multiple denoising iterations, the desired original image X0 is obtained, thus realizing the image generation capability of the pre-trained model.

[0134] Formula 1:

[0135]

[0136] Where x0 represents the original image, x t This represents the original image after adding noise, q(x) t |x0) represents the data x at time step t given the data x0. t The conditional probability distribution, ∫q(x 1:t |x0)dx 1:(t-1) This represents all intermediate states x1 to x2. t-1 The points, Let α represent the overall normal distribution (Gaussian distribution). t It is a time-dependent parameter, usually a noise ratio, used to control the amount of noise added to the image at each step. This represents the scaled portion of the original image, (1-α) t I represents the covariance matrix, and I is the identity matrix.

[0137] Formula 2:

[0138]

[0139] Among them, L γ (∈ θ ) represents the loss function. γ represents the sum from time step 1 to T, where T is the total number of diffusion steps. t This represents the weight at a time step t, used to assign different weights to different time steps in the loss function. Let q(x0) represent the expectation of the initial data x0, which is sampled from the data distribution q(x0). Indicates noise ∈ t The expectation, ∈ t From the standard normal distribution Mid-sampling, This represents the noise predicted at time step t. This represents the scaled portion of the original image. This represents the noise added at time step t, which, after scaling, is added to the scaled portion of the original image. It represents the square of the Euclidean norm (L2 norm), used to calculate the difference between the predicted noise and the actual noise.

[0140] In this embodiment, steps 405-406 above are used to predict based on first viewpoint features, third viewpoint features, and reference features to obtain second noise features. In this embodiment, by splicing the first viewpoint features and the second viewpoint features, the prior of the viewpoint image with texture defects is added to the viewpoint features. This makes the noise features predicted by the image correction model after the noise features are removed so that the viewpoint image is close to the first viewpoint image but the texture defects are corrected. This makes the prediction results more accurate and improves the correction quality.

[0141] 407. Based on the difference information between the first noise features and the second noise features of each of the multiple sets of first training samples, the image correction model is iteratively trained. The image correction model is used to predict the noise in the view map and correct the view map by removing the noise.

[0142] In this embodiment, the difference information between the first noise feature and the second noise feature can be a first loss function value, which indicates the difference between the first noise feature and the second noise feature. Optionally, the first loss function value can be a mean squared error loss function value or a cross-entropy loss function value, etc., and is not specifically limited here.

[0143] In the embodiments of this application, each iteration process uses one or at least two sets of first training samples. If each iteration process uses at least two sets of first training samples, the mean of the first loss function values ​​of the at least two sets of first training samples is determined, and the model parameters of the image correction model are adjusted based on the mean. It should be noted that since multiple sets of first training samples can be obtained through a three-dimensional model, optionally, multiple sets of first training samples corresponding to the same three-dimensional model can be used in each iteration process.

[0144] In this embodiment, the model parameters of the image correction model are iteratively adjusted based on the first loss function value until an iteration stopping condition is met. The iteration stopping condition includes at least one of the following: the number of iterations reaches the first iteration count; the first loss function value is less than or equal to a first threshold. The iteration stopping condition is the criterion for determining when to stop updating the model parameters based on the first loss function value; the first iteration count is the maximum number of iterations. The first threshold represents the minimum acceptable error of the first loss function value.

[0145] It should be noted that the image correction model includes a first encoding module, a second encoding module, a noise-adding module, a decoding module, and a Unet network. The decoding module is used to decode viewpoint features into a viewpoint map. The Unet network includes a convolutional module, an attention module, and a denoising module. Optionally, the first encoding module, the second encoding module, the noise-adding module, and the decoding module are pre-trained modules. If these modules are also included in the pre-trained modules, then when adjusting the model parameters based on the first loss function value, only the model parameters of the modules included in the Unet network need to be adjusted, without adjusting the model parameters outside the Unet network, thereby improving the model training efficiency.

[0146] For example, see Figure 6 , Figure 6 This is a flowchart illustrating the training process of an image correction model according to an embodiment of this application. A first-view image, a second-view image, and a reference image are input into the image correction model. The first-view image is processed to obtain first-view features. The second-view image is processed to obtain second-view features, and a first noise feature is added to the second-view features to obtain third-view features. The first-view features and the third-view features are concatenated to obtain concatenated features. The reference image is processed to obtain reference features. The concatenated features and the reference features are input into a Unet network, and prediction is performed through the Unet network to obtain second noise features.

[0147] This application provides a training method for an image correction model. During training, features of each image are first extracted using the image correction model. Then, noise features are added to the viewpoint features corresponding to the second viewpoint image after texture defect correction. The added noise features are then predicted by combining the viewpoint features corresponding to the first viewpoint image with texture defects and reference features. The image correction model is trained based on the difference between the predicted noise features and the actual added noise features, enabling the image correction model to accurately predict noise features. After training the image correction model in this way, for any viewpoint image with texture defects, the image correction model can predict the noise corresponding to the texture defects in that viewpoint image and correct the viewpoint image by removing the noise, without requiring manual correction, thus improving correction efficiency. Furthermore, when predicting noise features, first-view features and reference features are combined. Since the first-view features provide prior information about the first-view image, the predicted noise features are noise features that, after removal, can make the view image close to the first-view image but correct texture defects. Since the reference features provide appearance control information for the view image, the predicted noise features are noise features that, after removal, can make the view image match the overall appearance of the 3D model. In other words, this method makes the noise features predicted by the image correction model accurate, which can improve the image correction model's ability to correct texture defects and thus improve the correction quality.

[0148] See Figure 7 , Figure 7 This is a flowchart of an image correction method provided in one embodiment of this application. The image correction model used in this method can be an image correction model trained in any of the above embodiments. The execution subject of each step of this method can be a terminal or a server. Taking a server as an example, this method may include at least one of the following steps 701 to 702.

[0149] 701. Obtain the noise map, the view map of the 3D model, and the reference map. The texture of the 3D model in the view map has defects, and the reference map is a view map that provides the appearance features of the 3D model.

[0150] In this embodiment, the noise map can be a random noise map, such as a Gaussian noise map. The viewpoint map is similar to the first viewpoint map in step 301, and the reference map is similar to the reference map in step 301, so it will not be described again here.

[0151] 702. Input the noise map, viewpoint map, and reference map into the image correction model. Process the noise map, viewpoint map, and reference map respectively through the image correction model to obtain noise features, viewpoint features, and reference features. Make predictions based on the noise features, viewpoint features, and reference features to obtain predicted noise features that represent noise features other than viewpoint features in the noise features. Remove the predicted noise features from the noise features to obtain the corrected viewpoint features. Based on the corrected viewpoint features, obtain the corrected viewpoint map. The noise features, viewpoint features, and reference features are used to describe the noise map, viewpoint map, and reference map, respectively.

[0152] In this embodiment, the noise features and the viewpoint features have the same dimension, so that after removing the predicted noise features from the noise features, the corrected viewpoint features corresponding to the dimension of the viewpoint features are obtained. The corrected viewpoint features are also the representation of the corrected viewpoint map in the latent space.

[0153] This application provides an image correction method. When using an image correction model to correct a viewpoint image, the method first extracts features from each image using the image correction model. Then, it combines the viewpoint features corresponding to the viewpoint image with texture defects and reference features to predict noise features. After removing the predicted noise features, corrected viewpoint features are obtained. Based on these corrected viewpoint features, the corrected viewpoint image can be obtained without manual correction, thus improving correction efficiency. Furthermore, when predicting noise features, both viewpoint features and reference features are combined. Since viewpoint features provide prior information about the viewpoint image, the predicted noise features, after removal, can make the viewpoint image closer to the original viewpoint image while correcting texture defects. Since reference features provide appearance control information for the viewpoint image, the predicted noise features, after removal, can make the viewpoint image match the overall appearance of the 3D model. In other words, by using viewpoint features and reference features, the predicted noise features are accurate, improving the image correction model's ability to correct texture defects and thus improving correction quality.

[0154] See Figure 8 , Figure 8 This is a flowchart of an image correction method provided in one embodiment of this application. The image correction model used in this method can be an image correction model trained in any of the above embodiments. The execution subject of each step of this method can be a terminal or a server. Taking a server as an example, this method may include at least one of the following steps 801-806.

[0155] 801. Obtain the noise map, the view map of the 3D model, and the reference map. The texture of the 3D model in the view map has defects, and the reference map is a view map that provides the appearance features of the 3D model.

[0156] In the embodiments of this application, step 801 is the same as step 701, and will not be described again here.

[0157] 802. Input the noise map, view map and reference map into the image correction model. The first encoding module of the image correction model processes the noise map and view map respectively to obtain noise features and view features. The noise features are the representation of the noise map in the latent space, and the view features are the representation of the view map in the latent space.

[0158] In this embodiment of the application, the process of obtaining noise features and viewpoint features in step 802 is the same as the process of obtaining first viewpoint features and second viewpoint features in step 402, and will not be described again here.

[0159] 803. The reference image is processed by the second encoding module of the image correction model to obtain reference features, which are the representation of the reference image in the embedding space.

[0160] In this embodiment of the application, the process of obtaining the reference feature in step 803 is the same as the process of obtaining the reference feature in step 403, and will not be described again here.

[0161] 804. By combining noise features and viewpoint features through the image correction model, the splicing features are obtained.

[0162] In this embodiment of the application, the process of obtaining the splicing features in step 804 is the same as the process of obtaining the splicing features in step 405, and will not be described again here.

[0163] 805. Using an image correction model, predictions are made based on splicing features and reference features to obtain predicted noise features.

[0164] In some embodiments, the image correction model includes a convolution module, a cross-attention module, and a denoising module. Accordingly, the process described above, which uses the image correction model to predict noise features based on stitched features and reference features, includes the following steps: The stitched features are convolved using the convolution module of the image correction model to obtain processed stitched features; the convolution process is used for feature extraction from the stitched features and to limit the dimensionality of the stitched features to a preset range; The processed stitched features and reference features are processed using the attention module of the image correction model to obtain cross-attention features; The query parameter in the attention module is the processed stitched features, and the key and value parameters in the attention module are both reference features; The processed stitched features and cross-attention features are fused using the denoising module of the image correction model, and prediction is performed based on the fused features to obtain the predicted noise features.

[0165] In some embodiments, to improve the quality of the corrected viewpoint map, the process of predicting noise features other than the viewpoint features based on noise features, viewpoint features, and reference features to obtain predicted noise features, and removing the predicted noise features from the noise features to obtain corrected viewpoint features, further includes the following implementation: performing multiple first iteration calculations based on noise features, viewpoint features, and reference features until the first iteration stopping condition is met to obtain corrected viewpoint features; the multiple first iteration calculations include the following steps: in the first first iteration calculation, prediction is performed based on noise features, viewpoint features, and reference features. The first predicted noise feature is obtained, which represents the noise features other than the viewpoint feature. The first predicted noise feature is removed from the noise features to obtain the first corrected viewpoint feature. In the first iteration of the Mth calculation, prediction is performed based on the (M-1)th corrected viewpoint feature, the viewpoint feature, and the reference feature to obtain the Mth predicted noise feature. The Mth predicted noise feature is used to represent the noise features other than the viewpoint feature in the (M-1)th corrected viewpoint feature. The Mth predicted noise feature is removed from the (M-1)th corrected viewpoint feature to obtain the Mth corrected viewpoint feature, where M is an integer greater than 1.

[0166] The first iteration stopping condition can be that the number of iterations reaches the second iteration number. The second iteration number is the maximum number of iterations, which can be set as needed, such as any number between 20 and 50.

[0167] In this embodiment, the viewpoint image is denoised through multiple iterative cycles to improve the denoising effect, making the removal of noise in the viewpoint image more thorough, thus making the texture defects in the viewpoint image more thoroughly corrected, thereby improving the correction effect of the image correction model on the viewpoint image and improving the correction quality.

[0168] 806. The corrected viewpoint features are processed by the decoding module of the image correction model to obtain the corrected viewpoint image.

[0169] The corrected viewpoint feature is the latent space representation of the corrected viewpoint map. The decoding module decodes the corrected viewpoint feature to obtain the corrected viewpoint map. The dimensions of the corrected viewpoint map are the same as those of the original viewpoint map, such as 3*512*512.

[0170] The decoding module is a decoder, which can be a decoder in a variational autoencoder. Accordingly, the model parameters in the decoding module are pre-trained before training the image correction model, so the corrected viewpoint image can be obtained directly using the decoding module without further training of the decoding module, thereby improving the training efficiency of the image correction model.

[0171] For example, see Figure 9 , Figure 9 This is a flowchart of an image correction method provided in one embodiment of this application. The method involves inputting a noise map, a viewpoint map, and a reference map into an image correction model. The noise map is processed to obtain noise features, the viewpoint map is processed to obtain viewpoint features, and the reference map is processed to obtain reference features. Based on the noise features, viewpoint features, and reference features, a prediction is made to obtain predicted noise features. The predicted noise features are then removed from the noise features to obtain corrected noise features. Finally, the corrected viewpoint map is obtained based on the corrected noise features.

[0172] In some embodiments, after correcting the viewpoint map of the 3D model using an image correction model, the texture map of the 3D model can be further corrected using the corrected viewpoint map. This process includes the following implementation: using an image correction model, multiple viewpoint maps of the 3D model are corrected to obtain multiple corrected viewpoint maps. The multiple viewpoint maps are rendered based on the texture map of the 3D model, and the texture map includes textures at multiple locations on the 3D model; the texture map is corrected based on the multiple viewpoint maps and the multiple corrected viewpoint maps.

[0173] Among them, multiple view maps cover the overall appearance of the 3D model, that is, the textures on multiple view maps cover all the textures on the texture map, and thus the texture map can be fully corrected based on multiple view maps.

[0174] In this embodiment, since the view map is obtained by rendering the 3D model based on the texture map of the 3D model, the texture map can be corrected based on the difference information between the corrected view map and the uncorrected view map. The view map subsequently obtained based on the corrected texture map will then be close to the corrected view map. Thus, the view map obtained directly from the texture map no longer includes texture defects, eliminating the need for further correction and improving the efficiency of obtaining subsequent view maps.

[0175] In some embodiments, the process of correcting a texture map based on multiple viewpoint maps and multiple corrected viewpoint maps to obtain a corrected texture map includes the following implementation: performing multiple second-iteration calculations based on multiple viewpoint maps and multiple corrected viewpoint maps until the second-iteration stopping condition is met; the multiple second-iteration calculations include the following steps: in the first second-iteration calculation, determining the difference information between each viewpoint map and its corresponding corrected viewpoint map, and correcting the texture map based on the difference information between the multiple viewpoint maps and multiple corrected viewpoint maps; in the Nth second-iteration calculation, rendering the 3D model based on the corrected texture map in the N-1th second-iteration calculation and the multiple viewpoint maps corresponding to the multiple viewpoint maps, obtaining multiple viewpoint maps in the Nth time, determining the difference information between each viewpoint map in the Nth time and its corresponding corrected viewpoint map, and correcting the texture map in the N-1th second-iteration calculation based on the difference information between the multiple viewpoint maps in the Nth time and multiple corrected viewpoint maps, where N is an integer greater than 1.

[0176] Optionally, the 3D model is rendered differently based on the texture map, and correspondingly, backpropagation is performed based on the difference information between multiple view maps and multiple corrected view maps to correct the texture map.

[0177] In this embodiment, the texture map is corrected with the goal of minimizing the difference between the view map and the corrected view map. Optionally, the difference between each view map and its corresponding corrected view map is a second loss function value between them. This second loss function value can be a mean squared error loss function value or a cross-entropy loss function value, etc., and is not specifically limited here.

[0178] Optionally, the process of correcting the texture map based on the difference information between multiple viewpoint maps and multiple corrected viewpoint maps includes the following steps: correcting the texture map based on the sum of the second loss function values ​​between the multiple viewpoint maps and multiple corrected viewpoint maps. The process of determining the sum of the multiple second loss function values ​​is described in Formula 3 below.

[0179]

[0180] Where n represents the number of viewpoint images. renderi Let I represent the i-th viewpoint. gti Let X represent the corrected view map corresponding to the i-th view map, and let T represent the texture map. i This represents the i-th viewpoint. In this application, the viewpoint refers to the camera viewpoint, that is, the viewpoint map is obtained by rendering based on the i-th camera viewpoint and the texture map. Each pixel in the viewpoint map is sampled from the texture map according to the camera viewpoint. The viewpoint maps of different camera views are different, but they all correspond to the same texture map.

[0181] The second iteration stopping condition can be that the second loss function value converges, such as when the second loss function value is less than or equal to the second threshold. The second threshold represents the minimum acceptable error for the second loss function value.

[0182] In this embodiment, the original texture map of the 3D model can be corrected based on the corrected viewpoint map; alternatively, a texture map can be randomly initialized and corrected based on the corrected viewpoint map to obtain the corrected texture map. For example, see... Figure 10 , Figure 10 This is a flowchart of texture map correction provided in one embodiment of this application. A texture map is randomly initialized, and the texture map is iteratively corrected based on multiple viewpoint maps corrected by an image correction model until convergence is achieved, resulting in a texture map corrected by the 3D model.

[0183] For example, see Figure 11 , Figure 11 This is a comparison of the effects before and after texture map correction according to an embodiment of this application. When the initial texture map is entirely gray, the multiple corrected viewpoint maps and multiple viewpoint maps are shown in the first two rows of images. Since the rendering process involves sampling from the texture map, and the texture map is entirely gray at this point, the viewpoint maps are also entirely gray. After correcting the texture map, the multiple corrected viewpoint maps and multiple viewpoint maps are shown in the last two rows of images. Because the texture map has been corrected, the viewpoint maps obtained based on the texture map are almost identical to the corrected viewpoint maps. When the second loss function value reaches its minimum, i.e., convergence is achieved, the correction of the texture map is complete, and the texture map is output.

[0184] The method provided in this application can replace super-resolution without changing the image resolution, and can handle texture defects that super-resolution cannot handle, such as texture errors and lack of texture details. Furthermore, by using an input reference image as a control signal, the viewpoint image of the 3D model is optimized to obtain a detailed, aesthetically pleasing, and textured 3D model highly correlated with the input reference image. The entire process is short and efficient, requiring only 10 seconds. Moreover, the method provided in this application can be applied to a post-processing module to correct 3D models with blurred textures generated by other algorithms, improving the quality and usability of the generated 3D model and reducing production costs. It can also be decoupled from other algorithms, allowing for general and flexible application after various generation algorithms.

[0185] For example, see Figure 12 , Figure 12This is a flowchart of an image correction method provided in one embodiment of this application. The method involves inputting a reference image of a 3D model and a blurred viewpoint image into an image correction model. After multi-view rendering, batch viewpoint images are obtained. The texture image is then corrected based on the corrected viewpoint images, resulting in a 3D model corresponding to the corrected texture image—that is, a texture-optimized 3D model. The entire process is relatively short, and the facial details and texture clarity of the corrected 3D model are improved.

[0186] For example, see Figure 13 , Figure 13 This application provides a flowchart of texture map correction according to an embodiment. First, the 3D model is rendered from its texture map using multi-view rendering to obtain multiple view maps. These multiple view maps may include six full-body view maps and one face view map. Then, they are compared with a reference... Figure 1 The initial batch of input images is used to correct the model, resulting in seven corrected viewpoint images. These images are then combined with the texture coordinates of the texture map of the 3D model. Based on these seven corrected viewpoint images, the texture map is corrected to obtain the corrected texture map.

[0187] This application provides an image correction method. When using an image correction model to correct a viewpoint image, the method first extracts features from each image using the image correction model. Then, it combines the viewpoint features corresponding to the viewpoint image with texture defects and reference features to predict noise features. After removing the noise features, corrected viewpoint features are obtained. Based on these corrected viewpoint features, the corrected viewpoint image can be obtained without manual correction, thus improving correction efficiency. Furthermore, when predicting noise features, the method combines viewpoint features and reference features. Since viewpoint features provide prior information about the first viewpoint image, the predicted noise features, after removal, make the viewpoint image closer to the first viewpoint image while removing texture defects. Since reference features provide appearance control information for the viewpoint image, the predicted noise features, after removal, make the viewpoint image match the overall appearance of the 3D model. In other words, by using the first viewpoint features and reference features, the predicted noise features are accurate, improving the image correction model's ability to correct texture defects and thus improving correction quality.

[0188] The image correction method provided in this application embodiment can be applied to various scenarios. The following uses the game production scenario as an example for explanation. It is applied to the production process of three-dimensional models. The process includes the following steps (1)-(7).

[0189] (1) The game production terminal obtains the original texture map of the 3D model.

[0190] The game client includes virtual scenes and virtual objects that can be placed within these virtual scenes. The virtual scene can be a simulation of the real world, a semi-simulated / semi-fictional environment, or a purely fictional environment. The virtual scene can be any of a two-dimensional, 2.5-dimensional, or three-dimensional virtual scene; this application embodiment does not limit the dimension of the virtual scene. For example, the virtual scene may include the sky, land, and ocean, and the land may include environmental elements such as deserts and cities. Users can control virtual characters to move within the virtual scene. Optionally, the virtual scene can provide a battle environment for virtual objects, containing virtual resources available for use by the virtual objects. These resources may include virtual items needed for battle, virtual medicines needed for healing, virtual items needed for upgrades, and virtual coins needed for trading.

[0191] In addition, virtual objects include virtual characters, virtual buildings, virtual props, etc. Virtual characters can be virtual figures, virtual animals, virtual sprites, anime characters, etc. A virtual character can be a virtual avatar representing the user within the virtual scene. A virtual scene can include multiple virtual characters, each with its own shape and volume, occupying a portion of the space within the virtual scene. The 3D model of a virtual character can be a 3D character constructed based on 3D human skeleton technology; the same virtual character can display different appearances by wearing different skins. In some embodiments, virtual characters can also be implemented using 2.5D or 2D models; this application does not limit this.

[0192] A variety of different virtual characters can exist in a virtual scene. For example, virtual characters can include player characters controlled through client operations, and also non-player characters (NPCs) set up for interaction within the virtual scene. Optionally, the virtual character can be a virtual figure competing within the virtual scene. Optionally, the number of virtual characters participating in the interaction in the virtual scene can be preset or dynamically determined based on the number of clients joining the interaction.

[0193] Therefore, in the process of creating a game client, it is necessary to create a 3D model of the virtual object. This 3D model creation involves rendering the model based on texture maps to obtain the rendered 3D model.

[0194] First, the game development terminal acquires the original texture map of the 3D model, which contains texture defects. This texture map can be uploaded to the game development terminal by the game developers, or downloaded by the game development terminal from the game server.

[0195] (2) The game production terminal sends a texture map correction request to the game server. The texture map correction request carries the original texture map of the 3D model.

[0196] (3) The game server renders the 3D model from multiple perspectives based on the original texture map to obtain multiple perspective maps of the 3D model. The texture defects in the multiple perspective maps are corrected by image correction model to obtain corrected multiple perspective maps. The texture map of the 3D model is corrected based on the corrected multiple perspective maps to obtain the corrected texture map.

[0197] (4) The game server sends the corrected texture map to the game production terminal.

[0198] (5) The game production terminal receives the corrected texture map, renders the three-dimensional model based on the corrected texture map, obtains the rendered three-dimensional model, and displays the rendered three-dimensional model.

[0199] (6) In response to the confirmation operation of the rendered 3D model, the game production terminal uploads the rendered 3D model to the game server, or uploads the corrected texture map.

[0200] The game development terminal displays the rendered 3D model, allowing game developers to preview the rendering effect and determine if it meets requirements. If satisfied, they can trigger a confirmation step, indicating that the 3D model rendering is complete.

[0201] In another embodiment, if the game developers are not satisfied with the rendering effect of the 3D model, they can also trigger a rejection operation, thereby re-correcting the texture map of the 3D model and re-rendering it.

[0202] (7) The game server adds the rendered 3D model or the corrected texture map as the game resource of the 3D model to the game resource package of the game client.

[0203] The game server creates a game resource package for the game client, which includes various game resources required for the client's installation and operation. The game server then adds the rendered 3D model or the corrected texture to this resource package. After the game server releases the game client, terminals that download it can install and run the client based on this resource package, thus displaying the rendered 3D model of the virtual object within the game client.

[0204] Of course, the method provided in this application embodiment can also be applied to other scenarios of modifying texture maps, and this application embodiment does not limit this.

[0205] The following are embodiments of the apparatus described in this application, which can be used to execute the embodiments of the method described in this application. For details not disclosed in the apparatus embodiments of this application, please refer to the embodiments of the method described in this application.

[0206] Please refer to Figure 14 , Figure 14 This is a block diagram of a training apparatus for an image retouching model according to an embodiment of this application. The apparatus has the function of implementing the training method for the image retouching model described above. This function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be installed within a computer device. For example... Figure 14 As shown, the device may include:

[0207] The acquisition module 1401 is used to acquire multiple sets of first training samples. Each set of first training samples includes a first-view image, a second-view image, and a reference image of the 3D model. The texture of the 3D model in the first-view image has defects. The second-view image is the first-view image after the defects are corrected. The reference image is a view showing the appearance of the 3D model.

[0208] The input / output module 1402 is used to input a first-view image, a second-view image, and a reference image into an image correction model. The image correction model processes the first-view image, the second-view image, and the reference image respectively to obtain first-view features, second-view features, and reference features. A first noise feature is added to the second-view features to obtain third-view features. Based on the first-view features, third-view features, and reference features, a prediction is made to obtain a second noise feature used to represent the noise features in the third-view features. The first-view features, second-view features, and reference features are used to describe the first-view image, the second-view image, and the reference image, respectively.

[0209] The training module 1403 is used to iteratively train an image correction model based on the difference information between the first noise features and the second noise features of each of the multiple sets of first training samples. The image correction model is used to predict noise in the viewpoint map and correct the viewpoint map by removing noise.

[0210] In some embodiments, the input / output module 1402 is used for:

[0211] By combining the first-person perspective features and the third-person perspective features, the combined features are obtained.

[0212] The second noise feature is obtained by prediction based on the splicing features and the reference features.

[0213] In some embodiments, the input / output module 1402 is used for:

[0214] The convolution module of the image correction model performs convolution processing on the spliced ​​features to obtain the processed spliced ​​features. The convolution processing is used to extract features from the spliced ​​features and limit the dimensions of the spliced ​​features to a preset range.

[0215] The attention module of the image correction model processes the concatenated features and reference features to obtain cross-attention features. The query parameters in the attention module are the concatenated features, and the key and value parameters in the attention module are the reference features.

[0216] The image correction model's denoising module fuses the processed spliced ​​features and cross-attention features, and predictions are made based on the fused features to obtain the second noise feature.

[0217] In some embodiments, the acquisition module 1401 is further configured to:

[0218] Based on the pre-trained model, an image correction model is obtained. The pre-trained model is trained on multiple sets of second training samples. The model parameters of the convolution module corresponding to the second viewpoint features, the model parameters of the attention module, and the model parameters of the denoising module in the image correction model are the model parameters of the convolution module, attention module, and denoising module in the pre-trained model, respectively. The second training samples include sample images and sample images with added noise.

[0219] In some embodiments, the input / output module 1402 is used for:

[0220] The first encoding module of the image correction model processes the first view image and the second view image respectively to obtain the first view feature and the second view feature. The first view feature is the representation of the first view image in the latent space, and the second view feature is the representation of the second view image in the latent space.

[0221] The reference image is processed by the second encoding module of the image correction model to obtain reference features, which are the representation of the reference image in the embedding space.

[0222] This application provides a training apparatus for an image correction model. During training, the image correction model first extracts features from each image. Then, noise features are added to the viewpoint features corresponding to the second viewpoint image after texture defect correction. Next, the added noise features are predicted by combining the viewpoint features corresponding to the first viewpoint image with texture defects and reference features. The image correction model is trained based on the difference between the predicted noise features and the actual added noise features, enabling the image correction model to accurately predict noise features. After training the image correction model in this way, for any viewpoint image with texture defects, the image correction model can predict the noise corresponding to the texture defects in that viewpoint image and correct the viewpoint image by removing the noise, without requiring manual correction, thereby improving correction efficiency. Furthermore, when predicting noise features, the device combines first-viewpoint features and reference features. Since the first-viewpoint features provide prior information about the first-viewpoint image, the predicted noise features are noise features that, after removal, can make the viewpoint image close to the first-viewpoint image while correcting texture defects. Since the reference features provide appearance control information for the viewpoint image, the predicted noise features are noise features that, after removal, can make the viewpoint image match the overall appearance of the 3D model. In other words, this device makes the noise features predicted by the image correction model accurate, which can improve the image correction model's ability to correct texture defects and thus improve the correction quality.

[0223] Please refer to Figure 15 , Figure 15 This is a block diagram of an image correction apparatus according to an embodiment of this application. The apparatus has the function of implementing the above-described image correction method; the function can be implemented in hardware or by hardware executing corresponding software. The apparatus can be the computer device described above, or it can be installed within a computer device. For example... Figure 15 As shown, the device may include:

[0224] The acquisition module 1501 is used to acquire a noise map, a view map of the 3D model, and a reference map. The texture of the 3D model in the view map has defects, and the reference map is a view map that provides the appearance features of the 3D model.

[0225] The input / output module 1502 is used to input the noise map, view map, and reference map into the image correction model, process the noise map, view map, and reference map respectively through the image correction model to obtain noise features, view features, and reference features, perform prediction based on the noise features, view features, and reference features to obtain predicted noise features that represent noise features other than view features in the noise features, remove the predicted noise features from the noise features to obtain corrected view features, and obtain the corrected view map based on the corrected view features.

[0226] Among them, noise features, viewpoint features, and reference features are used to describe the noise map, viewpoint map, and reference map, respectively.

[0227] In some embodiments, the input / output module 1502 is used for:

[0228] By splicing noise features and viewpoint features, splicing features are obtained;

[0229] Prediction noise features are obtained by using splicing features and reference features.

[0230] In some embodiments, the input / output module 1502 is used for:

[0231] The convolution module of the image correction model performs convolution processing on the spliced ​​features to obtain the processed spliced ​​features. The convolution processing is used to extract features from the spliced ​​features and limit the dimensions of the spliced ​​features to a preset range.

[0232] The attention module of the image correction model processes the concatenated features and reference features to obtain cross-attention features. The query parameters in the attention module are the concatenated features, and the key and value parameters in the attention module are the reference features.

[0233] The image correction model's denoising module fuses the processed spliced ​​features and cross-attention features, and predictions are made based on the fused features to obtain the predicted noise features.

[0234] In some embodiments, the input / output module 1502 is used for:

[0235] The first encoding module of the image correction model processes the noise map and the view map respectively to obtain noise features and view features. The noise features are the representation of the noise map in the latent space, and the view features are the representation of the view map in the latent space.

[0236] The reference image is processed by the second encoding module of the image correction model to obtain reference features, which are the representation of the reference image in the embedding space.

[0237] In some embodiments, the input / output module 1502 is used for:

[0238] The image correction model's decoding module processes the corrected viewpoint features to obtain the corrected viewpoint image.

[0239] In some embodiments, the input / output module 1502 is used for:

[0240] Based on noise features, viewpoint features, and reference features, multiple first-iteration calculations are performed until the first-iteration stopping condition is met, resulting in corrected viewpoint features. These multiple first-iteration calculations include:

[0241] In the first iteration of the calculation, prediction is made based on noise features, view features and reference features to obtain the first predicted noise feature. The first predicted noise feature is used to represent the noise features other than the view features. The first predicted noise feature is removed from the noise features to obtain the first corrected view feature.

[0242] In the first iteration of the Mth calculation, prediction is made based on the (M-1)th corrected viewpoint feature, the viewpoint feature, and the reference feature to obtain the Mth predicted noise feature. The Mth predicted noise feature is used to represent the noise features other than the viewpoint feature in the (M-1)th corrected viewpoint feature. The Mth predicted noise feature is removed from the (M-1)th corrected viewpoint feature to obtain the Mth corrected viewpoint feature, where M is an integer greater than 1.

[0243] In some embodiments, the apparatus further includes:

[0244] The first correction module is used to correct multiple viewpoints of the 3D model based on the image correction model, so as to obtain multiple corrected viewpoints. The multiple viewpoints are rendered based on the texture map of the 3D model, and the texture map includes textures at multiple locations on the 3D model.

[0245] The second correction module is used to correct the texture map based on multiple view maps and multiple corrected view maps.

[0246] In some embodiments, the second correction module is configured to:

[0247] Based on multiple viewpoints and multiple corrected viewpoints, multiple second-iteration calculations are performed until the second-iteration stopping condition is met; these multiple second-iteration calculations include:

[0248] In the first iteration of the second calculation, the difference information between each view map and its corresponding corrected view map is determined. Based on the difference information between multiple view maps and multiple corrected view maps, the texture map is corrected.

[0249] In the Nth second iteration calculation, based on the corrected texture map and multiple view maps corresponding to multiple view maps in the (N-1)th second iteration calculation, the 3D model is rendered to obtain multiple view maps in the Nth iteration. The difference information between each view map in the Nth iteration and the corresponding corrected view map is determined. Based on the difference information between the multiple view maps in the Nth iteration and the multiple corrected view maps, the corrected texture map in the (N-1)th second iteration calculation is corrected, where N is an integer greater than 1.

[0250] This application provides an image correction device. When using an image correction model to correct a viewpoint image, the device first extracts features from each image using the image correction model. Then, it combines the viewpoint features corresponding to the viewpoint image with texture defects and reference features to predict noise features. After removing the predicted noise features, the corrected viewpoint features are obtained. Based on the corrected viewpoint features, the corrected viewpoint image can be obtained without manual correction, thus improving correction efficiency. Furthermore, when predicting noise features, the viewpoint features and reference features are combined. Since the viewpoint features provide prior information about the viewpoint image, the predicted noise features, after removal, can make the viewpoint image closer to the original viewpoint image while removing texture defects. Since the reference features provide appearance control information for the viewpoint image, the predicted noise features, after removal, can make the viewpoint image match the overall appearance of the 3D model. In other words, by using viewpoint features and reference features, the predicted noise features are accurate, which can improve the ability of the image correction model to correct texture defects, thereby improving the correction quality.

[0251] Please refer to Figure 16 This diagram illustrates a structural block diagram of a computer device 1600 provided in one embodiment of this application. The computer device 1600 may be a terminal device 10 or a server 20 in an implementation environment. The computer device 1600 is used to implement the training method or image correction method of the image correction model provided in the above embodiments. Specifically:

[0252] Typically, computer device 1600 includes a processor 1610 and memory 1620.

[0253] Processor 1610 may include one or more processing cores, such as a quad-core processor or a 16-core processor. Processor 1610 may be implemented using at least one hardware form selected from Digital Signal Processing (DSP), Field Programmable Gate Array (FPGA), and Programmable Logic Array (PLA). Processor 1610 may also include a main processor and a coprocessor. The main processor, also known as the central processing unit, is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1610 may include a GPU, which is responsible for executing the method steps provided in this application.

[0254] The memory 1620 may include one or more computer-readable storage media, which may be non-transitory. The memory 1620 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1620 is used to store a computer program configured to be executed by one or more processors (such as GPUs) to implement the training method or image correction method of the image correction model described above.

[0255] Those skilled in the art will understand that Figure 16 The structure shown does not constitute a limitation on the computer device 1600, and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0256] This application also provides a computer-readable storage medium storing a computer program, which is loaded and executed by a processor to implement a training method or an image correction method for any of the above implementations of the image correction model.

[0257] This application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium. The processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform a training method or an image correction method for an image correction model according to any of the above implementations.

[0258] In some embodiments, the computer program product involved in the present application may be deployed and executed on a computer device, or on multiple computer devices located in one location, or on multiple computer devices distributed in multiple locations and interconnected through a communication network.

[0259] It should be noted that the data collection and processing in this application should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0260] It should be understood that "multiple" as used herein refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. Furthermore, the step numbers described herein are merely illustrative of one possible execution order. In some other embodiments, the steps may not be executed in numerical order, such as two steps with different numbers being executed simultaneously, or two steps with different numbers being executed in the reverse order of the illustration. This application does not limit this.

[0261] All the above-mentioned optional technical solutions can be combined in any way to form the optional embodiments of this application, and will not be described in detail here. The above are only optional embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A training method for an image correction model, characterized in that, The method includes: Multiple sets of first training samples are obtained. Each set of first training samples includes a first view image, a second view image, and a reference image of the three-dimensional model. The texture of the three-dimensional model in the first view image has defects. The second view image is the first view image after the defects are corrected. The reference image is a view image showing the appearance of the three-dimensional model. The first viewpoint image, the second viewpoint image, and the reference image are input into an image correction model. The image correction model processes the first viewpoint image, the second viewpoint image, and the reference image respectively to obtain a first viewpoint feature, a second viewpoint feature, and a reference feature. A first noise feature is added to the second viewpoint feature to obtain a third viewpoint feature. Based on the first viewpoint feature, the third viewpoint feature, and the reference feature, a prediction is made to obtain a second noise feature that represents the noise feature in the third viewpoint feature. The first viewpoint feature, the second viewpoint image, and the reference feature are used to describe the first viewpoint image, the second viewpoint image, and the reference image, respectively. Based on the difference information between the first noise feature and the second noise feature of each of the multiple sets of first training samples, the image correction model is iteratively trained. The image correction model is used to predict the noise in the viewpoint map and correct the viewpoint map by removing the noise.

2. The method according to claim 1, characterized in that, The prediction based on the first viewpoint feature, the third viewpoint feature, and the reference feature to obtain a second noise feature representing the noise feature in the third viewpoint feature includes: By splicing the first viewpoint features and the third viewpoint features, a spliced ​​feature is obtained; The second noise feature is obtained by prediction based on the splicing features and the reference features.

3. The method according to claim 2, characterized in that, The prediction based on the splicing features and the reference features to obtain the second noise features includes: The convolution module of the image correction model performs convolution processing on the stitched features to obtain processed stitched features. The convolution processing is used to extract features from the stitched features and limit the dimensions of the stitched features to a preset range. The attention module of the image correction model processes the processed spliced ​​features and the reference features to obtain cross-attention features. The query parameter in the attention module is the processed spliced ​​features, and the key parameter and value parameter in the attention module are both the reference features. The image correction model's denoising module fuses the processed spliced ​​features and the cross-attention features, and prediction is performed based on the fused features to obtain the second noise feature.

4. The method according to claim 3, characterized in that, Before obtaining multiple sets of first training samples, the method further includes: The image correction model is obtained based on the pre-trained model. The pre-trained model is trained based on multiple sets of second training samples. The model parameters of the convolution module corresponding to the second viewpoint features, the model parameters of the attention module, and the model parameters of the denoising module in the image correction model are the model parameters of the convolution module, attention module, and denoising module in the pre-trained model, respectively. The second training samples include sample images and sample images with added noise.

5. The method according to claim 1, characterized in that, The process of processing the first viewpoint image, the second viewpoint image, and the reference image using the image correction model to obtain first viewpoint features, second viewpoint features, and reference features includes: The first encoding module of the image correction model processes the first view map and the second view map respectively to obtain the first view feature and the second view feature. The first view feature is the representation of the first view map in the latent space, and the second view feature is the representation of the second view map in the latent space. The reference image is processed by the second encoding module of the image correction model to obtain the reference features, which are the representation of the reference image in the embedding space.

6. An image correction method, characterized in that, The method includes: Obtain a noise map, a view of the 3D model, and a reference map. The texture of the 3D model in the view of the model has defects, and the reference map is a view of the model that provides the appearance features of the 3D model. The noise map, the viewpoint map, and the reference map are input into an image correction model. The image correction model processes the noise map, the viewpoint map, and the reference map respectively to obtain noise features, viewpoint features, and reference features. Based on the noise features, the viewpoint features, and the reference features, prediction is performed to obtain predicted noise features that represent noise features other than viewpoint features in the noise features. The predicted noise features are removed from the noise features to obtain corrected viewpoint features. Based on the corrected viewpoint features, the corrected viewpoint map is obtained. The noise feature, the viewpoint feature, and the reference feature are used to describe the noise map, the viewpoint map, and the reference map, respectively.

7. The method according to claim 6, characterized in that, The prediction based on the noise features, the viewpoint features, and the reference features to obtain predicted noise features representing noise features other than the viewpoint features includes: By splicing the noise features and the viewpoint features, a spliced ​​feature is obtained; Based on the splicing features and the reference features, prediction is performed to obtain the predicted noise features.

8. The method according to claim 7, characterized in that, The prediction based on the splicing features and the reference features to obtain the predicted noise features includes: The convolution module of the image correction model performs convolution processing on the stitched features to obtain processed stitched features. The convolution processing is used to extract features from the stitched features and limit the dimensions of the stitched features to a preset range. The attention module of the image correction model processes the processed spliced ​​features and the reference features to obtain cross-attention features. The query parameter in the attention module is the processed spliced ​​features, and the key parameter and value parameter in the attention module are both the reference features. The image correction model's denoising module fuses the processed splicing features and the cross-attention features, and prediction is performed based on the fused features to obtain the predicted noise features.

9. The method according to claim 6, characterized in that, The image correction model is used to process the noise map, the viewpoint map, and the reference map respectively to obtain noise features, viewpoint features, and reference features, including: The noise map and the view map are processed by the first encoding module of the image correction model to obtain the noise feature and the view feature. The noise feature is the representation of the noise map in the latent space, and the view feature is the representation of the view map in the latent space. The reference image is processed by the second encoding module of the image correction model to obtain the reference features, which are the representation of the reference image in the embedding space.

10. The method according to claim 9, characterized in that, The process of obtaining the corrected viewpoint map based on the corrected viewpoint features includes: The image correction model's decoding module processes the corrected viewpoint features to obtain the corrected viewpoint image.

11. The method according to claim 6, characterized in that, The process of predicting based on the noise features, the viewpoint features, and the reference features to obtain predicted noise features representing noise features other than the viewpoint features, and removing the predicted noise features from the noise features to obtain corrected viewpoint features, includes: Based on the noise features, the viewpoint features, and the reference features, multiple first-iteration calculations are performed until the first-iteration stopping condition is met, thereby obtaining the corrected viewpoint features; the multiple first-iteration calculations include: In the first iteration calculation, a prediction is made based on the noise feature, the view feature, and the reference feature to obtain the first predicted noise feature. The first predicted noise feature is used to represent the noise features other than the view feature in the noise features. The first predicted noise feature is removed from the noise features to obtain the first corrected view feature. In the first iteration of the Mth calculation, prediction is made based on the (M-1)th corrected view feature, the view feature, and the reference feature to obtain the Mth predicted noise feature. The Mth predicted noise feature is used to represent the noise features other than the view feature in the (M-1)th corrected view feature. The Mth predicted noise feature is removed from the (M-1)th corrected view feature to obtain the Mth corrected view feature, where M is an integer greater than 1.

12. The method according to claim 6, characterized in that, The method further includes: The image correction model is used to correct multiple viewpoints of the 3D model to obtain multiple corrected viewpoints. The multiple viewpoints are rendered based on the texture map of the 3D model, and the texture map includes textures at multiple locations on the 3D model. The texture map is corrected based on the multiple view maps and the multiple corrected view maps.

13. The method according to claim 12, characterized in that, The step of correcting the texture map based on the plurality of view maps and the plurality of corrected view maps includes: Based on the multiple viewpoints and the multiple corrected viewpoints, multiple second-iteration calculations are performed until the second-iteration stopping condition is met; the multiple second-iteration calculations include: In the first second iteration calculation, the difference information between each view map and its corresponding corrected view map is determined, and the texture map is corrected based on the difference information between the multiple view maps and the multiple corrected view maps; In the Nth second iteration calculation, based on the corrected texture map from the (N-1)th second iteration calculation and the multiple perspectives corresponding to the multiple perspective maps, the 3D model is rendered to obtain the Nth multiple perspective maps. The difference information between each perspective map in the Nth iteration and its corresponding corrected perspective map is determined. Based on the difference information between the multiple perspective maps in the Nth iteration and the multiple corrected perspective maps, the corrected texture map from the (N-1)th second iteration calculation is corrected, where N is an integer greater than 1.

14. A training device for an image correction model, characterized in that, The device includes: The acquisition module is used to acquire multiple sets of first training samples. Each set of first training samples includes a first view image, a second view image, and a reference image of the three-dimensional model. The texture of the three-dimensional model in the first view image has defects. The second view image is the first view image after the defects are corrected. The reference image is a view image showing the appearance of the three-dimensional model. The input / output module is used to input the first viewpoint image, the second viewpoint image, and the reference image into an image correction model, and process the first viewpoint image, the second viewpoint image, and the reference image through the image correction model to obtain a first viewpoint feature, a second viewpoint feature, and a reference feature. A first noise feature is added to the second viewpoint feature to obtain a third viewpoint feature. Based on the first viewpoint feature, the third viewpoint feature, and the reference feature, a prediction is made to obtain a second noise feature that represents the noise feature in the third viewpoint feature. The first viewpoint feature, the second viewpoint image, and the reference feature are used to describe the first viewpoint image, the second viewpoint image, and the reference image, respectively. The training module is used to iteratively train the image correction model based on the difference information between the first noise features and the second noise features of each of the multiple sets of first training samples. The image correction model is used to predict noise in the viewpoint map and correct the viewpoint map by removing noise.

15. An image correction device, characterized in that, The device includes: The acquisition module is used to acquire a noise map, a view map of the 3D model, and a reference map. The texture of the 3D model in the view map has defects, and the reference map is a view map that provides the appearance features of the 3D model. The input / output module is used to input the noise map, the viewpoint map, and the reference map into an image correction model, process the noise map, the viewpoint map, and the reference map through the image correction model to obtain noise features, viewpoint features, and reference features, predict the noise features based on the noise features, the viewpoint features, and the reference features to obtain predicted noise features that represent noise features other than viewpoint features, remove the predicted noise features from the noise features to obtain corrected viewpoint features, and obtain the corrected viewpoint map based on the corrected viewpoint features. The noise feature, the viewpoint feature, and the reference feature are used to describe the noise map, the viewpoint map, and the reference map, respectively.

16. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store a computer program, the computer program being loaded by the processor and executed as the training method of the image correction model according to any one of claims 1 to 5 or the image correction method according to any one of claims 6 to 13.

17. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for executing the training method of the image correction model according to any one of claims 1 to 5 or the image correction method according to any one of claims 6 to 13.

18. A computer program product, characterized in that, The computer program product includes a computer program stored in a computer-readable storage medium, a processor of a computer device reading the computer program from the computer-readable storage medium, and the processor executing the computer program to cause the computer device to perform the training method of the image correction model according to any one of claims 1 to 5 or the image correction method according to any one of claims 6 to 13.

Images