Data synthesis method and training method for monocular three-dimensional reconstruction based on synthesized data

Abstract

The invention relates to the technical field of three-dimensional reconstruction, and provides a data synthesis method and a training method for monocular three-dimensional reconstruction based on synthesized data, and the method comprises the steps: 1, rendering a picture containing a specific object into a plurality of pictures I corresponding to the object at different angles; step 2, obtaining a mask m of the object based on the picture I, and 3, guiding the picture generation model to generate a new picture which conforms to the text description and retains the structural information of the specific object by utilizing the picture I by utilizing the mask m and the additionally input text description. According to the method, a large number of 3D models and picture pairs are generated through different viewpoints, styles and background conditions, a rich training data set is provided, and the generalization ability of the monocular three-dimensional reconstruction model can be remarkably improved through the diversity. By randomly applying disturbance within a certain angle range of a plurality of viewpoints (R, T), the adaptive capacity of the model to different camera poses is improved, and the robustness of monocular three-dimensional reconstruction is enhanced.

Description

Technical Field

[0001] The present invention relates to the technical field of three-dimensional reconstruction, and in particular to a data synthesis method and a training method for performing monocular three-dimensional reconstruction based on synthesized data. Background Art

[0002] The present invention is to solve the problem of single vision Figure 3 It is designed to solve several key problems in dimensional modeling. Figure 3 In the context of single-view 3D reconstruction, traditional methods often face challenges such as insufficient diversity of training data, insufficient robustness to different camera poses and lighting conditions, and poor quality and details of the generated 3D models. How can we use more synthetic data to make single-view 3D models more realistic? Figure 3 How to achieve better dimensional modeling has become an urgent problem to be solved.

[0003] Currently, there are a series of text-to-image AI models that can generate high-quality image content. However, most of these methods can only generate a complete image, and the controllability is poor. There are obstacles to retaining the main body of the object and changing the background, retaining the background and changing the main body, or changing part of the main area. Back to single view Figure 3 In the context of 3D reconstruction, the inventors found in practice that factors such as the lighting / background of the environment where the object is placed will affect the single-view Figure 3 The network often has background bias. Some methods try to reconstruct 3D images by removing the object to be segmented through the cutout method, but this method cannot utilize the context information provided by the background information, and actually loses the scarce input information. How to obtain a data synthesis method that is centered on a specific object and can realistically enhance various conditions of the object in order to cope with the diverse input of users during the reasoning process is a technical problem that needs to be solved urgently. Summary of the invention

[0004] In order to achieve the above object, the present invention provides the following technical solutions:

[0005] The present invention provides a method for data synthesis, comprising:

[0006] The first step is to render a picture containing a specific object into multiple pictures I corresponding to the object at different angles;

[0007] The second step is to obtain a mask m of the object based on the image I, wherein the mask is the result of a semantic segmentation model, such as a SAM model, or a saliency segmentation model, such as a U2Net model, or a result of a tracking algorithm; or a mask manually input by a user;

[0008] In the third step, the mask m and the additional input text description are used to guide the image generation model to use image I to generate a new image that conforms to the text description and retains specific object structure information.

[0009] Furthermore, the method of rendering multiple images I at different angles in the first step is:

[0010] A three-dimensional field or a three-dimensional model obtained by optimizing some images with limited viewing angles is used to render multiple images I at different angles from a new perspective; or a three-dimensional model is rendered at any viewpoint to obtain multiple images I at different angles, such as a three-dimensional model represented by a mesh, voxel, or point cloud, or a three-dimensional model or neural radiation field is obtained by an algorithm for converting images to three-dimensional models, and then rendered at a new perspective to obtain multiple images I at different angles, such as Zero1-to-3 or Magic3D; Alternatively, a 3D new perspective synthesis model is used to synthesize a new perspective of an object image to obtain multiple images I at different angles.

[0011] Further, the specific process is:

[0012] (3.1) Input the image I into the pre-trained diffusion model (DDIM) and encode it into a hidden image code with a noisy signal;

[0013] (3.2) Iteratively improve image I by gradually reducing the hidden code of the image with noisy signals;

[0014] (3.3) The text prompt and the improved image I are input into the diffusion model to generate a new image that conforms to the text description; the text can come from user input or from multiple groups of prompts generated by the large language model (LLM) based on user prompts, and these prompts are used to guide the diffusion model to generate content that conforms to the text description.

[0015] Furthermore, in the second step, objects are separated from the background through mask separation. For objects without background, new images are generated through an image-to-image neural network. The image-to-image neural network can be a Diffusion network or a network containing a ControlNet structure. The generated new images are supervised and constrained by text and / or masks and / or depth maps and / or edge contour maps to obtain new images with changed styles and patterns.

[0016] Furthermore, in step (3.2), in each iterative diffusion process, the image pixels are modified according to the mask guidance. Specifically, the mask guides the diffusion by modifying or editing a region of the specified image, and the modification includes changing the color and texture of the selected region.

[0017] Furthermore, in step (3.2), the result after continuously reducing the noise during the diffusion model sampling process is The object foreground (interior) can be better edited, which is represented by the following equation:

[0018]

[0019] Among them, z t represents the implicit code after adding noise, y t It is the result after adding noise continuously during DDIM sampling, and m is the mask;

[0020] This step is performed N times, N>=1. If the edit needs to be more obvious, the number of steps can be increased until the user is satisfied with the edit of the background. With each iteration, the image will be further improved and the edit effect will become more obvious. The diffusion process ensures that the modification blends smoothly with the rest of the image.

[0021] Furthermore, in step (3.2), the result after continuously reducing the noise during the scattered model sampling process is The object background (exterior) can be better edited, which is represented by the following equation:

[0022]

[0023] Among them, z t represents the implicit code after adding noise, y t It is the result after adding noise continuously during DDIM sampling, and m is the mask;

[0024] This step is performed N times, N>=1. If the edit needs to be more obvious, the number of steps can be increased until the user is satisfied with the edit of the background. With each iteration, the image will be further improved and the edit effect will become more obvious. The diffusion process ensures that the modification blends smoothly with the rest of the image.

[0025] Furthermore, before step (3.2) is executed, the following steps are also included:

[0026] Traverse all network layers of the diffusion model and operate on the cross attention map or self-attention map, sample the mask m at the resolution of the cross attention map or self-attention map, and for the cross attention map, determine the pixel index J in the object foreground and background column according to whether the token points to the object foreground (inside) or background (outside) in and J out ; For the self-attention map, the pixel index J in the foreground (inside) and background (outside) of the object is determined according to whether the pixel belongs to the object in and J out , using J in and J out Get the mask attention map:

[0027]

[0028] Among them, *j in Indicates the column index pointer indicating the pixel in the foreground (inside) of the object;

[0029] *jout indicates the column index pointer indicating the pixel in the background (outside) of the object;

[0030] Indicates index is *j in The masked attention map of

[0031] Indicates index is *j in Attention map of

[0032] Indicates index is *j out The masked attention map of

[0033] Indicates that the index is *j out Attention map.

[0034] Furthermore, by using the mask attention map, a constrained noise Z is obtained at each denoising stage. cond , the noise without mask constraint is recorded as Z uncond ,calculate:

[0035] Z t-1 ←Z cond +W g *(Z cond -Z uncond );

[0036]

[0037] Among them, Z t-1 represents the noise without mask constraint at time t-1;

[0038] W g Indicates the unmasked constraint guidance scale;

[0039] Represents the noise after the mask attention calculation at time t-1;

[0040] Furthermore, when the image is a new perspective rendering result obtained by optimizing a three-dimensional field or a three-dimensional model through a partially limited perspective image, the first step to the third step are repeated, and different viewpoints (R, T) are selected in the first step, and different text combinations are selected in the third step to obtain a large number of 3D model-image pairs for the same model but under different lighting and background conditions.

[0041] The present invention also provides a training method for monocular 3D reconstruction based on synthetic diverse samples, comprising:

[0042] The first step is to add a large number of 3D model-image pairs obtained by the above method to the original data set, randomly sample or collect multiple viewpoints (R, T) according to natural distribution or according to pre-experimental results, or randomly apply perturbations within a certain angle range of one or more viewpoints (R, T) to improve the robustness to the camera pose, where R represents the rotation matrix and T represents the camera matrix;

[0043] In the second step, each image obtained in the first step is subjected to feature extraction through a convolutional neural network, and then further feature extraction is performed through a Transformer encoder to obtain normalized canonical space (normalized coordinate space) features; preferably, the convolutional neural network is a pre-trained ResNet or DeepLab, and the Transformer encoder is from Segformer. Preferably, when the resolution of the input image is 512*512*3 (RGB channels), 64*64*256 features are obtained through the first convolutional neural network, and then 64*64*256 features are obtained through the Transformer;

[0044] In the third step, the canonical space (normalized coordinate space) feature is passed through another Transformer encoder or a lightweight convolution decoder to obtain a triplane feature, wherein the triplane feature contains the three-dimensional information of the object. Preferably, the feature with a size of 64*64*256 in the second step is passed through the Transformer encoder in the third step to obtain a feature with a size of 256*256*96, wherein each plane has 32 channels of information;

[0045] The fourth step is to flatten the three-plane features and add time-related noise z t , and then input the text prompt, and use the cross attention to act on the diffusion model after passing through the encoder to gradually reduce the noise z t Denoising generates a three-dimensional model with reasonable three-dimensional information. Preferably, the three-plane features of size 256*256*96 are flattened into features of 256*768*32, and the three-plane representation of the three-dimensional model is generated after denoising. The present invention combines the perceptual compression capability of the convolutional neural network to compress the image into a KL regularized three-dimensional latent space, and then uses the Transformer to carry out a flexible three-dimensional perceptual attention mechanism between the three-dimensional information in the latent space, thereby enhancing the efficiency of information flow in the three-dimensional space, thereby promoting consistent geometric reconstruction.

[0046] Furthermore, in the third step, the three-plane features are rendered by a differentiable renderer, and the rendering result is compared with the input image. The comparison result is input into the loss function to participate in the training of the diffusion model. The comparison result is L2 loss and perception loss. An adversarial loss can also be added to become a self-learning 3D generation network. Specifically, the adversarial loss comes from the input view discriminator, which is used to ensure that the reconstruction of the input view is perceptually reasonable; another auxiliary new view discriminator is used to distinguish the input view rendering result from the new view rendering result. If the new view discriminator is allowed to distinguish the new view rendering result from the real picture, the reconstruction model will suffer from the problem of posterior collapse, and output a result that is unrelated to the input but high-fidelity to deceive the new view discriminator. Therefore, it is chosen to let the new view discriminator distinguish the input view rendering result from the new view rendering result, rather than distinguish the new view rendering result from the real picture, to avoid the problem of posterior collapse and ensure that the reconstruction model outputs a result related to the input.

[0047] Furthermore, it also includes extracting high-frequency information from each image in the first step through a filter or a convolutional neural network, and then adding or concatenating the high-frequency information with the canonical space features in the second step and inputting them into the Transformer encoder in the third step, so as to retain the high-frequency information and increase the fine-grained feature extraction effect of the image. The high-frequency information includes the eyes, hair, etc. of the figure / character model.

[0048] The present invention has the following beneficial effects:

[0049] (1) Improving the diversity and richness of single-view data: A large number of 3D models and image pairs generated from different viewpoints, styles, and background conditions provide a rich training dataset. This diversity can significantly improve the generalization ability of monocular 3D reconstruction models.

[0050] (2) Enhanced robustness to camera pose: By randomly applying perturbations within a certain angle range at multiple viewpoints (R, T), the model’s adaptability to different camera poses is improved, thereby enhancing the robustness of monocular 3D reconstruction.

[0051] (3) High-quality feature extraction and 3D information generation: Use pre-trained convolutional neural networks and Transformer encoders to extract normalized features, which are then further processed to generate three-plane features containing 3D information. This high-quality feature extraction process is a single-view Figure 3 Dimensional modeling provides a solid foundation.

[0052] (4) Accurate image editing and synthesis: Through mask guidance and attention mechanism, the color and texture of specific areas can be accurately modified during the image synthesis process to ensure the details and consistency of the generated images, which is crucial for generating high-quality 3D models.

[0053] (5) The present invention is provided by a user with a picture I containing a specific object, and based on the information contained in the picture I and the mask m of the object, combined with a text description, generates pictures containing the object with diverse backgrounds, which are more in line with the user's expectations.

[0054] (6) The present invention proposes a data synthesis method based on a diffusion model, which generates rich 3D models and image pairs through different viewpoints and lighting conditions, and combines mask segmentation and attention mechanism to achieve high-precision image editing and 3D information extraction. In addition, the diffusion model generation technology guided by text prompts can flexibly generate high-quality 3D content that meets specific descriptions, thereby significantly improving the single-view Figure 3 The effect and accuracy of dimensional reconstruction. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 This is a flow chart of data synthesis in Example 1.

[0056] Figure 2 It is a schematic diagram of a picture generated by the synthesis method of the present invention in Example 1.

[0057] Figure 3 This is a comparison diagram between the image generated by the synthesis method of the present invention in Example 1 and the image generated using text prompts.

[0058] Figure 4 This is a flowchart of the three-dimensional reconstruction training in Example 2. DETAILED DESCRIPTION

[0059] The specific implementation methods of the present invention are described in detail below in conjunction with the accompanying drawings. It should be pointed out that the embodiments are only specific explanations of the invention and should not be regarded as limitations of the invention. The purpose of the embodiments is to enable those skilled in the art to better understand and reproduce the technical solutions of the present invention. The protection scope of the present invention shall still be based on the scope defined by the claims.

[0060] Example 1

[0061] like Figure 1 As shown, this embodiment provides a method for data synthesis, including:

[0062] The first step is to render a picture containing a specific object into multiple pictures I corresponding to the object at different angles; specifically: multiple pictures I at different angles are obtained by rendering a three-dimensional field or a three-dimensional model from a new perspective obtained by optimizing some pictures with limited perspectives; or multiple pictures I at different angles are obtained by rendering a three-dimensional model at any viewpoint, such as a three-dimensional model represented by a mesh, voxel or point cloud, or a three-dimensional model or neural radiation field is obtained by an algorithm for converting pictures to three-dimensional models, and then rendered from a new perspective to obtain multiple pictures I at different angles, such as Zero1-to-3 or Magic3D; or multiple pictures I at different angles are obtained by synthesizing the new perspective of the object picture through a 3D new perspective synthesis model; wherein R represents the rotation matrix and T represents the camera matrix.

[0063] The second step is to obtain a mask m of the object based on the image I, wherein the mask is the result of a semantic segmentation model, such as a SAM model, or a saliency segmentation model, such as a U2Net model, or a result of a tracking algorithm; or a mask manually input by a user;

[0064] In the third step, the mask m and the additional input text description are used to guide the image generation model to use the image I to generate a new image that conforms to the text description and retains the specific object structure information. Specifically:

[0065] (3.1) Input the image I into the pre-trained diffusion model (DDIM) and encode it into an image latent code with a noise signal; if there is an object without background, a new image is generated through an image-to-image neural network. The image-to-image neural network can be a Diffusion network or a network with a ControlNet structure. The generated new image is supervised and constrained by text and / or mask and / or depth map and / or edge contour map to obtain a new image with changed style and pattern.

[0066] Before executing (3.2), it is necessary to traverse all network layers of the diffusion model and operate the cross-attention map or self-attention map, sample the mask m at the resolution of the cross-attention map or self-attention map, and for the cross-attention map, determine the pixel index J in the object foreground and background column according to whether the token points to the object foreground (inside) or background (outside) in and J out ; For the self-attention map, the pixel index J in the foreground (inside) and background (outside) of the object is determined according to whether the pixel belongs to the object in and J out , using J in and J out Get the mask attention map:

[0067]

[0068] Among them, *j in Indicates the column index pointer indicating the pixel in the foreground (inside) of the object;

[0069] *jout indicates the column index pointer indicating the pixel in the background (outside) of the object;

[0070] Indicates that the index is *j in The masked attention map of

[0071] Indicates that the index is *j in Attention map of

[0072] Indicates that the index is *j out The masked attention map of

[0073] Indicates that the index is *j out Attention map of

[0074] Using the mask attention map, we can get the constrained noise Z at each denoising stage cond , the noise without mask constraint is recorded as Z uncond ,calculate:

[0075] Z t-1 ←Z cond +W g *(Z cond -Z uncond );

[0076]

[0077] Among them, Z t-1 represents the noise without mask constraint at time t-1;

[0078] W g Indicates the unmasked constraint guidance scale;

[0079] Represents the noise after the mask attention calculation at time t-1;

[0080] (3.2) Iteratively improve image I by gradually reducing the hidden code of the image with noisy signals; in each iterative diffusion process, the image pixels are modified according to the guidance of the mask. Specifically, the mask guides the diffusion by modifying or editing the specified area of ​​the image, and the modification includes changing the color and texture of the selected area.

[0081] The result of continuously reducing noise during the diffusion model sampling process The object foreground (interior) can be better edited, which is represented by the following equation:

[0082]

[0083] Among them, z t represents the implicit code after adding noise, y t It is the result after adding noise continuously during DDIM sampling, and m is the mask;

[0084] This step is performed N times, N>=1. If the edit needs to be more obvious, the number of steps can be increased until the user is satisfied with the edit of the background. With each iteration, the image will be further improved and the edit effect will become more obvious. The diffusion process ensures that the modification blends smoothly with the rest of the image.

[0085] The result of continuously reducing noise during the sampling process of the scattered model The object background (exterior) can be better edited, which is represented by the following equation:

[0086]

[0087] Among them, z t represents the implicit code after adding noise, y t It is the result after adding noise continuously during DDIM sampling, and m is the mask;

[0088] This step is performed N times, N>=1. If the edit needs to be more obvious, the number of steps can be increased until the user is satisfied with the edit of the background. With each iteration, the image will be further improved and the edit effect will become more obvious. The diffusion process ensures that the modification blends smoothly with the rest of the image.

[0089] Through mask guidance and attention mechanisms, the color and texture of specific areas can be accurately modified during the image synthesis process to ensure the details and consistency of the generated images, which is crucial for generating high-quality 3D models.

[0090] (3.3) The text prompt is combined with the improved image I to generate a new image that matches the text description through a guided diffusion model; the text can come from user input or from multiple groups of prompts generated by a large language model (LLM) based on user prompts, and these prompts are used to guide the diffusion model to generate content that matches the text description.

[0091] When the image is a new perspective rendering result obtained by optimizing a three-dimensional field or a three-dimensional model through a partially limited perspective image, the first step to the third step are repeated, and different viewpoints (R, T) are selected in the first step, and different text combinations are selected in the third step to obtain a large number of 3D model-image pairs for the same model but under different lighting and background conditions.

[0092] like Figure 2 As shown, the image with a mask separated background is encoded into a neural radiation field using an algorithm for converting an image to a three-dimensional model. The radiation field can generate rendered images at any angle. The generated rendered image (Image I) is input into the diffusion model together with a text prompt "Toy motorcycles on the shelf" that specifies the required background to generate new images with changing styles and patterns.

[0093] In order to illustrate that the new image generated by the present invention is more accurate and more in line with the requirements of the text prompt, a comparison is made between the image generated by the method of the present invention and the image generated by only using the text prompt. The present invention uses the neural radiation field and the text prompt to render and generate images from multiple perspectives, such as Figure 3 As shown in the figure, the first column is the multiple perspective renderings and text prompts required in the present invention, the second to fourth columns are pictures generated by the method of the present invention, and the fifth column is a picture generated only by text prompts. It can be clearly seen from the figure that rich 3D models and picture pairs are generated by different viewpoints and lighting conditions, combined with mask segmentation and attention mechanisms, to achieve high-precision picture editing and three-dimensional information extraction. In addition, through the diffusion model generation technology guided by text prompts, high-quality three-dimensional content that meets specific descriptions can be flexibly generated, thereby significantly improving the single-view Figure 3 A large number of 3D models and image pairs generated from different viewpoints, styles, and background conditions provide a rich training dataset. This diversity can significantly improve the generalization ability of monocular 3D reconstruction models.

[0094] Example 2

[0095] like Figure 4 As shown, this embodiment provides a training method for monocular 3D reconstruction based on synthetic diverse samples, including:

[0096] The first step is to add a large number of 3D model-image pairs obtained using Example 1 in addition to the original data set, randomly sample or collect multiple viewpoints (R, T) according to natural distribution or according to preliminary experimental results, or randomly apply perturbations within a certain angle range of one or more viewpoints (R, T) to improve the robustness to the camera pose, where R represents the rotation matrix and T represents the camera matrix;

[0097] In the second step, each image obtained in the first step is subjected to feature extraction through a convolutional neural network, and then further feature extraction is performed through a Transformer encoder to obtain normalized canonical space (normalized coordinate space) features; preferably, the convolutional neural network is a pre-trained ResNet or DeepLab, and the Transformer encoder is from Segformer. Preferably, when the resolution of the input image is 512*512*3 (RGB channels), 64*64*256 features are obtained through the first convolutional neural network, and then 64*64*256 features are obtained through the Transformer;

[0098] In the third step, the canonical space (normalized coordinate space) feature is passed through another Transformer encoder or a lightweight convolution decoder to obtain a triplane feature, wherein the triplane feature contains the three-dimensional information of the object. Preferably, the feature with a size of 64*64*256 in the second step is passed through the Transformer encoder in the third step to obtain a feature with a size of 256*256*96, wherein each plane has 32 channels of information; the triplane feature is rendered by a differentiable renderer, and the rendering result is compared with the input image, and the comparison result is input into the loss function to participate in the training of the diffusion model. The comparison result is L2 loss and perception loss, and an adversarial loss can also be added to become a self-learning 3D generation network. Specifically, the adversarial loss comes from the input view discriminator, which is used to ensure that the reconstruction of the input view is perceptually reasonable; another auxiliary new view discriminator is used to distinguish between the input view rendering result and the new view rendering result. If the new view discriminator is allowed to distinguish between the new view rendering results and the real pictures, the reconstructed model will suffer from the problem of posterior collapse, and output high-fidelity results that are irrelevant to the input to deceive the new view discriminator. Therefore, we choose to let the new view discriminator distinguish between the input view rendering results and the new view rendering results, rather than distinguish between the new view rendering results and the real pictures, to avoid the problem of posterior collapse and ensure that the reconstructed model outputs results related to the input.

[0099] The fourth step is to flatten the three-plane features and add time-related noise z t , and then input the text prompt, pass through the encoder and use the cross attention to act on the diffusion model, gradually reducing the noise z tDenoising generates a three-dimensional model with reasonable three-dimensional information. Preferably, the three-plane features of size 256*256*96 are flattened into features of 256*768*32, and the three-plane representation of the three-dimensional model is generated after denoising. The present invention combines the perceptual compression capability of the convolutional neural network to compress the image into a KL regularized three-dimensional latent space, and then uses the Transformer to carry out a flexible three-dimensional perceptual attention mechanism between the three-dimensional information in the latent space, thereby enhancing the efficiency of information flow in the three-dimensional space, thereby promoting consistent geometric reconstruction.

[0100] The pre-trained convolutional neural network and Transformer encoder are used to extract normalized features, which are then further processed to generate three-dimensional features containing three-dimensional information. This high-quality feature extraction process is a single-view Figure 3 Dimensional modeling provides a solid foundation.

[0101] In addition, each image in the first step is filtered or a convolutional neural network to extract high-frequency information, and then added or concatenated with the canonical space features in the second step and input into the Transformer encoder in the third step. The high-frequency information is retained to increase the effect of fine-grained feature extraction of the image. High-frequency information includes the eyes and hair of the figure / character model.

[0102] A large number of 3D models and image pairs generated from different viewpoints, styles, and background conditions provide a rich training data set. This diversity can significantly improve the generalization ability of the monocular 3D reconstruction model. And by randomly applying perturbations within a certain angle range of multiple viewpoints (R, T), the model's adaptability to different camera poses is improved, and the robustness of monocular 3D reconstruction is enhanced.

[0103] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications falling within the scope of the present application.

[0104] It should be noted that the technical features not described in detail in the present invention can be implemented by any existing technology.

Claims

1. A method for data synthesis, characterized in that: include: The first step is to render a picture containing a specific object into multiple pictures I corresponding to the object at different angles; The second step is to obtain a mask m of the object based on the image I, wherein the mask is a semantic segmentation model, a saliency segmentation model, a result of a tracking algorithm, or a mask manually input by a user; In the third step, the mask m and the additional input text description are used to guide the image generation model to use image I to generate a new image that conforms to the text description and retains specific object structure information.

2. A method for data synthesis according to claim 1, characterized in that: The method for rendering multiple images I at different angles in the first step is: A new perspective rendering of a three-dimensional field or a three-dimensional model is performed by optimizing some images with limited perspectives to obtain multiple images I with different perspectives; Or a three-dimensional model is rendered at any viewpoint to obtain multiple images I at different angles; or a three-dimensional model or neural radiation field is obtained through an algorithm for converting images to three-dimensional models and then rendered from a new perspective to obtain multiple images I at different angles; or a new perspective synthesis model is used to synthesize the new perspective of the object image to obtain multiple images I at different angles.

3. A method for data synthesis according to claim 1, characterized in that: The specific process of the third step is: (3.1) Input the image I into the pre-trained diffusion model and encode it into a hidden image code with a noisy signal; (3.2) Iteratively improve image I by gradually reducing the hidden code of the image with noisy signals; (3.3) The text prompt is combined with the improved image I to generate a new image that matches the text description through the guided diffusion model.

4. A method for data synthesis according to claim 3, characterized in that: In the second step, objects are separated from the background through mask separation. For objects without background, new images are generated through an image-to-image neural network. The generated new images are supervised and constrained by text and / or masks and / or depth maps and / or edge contour maps to obtain new images with changed styles and patterns.

5. A method for data synthesis according to claim 3, characterized in that: In step (3.2), during each iterative diffusion process, the image pixels are modified according to the mask guidance. Specifically, the mask guides the diffusion by modifying or editing a specified area of ​​the image. The modification includes changing the color and texture of the selected area.

6. A method for data synthesis according to claim 3, characterized in that: In step (3.2), the result after continuously reducing the noise during the diffusion model sampling process is Better editing of object foreground is represented by the following equation: Among them, z t represents the implicit code after adding noise, y t It is the result after adding noise continuously during DDIM sampling, and m is the mask; This step is performed N times, N>=1. If the edit needs to be more obvious, the number of steps can be increased until the user is satisfied with the edit of the background. With each iteration, the image will be further improved and the edit effect will become more obvious. The diffusion process ensures that the modification blends smoothly with the rest of the image.

7. A method for data synthesis according to claim 3, characterized in that: The result after continuously reducing the noise during the diffusion model sampling process in step (3.2) Better editing of object background is represented by the following equation: Among them, z t represents the implicit code after adding noise, y t It is the result after adding noise continuously during DDIM sampling, and m is the mask; This step is performed N times, N>=1. If the edit needs to be more obvious, the number of steps can be increased until the user is satisfied with the edit of the background. With each iteration, the image will be further improved and the edit effect will become more obvious. The diffusion process ensures that the modification blends smoothly with the rest of the image.

8. A method for data synthesis according to claim 3, characterized in that: Before step (3.2) is executed, it also includes: Traverse all network layers of the diffusion model and operate on the cross attention map or self-attention map, sample the mask m at the resolution of the cross attention map or self-attention map, and for the cross attention map, determine the pixel in the object foreground and background column index J according to whether the token points to the object or the background in and J out ; For the self-attention map, the pixel index J in the foreground and back of the object is determined according to whether the pixel belongs to the object in and J out , using J in and J out Get the mask attention map: Among them, *j in Indicates the column index pointer representing the pixel in the foreground of the object; *jout indicates the column index pointer of the pixel in the object background; Indicates that the index is *j in The masked attention map of Indicates that the index is *j in Attention map of Indicates that the index is *j out The masked attention map of Indicates that the index is *j out Attention map.

9. A method for data synthesis according to claim 8, characterized in that: Using the mask attention map, we can obtain the constrained noise Z at each denoising stage cond , the noise without mask constraint is recorded as Z uncond ,calculate: WITH t-1 ←From cond +W g *(WITH cond -WITH uncond ); Among them, Z t-1 represents the noise without mask constraint at time t-1; W g Indicates the unmasked constraint guidance scale; Represents the noise after mask attention calculation at time t-1.

10. A method for data synthesis according to claim 1, characterized in that: When the image is a new perspective rendering result obtained by optimizing a three-dimensional field or a three-dimensional model through a partially limited perspective image, the first step to the third step are repeated, and different viewpoints (R, T) are selected in the first step, and different text combinations are selected in the third step to obtain a large number of 3D model-image pairs for the same model but under different lighting and background conditions.

11. A training method for monocular 3D reconstruction based on synthetic diverse samples, characterized in that: include: The first step is to add a large number of 3D model-image pairs obtained by the method described in any one of claims 1 to 10 to the original data set, randomly sample or collect multiple viewpoints (R, T) according to natural distribution or according to preliminary experimental results, or randomly apply perturbations within a certain angle range of one or more viewpoints (R, T) to improve the robustness to the camera pose, where R represents the rotation matrix and T represents the camera matrix; In the second step, each image obtained in the first step is passed through a convolutional neural network for feature extraction, and then further feature extraction is performed through a Transformer encoder to obtain normalized canonical space features; The third step is to pass the canonical space feature through another Transformer encoder or a lightweight convolution decoder to obtain a three-plane feature, which contains the three-dimensional information of the object. The fourth step is to flatten the three-plane features and add time-related noise z t , and then input the text prompt, and use the cross attention to act on the diffusion model after passing through the encoder to gradually reduce the noise z t Denoising generates a 3D model with reasonable 3D information.

12. The training method for monocular 3D reconstruction based on synthetic diverse samples according to claim 11, characterized in that: In the third step, the three-plane features are rendered by a differentiable renderer, the rendering results are compared with the input images, and the comparison results are input into the loss function to participate in the training of the diffusion model.

13. The training method for monocular 3D reconstruction based on synthetic diverse samples according to claim 11, characterized in that: It also includes extracting high-frequency information from each image in the first step through a filter or a convolutional neural network, and then adding or connecting the high-frequency information with the canonical space features in the second step and inputting them into the Transformer encoder in the third step, so as to retain the high-frequency information and increase the fine-grained feature extraction effect of the image.

Images