A three-dimensional model data processing method, system, product, equipment and medium

Abstract

The present invention discloses a data processing method, system, product, equipment and medium for a three-dimensional model, and relates to the field of three-dimensional modeling. In order to solve the problem of inconsistency of multiple perspectives of three-dimensional editing caused by multiple iterative editing, the method includes determining the source three-dimensional model corresponding to the current iteration, obtaining the current single-perspective editing image of the source three-dimensional model edited according to the guiding text; generating a multi-perspective editing image group based on the current single-perspective editing image, the multi-perspective editing image group including the editing image of the front view, the editing image of the left view and the editing image of the right view; generating a dense perspective editing image matrix based on the multi-perspective editing image group; and processing the source three-dimensional model using the editing image matrix to obtain the edited three-dimensional model corresponding to the current iteration. The present invention can improve the consistency of editing between different perspectives, reconstruct the edited three-dimensional model based on the editing image matrix, and improve the editing quality and editing efficiency of the three-dimensional model.

Description

Technical Field

[0001] The present invention relates to the field of three-dimensional modeling, and in particular to a data processing method, system, product, equipment and medium for a three-dimensional model. Background Art

[0002] Three-dimensional models play a key role in applications and industries such as augmented / virtual reality, film / game production, and artistic creation. Currently, the most direct and efficient way to create a three-dimensional model is to change the geometry or appearance of an existing three-dimensional model through a three-dimensional editing method based on text description, thereby generating a variety of three-dimensional models. In order to complete the editing of the target editing area while retaining the non-editing area, it is necessary to edit the image of the three-dimensional model in accordance with the text description, and then promote the image editing result to a three-dimensional model to achieve three-dimensional editing. When promoting the image editing result to a three-dimensional model, multiple iterative editing is required. Since each iteration process only edits one image, multiple iterations will cause the problem of multi-perspective inconsistency in three-dimensional editing, affecting the editing efficiency and quality of the three-dimensional model.

[0003] Therefore, how to provide a solution to the above technical problems is a problem that those skilled in the art need to solve at present. Summary of the invention

[0004] The purpose of the present invention is to provide a data processing method, system, product, equipment and medium for a three-dimensional model, which can improve the consistency of editing between different perspectives, reconstruct the edited three-dimensional model based on the editing image matrix, and improve the editing quality and editing efficiency of the three-dimensional model.

[0005] In order to solve the above technical problems, the present invention provides a data processing method of a three-dimensional model, comprising:

[0006] Determine a source three-dimensional model corresponding to a current iteration, and obtain a current single-view edited image of the source three-dimensional model edited according to the guidance text;

[0007] Generate a multi-view editing image group based on the current single-view editing image, wherein the multi-view editing image group includes an editing image of a front view, an editing image of a left view, and an editing image of a right view;

[0008] Generate a dense perspective editing image matrix based on the multi-perspective editing image group;

[0009] The source three-dimensional model is processed using the edited image matrix to obtain an edited three-dimensional model corresponding to the current iteration.

[0010] Optionally, the data processing method of the three-dimensional model further includes:

[0011] Pre-created multi-view generator;

[0012] The process of generating a multi-view editing image group based on the current single-view editing image comprises:

[0013] The current single-view editing image is input into the multi-view generator to obtain a multi-view editing image group.

[0014] Optionally, the process of pre-creating a multi-view generator includes:

[0015] Construct a multi-view generation network;

[0016] Generate multiple to-be-selected editing images of different viewing angles based on the front view editing image, wherein the multiple to-be-selected editing images correspond to different azimuth angles;

[0017] Determine a plurality of left view editing images and a plurality of right view editing images; the left view editing images are to-be-selected editing images whose azimuth angles are within the left view selection range, and the right view editing images are to-be-selected editing images whose azimuth angles are within the right view selection range;

[0018] A plurality of triplets are formed based on the front view editing image, a plurality of the left view editing images and a plurality of the right view editing images; each of the triplets includes the front view editing image, any one of the left view editing images and any one of the right view editing images, and the left view editing images or the right view editing images in any two of the triplets are different;

[0019] The multi-view generation network is trained according to the plurality of triplets to obtain a multi-view generator.

[0020] Optionally, the data processing method of the three-dimensional model further includes:

[0021] Pre-create dense view generator;

[0022] The process of generating a dense perspective editing image matrix based on the multi-perspective editing image group includes:

[0023] The multi-view edited image group is input into the dense view generator to obtain the dense view edited image matrix.

[0024] Optionally, the process of pre-creating a dense view generator includes:

[0025] Establishing a dense view generation network; the dense view generation network includes a video generation diffusion network and a view adaptation module;

[0026] Acquire a first training data set obtained by rotating a camera around a target object based on a fixed trajectory and a second training data set obtained by rotating a camera around the target object based on a random trajectory; the target object is an object in the source three-dimensional model;

[0027] The dense perspective generation network is trained using the first training data set. When the number of training times reaches a preset number, the video generation diffusion network is fixed, and the perspective adaptation module is trained using the second training data set until the training requirements are met to obtain a dense perspective generator.

[0028] Optionally, the data processing method of the three-dimensional model further includes:

[0029] Sampling the fixed trajectory to obtain a plurality of sampling points;

[0030] For each of the sampling points, adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle;

[0031] At least one random trajectory is obtained according to all the new azimuth angles and / or the new elevation angles.

[0032] Optionally, for each of the sampling points, a process of adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle includes:

[0033] For each of the sampling points, random noise is added to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

[0034] Optionally, for each of the sampling points, a process of adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle includes:

[0035] For each of the sampling points, a first preset interval is added to the azimuth angle corresponding to the sampling point, or a second preset interval is reduced to the azimuth angle corresponding to the sampling point, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

[0036] Optionally, after the source three-dimensional model is processed by using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration, the three-dimensional model data processing method further includes:

[0037] Determine whether the current iteration satisfies the iteration end condition;

[0038] If not, the edited three-dimensional model is used as the source three-dimensional model of the next iteration for the next iteration;

[0039] If so, the edited three-dimensional model is used as a target three-dimensional model that conforms to the description of the guidance text.

[0040] Optionally, the process of obtaining a current single-view edited image of the source three-dimensional model edited according to the guidance text includes:

[0041] Acquire a current single-view image of the source three-dimensional model at a selected view angle of the current iteration;

[0042] The current single-view image is edited according to the guide text to obtain a current single-view edited image.

[0043] Optionally, the process of obtaining a current single-view image of the source 3D model at the selected view angle of the current iteration includes:

[0044] Determine, based on the selected viewing angle of the current iteration, a query point corresponding to each pixel point in the three-dimensional space of the source three-dimensional model;

[0045] For each query point, obtaining the color of the pixel point corresponding to the query point according to a three-dimensional Gaussian function and a distance between the query point and a center point of the source three-dimensional model;

[0046] A current single-view image of the source three-dimensional model at a selected viewing angle is obtained based on the colors of all the pixel points.

[0047] Optionally, the process of processing the source three-dimensional model using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration includes:

[0048] Determining a first reconstruction loss value using the edited image matrix;

[0049] The source three-dimensional model is adjusted based on the first reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

[0050] Optionally, the process of determining the first reconstruction loss value by using the edited image matrix includes:

[0051] A first reconstruction loss value is calculated based on a first reconstruction loss function, where the first reconstruction loss function is: ;

[0052] Among them, L rec is the first reconstruction loss value, I e is any edit image in the edit image matrix, Based on I e The first reference image is obtained by rendering the source 3D model at the same viewing angle, M is an editing mask from a 3D editing frame specified by a user to the editing image, is the element-wise multiplication sign.

[0053] Optionally, the process of adjusting the source three-dimensional model based on the first reconstruction loss value to obtain the edited three-dimensional model corresponding to the current iteration includes:

[0054] Adjusting the source 3D model according to the first reconstruction loss value to obtain a rough edited 3D model;

[0055] Obtaining a pseudo-truth image matrix based on the rough edited three-dimensional model and the three-dimensional edit frame specified by the user;

[0056] Determining a second reconstruction loss value using the pseudo-true value image matrix;

[0057] The source three-dimensional model is adjusted based on the second reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

[0058] Optionally, the process of determining the second reconstruction loss value using the pseudo-true value image matrix includes:

[0059] A second reconstruction loss value is calculated based on a second reconstruction loss function, where the second reconstruction loss function is: ;

[0060] Among them, L rec ' is the second reconstruction loss value, I r is any pseudo-true value image in the pseudo-true value image matrix, Based on I r The second reference image is rendered from the same viewing angle to the 3D edited model in the optimization phase.

[0061] Optionally, the process of obtaining a pseudo-true value image matrix based on the rough edited three-dimensional model and the three-dimensional edit box specified by the user includes:

[0062] For each image pose, a first image corresponding to the image pose is obtained by rendering using the rough edited 3D model, a second image corresponding to the image pose is obtained by rendering using the source 3D model, an edit region mask is obtained based on the rough edited 3D model and a 3D edit frame specified by a user, and a pseudo-truth image corresponding to the image pose is obtained according to the first image, the second image, and the edit region mask;

[0063] Based on the pseudo-true value images corresponding to the plurality of image postures, a pseudo-true value edited image matrix is ​​obtained.

[0064] Optionally, the process of obtaining a pseudo true value image corresponding to the image posture according to the first image, the second image, and the edited area mask includes:

[0065] A pseudo true value image corresponding to the image posture is obtained based on a first relational expression, wherein the first relational expression is: ;

[0066] Among them, I r is the pseudo-truth image corresponding to any of the image poses, is the first image, I s For the second image, is the edit region mask, is the element-wise multiplication sign.

[0067] In order to solve the above technical problems, the present application also provides a three-dimensional model data processing system, including:

[0068] A first determination module is used to determine a source three-dimensional model corresponding to a current iteration, and obtain a current single-view edited image of the source three-dimensional model edited according to the guidance text;

[0069] A first generating module, configured to generate a multi-view editing image group based on the current single-view editing image, wherein the multi-view editing image group includes an editing image of a front view, an editing image of a left view, and an editing image of a right view;

[0070] A second generating module, configured to generate a dense perspective editing image matrix based on the multi-perspective editing image group;

[0071] A processing module is used to process the source three-dimensional model using the edited image matrix to obtain an edited three-dimensional model corresponding to the current iteration.

[0072] To solve the above technical problems, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the three-dimensional model data processing method described in any one of the above items.

[0073] In order to solve the above technical problems, the present application also provides an electronic device, including:

[0074] Memory for storing computer programs;

[0075] A processor is used to implement the steps of any of the above three-dimensional model data processing methods when executing the computer program.

[0076] In order to solve the above technical problems, the present application also provides a non-volatile storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the data processing method of the three-dimensional model described in any one of the above items are implemented.

[0077] The present invention provides a data processing method for a three-dimensional model. After a source three-dimensional model is edited according to a guidance text to obtain a current single-view editing image, a multi-view editing image group corresponding to the current single-view editing image is first generated, including a front view, a left view and a right view. Then, based on the editing images of the front view, the left view and the right view, a dense perspective editing image matrix is ​​generated to improve the consistency of editing between different perspectives. The edited three-dimensional model is reconstructed based on the editing image matrix, thereby improving the editing quality and editing efficiency of the three-dimensional model.

[0078] The present invention also provides a three-dimensional model data processing system, a computer program product, an electronic device and a non-volatile storage medium, which have the same beneficial effects as the above-mentioned three-dimensional model data processing method. BRIEF DESCRIPTION OF THE DRAWINGS

[0079] In order to more clearly illustrate the embodiments of the present invention, the following briefly introduces the drawings required for use in the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0080] Figure 1 A flowchart of the steps of a three-dimensional model editing method provided by the present invention;

[0081] Figure 2 A schematic diagram of a source three-dimensional model of a toy dog ​​provided by the present invention;

[0082] Figure 3 A schematic diagram of a single-view image obtained by rendering a source three-dimensional model provided by the present invention at a viewing angle corresponding to a front view;

[0083] Figure 4 A schematic diagram of a single-view editing image provided by the present invention;

[0084] Figure 5 A schematic diagram of generating a multi-view image provided by the present invention;

[0085] Figure 6 A schematic diagram of generating a generalized dense perspective image provided by the present invention;

[0086] Figure 7 A flowchart of editing a three-dimensional model based on dense perspective generation provided by the present invention;

[0087] Figure 8 A schematic diagram of a three-dimensional model editing interactive interface provided by the present invention;

[0088] Fig. 9A schematic diagram of the structure of a three-dimensional model editing system provided by the present invention;

[0089] Fig.10 A schematic diagram of the structure of an electronic device provided by the present invention;

[0090] Fig.11 A schematic diagram of the structure of a non-volatile storage medium provided by the present invention. DETAILED DESCRIPTION

[0091] The core of the present invention is to provide a data processing method, system, product, equipment and medium for a three-dimensional model, which can improve the consistency of editing between different perspectives, reconstruct the edited three-dimensional model based on the editing image matrix, and improve the editing quality and editing efficiency of the three-dimensional model.

[0092] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0093] First, please refer to Figure 1 The present invention provides a three-dimensional model data processing method, comprising:

[0094] S101: Determine a source 3D model corresponding to a current iteration, and obtain a current single-view edited image of the source 3D model edited according to a guidance text;

[0095] In this embodiment, it is considered that in the three-dimensional editing based on text description, multiple iterations can be performed to gradually refine the three-dimensional model and improve the precision and realism of the three-dimensional model. For the convenience of explanation, this embodiment takes any one of the iterations as an example to illustrate the data processing method of the three-dimensional model, and the data processing methods of the three-dimensional models of other iterations are similar. For the current iteration, if the current iteration is the first iteration, the source three-dimensional model corresponding to the current iteration is the initial three-dimensional model, that is, the three-dimensional model that has not been iteratively processed. If the current iteration is not the first iteration, then the source three-dimensional model corresponding to the current iteration is the edited three-dimensional model corresponding to the previous iteration. Each iteration is based on the result of the previous iteration, which ensures the continuity of the editing operation and reduces the repeated workload. Through step-by-step iteration, the three-dimensional model can be adjusted and optimized more accurately. Exemplarily, assuming that the current iteration is the second iteration, the source three-dimensional model corresponding to the second iteration is the edited three-dimensional model corresponding to the first iteration.

[0096] In an exemplary embodiment, the process of obtaining a current single-perspective edited image of a source three-dimensional model edited according to a guidance text includes: determining a selected perspective of a current iteration, obtaining a current single-perspective image of the source three-dimensional model at the selected perspective, and editing the current single-perspective image according to the guidance text to obtain the current single-perspective edited image.

[0097] It can be understood that a specific perspective is first selected to render the source 3D model to obtain a front view, and the two-dimensional image (single-perspective image) of the source 3D model at the specified perspective is edited in accordance with the description of the guidance text to generate the current single-perspective editing image corresponding to the current iteration, so as to facilitate the precise adjustment and optimization of the detail features at the perspective according to the guidance text in the current iteration, thereby improving the overall refinement of the model editing.

[0098] Specifically, the process of obtaining the current single-view image of the source three-dimensional model at a selected viewing angle includes: based on the selected viewing angle, determining the query point corresponding to each pixel point in the three-dimensional space of the source three-dimensional model; for each query point, obtaining the color of the pixel point corresponding to the query point according to the three-dimensional Gaussian function and the distance between the query point and the center point of the source three-dimensional model; and obtaining the current single-view image of the source three-dimensional model at the selected viewing angle based on the colors of all pixel points.

[0099] In this embodiment, the edited three-dimensional model g is represented by a three-dimensional Gaussian. s ={g1,…,g n}, where the i-th three-dimensional Gaussian point g i ={μ, λ, c, a}, i∈{1,…,N}, Indicates the location of the center point, Represents the covariance matrix, and the color of each pixel is given by Indicates that transparency , represents the three-dimensional real space, represents the seven-dimensional real space, then the three-dimensional Gaussian representation is:

[0100] ;

[0101] Where x represents the distance between the query point and the center point.

[0102] Then, the color of a pixel in the single-view image Is of the source 3D model obtained based on rendering is:

[0103] , ;c i Represents the color vector of the i-th three-dimensional Gaussian point, a i represents the transparency of the i-th three-dimensional Gaussian point, G(x) represents the Gaussian function, Indicates an intermediate parameter.

[0104] The current single-view image Is of the source 3D model can be generated based on the colors of all pixels. Figure 2 and Figure 3 As shown, Figure 2 A source three-dimensional model including a toy dog ​​in this embodiment is shown. Figure 3 A single-view image of a toy dog ​​rendered from the source three-dimensional model at a viewing angle corresponding to the front view is shown.

[0105] Then, the current single-view image is edited to obtain a single-view edited image that meets the description of the instruction text. According to the input edited text, the single-view image of the source 3D model is edited using the instruction text and the image editing method InstructPix2Pix to obtain a single-view edited image that meets the description of the instruction text. Assuming that the instruction text is a toy dog ​​with a ball under its feet, the single-view edited image can refer to Figure 4 shown.

[0106] S102: generating a multi-view editing image group based on the current single-view editing image, where the multi-view editing image group includes an editing image of a front view, an editing image of a left view, and an editing image of a right view;

[0107] In this embodiment, the multi-view editing image group includes an editing image corresponding to the front view, that is, the current single-view editing image in this embodiment, and then obtains the editing image of the left view and the editing image corresponding to the right view corresponding to the current single-view editing image. It can be understood that the multi-view editing image group helps to maintain the consistency of the model under different viewing angles, ensuring that the model conforms to the description of the guidance text at all angles.

[0108] The perspective of the left view may refer to a view obtained from a specific perspective, or may refer to a series of perspectives, i.e., the left view may include multiple views obtained by observing the left side of the object from different heights or different distances, and correspondingly, the perspective of the right view may refer to a view obtained from a specific perspective, or may refer to a series of perspectives, i.e., the right view may include multiple views obtained by observing the right side of the object from different heights or different distances. The selection may be made according to actual engineering needs, and this embodiment is not specifically limited here.

[0109] S103: generating a dense perspective editing image matrix based on the multi-perspective editing image group;

[0110] In this embodiment, a group of candidate editing images of dense perspectives are generated for the front view in the multi-perspective editing image group, a group of candidate editing images of dense perspectives are generated for the left view in the perspective editing image group, and a group of candidate editing images of dense perspectives are generated for the right view in the perspective editing image group, and an editing image matrix is ​​constructed based on all the above candidate editing images. It can be understood that the dense perspective provides a more detailed view, which is convenient for capturing the slight changes and details of the three-dimensional model at different angles, thereby improving the accuracy of three-dimensional model editing. By generating a group of dense perspective images for each main perspective (front view, left view, right view), it is convenient to ensure the consistency and integrity of the three-dimensional model in all directions. In the iterative editing process, the editing effects at different perspectives can be quickly compared, thereby accelerating the editing process and reducing the number of iterations.

[0111] S104: Processing the source three-dimensional model using the edited image matrix to obtain an edited three-dimensional model corresponding to the current iteration.

[0112] In this embodiment, the source 3D model is optimized using the edited image matrix obtained in S103 to obtain the edited 3D model corresponding to the current iteration. It can be understood that if the iteration is not completed, the edited 3D model corresponding to the current iteration is the source 3D model corresponding to the next iteration. It can be understood that since the image editing matrix composed of edited images under dense viewing angles corresponding to multiple views is edited in a single iteration process, and the editing consistency between different viewing angles in a single iteration process is improved, the number of iterations can be reduced and the iteration efficiency can be improved.

[0113] It can be seen that in this embodiment, after the source three-dimensional model is edited according to the guidance text to obtain the current single-perspective editing image, a multi-perspective editing image group corresponding to the current single-perspective editing image is first generated, including a front view, a left view and a right view, and then a dense perspective editing image matrix is ​​generated based on the editing images of the front view, the left view and the right view, so as to improve the consistency of editing between different perspectives, and reconstruct the edited three-dimensional model based on the editing image matrix, thereby improving the editing quality and editing efficiency of the three-dimensional model.

[0114] Based on the above embodiments:

[0115] In an exemplary embodiment, the data processing method of the three-dimensional model further includes:

[0116] Pre-created multi-view generator;

[0117] The process of generating a multi-view editing image group based on the current single-view editing image includes:

[0118] The current single-view edited image is input into the multi-view generator to obtain a multi-view edited image group.

[0119] In this embodiment, a multi-view generator is pre-established, and the current single-view editing image is input into the multi-view generator, so that multiple viewing angles of selected editing images can be quickly generated, thereby improving editing efficiency. Generating the selected editing images under different viewing angles can better ensure the consistency of viewing angles, reduce the number of iterations, and speed up the editing process.

[0120] In an exemplary embodiment, the process of pre-creating a multi-view generator includes:

[0121] Construct a multi-view generation network;

[0122] Generate multiple selected editing images at multiple viewing angles based on the front view editing image, where the multiple selected editing images correspond to different azimuth angles;

[0123] Determine a plurality of left view editing images and a plurality of right view editing images; the left view editing images are to-be-selected editing images whose azimuth angles are within the left view selection range, and the right view editing images are to-be-selected editing images whose azimuth angles are within the right view selection range;

[0124] A plurality of triplets are formed based on the front view editing image, the plurality of left view editing images and the plurality of right view editing images; each triplet includes the front view editing image, any left view editing image and any right view editing image, and the left view editing images or the right view editing images in any two triplets are different;

[0125] A multi-view generation network is trained based on multiple triplets to obtain a multi-view generator.

[0126] In this embodiment, a multi-view generation network is constructed, with the MVDream 2D ​​UNet structure as the backbone network, and each layer is composed of a convolutional layer, a 3D self-attention layer, and a 2D cross-attention layer. In this embodiment, the generation process of the multi-view generation fine-tuning dataset is first introduced, specifically including: constructing a dataset, inputting the front view x f , generate 18 viewing angles of 360° around the source 3D model, randomly select an azimuth angle within the 45° azimuth angle β∈[-45°, 0] (i.e., within the left view selection range) from the left offset of the front view, and obtain the left view x l , accordingly, select the right view x of the corresponding azimuth in the right view selection range r , forming a triple (x f , x l , x r ), iterate the above process several times to generate enough triplets to train the multi-view generation network and obtain the multi-view generator. The schematic diagram of the multi-view generator can be found in Figure 5As shown. In practical applications, given the current single-view edited image, it is input into the multi-view generator to directly generate edited images of the front view, left and right side views. It can be understood that the process of generating triplets through multiple iterations can quickly build a large amount of training data, which helps to accelerate the training process of the network. Randomly selecting azimuths within a preset range enables the network to learn better generalization capabilities in an uncertain environment, so that it can perform more robustly in practical applications. The use of a structured data organization form such as triplets helps the network understand the intrinsic connection between the front view and the left and right views, thereby improving the accuracy of perspective generation. The multi-view generator will be able to generate a series of high-quality multi-view images based on a single-view input image, which will greatly promote the process of editing and iterating three-dimensional models based on text descriptions.

[0127] In an exemplary embodiment, the data processing method of the three-dimensional model further includes:

[0128] Pre-create dense view generator;

[0129] The process of generating a dense perspective editing image matrix based on a multi-perspective editing image group includes:

[0130] The multi-view edited image group is input into the dense view generator to obtain the dense view edited image matrix.

[0131] In this embodiment, a dense view generator is pre-established, and a multi-view editing image group is input into the dense view generator, so that multiple viewpoints of selected editing images for each view (front view, left view or right view) can be quickly generated, and there is no need to perform separate viewpoint rendering on the editing image of each view, thereby improving editing efficiency. Generating the selected editing images under different viewpoints can better evaluate the performance of each view under different viewpoints, ensure the consistency of viewpoints, reduce the number of iterations, and speed up the editing process.

[0132] In an exemplary embodiment, the process of pre-creating a dense view generator includes:

[0133] Establish a dense view generation network; the dense view generation network includes a video generation diffusion network and a view adaptation module;

[0134] Acquire a first training data set obtained by rotating the camera around the target object for one cycle based on a fixed trajectory and a second training data set obtained by rotating the camera around the target object for one cycle based on a random trajectory; the target object is an object in the source three-dimensional model;

[0135] The dense view generation network is trained using the first training data set. When the training times reach a preset number, the video is fixed to generate the diffusion network, and the view adaptation module is trained using the second training data set until the training requirements are met to obtain a dense view generator.

[0136] In this embodiment, the process of establishing a dense perspective generator is described. First, a dense perspective generation network with perspective adaptation is constructed to optimize the traditional video diffusion model so that it can adapt to different perspective inputs. The network structure follows the three-dimensional UNet (U-shaped Network) structure of the video diffusion model. Each layer is a spatial convolution layer, a temporal convolution layer, a self-attention layer, and a cross-attention layer. Add a perspective adaptation module, that is, add a LoRA (Low-Rank Adaptation) layer to each cross-attention layer to construct a dense perspective generator, such as Figure 6 shown.

[0137] On a fixed trajectory, the camera is controlled to rotate around the object of the source 3D model at a regularly spaced azimuth angle at the same elevation angle as the single-view editing image (i.e., the front view editing image, the left view editing image, or the right view editing image). The fixed trajectory is represented by S = (s1,…, s g ), images at each azimuth can be obtained, and the first training data set is constructed based on the images at each azimuth on the fixed trajectory. Generally, 18 azimuths can be selected. Considering that the fixed trajectory cannot obtain information about the top or bottom of the object, this embodiment also sets a random trajectory. The spacing of the azimuth of the random trajectory is irregular, and the elevation angle of each observation is also randomly selected. The random trajectory is expressed as D=(d1,…,d g ), and then acquire images at different azimuths and elevations based on each random trajectory to obtain a second training data set. The constructed generalized dense view generation network (including the video generation diffusion network and the view adaptation module) is trained using the first training data set based on the fixed trajectory to reach a certain number of training times. Then, the fixed video generation diffusion network part remains unchanged, and the view adaptation module is trained with the second training data set generated by the random trajectory, thereby ensuring that the editing adaptation module focuses on learning the generation of free viewpoints while reducing the amount of training. After training, given an input image of any viewpoint, the generalized dense view generator can generate a corresponding set of dense view images.

[0138] In an exemplary embodiment, the data processing method of the three-dimensional model further includes:

[0139] Sampling a fixed trajectory to obtain multiple sampling points;

[0140] For each sampling point, adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle;

[0141] At least one random trajectory is obtained according to all new azimuth angles and / or new elevation angles.

[0142] In an exemplary embodiment, for each sampling point, a process of adding noise to an azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to an elevation angle corresponding to the sampling point to obtain a new elevation angle includes:

[0143] For each sampling point, random noise is added to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

[0144] In an exemplary embodiment, for each sampling point, a process of adding noise to an azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to an elevation angle corresponding to the sampling point to obtain a new elevation angle includes:

[0145] For each sampling point, a first preset interval is added to the azimuth angle corresponding to the sampling point, or a second preset interval is reduced to the azimuth angle corresponding to the sampling point, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

[0146] In this embodiment, when generating a random trajectory, the fixed trajectory may be sampled and a small random noise may be added to the azimuth. Specifically, this may be achieved by adding a small random number to the original azimuth. For example, if the original azimuth is 120°, a random number between [-10°, 10°] may be added, and the new azimuth is a value between 110° and 130°. In another embodiment, the azimuth corresponding to the sampling point may be increased by a first preset interval, or the azimuth corresponding to the sampling point may be decreased by a second preset interval, that is, for each sampling point, its current azimuth is increased by a preset value (first preset interval). For example, if the current azimuth is 30 degrees and the preset interval is 15 degrees, the new azimuth will be 45 degrees. Conversely, the azimuth of each sampling point is decreased by a preset value (second preset interval). Continuing with the above example, if it is decreased by 15 degrees, the new azimuth will be 15 degrees. In another embodiment, a random weighted combination of sine waves of different frequencies may be added to the elevation angle, that is, multiple sine waves may be superimposed in each direction, each sine wave has a different frequency and amplitude. Assume that there is a sine wave with a frequency of 0.5 Hz and a sine wave with a frequency of 1 Hz, and their amplitudes at each sampling point are random. By adjusting the parameters of the random noise and the sine wave, it can be ensured that the trajectory of the camera ends when it cycles to the same azimuth and elevation as the conditional image, that is, the scene captured by the camera in the last image is similar to the scene in the conditional image. Due to the addition of random noise and sine waves, the motion trajectory of the camera is smooth in time, avoiding obvious jumps or unnatural changes in the generated trajectory. Through different combinations of random noise and sine waves, a variety of trajectories can be generated, increasing the diversity of the generated images. In combination with the above embodiment, as another optional embodiment, when constructing a data set for a multi-view generator, this embodiment can use a dense perspective generator, that is, the front view x f The dense view generator is input to generate 18 view images of 360° around the source 3D model, and the multi-view generator is trained so that the generated multi-view images are visually consistent with the images generated by the dense view generator.

[0147] In an exemplary embodiment, after the source three-dimensional model is processed using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration, the three-dimensional model data processing method further includes:

[0148] Determine whether the current iteration meets the iteration end condition;

[0149] If not, the edited 3D model is used as the source 3D model for the next iteration;

[0150] If so, the edited three-dimensional model is used as the target three-dimensional model that meets the description of the guidance text.

[0151] In this embodiment, considering that multiple iterations are required to obtain the target three-dimensional model that meets the description of the guidance text, it is necessary to determine whether the current iteration meets the iteration end condition in this embodiment. If the iteration end condition is not met, the edited three-dimensional model is used as the source three-dimensional model for the next iteration. If the iteration end condition is met, the edited three-dimensional model of the current iteration is directly output as the target three-dimensional model. In this embodiment, the determination of the iteration end condition avoids unnecessary excessive iterations, saves computing resources and time, and improves editing efficiency. On the other hand, it can ensure that the output three-dimensional model meets the description of the guidance text.

[0152] In an exemplary embodiment, the process of processing the source three-dimensional model using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration includes:

[0153] determining a first reconstruction loss value using the edited image matrix;

[0154] The source three-dimensional model is adjusted based on the first reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

[0155] In this embodiment, the first reconstruction loss value is used to characterize the difference between the image of the edited image matrix and the reference image obtained by rendering the source three-dimensional model at the same perspective. By adjusting the parameters of the source three-dimensional model according to the first reconstruction loss value, the edited three-dimensional model corresponding to the current iteration can be obtained.

[0156] Specifically, the edited 3D model is still represented by a 3D Gaussian. The edited 3D model g e ={g e1 , …, g en}, where the i-th three-dimensional Gaussian point g ei ={μ e , e , c e , a e}, i∈{1,…,N}, Indicates the location of the center point, Represents the covariance matrix, and the color of each pixel is given by Indicates that transparency , then the three-dimensional Gaussian expression is:

[0157] ;

[0158] Where x represents the distance between the query point and the center point.

[0159] Then, based on the rendered source 3D model single-view image I e The color of a pixel is:

[0160] , .

[0161] It can be understood that if the current iteration is the first iteration, before adjusting the source three-dimensional model based on the first reconstruction loss value, it also includes performing an initialization operation on the edited three-dimensional model.

[0162] In an exemplary embodiment, the process of determining the first reconstruction loss value using the edited image matrix includes:

[0163] The first reconstruction loss value is calculated based on the first reconstruction loss function. The first reconstruction loss function is: ;

[0164] Among them, L rec is the first reconstruction loss value, I e For any edit image in the edit image matrix, Based on I e The first reference image is obtained by rendering the source 3D model from the same viewing angle, M is the editing mask from the 3D editing frame specified by the user to the editing image, is the element-wise multiplication sign.

[0165] Specifically, the first reconstruction loss function is used to calculate the sum of squares of pixel differences in a specified area between the edited image and the first reference image in the edited image matrix under the same viewing angle, which gives a quantitative measure of the impact of the editing operation on the image. This value is called the first reconstruction loss because it reflects the accuracy of the reconstruction (or editing) process. In the optimization process, minimizing this loss value can help find a better editing strategy or parameter set, thereby producing an editing result that is closer to the expected effect. For M, M is an edit mask used to indicate which pixels are in the edited image I e Which pixels should be considered in the calculation of the first reconstruction loss value, and which pixels should not be considered. This mask is specified by the user, and it represents the mapping of the three-dimensional area that the user wants to edit in the edited image. Specifically, M is a binary image, and pixels with a value of 1 in M ​​indicate that these pixels should be included in the scope of influence of the editing operation, that is, the editing effect of these pixels should be considered in the calculation of the first reconstruction loss value. Pixels with a value of 0 in M ​​indicate that these pixels should not be affected by the editing operation, that is, the editing effect of these pixels should be ignored and not included in the calculation of the first reconstruction loss value. Considering that the first reconstruction loss function calculates the edited image I e and reference image However, this difference is only calculated in the area specified by the edit mask M, that is, only the edited image I corresponding to the pixels with a value of 1 in M e and reference image The difference will be considered in Lrec The advantage of this method is that it allows users to specify the scope of influence of the editing operation, thereby ensuring that the editing operation takes effect only in the area that the user cares about, rather than being applied uniformly to the entire image. This can more effectively control the effect of the editing and improve the accuracy and efficiency of the editing.

[0166] In an exemplary embodiment, the process of adjusting the source 3D model based on the first reconstruction loss value to obtain the edited 3D model corresponding to the current iteration includes:

[0167] The source 3D model is adjusted according to the first reconstruction loss value to obtain a rough edited 3D model;

[0168] A pseudo-truth image matrix is ​​obtained based on a rough edited 3D model and a user-specified 3D edit box;

[0169] Determining a second reconstruction loss value using the pseudo-true value image matrix;

[0170] The source three-dimensional model is adjusted based on the second reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

[0171] In this embodiment, a two-stage three-dimensional Gaussian reconstruction method is used to reconstruct the edited three-dimensional model according to the given dense perspective image editing matrix. Specifically, the source three-dimensional model is first adjusted by the first reconstruction loss value to obtain a rough edited three-dimensional model, and a pseudo-true value image matrix is ​​generated based on the rough edited three-dimensional model and the user-specified three-dimensional editing box. It can be understood that the pseudo-true value image matrix contains the image information of the user-specified three-dimensional editing box, which is obtained based on the rendering of the rough edited three-dimensional model and is used for the subsequent calculation of the second reconstruction loss value. The second reconstruction loss value is determined using the pseudo-true value image matrix, and the source three-dimensional model is adjusted based on the second reconstruction loss value to obtain the edited three-dimensional model corresponding to the current iteration, thereby improving the quality of three-dimensional Gaussian reconstruction. Through the above two-stage three-dimensional Gaussian reconstruction method, the quality of three-dimensional model reconstruction can be gradually improved, so that the edited three-dimensional model is more in line with the user's guidance description. The source three-dimensional model is adjusted using two reconstruction loss values, and each adjustment is based on the result of the previous time, so as to gradually approach the ideal editing effect.

[0172] In an exemplary embodiment, the process of determining the second reconstruction loss value using the pseudo-true value image matrix includes:

[0173] The second reconstruction loss value is calculated based on the second reconstruction loss function. The second reconstruction loss function is: Among them, L rec ' is the second reconstruction loss value, I r is any pseudo-true image in the pseudo-true image matrix, Based on I rThe second reference image is rendered from the same viewing angle to the 3D edited model in the optimization phase.

[0174] Specifically, we first calculate the pixel value difference between the pseudo-true value image and the corresponding second reference image in the pseudo-true value image under the same viewing angle, square the difference of each pixel, and obtain an error matrix. This process will amplify the larger difference, so that more obvious errors have a larger weight in the loss function. Finally, by averaging the entire error matrix, we get the second loss value L rec '.

[0175] In an exemplary embodiment, the process of obtaining a pseudo-truth image matrix based on a coarse edited 3D model and a user-specified 3D edit box includes:

[0176] For each image pose, a first image corresponding to the image pose is obtained by rendering using a rough edited 3D model, a second image corresponding to the image pose is obtained by rendering using a source 3D model, an edit region mask is obtained based on the rough edited 3D model and a 3D edit frame specified by a user, and a pseudo-truth image corresponding to the image pose is obtained according to the first image, the second image, and the edit region mask;

[0177] Based on the pseudo-truth images corresponding to the poses of the multiple images, a pseudo-truth edited image matrix is ​​obtained.

[0178] The process of obtaining a pseudo-true value image corresponding to the image posture according to the first image, the second image, and the edited region mask includes:

[0179] Based on the first relational expression, the pseudo-true value image corresponding to the image posture is obtained. The first relational expression is: ;

[0180] Among them, I r is the pseudo-truth image corresponding to any image posture, is the first image, I s For the second image, To edit the area mask, is the element-wise multiplication sign.

[0181] In this embodiment, the first image is obtained by rendering the rough edited 3D model. , and then use the source 3D model to render the second image I s , and then use the rough edited 3D model and the user-defined 3D edit box B to determine which 3D Gaussian points are in the 3D edit box, and render them to obtain a more accurate edit area mask , and then based on the pseudo-true value image under each image posture of the first relationship, continuously change the rendered image posture, repeat the above steps, obtain the pseudo-true value editing image matrix, and then use the reconstruction loss training to obtain the optimized 3D editing model.

[0182] As an optional embodiment, an editing platform that is easy for users to operate is also provided, which is used to respond to the operation instructions and guidance texts input by users, and generate a target 3D model that conforms to the description of the guidance text based on the source 3D model. Specifically, in order to assist users in completing the editing, a Web-based 3D model editing platform is constructed. The interface diagram is shown in FIG. Figure 7 As shown, click "Please enter the source 3D model", a 3D model input box will pop up, which supports the input of a 3D Gaussian model; click "Please enter the editing text prompt", a text input box will pop up, enter the text; after that, the background will automatically perform single-view image editing, two-stage editing image matrix generation and 3D editing model training, and pop up a prompt when the training is completed in each stage; finally, the 3D editing model is output.

[0183] In summary, combined Figure 8 , this application proposes a 3D model editing scheme based on dense perspective generation, which integrates single-perspective editing and dense perspective generation, realizes multi-perspective consistency editing in a forward process, and improves editing quality and speed. This application also proposes a perspective-adaptive dense perspective generation method to realize dense perspective generation for any input perspective. Based on the pre-trained video generator, a dense perspective generator is constructed by adding a perspective adaptation module. A data set is constructed according to fixed trajectories and random trajectories, and the dense perspective generator is trained so that it can generate a 360° dense perspective image around the object for any input perspective. This application also proposes a two-stage dense perspective editing method to generate a dense perspective editing image matrix to improve the consistency of editing between different perspectives. Input a single-perspective editing picture, use a restricted multi-perspective generator to obtain multiple perspective editing pictures (generally 4), and then use a perspective-adaptive dense perspective generator to generate a dense perspective editing image matrix. The present application also proposes a restricted multi-view generation method, constructs a restricted multi-view generator, and uses a pre-trained video generator for distillation, thereby ensuring the visual consistency between the edited images obtained by the multi-view generator and the edited images obtained by the video generator. Moreover, to ensure that the restricted multi-view generates images, only multiple viewpoints within 45° to the left and right of the front view are generated, further ensuring the consistency of the edited multi-view. The present application also proposes a two-stage three-dimensional Gaussian reconstruction method, which reconstructs the edited three-dimensional model based on a given dense perspective image editing matrix, thereby improving the quality of three-dimensional Gaussian reconstruction.

[0184] Second, please refer to Fig. 9 The present application also provides a three-dimensional model data processing system, including:

[0185] A first determination module 11 is used to determine a source 3D model corresponding to a current iteration, and obtain a current single-view edited image of the source 3D model edited according to the guidance text;

[0186] A first generating module 12, configured to generate a multi-view editing image group based on the current single-view editing image, wherein the multi-view editing image group includes an editing image of a front view, an editing image of a left view, and an editing image of a right view;

[0187] A second generating module 13, configured to generate a dense perspective editing image matrix based on the multi-perspective editing image group;

[0188] The processing module 14 is used to process the source three-dimensional model using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration.

[0189] It can be seen that in this embodiment, after the source three-dimensional model is edited according to the guidance text to obtain the current single-perspective editing image, a multi-perspective editing image group corresponding to the current single-perspective editing image is first generated, including a front view, a left view and a right view, and then a dense perspective editing image matrix is ​​generated based on the editing images of the front view, the left view and the right view, so as to improve the consistency of editing between different perspectives, and reconstruct the edited three-dimensional model based on the editing image matrix, thereby improving the editing quality and editing efficiency of the three-dimensional model.

[0190] In an exemplary embodiment, the data processing system for the three-dimensional model further includes:

[0191] A first pre-creation module, used to pre-create a multi-view generator;

[0192] The process of generating a multi-view editing image group based on the current single-view editing image includes:

[0193] The current single-view edited image is input into the multi-view generator to obtain a multi-view edited image group.

[0194] In an exemplary embodiment, the process of pre-creating a multi-view generator includes:

[0195] Construct a multi-view generation network;

[0196] Generate multiple selected editing images at multiple viewing angles based on the front view editing image, where the multiple selected editing images correspond to different azimuth angles;

[0197] Determine a plurality of left view editing images and a plurality of right view editing images; the left view editing images are to-be-selected editing images whose azimuth angles are within the left view selection range, and the right view editing images are to-be-selected editing images whose azimuth angles are within the right view selection range;

[0198] A plurality of triplets are formed based on the front view editing image, the plurality of left view editing images and the plurality of right view editing images; each triplet includes the front view editing image, any left view editing image and any right view editing image, and the left view editing images or the right view editing images in any two triplets are different;

[0199] A multi-view generation network is trained based on multiple triplets to obtain a multi-view generator.

[0200] In an exemplary embodiment, the data processing system for the three-dimensional model further includes:

[0201] A second pre-creation module, used to pre-create a dense perspective generator;

[0202] The process of generating a dense perspective editing image matrix based on a multi-perspective editing image group includes:

[0203] The multi-view edited image group is input into the dense view generator to obtain the dense view edited image matrix.

[0204] In an exemplary embodiment, the process of pre-creating a dense view generator includes:

[0205] Establish a dense view generation network; the dense view generation network includes a video generation diffusion network and a view adaptation module;

[0206] Acquire a first training data set obtained by rotating the camera around the target object for one cycle based on a fixed trajectory and a second training data set obtained by rotating the camera around the target object for one cycle based on a random trajectory; the target object is an object in the source three-dimensional model;

[0207] The dense view generation network is trained using the first training data set. When the training times reach a preset number, the video is fixed to generate the diffusion network, and the view adaptation module is trained using the second training data set until the training requirements are met to obtain a dense view generator.

[0208] In an exemplary embodiment, the data processing system for the three-dimensional model further includes:

[0209] A sampling module, used to sample a fixed trajectory to obtain multiple sampling points;

[0210] A second acquisition module is used for adding noise to the azimuth angle corresponding to each sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle;

[0211] The third acquisition module is used to obtain at least one random trajectory according to all new azimuth angles and / or new elevation angles.

[0212] In an exemplary embodiment, for each sampling point, a process of adding noise to an azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to an elevation angle corresponding to the sampling point to obtain a new elevation angle includes:

[0213] For each sampling point, random noise is added to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

[0214] In an exemplary embodiment, for each sampling point, a process of adding noise to an azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to an elevation angle corresponding to the sampling point to obtain a new elevation angle includes:

[0215] For each sampling point, a first preset interval is added to the azimuth angle corresponding to the sampling point, or a second preset interval is reduced to the azimuth angle corresponding to the sampling point, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

[0216] In an exemplary embodiment, after the source three-dimensional model is processed using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration, the data processing system of the three-dimensional model further includes:

[0217] Determine whether the current iteration meets the iteration end condition;

[0218] If not, the edited 3D model is used as the source 3D model for the next iteration;

[0219] If so, the edited three-dimensional model is used as the target three-dimensional model that meets the description of the guidance text.

[0220] In an exemplary embodiment, the process of obtaining a current single-view edited image of a source 3D model edited according to a guide text includes:

[0221] Obtain a current single-view image of the source 3D model at a selected viewpoint of the current iteration;

[0222] The current single-view image is edited according to the guidance text to obtain the current single-view edited image.

[0223] In an exemplary embodiment, the process of obtaining a current single-view image of the source 3D model at a selected view angle of a current iteration includes:

[0224] Based on the selected viewing angle of the current iteration, determine the query point corresponding to each pixel point in the three-dimensional space of the source three-dimensional model;

[0225] For each query point, the color of the pixel corresponding to the query point is obtained according to the three-dimensional Gaussian function and the distance between the query point and the center point of the source three-dimensional model;

[0226] The current single-view image of the source 3D model at the selected viewpoint is obtained based on the colors of all pixels.

[0227] In an exemplary embodiment, the process of processing the source three-dimensional model using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration includes:

[0228] determining a first reconstruction loss value using the edited image matrix;

[0229] The source three-dimensional model is adjusted based on the first reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

[0230] In an exemplary embodiment, the process of determining the first reconstruction loss value using the edited image matrix includes:

[0231] The first reconstruction loss value is calculated based on the first reconstruction loss function. The first reconstruction loss function is: ;

[0232] Among them, L rec is the first reconstruction loss value, I e For any edit image in the edit image matrix, Based on I e The first reference image is obtained by rendering the source 3D model from the same viewing angle, M is the editing mask from the 3D editing frame specified by the user to the editing image, is the element-wise multiplication sign.

[0233] In an exemplary embodiment, the process of adjusting the source 3D model based on the first reconstruction loss value to obtain the edited 3D model corresponding to the current iteration includes:

[0234] The source 3D model is adjusted according to the first reconstruction loss value to obtain a rough edited 3D model;

[0235] A pseudo-truth image matrix is ​​obtained based on a rough edited 3D model and a user-specified 3D edit box;

[0236] Determining a second reconstruction loss value using the pseudo-true value image matrix;

[0237] The source three-dimensional model is adjusted based on the second reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

[0238] In an exemplary embodiment, the process of determining the second reconstruction loss value using the pseudo-true value image matrix includes:

[0239] The second reconstruction loss value is calculated based on the second reconstruction loss function. The second reconstruction loss function is: ;

[0240] Among them, L rec' is the second reconstruction loss value, I r is any pseudo-true image in the pseudo-true image matrix, Based on I r The second reference image is rendered from the same viewing angle to the 3D edited model in the optimization phase.

[0241] In an exemplary embodiment, the process of obtaining a pseudo-truth image matrix based on a coarse edited 3D model and a user-specified 3D edit box includes:

[0242] For each image pose, a first image corresponding to the image pose is obtained by rendering using a rough edited 3D model, a second image corresponding to the image pose is obtained by rendering using a source 3D model, an edit region mask is obtained based on the rough edited 3D model and a 3D edit frame specified by a user, and a pseudo-truth image corresponding to the image pose is obtained according to the first image, the second image, and the edit region mask;

[0243] Based on the pseudo-truth images corresponding to the poses of the multiple images, a pseudo-truth edited image matrix is ​​obtained.

[0244] In an exemplary embodiment, the process of obtaining a pseudo-truth image corresponding to the image posture according to the first image, the second image, and the edited region mask includes:

[0245] Based on the first relational expression, the pseudo-true value image corresponding to the image posture is obtained. The first relational expression is: ;

[0246] Among them, I r is the pseudo-truth image corresponding to any image posture, is the first image, I s For the second image, To edit the area mask, is the element-wise multiplication sign.

[0247] In a third aspect, the present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of the three-dimensional model data processing method described in any of the above embodiments.

[0248] For an introduction to a computer program product provided by the present application, please refer to the above embodiments, and the present application will not go into details here.

[0249] A computer program product provided in the present application has the same beneficial effects as the above-mentioned three-dimensional model data processing method.

[0250] Fourth, please refer to Fig.10 , the present application also provides an electronic device, including:

[0251] A memory 21, used for storing computer programs;

[0252] The processor 22 is used to implement the steps of the three-dimensional model data processing method described in any of the above embodiments when executing the computer program.

[0253] The electronic device also includes:

[0254] The input interface 23 is connected to the processor 22 via the communication bus 26, and is used to obtain the computer programs, parameters and instructions imported from the outside, and save them in the memory 21 under the control of the processor 22. The input interface can be connected to an input device to receive parameters or instructions manually input by the user. The input device can be a touch layer covered on the display screen, or a key, trackball or touchpad set on the terminal housing.

[0255] The display unit 24 is connected to the processor 22 via the communication bus 26 and is used to display the data sent by the processor 22. The display unit can be a liquid crystal display or an electronic ink display.

[0256] The network port 25 is connected to the processor 22 via the communication bus 26, and is used to communicate with various external terminal devices. The communication technology used in the communication connection can be a wired communication technology or a wireless communication technology, such as mobile high-definition link technology, universal serial bus, high-definition multimedia interface, wireless fidelity technology, Bluetooth communication technology, low-power Bluetooth communication technology, communication technology based on IEEE802.11s, etc.

[0257] For an introduction to an electronic device provided in the present application, please refer to the above-mentioned embodiments, and the present application will not go into details here.

[0258] The electronic device provided in the present application has the same beneficial effects as the above-mentioned three-dimensional model data processing method.

[0259] Fifth, please refer to Fig.11 The present application also provides a non-volatile storage medium 30, on which a computer program 31 is stored. When the computer program 31 is executed by a processor, the steps of the three-dimensional model data processing method described in any of the above embodiments are implemented.

[0260] Among them, the non-volatile storage medium 30 may include: ROM (Read-only memory), PROM (Programmable read-only memory), EAROM (Electrically alterable read only memory), EPROM (Erasable programmable read only memory), EEPROM (Electrically erasable programmable read only memory), Flash memory and other media that can store program codes.

[0261] For an introduction to a non-volatile storage medium provided in the present application, please refer to the above-mentioned embodiments, and the present application will not go into details here.

[0262] The non-volatile storage medium provided in the present application has the same beneficial effects as the above-mentioned three-dimensional model data processing method.

[0263] It should also be noted that, in this specification, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "comprises", "comprising" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the statement "comprising a ..." does not exclude the presence of other identical elements in the process, method, article or device including the element.

[0264] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present invention. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but rather to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A data processing method for a three-dimensional model, characterized in that: include: Determine a source three-dimensional model corresponding to a current iteration, and obtain a current single-view edited image of the source three-dimensional model edited according to the guidance text; Generate a multi-view editing image group based on the current single-view editing image, the multi-view editing image group including an editing image of a front view of the current single-view editing image, an editing image of a left view of the current single-view editing image, and an editing image of a right view of the current single-view editing image; Generate a dense perspective editing image matrix based on the multi-perspective editing image group; Processing the source three-dimensional model using the edited image matrix to obtain an edited three-dimensional model corresponding to the current iteration; The data processing method of the three-dimensional model also includes: Establishing a dense view generation network; the dense view generation network includes a video generation diffusion network and a view adaptation module; Acquire a first training data set obtained by rotating a camera around a target object based on a fixed trajectory and a second training data set obtained by rotating a camera around the target object based on a random trajectory; the target object is an object in the source three-dimensional model; The dense view generation network is trained using the first training data set. When the number of training times reaches a preset number, the video generation diffusion network is fixed, and the view adaptation module is trained using the second training data set until the training requirements are met, thereby obtaining a dense view generator. The process of generating a dense perspective editing image matrix based on the multi-perspective editing image group includes: The multi-view edited image group is input into the dense view generator to obtain the dense view edited image matrix.

2. The data processing method of the three-dimensional model according to claim 1, characterized in that: The data processing method of the three-dimensional model also includes: Pre-created multi-view generator; The process of generating a multi-view editing image group based on the current single-view editing image comprises: The current single-view editing image is input into the multi-view generator to obtain a multi-view editing image group.

3. The data processing method of the three-dimensional model according to claim 2, characterized in that: The process of pre-creating a multi-view generator includes: Construct a multi-view generation network; Generate multiple to-be-selected editing images of different viewing angles based on the front view editing image, wherein the multiple to-be-selected editing images correspond to different azimuth angles; Determine a plurality of left view editing images and a plurality of right view editing images; the left view editing images are to-be-selected editing images whose azimuth angles are within the left view selection range, and the right view editing images are to-be-selected editing images whose azimuth angles are within the right view selection range; A plurality of triplets are formed based on the front view editing image, a plurality of the left view editing images and a plurality of the right view editing images; each of the triplets includes the front view editing image, any one of the left view editing images and any one of the right view editing images, and the left view editing images or the right view editing images in any two of the triplets are different; The multi-view generation network is trained according to the plurality of triplets to obtain a multi-view generator.

4. The data processing method of the three-dimensional model according to claim 1, characterized in that: The data processing method of the three-dimensional model also includes: Sampling the fixed trajectory to obtain a plurality of sampling points; For each of the sampling points, adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle; At least one random trajectory is obtained according to all the new azimuth angles and / or the new elevation angles.

5. The data processing method of the three-dimensional model according to claim 4, characterized in that: For each of the sampling points, a process of adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle includes: For each of the sampling points, random noise is added to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

6. The data processing method of the three-dimensional model according to claim 4, characterized in that: For each of the sampling points, a process of adding noise to the azimuth angle corresponding to the sampling point to obtain a new azimuth angle, and / or adding noise to the elevation angle corresponding to the sampling point to obtain a new elevation angle includes: For each of the sampling points, a first preset interval is added to the azimuth angle corresponding to the sampling point, or a second preset interval is reduced to the azimuth angle corresponding to the sampling point, and / or a random weighted combination of sine waves of different frequencies is added to the elevation angle corresponding to the sampling point to obtain a new elevation angle.

7. The data processing method of the three-dimensional model according to claim 1, characterized in that: After the source three-dimensional model is processed by using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration, the data processing method of the three-dimensional model further includes: Determine whether the current iteration satisfies the iteration end condition; If not, the edited three-dimensional model is used as the source three-dimensional model of the next iteration for the next iteration; If so, the edited three-dimensional model is used as a target three-dimensional model that conforms to the description of the guidance text.

8. The data processing method of the three-dimensional model according to claim 1, characterized in that: The process of obtaining the current single-view edited image of the source 3D model edited according to the guidance text includes: Acquire a current single-view image of the source three-dimensional model at a selected view angle of the current iteration; The current single-view image is edited according to the guide text to obtain a current single-view edited image.

9. The data processing method of the three-dimensional model according to claim 8, characterized in that: The process of obtaining the current single-view image of the source 3D model at the selected view angle of the current iteration includes: Determine, based on the selected viewing angle of the current iteration, a query point corresponding to each pixel point in the three-dimensional space of the source three-dimensional model; For each query point, obtaining the color of the pixel point corresponding to the query point according to a three-dimensional Gaussian function and a distance between the query point and a center point of the source three-dimensional model; A current single-view image of the source three-dimensional model at a selected viewing angle is obtained based on the colors of all the pixel points.

10. The data processing method of the three-dimensional model according to any one of claims 1 to 9, characterized in that: The process of processing the source three-dimensional model by using the edited image matrix to obtain the edited three-dimensional model corresponding to the current iteration includes: Determining a first reconstruction loss value using the edited image matrix; The source three-dimensional model is adjusted based on the first reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

11. The three-dimensional model data processing method according to claim 10, characterized in that: The process of determining the first reconstruction loss value by using the edited image matrix includes: A first reconstruction loss value is calculated based on a first reconstruction loss function, where the first reconstruction loss function is: ; Among them, L rec is the first reconstruction loss value, I e is any edit image in the edit image matrix, Based on I e The first reference image is obtained by rendering the source 3D model at the same viewing angle, M is an editing mask from a 3D editing frame specified by a user to the editing image, is the element-wise multiplication sign.

12. The three-dimensional model data processing method according to claim 10, characterized in that: The process of adjusting the source three-dimensional model based on the first reconstruction loss value to obtain the edited three-dimensional model corresponding to the current iteration includes: Adjusting the source 3D model according to the first reconstruction loss value to obtain a rough edited 3D model; Obtaining a pseudo-truth image matrix based on the rough edited three-dimensional model and the three-dimensional edit frame specified by the user; Determining a second reconstruction loss value using the pseudo-true value image matrix; The source three-dimensional model is adjusted based on the second reconstruction loss value to obtain an edited three-dimensional model corresponding to the current iteration.

13. The data processing method of the three-dimensional model according to claim 12, characterized in that: The process of determining the second reconstruction loss value using the pseudo-true value image matrix includes: A second reconstruction loss value is calculated based on a second reconstruction loss function, where the second reconstruction loss function is: ; Among them, L rec ' is the second reconstruction loss value, I r is any pseudo-true value image in the pseudo-true value image matrix, Based on I r The second reference image is rendered from the same viewing angle to the 3D edited model in the optimization phase.

14. The data processing method of the three-dimensional model according to claim 12, characterized in that: The process of obtaining a pseudo-true value image matrix based on the rough edited 3D model and the user-specified 3D edit frame includes: For each image pose, a first image corresponding to the image pose is obtained by rendering using the rough edited 3D model, a second image corresponding to the image pose is obtained by rendering using the source 3D model, an edit region mask is obtained based on the rough edited 3D model and a 3D edit frame specified by a user, and a pseudo-truth image corresponding to the image pose is obtained according to the first image, the second image, and the edit region mask; Based on the pseudo-true value images corresponding to the plurality of image postures, a pseudo-true value edited image matrix is ​​obtained.

15. The data processing method of the three-dimensional model according to claim 14, characterized in that: The process of obtaining a pseudo true value image corresponding to the image posture according to the first image, the second image, and the edited area mask includes: A pseudo true value image corresponding to the image posture is obtained based on a first relational expression, wherein the first relational expression is: ; Among them, I r is the pseudo-truth image corresponding to any of the image poses, is the first image, I s For the second image, is the edit region mask, is the element-wise multiplication sign.

16. A three-dimensional model data processing system, characterized in that: include: A first determination module is used to determine a source three-dimensional model corresponding to a current iteration, and obtain a current single-view edited image of the source three-dimensional model edited according to the guidance text; A first generating module, configured to generate a multi-view editing image group based on the current single-view editing image, wherein the multi-view editing image group includes an editing image of a front view of the current single-view editing image, an editing image of a left view of the current single-view editing image, and an editing image of a right view of the current single-view editing image; A second generating module, configured to generate a dense perspective editing image matrix based on the multi-perspective editing image group; A processing module, used for processing the source three-dimensional model using the edited image matrix to obtain an edited three-dimensional model corresponding to the current iteration; The data processing method of the three-dimensional model also includes: A second pre-creation module is used to establish a dense perspective generation network; the dense perspective generation network includes a video generation diffusion network and a perspective adaptation module; a first training data set obtained by a camera rotating around a target object based on a fixed trajectory and a second training data set obtained by a camera rotating around the target object based on a random trajectory; the target object is an object in the source three-dimensional model; the dense perspective generation network is trained using the first training data set, and when the number of training times reaches a preset number, the video generation diffusion network is fixed, and the perspective adaptation module is trained using the second training data set until the training requirements are met, thereby obtaining a dense perspective generator; The process of generating a dense perspective editing image matrix based on the multi-perspective editing image group includes: The multi-view edited image group is input into the dense view generator to obtain the dense view edited image matrix.

17. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the three-dimensional model data processing method described in any one of claims 1-15 are implemented.

18. An electronic device, characterized in that: include: Memory for storing computer programs; A processor, used to implement the steps of the three-dimensional model data processing method according to any one of claims 1 to 15 when executing the computer program.

19. A non-volatile storage medium, characterized in that: The non-volatile storage medium stores a computer program, which, when executed by a processor, implements the steps of the three-dimensional model data processing method according to any one of claims 1 to 15.

Images