Methods, apparatus, storage media and computer equipment for generating 3D object models

Abstract

The method, apparatus, storage medium, and computer equipment for generating three-dimensional object models provided in this application include: acquiring a single view and determining multiple views, predicting the normal maps of the multiple views to obtain predicted normal maps, iteratively updating a determined initial three-dimensional model to obtain a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model, then adding noise to the multiple views, and combining the single view and the rendered normal map to generate new multiple views, continuing to execute the steps of determining multiple views and subsequent steps until the latest multiple views and the latest target three-dimensional model are obtained when preset conditions are met. During this process, the rendered normal map obtained based on the target three-dimensional model has spatial consistency, so that subsequent generation of two-dimensional multiple views conforming to the spatial consistency of the three-dimensional structure is possible, and this multiple view is used as the initial data for the next iteration, continuously improving the consistency and accuracy of the target three-dimensional model through iteration.

Description

Technical Field

[0001] This application relates to the field of 3D modeling technology, and in particular to a method, apparatus, storage medium and computer equipment for generating 3D object models. Background Technology

[0002] With the rapid development of information technology, 3D object generation technology has been widely applied in various fields such as virtual reality, augmented reality, game development, film and television special effects, and product design. Traditional 3D modeling methods typically rely on specialized software and manual operation, which is time-consuming, labor-intensive, and costly. In recent years, with the advancement of deep learning technology, automatic or semi-automatic 3D object model generation methods have gradually become the mainstream generation approach. This method can reconstruct and restore the 3D structure of objects from 2D images, greatly improving the efficiency and quality of 3D modeling.

[0003] However, in the process of reconstructing a 3D object model based on a 2D image, since each step of the process is based on a 2D image, it is difficult to guarantee the consistency of the generated 3D object model from different perspectives, resulting in low accuracy of the generated 3D object model. Summary of the Invention

[0004] The purpose of this application is to at least solve one of the aforementioned technical defects, particularly the technical defect in the prior art where, in the process of reconstructing a three-dimensional object model based on a two-dimensional image, since each step of the process is based on a two-dimensional image, it is difficult to guarantee the consistency of the generated three-dimensional object model under various viewpoints, resulting in a low accuracy of the generated three-dimensional object model.

[0005] In a first aspect, this application provides a method for generating a three-dimensional object model, the method comprising:

[0006] Obtain a single view of the desired 3D object;

[0007] Determine multiple views and predict the normal maps of the multiple views to obtain predicted normal maps;

[0008] An initial 3D model is determined, and the initial 3D model is iteratively updated according to the predicted normal map to obtain a target 3D model. Then, the target 3D model is rendered to obtain a rendering normal map corresponding to the target 3D model.

[0009] Noise is superimposed on the multi-view, and a new multi-view is generated based on the multi-view with superimposed noise, the single view, and the rendering normal map. The steps of determining the multi-view and subsequent steps are continued until the preset conditions are met to obtain the latest multi-view and the latest target 3D model.

[0010] Based on the latest multi-view model, texture mapping is performed on the latest target 3D model to obtain a 3D object model.

[0011] In one embodiment, determining multiple views and predicting the normal maps of the multiple views to obtain predicted normal maps includes:

[0012] Determine if a newest multiview exists. If it does, use the newest multiview as the determined multiview.

[0013] If it does not exist, information is extracted from the single view to obtain description information and key information. The description information, the key information and the preset noise ratio are then input into the preset multi-view diffusion model to generate a multi-view corresponding to the single view. The generated multi-view is then used as the determined multi-view.

[0014] The multi-view is input into a preset normal prediction network to predict the normal map of the multi-view based on the normal prediction network, thereby obtaining the predicted normal map; wherein, the normal prediction network is trained using the multi-view rendered from a real 3D model and its corresponding normal map as training data.

[0015] In one embodiment, determining an initial 3D model and iteratively updating the initial 3D model based on the predicted normal map includes:

[0016] In one iteration, the initial 3D model corresponding to this iteration is determined, and the initial 3D model is rendered to obtain the rendering normal map corresponding to the initial 3D model;

[0017] Based on the predicted normal map and the rendered normal map of the initial 3D model, a loss function is constructed for this iteration, and the initial 3D model is updated according to the loss function. The updated initial 3D model is then used as the corresponding initial 3D model in the next iteration.

[0018] Determine whether the current iteration round has reached the preset iteration round. If not, proceed to the next iteration round. If it has, determine the initial 3D model completed by the iteration as the target 3D model.

[0019] In one embodiment, rendering the target 3D model to obtain a rendering normal map corresponding to the target 3D model includes:

[0020] Calculate the normal vector corresponding to each vertex of the target 3D model to obtain the normal vector corresponding to the target 3D model;

[0021] The target 3D model and its corresponding normal vector are projected onto a 2D plane to obtain a normal projection map, and the normal projection map is rendered to obtain a rendered normal map.

[0022] In one embodiment, the step of adding noise to the multi-view and generating a new multi-view based on the multi-view with added noise, the single view, and the rendered normal map includes:

[0023] Determine the noise ratio to be added in this round, and superimpose noise onto the multi-view according to the noise ratio; wherein, the noise ratio superimposed in each round decreases as the number of rounds increases;

[0024] Obtain a preset multi-view diffusion model and extract information from the single view to obtain single-view information;

[0025] The noise-stamped multi-view, the single-view information, and the rendering normal map are input into the multi-view diffusion model, so that the multi-view diffusion model generates a multi-view that matches the single-view information and the rendering normal map based on the noise-stamped multi-view, thereby obtaining a new multi-view.

[0026] In one embodiment, determining the proportion of noise to be added in this round includes:

[0027] When the target 3D model is obtained, the loss function value used to form the target 3D model is acquired;

[0028] Obtain a preset mapping function and input the loss function value into the mapping function to determine the proportion of noise to be added in this round.

[0029] In one embodiment, the step of performing texture mapping on the latest target 3D model based on the latest multi-view to obtain a 3D object model includes:

[0030] Get the latest pixel information of each view in the multi-view;

[0031] The pixel information of each view is mapped to the corresponding position of the latest target 3D model in turn to obtain the 3D model to be optimized.

[0032] Determine whether the 3D model to be optimized has a blind spot. If it does not, then the 3D model to be optimized is determined to be a 3D object model.

[0033] If it exists, the texture features of the surrounding area of ​​the blind spot in the 3D model to be optimized are obtained, and the texture of the blind spot in the 3D model to be optimized is filled according to the texture features to obtain the 3D object model.

[0034] Secondly, this application provides a three-dimensional object model generation apparatus, the apparatus comprising:

[0035] The single-view acquisition module is used to acquire a single view of the 3D object to be generated.

[0036] The normal prediction module is used to determine multiple views and predict the normal maps of the multiple views to obtain predicted normal maps.

[0037] The model update module is used to determine an initial 3D model, and iteratively update the initial 3D model according to the predicted normal map to obtain a target 3D model. Then, the target 3D model is rendered to obtain a rendering normal map corresponding to the target 3D model.

[0038] The iterative optimization module is used to superimpose noise onto the multi-view and generate a new multi-view based on the multi-view after superimposing noise, the single view, and the rendered normal map. The normal prediction module, the model update module, and the iterative optimization module are then executed until the preset conditions are met to obtain the latest multi-view and the latest target 3D model.

[0039] The model generation module is used to perform texture mapping on the latest target 3D model based on the latest multi-view to obtain a 3D object model.

[0040] Thirdly, this application provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the three-dimensional object model generation method as described in any of the above embodiments.

[0041] Fourthly, this application provides a computer device, including: one or more processors, and a memory;

[0042] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, they perform the steps of the three-dimensional object model generation method as described in any of the above embodiments.

[0043] As can be seen from the above technical solutions, the embodiments of this application have the following advantages:

[0044] This application provides a method, apparatus, storage medium, and computer device for generating 3D object models. The method includes: acquiring a single view of the desired 3D object and determining multiple views, predicting the normal maps of the multiple views to obtain predicted normal maps; then determining an initial 3D model and iteratively updating the initial 3D model based on the predicted normal maps to obtain a target 3D model; rendering the target 3D model to obtain a rendered normal map corresponding to the target 3D model; then adding noise to the multiple views and generating new multiple views based on the noise-added multiple views, single views, and rendered normal maps; continuing the process of determining multiple views and subsequent steps until preset conditions are met to obtain the latest multiple views and the latest target 3D model; in this process, the target 3D model is used as an intermediate medium, and the rendered normal map obtained based on the target 3D model has spatial consistency, so that subsequent generation of 2D multiple views conforming to the spatial consistency of the 3D structure is possible, and this multiple view is used as the initial data for the next iteration, continuously improving the spatial structural consistency and model accuracy of the target 3D model through iteration; finally, texture mapping is performed on the latest target 3D model based on the latest multiple views to obtain a high-quality 3D object model. Attached Figure Description

[0045] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0046] Figure 1 A flowchart illustrating a method for generating a three-dimensional object model provided in an embodiment of this application;

[0047] Figure 2 A flowchart illustrating the method for generating a three-dimensional object model provided in an embodiment of this application;

[0048] Figure 3 A schematic diagram illustrating the process of determining multiple views and predicting the normal diagrams of the multiple views provided in this application embodiment;

[0049] Figure 4 A schematic diagram illustrating the process of determining an initial three-dimensional model and iteratively updating the initial three-dimensional model based on the predicted normal map, as provided in the embodiments of this application.

[0050] Figure 5 This is a flowchart illustrating the process of adding noise to multiple views and generating a new multiple view based on the noise-added multiple views, a single view, and a rendered normal map, as provided in the embodiments of this application.

[0051] Figure 6 A flowchart illustrating the process of determining corresponding multiple views from a single view, provided for embodiments of this application;

[0052] Figure 7 A flowchart illustrating the iterative optimization process during the generation of a 3D object model provided in this application embodiment;

[0053] Figure 8 This is a schematic diagram of the structure of a three-dimensional object model generation device provided in an embodiment of this application;

[0054] Figure 9 This is an internal structural diagram of a computer device provided in an embodiment of this application. Detailed Implementation

[0055] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0056] In one embodiment, this application provides a method for generating a three-dimensional object model. The following embodiments illustrate the application of this method to a server. It is understood that the three-dimensional object model generation method can be executed by a single server or by a server cluster consisting of multiple servers, and this application does not impose any specific limitations on this.

[0057] Specifically, the process involves predicting multiple views from a reference image, then predicting normal maps from these multiple views, and finally generating a 3D model from the normal maps. These steps are all performed at the image level. The spatial consistency of the various views in the multiple views cannot be guaranteed in the 3D structure. In other words, the resulting multiple views are not truly projected views of the 3D model; identical elements from different viewpoints may not be in the same position when mapped back to 3D space. Therefore, this approach cannot directly generate accurate meshes. Directly using these multiple views as texture sources also fails to accurately project the image into 3D. Due to insufficient spatial consistency across the multiple views, the projected textures will exhibit misalignment and overlap. Therefore, an accurate, high-quality 3D model cannot be obtained. Based on this, if... Figure 1 As shown, this application provides a method for generating a three-dimensional object model, the method comprising:

[0058] S101: Obtain a single view of the desired 3D object.

[0059] In this step, when a user needs to generate a 3D model of a certain 3D object, they can obtain a single view of the 3D object and initiate model generation commands sequentially. When the server receives the model generation command, it obtains the single view of the 3D object to be generated based on the command, so that the single view can be used as the base data for the subsequent 3D object model generation task.

[0060] It is understood that a single view refers to an RGB image obtained from observing an object from a specific angle, which is only a single angle. A single view can include a front view, a left view, a right view, etc. This application does not impose specific limitations on this, but for the sake of information richness, a front view of a three-dimensional object can be used as a single view.

[0061] S102: Determine the multiple views and predict the normal map of the multiple views to obtain the predicted normal map.

[0062] In this step, the method for determining the multi-view is determined based on the current iteration round. In the first round, since S104 has not yet been completed (i.e., no new multi-view has been generated), the multi-view is generated and determined based on the single view. In subsequent rounds, since new multi-views already exist, the latest multi-view is determined as the multi-view in this step. Next, the normals of each view in the multi-view can be predicted to obtain the predicted normal map corresponding to the multi-view.

[0063] It is understandable that the above-mentioned rounds refer to the number of rounds formed by the iterations between S102 and S106.

[0064] In this context, a multi-view image refers to an RGB image composed of the front, left, right, and rear views of a 3D object. A predicted normal map is an image composed of the normal maps of the front, left, right, and rear views of the 3D object. A normal map represents the normal to each point of the object, and the direction of these normals is represented by RGB color channels. The red, green, and blue channels in the normal map represent the X, Y, and Z components of the normal, respectively. It can be understood that normal maps can reflect the surface structure of a mesh and can also be used to enhance the surface details of a 3D model. Therefore, normal maps can be used to simulate higher levels of surface detail visually.

[0065] S103: Determine the initial 3D model, and iteratively update the initial 3D model according to the predicted normal map to obtain the target 3D model. Then, render the target 3D model to obtain the rendering normal map corresponding to the target 3D model.

[0066] In this step, the method for determining the initial 3D model is clear based on the current iteration round. In the first round, a 3D model can be initialized as the initial 3D model, for example, a 3D sphere model. In subsequent rounds, the target 3D model obtained in the previous round can be used as the initial 3D model. When the initial 3D model is determined, iterative updates are performed on the initial 3D model based on the predicted normal map and the normal map obtained from each iteration's rendering. After the iteration update is completed, the target 3D model is obtained. At this point, the target 3D model is rendered to obtain its rendered normal map.

[0067] Specifically, the aforementioned rounds refer to the number of rounds formed by the iterations between S102 and S106, and the iteration refers to the process of updating the initial 3D model. In other words, a complete iterative update process is performed on the initial 3D model in each round.

[0068] Among them, the rendered normal map refers to the normal map obtained based on the 3D model. It can be understood that both the rendered normal map and the predicted normal map are normal maps. The difference is that the former is obtained based on the 3D model, while the latter is predicted based on the 2D image. Therefore, the rendered normal map has higher spatial consistency than the predicted normal map.

[0069] S104: Add noise to the multi-view and generate a new multi-view based on the multi-view with added noise, the single view, and the rendered normal map.

[0070] In this step, noise is added to the multi-view. Then, the multi-view with superimposed noise, the single view, and the rendered normal map can be input into a preset multi-view diffusion model to generate a new multi-view. The multi-view diffusion model is used to generate the corresponding multi-view based on the input content.

[0071] It is understandable that adding noise to the multiview can control the range of variation of the final generated multiview compared to the input multiview. When the noise level is low, the variation of the generated multiview is also small. In this case, the multiview diffusion model will be more inclined to generate results that are more similar to the input multiview when making predictions.

[0072] Specifically, since the generation of new multi-views also combines single-views and rendering normal maps, the guiding role of single-views and rendering normal maps can be fully utilized, so that the generation of new multi-views can be carried out in a direction that conforms to the original data and the consistency of the model structure space.

[0073] S105: Determine whether the preset conditions are met.

[0074] The preset condition refers to the stopping condition of each round in the process of generating the 3D object model. The preset condition can be set to reach a preset round, or it can be set to the quality of the latest target 3D model reaching the expected level. This application does not impose specific restrictions on this.

[0075] S106: If not satisfied, continue with the steps to determine multiple views and subsequent steps.

[0076] If the preset conditions are not met, you can return to the step of determining multiple views and continue execution.

[0077] S107: If satisfied, obtain the latest multi-view and the latest target 3D model.

[0078] When the preset conditions are met, the latest multi-view and the latest target 3D model are obtained, that is, the multi-view and target 3D model generated in this round when the preset conditions are met.

[0079] S108: Based on the latest multi-view, perform texture mapping on the latest target 3D model to obtain a 3D object model.

[0080] In this step, texture mapping is performed on the latest target 3D model based on the latest multi-view. At this time, texture features (such as texture, color, etc.) in the multi-view can be filled into the latest target 3D model to obtain a 3D object model.

[0081] In this embodiment, through the iterations of S102-S106 described above, since the new multi-view obtained in each iteration will better conform to the spatial consistency of the 3D object than the multi-view in the previous iteration, the prediction of the normal map will also be more accurate than that obtained in the previous iteration. Based on this, the initial 3D model is optimized to obtain a more accurate target 3D model, and then the rendering normal map of the target 3D model is determined. Based on this continuous iteration process, a high-precision and high-quality target 3D model and multi-views that meet the spatial consistency requirements of the 3D object can be obtained. This makes the final generated 3D object model have excellent performance in terms of spatial consistency, thereby improving the quality and accuracy of the generated 3D object model.

[0082] In one example, such as Figure 2 As shown, Figure 2 This is a flowchart illustrating a method for generating a three-dimensional object model provided in an embodiment of this application. Figure 2 In the process, the output optimized multi-view is the new multi-view generated in S104, and the 3D mesh is the target 3D model. During the iteration process, the optimized multi-view will be used as the multi-view for the next iteration. Finally, the optimized multi-view can be used to perform texture mapping on the generated 3D mesh (the latest target 3D model) to obtain a 3D object model.

[0083] This application provides a method, apparatus, storage medium, and computer device for generating 3D object models. The method includes: acquiring a single view of the desired 3D object and determining multiple views, predicting the normal maps of the multiple views to obtain predicted normal maps; then determining an initial 3D model and iteratively updating the initial 3D model based on the predicted normal maps to obtain a target 3D model; rendering the target 3D model to obtain a rendered normal map corresponding to the target 3D model; then adding noise to the multiple views and generating new multiple views based on the noise-added multiple views, single views, and rendered normal maps; continuing the process of determining multiple views and subsequent steps until preset conditions are met to obtain the latest multiple views and the latest target 3D model; in this process, the target 3D model is used as an intermediate medium, and the rendered normal map obtained based on the target 3D model has spatial consistency, so that subsequent generation of 2D multiple views conforming to the spatial consistency of the 3D structure is possible, and this multiple view is used as the initial data for the next iteration, continuously improving the spatial structural consistency and model accuracy of the target 3D model through iteration; finally, texture mapping is performed on the latest target 3D model based on the latest multiple views to obtain a high-quality 3D object model.

[0084] like Figure 3 As shown, in one embodiment, determining multiple views and predicting the normal maps of the multiple views to obtain predicted normal maps includes:

[0085] S201: Determine if a new multiview exists.

[0086] In this step, by determining whether a new multi-view exists, we can know whether the current 3D object model generation process has gone through S104. If it does, then a new multi-view exists; if it does not, then a new multi-view does not exist.

[0087] S202: If it exists, then the latest multiview is used as the determined multiview.

[0088] When a latest multiview exists, that latest multiview can be used as the determined multiview.

[0089] S203: If it does not exist, extract information from the single view to obtain description information and key information, and input the description information, key information and preset noise ratio into the preset multi-view diffusion model to generate a multi-view corresponding to the single view. Then, use the generated multi-view as the determined multi-view.

[0090] When there is no latest multi-view, information extraction is performed on the single view. The label extractor can be used to extract the description information of the single view, and the IP-Adapter network can be used to extract the key information of the single view. Then, the description information, key information and preset noise ratio are input into the preset multi-view diffusion model to obtain the multi-view output by the multi-view diffusion model, and then the multi-view is used as the determined multi-view.

[0091] Furthermore, the training data for the multi-view diffusion model consists of a large number of 3D models, combined with random use of different ambient lighting parameters for multi-view rendering. In the process of generating training data based on the 3D models, this scheme adopts an orthogonal 4-view format and a horizontal coordinate system, i.e., four camera perspectives to obtain multi-view images of a unified object. In practical use, different perspective settings can be used, such as 6 perspectives. Using any azimuth angle as the input perspective, the camera is rotated around the object at the same elevation angle. Assuming the camera intrinsic parameters are fixed, only the camera extrinsic parameters are considered. The camera is oriented towards the origin of the coordinate system, and the rotation angles at the azimuth angle are [0, 90, 180, 270] degrees, generating four camera perspectives. Then, the rendering engine renders the four views of the object from these perspectives. For each object, assuming there are N preset lighting parameters, M preset initial azimuth angles, and K preset elevation angles, a maximum of N*M*K different multi-view data can be generated. It is evident that each 3D model can generate multiple sets of different multi-view data.

[0092] The label extractor is used to extract representative labels from the input data to generate descriptive information about the input data. The IP-Adapter network is an adapter for the text-to-image diffusion model. The descriptive information refers to the information describing the key features of a single view. The key information refers to the information formed by the image features of a single view.

[0093] S204: Input the multiple views into the preset normal prediction network to predict the normal map of the multiple views based on the normal prediction network, and obtain the predicted normal map.

[0094] The normal prediction network is trained using multiple views rendered from a real 3D model and their corresponding normal maps as training data.

[0095] In this step, a preset normal prediction network is determined, and then the determined multi-view is input into the normal prediction network. Since the normal prediction network can be used to predict the normals of each view in the multi-view, the normal map of the multi-view is predicted based on the normal prediction network to obtain the predicted normal map corresponding to the multi-view.

[0096] Specifically, since the normal prediction network is trained using multiple views rendered from a real 3D model and their corresponding normal maps as training data, these training samples possess spatial consistency. Compared to normal prediction based on 2D images, the predicted normal maps output by the normal prediction network trained in this application have higher spatial consistency. Thus, by determining multiple views and inputting them into the aforementioned normal prediction model, a predicted normal map with high spatial consistency is obtained, thereby improving the quality of the subsequently synthesized target 3D model.

[0097] like Figure 4 As shown, in one embodiment, an initial 3D model is determined, and the initial 3D model is iteratively updated based on the predicted normal map, including:

[0098] S301: In one iteration, determine the initial 3D model corresponding to this iteration, and render the initial 3D model to obtain the rendering normal map corresponding to the initial 3D model.

[0099] In this step, during the iteration of the initial 3D model, if it is the first iteration, a sphere model is initialized as the corresponding initial 3D model for this iteration; if it is not the first iteration, the initial 3D model updated in the previous iteration is used as the corresponding initial 3D model for this iteration. Then, the initial 3D model is rendered to obtain its corresponding rendering normal map. For an explanation of this process, please refer to S103 and its corresponding sub-sections.

[0100] S302: Construct the loss function for this iteration based on the predicted normal map and the rendered normal map of the initial 3D model, update the initial 3D model according to the loss function, and use the updated initial 3D model as the corresponding initial 3D model in the next iteration round.

[0101] In this step, the loss function for this iteration is constructed based on the predicted normal map and the rendered normal map determined in S301. Then, the initial 3D model is updated based on the constructed loss function, and the updated initial 3D model serves as the corresponding initial 3D model for the next iteration. It is important to note that the initial 3D model is updated and the rendered normal map is re-rendered in each iteration.

[0102] In one example, the loss function can be constructed using the L2 loss function, whose expression is as follows:

[0103]

[0104] In the formula, L2loss represents the loss function value, and M represents the alpha channel, i.e., the foreground region. This indicates the rendering of the normal map. This represents the predicted normal plot.

[0105] Specifically, as can be seen from the defined loss function, the optimization objective when updating the initial 3D model is the overall average error between the rendered normal map and the predicted normal map. Therefore, even if there are regions inconsistent in the 3D space in the predicted normal map, a relatively average target 3D model can be obtained through gradient descent optimization.

[0106] S303: Determine whether the current iteration round has reached the preset iteration round.

[0107] The preset iteration rounds are empirical values; in one example, the preset iteration rounds can be set to 100.

[0108] S304: If not achieved, proceed to the next iteration round.

[0109] S305: If achieved, the initial 3D model completed in the iteration will be determined as the target 3D model.

[0110] In this embodiment, the initial 3D model is iteratively updated by optimizing the overall average error of the rendered normal map and the predicted normal map, and the rendered normal map is updated during the iterative update process, thereby continuously optimizing the initial 3D model. When the iteration is completed, the target 3D model is obtained to ensure the generation quality of the target 3D model.

[0111] In one embodiment, rendering the target 3D model to obtain a rendering normal map corresponding to the target 3D model includes:

[0112] S1: Calculate the normal vector corresponding to each vertex of the target 3D model to obtain the normal vector corresponding to the target 3D model.

[0113] S2: Project the target 3D model and its corresponding normal vector onto a 2D plane to obtain a normal projection map, and render the normal projection map to obtain a rendered normal map.

[0114] The normal vector is the vector that is perpendicular to the tangent plane of the corresponding vertex.

[0115] Specifically, the target 3D model can be rendered using a differentiable renderer to obtain a rendering normal map. The differentiable renderers used include, but are not limited to, DIRT, Stannum, Mitsuba 2, etc. This application does not impose any specific restrictions on this.

[0116] In this embodiment, by calculating the normal vector corresponding to each vertex of the target 3D model, multiple normal vectors corresponding to the target 3D model are obtained. These normal vectors and the target 3D model are then projected onto a 2D plane to obtain a normal projection map. This normal projection map is then rendered to obtain a rendered normal map. This method allows for the generation of a rendered normal map based on a real 3D model, resulting in accurate spatial consistency. This facilitates the subsequent generation of multiple views that better meet spatial consistency requirements, further optimizing the model and improving the spatial consistency and quality of the final 3D object model.

[0117] like Figure 5 As shown, in one embodiment, noise is superimposed on multiple views, and a new multiple view is generated based on the multi-view with superimposed noise, a single view, and a rendered normal map, including:

[0118] S401: Determine the noise ratio to be added in this round, and overlay noise onto the multi-view according to the noise ratio.

[0119] The proportion of noise superimposed in each round decreases as the number of rounds increases. This is understandable because the spatial consistency of the multi-view prediction is poor in the early stages, requiring more corrections, thus resulting in a larger proportion of noise superimposed in the early stages. Conversely, as the spatial consistency of the multi-view prediction improves in the later stages, fewer corrections are needed, leading to a smaller proportion of noise superimposed in the later stages.

[0120] In this step, the proportion of noise to be added in this round is first determined, and then noise is superimposed on the multi-view according to this noise proportion. It should be noted that the round in this embodiment refers to the number of rounds formed by the iterations between S102 and S106.

[0121] Furthermore, in determining the noise ratio to be added in this round, the noise ratio to be added in this round can be determined based on the pre-set noise ratio of each round, or a rule for dynamically determining the noise ratio can be set, and the noise ratio to be added in this round can be determined according to this rule. This application does not impose specific limitations in this regard.

[0122] S402: Obtain the preset multi-view diffusion model and extract information from the single view to obtain single-view information.

[0123] S403: Input the multi-view and single-view information and the rendered normal map after superimposing noise into the multi-view diffusion model so that the multi-view diffusion model can generate a multi-view that matches the single-view information and the rendered normal map based on the multi-view after superimposing noise, so as to obtain a new multi-view.

[0124] Specifically, single-view information can ensure that the final generated multi-view maintains information consistency with the original information, while the normal control network injects the rendering normal map into the multi-view diffusion model, thus controlling the multi-view diffusion model to generate a multi-view RBG image with a normal structure that conforms to the rendering normal map.

[0125] In this embodiment, the multi-view with added noise is used as the basis for generating new multi-views. In this process, instead of directly generating multi-views from pure noise, the latest multi-view is noise-added to obtain a noisy map, which is then used as input to the multi-view diffusion model. This reduces the diffusion steps and improves the efficiency of 3D object model generation. Furthermore, the guiding role of single-view information and rendering normal maps in the multi-view generation process ensures the quality and spatial consistency of the generated 3D object model.

[0126] In one embodiment, determining the proportion of noise to be added in this round includes:

[0127] S1: When the target 3D model is obtained, obtain the loss function value when the target 3D model is formed.

[0128] S2: Obtain the preset mapping function and input the loss function value into the mapping function to determine the proportion of noise to be added in this round.

[0129] The mapping function is used to reflect the correspondence between the loss function value and the noise ratio.

[0130] Specifically, since the diffusion steps in generating multiple views decrease as the proportion of superimposed noise decreases, the noise proportion can be minimized as much as possible while ensuring that the superimposed noise proportion is sufficient for multi-view generation, thereby improving the efficiency of 3D object model generation. Based on this, the correspondence between the loss function value and the noise proportion can be determined in advance through testing and other methods to obtain the mapping function.

[0131] Understandably, when the predicted normal map and the rendered normal map cannot be perfectly fitted when the target 3D model is finally determined, the loss function value will remain at a large value, indicating that there is a high degree of inconsistency in the generated multiple views. At this time, a larger noise ratio is required. Therefore, the loss function value is used to determine the most appropriate noise ratio. On the one hand, it can ensure the spatial consistency between the final generated multiple views and the latest target 3D model. On the other hand, it can also minimize the diffusion steps while ensuring spatial consistency, saving resources and improving the model generation efficiency.

[0132] In one embodiment, texture mapping is performed on the latest target 3D model based on the latest multi-view to obtain a 3D object model, including:

[0133] S1: Get the latest pixel information of each view in the multi-view.

[0134] S2: Map the pixel information of each view to the corresponding position of the latest target 3D model in sequence to obtain the 3D model to be optimized.

[0135] Specifically, the pixel information of each view in the latest multi-view is mapped onto the corresponding position of the latest target 3D model to obtain the 3D model to be optimized.

[0136] S3: Determine if there is a blind spot in the 3D model to be optimized. If not, then the 3D model to be optimized is determined as a 3D object model.

[0137] Among them, the blind spot refers to the area of ​​an object's surface that cannot be observed due to the object's own obstruction or the limitation of the shooting angle.

[0138] S4: If it exists, obtain the texture features of the area surrounding the blind spot in the 3D model to be optimized, and fill the blind spot in the 3D model to be optimized with texture based on the texture features to obtain the 3D object model.

[0139] Specifically, when there is a blind spot in the 3D model to be optimized, the texture features of the area surrounding the blind spot in the 3D model to be optimized are obtained, and these texture features are used to fill the blind spot in the 3D model to be optimized, so as to obtain the final 3D object model.

[0140] In this embodiment, since the latest multi-view and the latest target 3D model have high spatial consistency, texture mapping of the latest target 3D model based on the latest multi-view can produce a high-quality 3D object model. Furthermore, the problem of blind spots in the 3D model is also considered. For the blind spots, texture filling is performed using color diffusion, thereby improving the generation quality of the 3D object model.

[0141] In one example, such as Figure 6 and Figure 7 As shown, Figure 6 This is a flowchart illustrating how a single view determines the corresponding multiple views, as provided in an embodiment of this application. Figure 7 This is a flowchart illustrating the iterative optimization process during the generation of a 3D object model, as provided in an embodiment of this application.

[0142] exist Figure 6 In this context, the input reference image is a single view, and the initial noise is the preset noise. Figure 7 In this context, the synthesized 3D mesh is the target 3D model, and the input multi-view is the determined multi-view. Figure 7 In the iterative optimization process, the output multiview can be understood as the input multiview in the next iteration.

[0143] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0144] The three-dimensional object model generation apparatus provided in the embodiments of this application is described below. The three-dimensional object model generation apparatus described below can be referred to in correspondence with the three-dimensional object model generation method described above.

[0145] like Figure 8 As shown, this application provides a three-dimensional object model generation device 500, the device comprising:

[0146] The single-view acquisition module 501 is used to acquire a single view of the three-dimensional object to be generated.

[0147] The normal prediction module 502 is used to determine multiple views and predict the normal map of the multiple views to obtain the predicted normal map.

[0148] The model update module 503 is used to determine the initial three-dimensional model, and iteratively update the initial three-dimensional model according to the predicted normal map to obtain the target three-dimensional model. After obtaining the target three-dimensional model, the target three-dimensional model is rendered to obtain the rendering normal map corresponding to the target three-dimensional model.

[0149] The iterative optimization module 504 is used to overlay noise onto the multi-view and generate a new multi-view based on the multi-view, single view and rendered normal map after the noise is overlaid. The normal prediction module, model update module and iterative optimization module are then executed until the preset conditions are met and the latest multi-view and the latest target 3D model are obtained.

[0150] The model generation module 505 is used to perform texture mapping on the latest target 3D model based on the latest multi-view to obtain a 3D object model.

[0151] In one embodiment, the normal prediction module includes:

[0152] The first determination submodule is used to determine whether there is a latest multiview. If there is, the latest multiview is used as the determined multiview.

[0153] The second determining submodule is used to extract information from the single view if it does not exist, obtain description information and key information, and input the description information, key information and preset noise ratio into the preset multi-view diffusion model to generate a multi-view corresponding to the single view. The generated multi-view is then used as the determined multi-view.

[0154] The normal prediction submodule is used to input multiple views into a preset normal prediction network, so as to predict the normal map of the multiple views based on the normal prediction network and obtain the predicted normal map; wherein, the normal prediction network is trained using multiple views rendered from real 3D models and their corresponding normal maps as training data.

[0155] In one embodiment, the model update module includes:

[0156] The normal rendering submodule is used to determine the initial 3D model corresponding to the current iteration in one iteration, and to render the initial 3D model to obtain the rendering normal map corresponding to the initial 3D model.

[0157] The model update submodule is used to construct the loss function in the current iteration based on the predicted normal map and the rendered normal map of the initial 3D model, update the initial 3D model according to the loss function, and use the updated initial 3D model as the corresponding initial 3D model in the next iteration round.

[0158] The judgment submodule is used to determine whether the current iteration round has reached the preset iteration round. If it has not reached the preset iteration round, it will proceed to the next iteration round. If it has reached the preset iteration round, it will determine the initial 3D model that has been completed in the iteration as the target 3D model.

[0159] In one embodiment, the model update module includes:

[0160] The normal vector calculation submodule is used to calculate the normal vector corresponding to each vertex of the target 3D model, so as to obtain the normal vector corresponding to the target 3D model;

[0161] The vector projection submodule is used to project the target 3D model and its corresponding normal vectors onto a 2D plane to obtain a normal projection map, and then render the normal projection map to obtain a rendered normal map.

[0162] In one embodiment, the iterative optimization module includes:

[0163] The noise determination submodule is used to determine the proportion of noise to be added in the current round and to overlay noise onto the multi-view according to the noise proportion; wherein, the proportion of noise overlaid in each round decreases as the number of rounds increases;

[0164] The information extraction submodule is used to obtain a preset multi-view diffusion model and extract information from a single view to obtain single-view information.

[0165] The view generation submodule is used to input the multi-view and single-view information after superimposed noise and the rendered normal map into the multi-view diffusion model, so that the multi-view diffusion model can generate a multi-view that matches the single-view information and rendered normal map based on the multi-view after superimposed noise, so as to obtain a new multi-view.

[0166] In one embodiment, the noise determination submodule includes:

[0167] The function value acquisition unit is used to acquire the loss function value when the target 3D model is obtained;

[0168] The noise determination unit is used to obtain a preset mapping function and input the loss function value into the mapping function to determine the proportion of noise to be added in this round.

[0169] In one embodiment, the model generation module includes:

[0170] The information acquisition submodule is used to acquire the latest pixel information of each view in the multi-view;

[0171] The information mapping submodule is used to sequentially map the pixel information of each view to the corresponding position of the latest target 3D model in order to obtain the 3D model to be optimized.

[0172] The blind spot detection submodule is used to determine whether there is a visual blind spot in the 3D model to be optimized. If there is no blind spot, the 3D model to be optimized is determined to be a 3D object model.

[0173] The texture filling submodule, if present, is used to obtain the texture features of the area surrounding the viewpoint blind spot in the 3D model to be optimized, and to fill the viewpoint blind spot in the 3D model to be optimized with texture based on the texture features to obtain the 3D object model.

[0174] The division of modules in the above-described 3D object model generation device is merely illustrative. In other embodiments, the 3D object model generation device can be divided into different modules as needed to complete all or part of the functions of the above-described 3D object model generation device. Each module in the above-described 3D object model generation device can be implemented entirely or partially through software, hardware, or a combination thereof. Each module can be embedded in or independent of the processor in a computer device in hardware form, or it can be stored in the memory of a computer device in software form, so that the processor can call and execute the operations corresponding to each module.

[0175] In one embodiment, this application also provides a storage medium storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the three-dimensional object model generation method as described in any of the above embodiments.

[0176] In one embodiment, this application also provides a computer device storing computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the three-dimensional object model generation method as described in any of the above embodiments.

[0177] Indicatively, such as Figure 9 As shown, Figure 9 This is a schematic diagram of the internal structure of a computer device 600 provided in an embodiment of this application. The computer device 600 can be provided as a server. (Refer to...) Figure 9 The computer device 600 includes a processing component 602, which further includes one or more processors, and memory resources represented by memory 601 for storing instructions, such as application programs, that can be executed by the processing component 602. The application programs stored in memory 601 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 602 is configured to execute instructions to perform the three-dimensional object model generation method of any of the above embodiments.

[0178] The computer device 600 may also include a power supply component 603 configured to perform power management of the computer device 600, a wired or wireless network interface 604 configured to connect the computer device 600 to a network, and an input / output (I / O) interface 605. The computer device 600 may operate on an operating system stored in memory 601, such as Windows Server™, Mac OS X™, Unix™, Linux™, Free BSD™, or similar.

[0179] Those skilled in the art will understand that Figure 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different component arrangement.

[0180] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising a…" does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element. In this document, the singular forms "a," "an," and "the" may also include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising / including” or “having” specify the presence of the stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the presence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. Meanwhile, the term “and / or” as used in this specification includes any and all combinations of the associated listed items.

[0181] The various embodiments in this specification are described in a progressive manner. Each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can be referred to each other.

[0182] The above description of the disclosed embodiments enables those skilled in the art to make or use this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a three-dimensional object model, characterized in that, The method includes: Obtain a single view of the desired 3D object; Determine multiple views and predict the normal maps of the multiple views to obtain predicted normal maps; An initial 3D model is determined, and the initial 3D model is iteratively updated based on the predicted normal map to obtain a target 3D model. The target 3D model is then rendered to obtain a rendered normal map corresponding to the target 3D model. The initial 3D model is iteratively updated by optimizing the overall average error between the rendered normal map and the predicted normal map, and the rendered normal map is updated during the iterative update process to continuously optimize the initial 3D model. When the iteration is completed, the target 3D model is obtained. Noise is superimposed on the multi-view, and a new multi-view is generated based on the multi-view with superimposed noise, the single view, and the rendering normal map. The steps of determining the multi-view and subsequent steps are continued until the preset conditions are met to obtain the latest multi-view and the latest target 3D model. Based on the latest multi-view model, texture mapping is performed on the latest target 3D model to obtain a 3D object model.

2. The method for generating a three-dimensional object model according to claim 1, characterized in that, The process of determining multiple views and predicting the normal maps of the multiple views to obtain predicted normal maps includes: Determine if a newest multiview exists. If it does, use the newest multiview as the determined multiview. If it does not exist, information is extracted from the single view to obtain description information and key information. The description information, the key information and the preset noise ratio are then input into the preset multi-view diffusion model to generate a multi-view corresponding to the single view. The generated multi-view is then used as the determined multi-view. The multi-view is input into a preset normal prediction network to predict the normal map of the multi-view based on the normal prediction network, thereby obtaining the predicted normal map; wherein, the normal prediction network is trained using the multi-view rendered from a real 3D model and its corresponding normal map as training data.

3. The method for generating a three-dimensional object model according to claim 1, characterized in that, The process of determining an initial 3D model and iteratively updating the initial 3D model based on the predicted normal map includes: In one iteration, the initial 3D model corresponding to this iteration is determined, and the initial 3D model is rendered to obtain the rendering normal map corresponding to the initial 3D model; Based on the predicted normal map and the rendered normal map of the initial 3D model, a loss function is constructed for this iteration, and the initial 3D model is updated according to the loss function. The updated initial 3D model is then used as the corresponding initial 3D model in the next iteration. Determine whether the current iteration round has reached the preset iteration round. If not, proceed to the next iteration round. If it has, determine the initial 3D model completed by the iteration as the target 3D model.

4. The method for generating a three-dimensional object model according to claim 1, characterized in that, The step of rendering the target 3D model to obtain a rendering normal map corresponding to the target 3D model includes: Calculate the normal vector corresponding to each vertex of the target 3D model to obtain the normal vector corresponding to the target 3D model; The target 3D model and its corresponding normal vector are projected onto a 2D plane to obtain a normal projection map, and the normal projection map is rendered to obtain a rendered normal map.

5. The method for generating a three-dimensional object model according to any one of claims 1 to 4, characterized in that, The step of adding noise to the multi-view and generating a new multi-view based on the multi-view with added noise, the single view, and the rendering normal map includes: Determine the noise ratio to be added in this round, and superimpose noise onto the multi-view according to the noise ratio; wherein, the noise ratio superimposed in each round decreases as the number of rounds increases; Obtain a preset multi-view diffusion model and extract information from the single view to obtain single-view information; The noise-stamped multi-view, the single-view information, and the rendering normal map are input into the multi-view diffusion model, so that the multi-view diffusion model generates a multi-view that matches the single-view information and the rendering normal map based on the noise-stamped multi-view, thereby obtaining a new multi-view.

6. The method for generating a three-dimensional object model according to claim 5, characterized in that, Determining the proportion of noise to be added in this round includes: When the target 3D model is obtained, the loss function value used to form the target 3D model is acquired; Obtain a preset mapping function and input the loss function value into the mapping function to determine the proportion of noise to be added in this round.

7. The method for generating a three-dimensional object model according to claim 1, characterized in that, The process of performing texture mapping on the latest target 3D model based on the latest multi-view to obtain a 3D object model includes: Get the latest pixel information of each view in the multi-view; The pixel information of each view is mapped to the corresponding position of the latest target 3D model in turn to obtain the 3D model to be optimized. Determine whether the 3D model to be optimized has a blind spot. If it does not, then the 3D model to be optimized is determined to be a 3D object model. If it exists, the texture features of the surrounding area of ​​the blind spot in the 3D model to be optimized are obtained, and the texture of the blind spot in the 3D model to be optimized is filled according to the texture features to obtain the 3D object model.

8. A three-dimensional object model generation device, characterized in that, The device includes: The single-view acquisition module is used to acquire a single view of the 3D object to be generated. The normal prediction module is used to determine multiple views and predict the normal maps of the multiple views to obtain predicted normal maps. The model update module is used to determine an initial 3D model, and iteratively update the initial 3D model according to the predicted normal map to obtain a target 3D model. Then, the target 3D model is rendered to obtain a rendered normal map corresponding to the target 3D model. In this process, the overall average error between the rendered normal map and the predicted normal map is optimized to iteratively update the initial 3D model, and the rendered normal map is updated during the iterative update process to continuously optimize the initial 3D model. When the iteration is completed, the target 3D model is obtained. The iterative optimization module is used to superimpose noise onto the multi-view and generate a new multi-view based on the multi-view after superimposing noise, the single view, and the rendered normal map. The normal prediction module, the model update module, and the iterative optimization module are then executed until the preset conditions are met to obtain the latest multi-view and the latest target 3D model. The model generation module is used to perform texture mapping on the latest target 3D model based on the latest multi-view to obtain a 3D object model.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions that, when executed by one or more processors, cause the one or more processors to perform the steps of the three-dimensional object model generation method as described in any one of claims 1 to 7.

10. A computer device, characterized in that, include: One or more processors, and memory; The memory stores computer-readable instructions, which, when executed by the one or more processors, perform the steps of the three-dimensional object model generation method as described in any one of claims 1 to 7.

Images