Three-dimensional object model generation method and device, storage medium and computer equipment

Abstract

The invention provides a three-dimensional object model generation method and device, a storage medium and computer equipment, and the method comprises the steps: obtaining a single view, determining multiple views, carrying out the prediction of a normal diagram of the multiple views, obtaining a prediction normal diagram, carrying out the iteration updating of a determined initial three-dimensional model, obtaining a target three-dimensional model, and carrying out the calculation of the target three-dimensional model. The target three-dimensional model is rendered to obtain a rendering normal diagram corresponding to the target three-dimensional model, then noise superposition is performed on the multiple views, a new multiple view is generated by combining the single view and the rendering normal diagram, and the step of determining the multiple views and the subsequent steps are continuously executed. And obtaining the latest multi-view and the latest target three-dimensional model until a preset condition is met. In the process, a rendering normal graph obtained based on the target three-dimensional model has spatial consistency, so that a two-dimensional multi-view conforming to the spatial consistency of the three-dimensional structure is subsequently generated, the multi-view serves as initial data of next iteration, and the consistency and precision of the target three-dimensional model are continuously improved through iteration.

Description

Technical Field

[0001] The present application relates to the technical field of three-dimensional modeling, and in particular to a method, device, storage medium and computer equipment for generating a three-dimensional object model. Background Art

[0002] With the rapid development of information technology, 3D object generation technology has been widely used in many fields such as virtual reality, augmented reality, game development, film and television special effects, and product design. Traditional 3D modeling methods usually rely on professional software and manual operations, which are time-consuming, labor-intensive, and costly. In recent years, with the advancement of deep learning technology, automatic or semi-automatic 3D object model generation methods have gradually become the mainstream generation method. This method can reconstruct and restore the 3D structure of an object from a 2D image, greatly improving the efficiency and quality of 3D modeling.

[0003] However, in the process of reconstructing a three-dimensional object model based on a two-dimensional image, since each step of the process is completed based on a two-dimensional picture, it is difficult to ensure the consistency of the generated three-dimensional object model under various viewing angles, resulting in low accuracy of the generated three-dimensional object model. Summary of the invention

[0004] The purpose of the present application is to solve at least one of the above-mentioned technical defects, especially in the process of reconstructing a three-dimensional object model based on a two-dimensional image in the prior art, since each step of the process is completed based on the two-dimensional picture, it is difficult to ensure the consistency of the generated three-dimensional object model under various viewing angles, resulting in a technical defect of low accuracy of the generated three-dimensional object model.

[0005] In a first aspect, the present application provides a method for generating a three-dimensional object model, the method comprising:

[0006] Obtain a single view of the three-dimensional object to be generated;

[0007] Determining multiple views, and predicting normal maps of the multiple views to obtain predicted normal maps;

[0008] Determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model;

[0009] Superimposing noise on the multiple views, and generating a new multiple view according to the multiple views after superimposing noise, the single view and the rendered normal map, and continuing to perform the step of determining the multiple views and subsequent steps until the latest multiple views and the latest target three-dimensional model are obtained when preset conditions are met;

[0010] The latest target 3D model is texture mapped based on the latest multi-views to obtain a 3D object model.

[0011] In one embodiment, determining multiple views and predicting normal maps of the multiple views to obtain predicted normal maps includes:

[0012] Determine whether there is a latest multi-view currently, and if so, use the latest multi-view currently as the determined multi-view;

[0013] If not, extracting information from the single view to obtain description information and key information, and inputting the description information, the key information and a preset noise ratio into a preset multi-view diffusion model to generate a multi-view corresponding to the single view, and using the generated multi-view as the determined multi-view;

[0014] The multiple views are input into a preset normal prediction network to predict the normal maps of the multiple views based on the normal prediction network to obtain predicted normal maps; wherein the normal prediction network is trained using multiple views rendered by a real three-dimensional model and their corresponding normal maps as training data.

[0015] In one embodiment, determining the initial three-dimensional model and iteratively updating the initial three-dimensional model according to the predicted normal map includes:

[0016] In one iteration, an initial three-dimensional model corresponding to this iteration is determined, and the initial three-dimensional model is rendered to obtain a rendering normal map corresponding to the initial three-dimensional model;

[0017] Constructing a loss function in this iteration based on the predicted normal map and the rendered normal map of the initial three-dimensional model, updating the initial three-dimensional model according to the loss function, and using the updated initial three-dimensional model as the corresponding initial three-dimensional model in the next iteration round;

[0018] It is determined whether the current iteration round reaches the preset iteration round. If not, the next iteration round is entered. If reached, the initial three-dimensional model after the iteration is completed is determined as the target three-dimensional model.

[0019] In one of the embodiments, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model includes:

[0020] Calculating a normal vector corresponding to each vertex of the target three-dimensional model to obtain a normal vector corresponding to the target three-dimensional model;

[0021] The target three-dimensional model and its corresponding normal vector are projected onto a two-dimensional plane to obtain a normal projection map, and the normal projection map is rendered to obtain a rendered normal map.

[0022] In one embodiment, the step of superimposing noise on the multiple views and generating new multiple views according to the multiple views after superimposing noise, the single view and the rendered normal map comprises:

[0023] Determine the noise ratio to be added in this round, and superimpose noise on the multiple views according to the noise ratio; wherein the noise ratio superimposed in each round decreases as the round increases;

[0024] Acquire a preset multi-view diffusion model, and extract information from the single view to obtain single view information;

[0025] The multi-view after superimposing noise, the single view information and the rendered normal map are input into the multi-view diffusion model, so that the multi-view diffusion model generates a multi-view matching the single view information and the rendered normal map based on the multi-view after superimposing noise, so as to obtain a new multi-view.

[0026] In one embodiment, determining the noise ratio to be added in this round includes:

[0027] When the target three-dimensional model is obtained, obtaining a loss function value when forming the target three-dimensional model;

[0028] Obtain a preset mapping function and input the loss function value into the mapping function to determine the noise ratio to be added in this round.

[0029] In one embodiment, the step of performing texture mapping on the latest target three-dimensional model based on the latest multi-views to obtain a three-dimensional object model includes:

[0030] Get the latest pixel information of each view in the multi-view;

[0031] Mapping the pixel information of each view to the corresponding position of the latest target three-dimensional model in turn to obtain the three-dimensional model to be optimized;

[0032] Determine whether the three-dimensional model to be optimized has a viewing angle blind spot, and if not, determine the three-dimensional model to be optimized as a three-dimensional object model;

[0033] If it exists, the texture features of the surrounding area of ​​the viewing blind area in the three-dimensional model to be optimized are obtained, and the viewing blind area in the three-dimensional model to be optimized is texture filled according to the texture features to obtain a three-dimensional object model.

[0034] In a second aspect, the present application provides a three-dimensional object model generation device, the device comprising:

[0035] A single view acquisition module, used to acquire a single view of the three-dimensional object to be generated;

[0036] A normal prediction module, used for determining multiple views and predicting normal maps of the multiple views to obtain predicted normal maps;

[0037] A model updating module, used for determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, and after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model;

[0038] an iterative optimization module, configured to superimpose noise on the multi-views, generate a new multi-view according to the multi-views after superimposing noise, the single view and the rendered normal map, and continue to execute the single view acquisition module, the normal prediction module, the model update module and the iterative optimization module until the latest multi-views and the latest target three-dimensional model are acquired when preset conditions are met;

[0039] The model generation module is used to perform texture mapping on the latest target three-dimensional model based on the latest multi-views to obtain a three-dimensional object model.

[0040] In a third aspect, the present application provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the three-dimensional object model generation method described in any of the above embodiments.

[0041] In a fourth aspect, the present application provides a computer device, comprising: one or more processors, and a memory;

[0042] The memory stores computer-readable instructions, and when the one or more processors execute the computer-readable instructions, the steps of the method for generating a three-dimensional object model as described in any one of the above embodiments are performed.

[0043] It can be seen from the above technical solutions that the embodiments of the present application have the following advantages:

[0044] The present application provides a method, device, storage medium and computer device for generating a three-dimensional object model. The method includes: obtaining a single view of the three-dimensional object to be generated, determining multiple views, and predicting the normal map of the multiple views to obtain a predicted normal map; then determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model, then superimposing noise on the multiple views, and generating a new multi-view according to the multi-view, single view and rendered normal map after superimposing noise, continuing to perform the step of determining the multiple views and its subsequent steps until the latest multi-view and the latest target three-dimensional model are obtained when the preset conditions are met; in this process, the target three-dimensional model is used as an intermediate medium, and the rendered normal map obtained based on the target three-dimensional model has spatial consistency, so that a two-dimensional multi-view that meets the spatial consistency of the three-dimensional structure is subsequently generated, and the multi-view is used as the initial data for the next iteration, and the spatial structure consistency and model accuracy of the target three-dimensional model are continuously improved through iteration. Finally, texture mapping is performed on the latest target three-dimensional model based on the latest multi-view, so as to obtain a high-quality three-dimensional object model. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0046] Figure 1 A schematic diagram of a flow chart of a method for generating a three-dimensional object model provided in an embodiment of the present application;

[0047] Figure 2 A flowchart of a method for generating a three-dimensional object model according to an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a process for determining multiple views and predicting normal maps of multiple views provided in an embodiment of the present application;

[0049] Figure 4 A schematic diagram of a process for determining an initial three-dimensional model and iteratively updating the initial three-dimensional model according to a predicted normal map provided in an embodiment of the present application;

[0050] Figure 5 A schematic diagram of a process for superimposing noise on multiple views, and generating new multiple views based on the multiple views after superimposing noise, a single view, and a rendered normal map provided in an embodiment of the present application;

[0051] Figure 6 An example diagram of a process for determining corresponding multiple views from a single view provided in an embodiment of the present application;

[0052] Figure 7 An example diagram of an iterative optimization process in the three-dimensional object model generation process provided in an embodiment of the present application;

[0053] Figure 8 A schematic diagram of the structure of a three-dimensional object model generation device provided in an embodiment of the present application;

[0054] Fig. 9 An internal structure diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0055] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0056] In one embodiment, the present application provides a method for generating a 3D object model. The following embodiment is described by applying the method to a server. It can be understood that the method for generating a 3D object model can be executed by a single server or a server cluster composed of multiple servers, and the present application does not make any specific restrictions on this.

[0057] Specifically, multiple views are predicted through the reference image, and then the normal map is predicted through the multiple views, and finally the three-dimensional model is generated through the normal map. These steps are all performed at the image level, and the spatial consistency of the multiple views in the three-dimensional structure cannot be guaranteed. In other words, the multi-views obtained in this way are not multi-views projected from the true three-dimensional model. The same elements between different perspectives may not have the same position when mapped back to the three-dimensional space. Therefore, it is impossible to directly generate an accurate mesh grid through this solution. Directly using the multi-views obtained in this way as the source of the texture cannot accurately project the image to 3D. The texture after projection will be misaligned and overlapped due to the insufficient spatial consistency of the multiple views. Therefore, it is impossible to obtain an accurate and high-quality three-dimensional model. Based on this, if Figure 1 As shown, the present application provides a method for generating a three-dimensional object model, the method comprising:

[0058] S101: Obtain a single view of a three-dimensional object to be generated.

[0059] In this step, when the user needs to generate a 3D model of a 3D object, a single view of the 3D object can be obtained, and model generation instructions can be initiated in sequence. When the server receives the model generation instruction, it obtains the single view of the 3D object to be generated according to the model generation instruction, so that the single view can be used as basic data for the 3D object model generation task.

[0060] It can be understood that a single view refers to an RBG image of only a single angle obtained by observing an object from a specific angle. A single view may include a front view, a left view, a right view, etc. This application does not impose specific restrictions on this, but for the sake of information richness, a single view may use a front view of a three-dimensional object.

[0061] S102: Determine multiple views, and predict normal maps of the multiple views to obtain predicted normal maps.

[0062] In this step, the method of determining the multi-view can be clarified according to the current iteration round. In the first round, since S104 has not been passed, that is, no new multi-view has been generated, the multi-view is generated and determined according to the single view. In the non-first round, a new multi-view already exists at this time, and the latest multi-view can be determined as the multi-view in this step. Then, the normal of each view in the multi-view can be predicted to obtain a predicted normal map corresponding to the multi-view.

[0063] It can be understood that the above rounds refer to the number of rounds formed by the iterations between S102-S106.

[0064] Among them, multi-view refers to an RBG image composed of the front view, left view, right view and back view of a three-dimensional object, and the predicted normal map refers to an image composed of the normal maps of the front view, left view, right view and back view of a three-dimensional object. The normal map can represent the normal of each point of the object, and the direction of these normals is represented by the RGB color channel. The red, green and blue channels in the normal map represent the X, Y and Z components of the normal, respectively. It can be understood that the normal map can reflect the surface structure of the mesh and can also be used to enhance the surface details of the three-dimensional model. Therefore, the normal map can simulate higher surface details in visual effects.

[0065] S103: determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, and after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model.

[0066] In this step, the method for determining the initial three-dimensional model can be clarified according to the current iteration round. In the first round, a three-dimensional model can be initialized as the initial three-dimensional model, for example, a three-dimensional sphere model can be initialized as the initial three-dimensional model. In a non-first round, the target three-dimensional model obtained in the previous round can be used as the initial three-dimensional model. When the initial three-dimensional model is determined, the initial three-dimensional model is iteratively updated according to the predicted normal map and the normal map obtained by each iterative rendering. When the iterative update is completed, the target three-dimensional model is obtained. At this time, the target three-dimensional model is rendered to obtain the rendered normal map of the target three-dimensional model.

[0067] Specifically, the above rounds refer to the number of rounds formed by the iterations between S102-S106, and iteration refers to the update of the initial three-dimensional model. In other words, a complete iterative update process is performed on the initial three-dimensional model in each round.

[0068] Among them, the rendered normal map refers to the normal map obtained based on the three-dimensional model. It can be understood that both the rendered normal map and the predicted normal map are normal maps. The difference is that the former is obtained based on the three-dimensional model, while the latter is predicted based on the two-dimensional image. Therefore, the rendered normal map will have higher spatial consistency than the predicted normal map.

[0069] S104: superimposing noise on the multi-views, and generating new multi-views according to the multi-views after superimposing noise, the single view, and the rendered normal map.

[0070] In this step, noise is added to the multi-view, and then the multi-view, single view and rendered normal map after superimposing noise can be input into a preset multi-view diffusion model to generate a new multi-view. The multi-view diffusion model is used to generate corresponding multi-views according to the input content.

[0071] It can be understood that adding noise to the multi-view can control the range of variation of the finally generated new multi-view compared to the input multi-view. When the degree of noise addition is low, the variation of the generated multi-view is also small. At this time, the multi-view diffusion model will be more inclined to generate results that are more similar to the input multi-view when predicting.

[0072] Specifically, since a single view and a rendered normal map are also combined when generating a new multi-view, the guiding role of the single view and the rendered normal map can be fully utilized, so that in the process of generating a new multi-view, the generation can be carried out in a direction that is consistent with the original data and the spatial consistency of the model structure.

[0073] S105: Determine whether a preset condition is met.

[0074] Among them, the preset condition refers to the round stopping condition in the process of generating the three-dimensional object model. The preset condition can be set to reach the preset round, or the preset condition can be set to the quality of the current latest target three-dimensional model reaches the expectation. This application does not impose any specific restrictions on this.

[0075] S106: If not satisfied, continue to execute the step of determining multiple views and subsequent steps.

[0076] When the preset condition is not met, the process may return to the step of determining multiple views and continue the process.

[0077] S107: If satisfied, obtain the latest multi-view and the latest target 3D model.

[0078] When the preset conditions are met, the latest multi-view and the latest target 3D model are obtained, that is, the multi-view and the target 3D model generated in this round that meets the preset conditions.

[0079] S108: Perform texture mapping on the latest target three-dimensional model based on the latest multi-view to obtain a three-dimensional object model.

[0080] In this step, texture mapping is performed on the latest target 3D model according to the latest multi-views. At this time, the texture features (such as texture, color, etc.) in the multi-views can be filled into the latest target 3D model to obtain a 3D object model.

[0081] In this embodiment, through the iterations of S102-S106 above, since the new multi-views obtained in each iteration will be more consistent with the spatial consistency of the three-dimensional object than the multi-views in the previous iteration, the prediction of the normal map will also be more accurate than that obtained in the previous iteration. Based on this, the initial three-dimensional model is optimized to obtain a target three-dimensional model with higher accuracy, and then the rendering normal map of the target three-dimensional model is determined. Based on this continuous iterative process, a high-precision and high-quality target three-dimensional model and multi-views that meet the spatial consistency requirements of the three-dimensional object can be finally obtained, so that the finally generated three-dimensional object model has excellent performance in terms of spatial consistency, thereby improving the quality and accuracy of the generated three-dimensional object model.

[0082] In one example, if Figure 2 As shown, Figure 2 This is a flowchart of a method for generating a three-dimensional object model provided in an embodiment of the present application. Figure 2 In the process of iteration, the optimized multi-view output is the new multi-view generated in S104, and the three-dimensional mesh is the target three-dimensional model. During the iteration process, the optimized multi-view will be used as the multi-view of the next iteration for iteration. The optimized multi-view finally obtained can perform texture mapping on the generated three-dimensional mesh (the latest target three-dimensional model) to obtain a three-dimensional object model.

[0083] The present application provides a method, device, storage medium and computer device for generating a three-dimensional object model. The method includes: obtaining a single view of the three-dimensional object to be generated, determining multiple views, and predicting the normal map of the multiple views to obtain a predicted normal map; then determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model, then superimposing noise on the multiple views, and generating a new multi-view according to the multi-view, single view and rendered normal map after superimposing noise, continuing to perform the step of determining the multiple views and its subsequent steps until the latest multi-view and the latest target three-dimensional model are obtained when the preset conditions are met; in this process, the target three-dimensional model is used as an intermediate medium, and the rendered normal map obtained based on the target three-dimensional model has spatial consistency, so that a two-dimensional multi-view that meets the spatial consistency of the three-dimensional structure is subsequently generated, and the multi-view is used as the initial data for the next iteration, and the spatial structure consistency and model accuracy of the target three-dimensional model are continuously improved through iteration. Finally, texture mapping is performed on the latest target three-dimensional model based on the latest multi-view, so as to obtain a high-quality three-dimensional object model.

[0084] like Figure 3 As shown, in one embodiment, determining multiple views and predicting normal maps of the multiple views to obtain predicted normal maps includes:

[0085] S201: Determine whether there is a latest multi-view.

[0086] In this step, by judging whether there is the latest multi-view currently, it can be known whether the current 3D object model generation process has gone through S104. If so, there is the latest multi-view; if not, there is no latest multi-view.

[0087] S202: If it exists, taking the current latest multi-view as the determined multi-view.

[0088] When the latest multi-view exists, the latest multi-view may be used as the determined multi-view.

[0089] S203: If it does not exist, extract information from the single view to obtain description information and key information, and input the description information, key information and preset noise ratio into a preset multi-view diffusion model to generate a multi-view corresponding to the single view, and then use the generated multi-view as the determined multi-view.

[0090] When the latest multi-view does not exist, information extraction is performed on the single view. The label extractor can be used to extract the description information of the single view, and the IP-Adapter network can be used to extract the key information of the single view. Then, the description information, the key information and the preset noise ratio are input into the preset multi-view diffusion model to obtain the multi-view output by the multi-view diffusion model, and then the multi-view is used as the determined multi-view.

[0091] Furthermore, the training data of the multi-view diffusion model is obtained by a large number of three-dimensional models, combined with multi-view rendering using different random ambient lighting parameters. In the process of generating training data based on the three-dimensional model, this solution adopts the orthogonal 4-view form and the horizontal coordinate system, that is, 4 camera perspectives to obtain a multi-view image of a unified object. Different perspective settings can be used in actual use, such as 6 perspectives. With any azimuth angle as the input perspective, the camera is rotated around the object at the same altitude angle. Assuming that the camera intrinsic parameters are fixed, only the camera extrinsic parameters are considered, the camera is facing the origin of the coordinate system, and the rotation angles in the azimuth are [0, 90, 180, 270] degrees, generating 4 camera perspectives, and then rendering the 4 views of the object under the perspective through the rendering engine. For each object, assuming that there are N preset lighting parameters, and the camera presets M initial azimuth angles and K altitude angles, a maximum of N*M*K different multi-view data can be generated. It can be seen that each three-dimensional model can generate multiple sets of different multi-view data.

[0092] Among them, the label extractor is used to extract representative labels from the input data to generate descriptive information about the input data, the IP-Adapter network is an adapter for the text-to-image diffusion model, the descriptive information refers to the information describing the key features of a single view, and the key information refers to the information formed by the image features of a single view.

[0093] S204: Inputting the multiple views into a preset normal prediction network to predict the normal maps of the multiple views based on the normal prediction network to obtain predicted normal maps.

[0094] Among them, the normal prediction network is trained using multiple views rendered by real 3D models and their corresponding normal maps as training data.

[0095] In this step, a preset normal prediction network is determined, and then the determined multi-views are input into the normal prediction network. Since the normal prediction network can be used to predict the normals of each view in the multi-views, the normal maps of the multi-views are predicted based on the normal prediction network to obtain the predicted normal maps corresponding to the multi-views.

[0096] Specifically, since the normal prediction network is trained using multiple views rendered by real three-dimensional models and their corresponding normal maps as training data, these training samples have spatial consistency. Compared with normal prediction based on two-dimensional image level, the predicted normal map output by the normal prediction network trained in this application has higher spatial consistency. In this way, by determining multiple views and inputting the multiple views into the above-mentioned normal prediction model, a predicted normal map with higher spatial consistency is obtained, thereby improving the quality of the target three-dimensional model synthesized subsequently.

[0097] like Figure 4 As shown, in one embodiment, determining an initial three-dimensional model and iteratively updating the initial three-dimensional model according to the predicted normal map includes:

[0098] S301: In one iteration, an initial three-dimensional model corresponding to this iteration is determined, and the initial three-dimensional model is rendered to obtain a rendering normal map corresponding to the initial three-dimensional model.

[0099] In this step, when the initial three-dimensional model is iterated, if it is the first iteration, a sphere model is initialized as the initial three-dimensional model corresponding to this iteration, and if it is not the first iteration, the initial three-dimensional model updated in the previous iteration is used as the initial three-dimensional model corresponding to this iteration. Then, the initial three-dimensional model is rendered to obtain its corresponding rendering normal map. For the description of this process, please refer to the corresponding description in S103 and its subordinates.

[0100] S302: constructing a loss function in this iteration based on the predicted normal map and the rendered normal map of the initial three-dimensional model, updating the initial three-dimensional model according to the loss function, and using the updated initial three-dimensional model as the corresponding initial three-dimensional model in the next iteration round.

[0101] In this step, the loss function in this iteration is constructed based on the predicted normal map and the rendered normal map determined in S301, and then the initial three-dimensional model is updated based on the constructed loss function, and the updated initial three-dimensional model is used as the corresponding initial three-dimensional model in the next iteration. It should be noted that each iteration will update the initial three-dimensional model and re-render to obtain the rendered normal map.

[0102] In one example, the loss function can be constructed using the L2loss loss function, which is expressed as follows:

[0103]

[0104] In the formula, L2loss represents the loss function value, M represents the alpha channel, that is, the foreground area, Represents the rendered normal map, Represents the predicted normal map.

[0105] Specifically, from the defined loss function, we can see that the optimization goal when updating the initial three-dimensional model is the overall average error between the rendered normal map and the predicted normal map. Therefore, even if there are areas of three-dimensional spatial inconsistency in the predicted normal map, a relatively average target three-dimensional model can be obtained through gradient descent optimization.

[0106] S303: Determine whether the current iteration round reaches the preset iteration round.

[0107] The preset iteration round is an empirical value. In an example, the preset iteration round can be set to 100.

[0108] S304: If not reached, proceed to the next iteration round.

[0109] S305: If the target is reached, the initial three-dimensional model after iteration is determined as the target three-dimensional model.

[0110] In this embodiment, the initial three-dimensional model is iteratively updated by optimizing the overall average error between the rendered normal map and the predicted normal map, and the rendered normal map is updated during the iterative update process, so as to continuously optimize the initial three-dimensional model and obtain the target three-dimensional model when the iteration is completed to ensure the generation quality of the target three-dimensional model.

[0111] In one embodiment, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model includes:

[0112] S1: Calculate the normal vector corresponding to each vertex of the target three-dimensional model to obtain the normal vector corresponding to the target three-dimensional model.

[0113] S2: Project the target three-dimensional model and its corresponding normal vector onto a two-dimensional plane to obtain a normal projection map, and render the normal projection map to obtain a rendered normal map.

[0114] The normal vector is the vector perpendicular to the tangent plane of the corresponding vertex.

[0115] Specifically, the target three-dimensional model can be rendered by a differentiable renderer to obtain a rendered normal map. The differentiable renderers used include but are not limited to DIRT, Stannum, Mitsuba 2, etc., and this application does not make any specific restrictions on this.

[0116] In this embodiment, the normal vector corresponding to each vertex of the target three-dimensional model is calculated to obtain multiple normal vectors corresponding to the target three-dimensional model, and then these normal vectors and the target three-dimensional model are projected onto a two-dimensional plane to obtain a normal projection map, and then the normal projection map is rendered to obtain a rendered normal map. In this way, a rendered normal map can be obtained by normal projection based on a real three-dimensional model, which has accurate spatial consistency, so as to facilitate the subsequent generation of multiple views that better meet the spatial consistency requirements, so as to further optimize the model and improve the spatial consistency and model quality of the final three-dimensional object model.

[0117] like Figure 5 As shown, in one embodiment, noise is superimposed on multiple views, and new multiple views are generated according to the multiple views after the noise is superimposed, the single view, and the rendered normal map, including:

[0118] S401: Determine the noise ratio to be added in this round, and superimpose noise on multiple views according to the noise ratio.

[0119] The proportion of noise superimposed in each round decreases as the number of rounds increases. It can be understood that since the spatial consistency of multiple views is poor in the early prediction, more content needs to be corrected, so the proportion of noise superimposed in the early stage is large, and after the spatial consistency of multiple views is improved in the later stage, less content needs to be corrected, so the proportion of noise superimposed in the later stage is small.

[0120] In this step, the noise ratio to be added in this round is first determined, and then noise is superimposed on multiple views according to the noise ratio. It should be noted that the round in this embodiment refers to the number of rounds formed by the iterations between S102-S106.

[0121] Furthermore, in the process of determining the noise ratio to be added in this round, the noise ratio to be added in this round can be determined according to the preset noise ratio of each round, and a rule for dynamically determining the noise ratio can be set, and the noise ratio to be added in this round is determined according to the rule. This application does not make specific restrictions on this.

[0122] S402: Obtain a preset multi-view diffusion model, and extract information from a single view to obtain single view information.

[0123] S403: Input the multi-view after superimposing noise, the single view information and the rendered normal map into the multi-view diffusion model, so that the multi-view diffusion model generates a multi-view matching the single view information and the rendered normal map based on the multi-view after superimposing noise, so as to obtain a new multi-view.

[0124] Specifically, the single-view information can ensure that the final generated multi-view maintains information consistency with the original information, and the rendered normal map is injected into the multi-view diffusion model through the normal control network, so that the multi-view diffusion model can be controlled to generate a multi-view RBG image with a normal structure that conforms to the rendered normal map.

[0125] In this embodiment, the multi-view after superimposing noise is used as the basis for generating new multi-views. In this process, multi-views can be generated not directly from pure noise, but the current latest multi-view is subjected to noise processing to obtain a noisy image and use it as the input of the multi-view diffusion model. In this way, the diffusion steps can be reduced and the efficiency of generating the three-dimensional object model can be improved. In addition, the guiding role played by the single view information and the rendered normal map in the process of generating multiple views can ensure the generation quality and spatial consistency of the three-dimensional object model.

[0126] In one embodiment, determining the noise ratio to be added in this round includes:

[0127] S1: When the target three-dimensional model is obtained, the loss function value when the target three-dimensional model is formed is obtained.

[0128] S2: Obtain a preset mapping function and input the loss function value into the mapping function to determine the noise ratio to be added in this round.

[0129] Among them, the mapping function is used to reflect the corresponding relationship between the loss function value and the noise ratio.

[0130] Specifically, since the diffusion steps for generating multiple views will be reduced as the superimposed noise ratio decreases, the noise ratio can be reduced as much as possible while ensuring that the superimposed noise ratio can meet the multi-view generation, thereby improving the efficiency of generating the three-dimensional object model. Based on this, the corresponding relationship between the loss function value and the noise ratio can be determined in advance through testing and other methods, thereby obtaining a mapping function.

[0131] It is understandable that when the predicted normal map and the rendered normal map cannot be completely fitted when the target three-dimensional model is finally determined, the loss function value will maintain a large value, indicating that the generated multi-views have high inconsistency. At this time, a larger noise ratio is required. Therefore, the most appropriate noise ratio is determined by the loss function value. On the one hand, it can ensure the spatial consistency of the finally generated multi-views and the latest target three-dimensional model. On the other hand, it can also reduce the diffusion steps as much as possible while ensuring spatial consistency, saving resources to improve model generation efficiency.

[0132] In one embodiment, texture mapping is performed on the latest target three-dimensional model based on the latest multi-view to obtain a three-dimensional object model, including:

[0133] S1: Obtain the latest pixel information of each view in the multi-view.

[0134] S2: Mapping pixel information of each view to a corresponding position of the latest target three-dimensional model in sequence to obtain a three-dimensional model to be optimized.

[0135] Specifically, the pixel information of each view in the latest multi-view is mapped to the corresponding position of the latest target three-dimensional model to obtain the three-dimensional model to be optimized.

[0136] S3: Determine whether the three-dimensional model to be optimized has a viewing blind spot. If not, determine the three-dimensional model to be optimized as a three-dimensional object model.

[0137] The blind spot refers to the surface area of ​​an object that cannot be observed due to the object's own occlusion or the limitation of the shooting angle.

[0138] S4: If it exists, then obtain the texture features of the surrounding area of ​​the viewing blind area in the three-dimensional model to be optimized, and perform texture filling on the viewing blind area in the three-dimensional model to be optimized according to the texture features to obtain a three-dimensional object model.

[0139] Specifically, when there is a viewing blind spot in the three-dimensional model to be optimized, the texture features of the surrounding area of ​​the viewing blind spot in the three-dimensional model to be optimized are obtained, and these texture features are used to fill the viewing blind spot in the three-dimensional model to be optimized to obtain the final three-dimensional object model.

[0140] In this embodiment, since the latest multi-view and the latest target three-dimensional model both have high spatial consistency, a high-quality three-dimensional object model can be obtained by texture mapping the latest target three-dimensional model based on the latest multi-view. The problem of viewing blind spots in the three-dimensional model is also considered, and color diffusion is used to fill the texture in the viewing blind spots, thereby improving the generation quality of the three-dimensional object model.

[0141] In one example, if Figure 6 and Figure 7 As shown, Figure 6 An example diagram of a process for determining corresponding multiple views from a single view provided in an embodiment of the present application. Figure 7 This is an example diagram of the iterative optimization process in the three-dimensional object model generation process provided in an embodiment of the present application.

[0142] exist Figure 6 In , the reference image is input as a single view, and the initialization noise is the preset noise. Figure 7 In the above, the synthesized 3D mesh is the target 3D model, and the input multi-view is the determined multi-view. Figure 7 In the iterative optimization process, the output multi-view can be understood as the input multi-view in the next iteration.

[0143] It should be understood that, although the various steps in the flowcharts involved in the above-mentioned embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps does not have a strict order restriction, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above-mentioned embodiments can include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily to be carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0144] The following is a description of a three-dimensional object model generation device provided in an embodiment of the present application. The three-dimensional object model generation device described below and the three-dimensional object model generation method described above can refer to each other.

[0145] like Figure 8 As shown, the present application provides a three-dimensional object model generation device 500, the device comprising:

[0146] A single view acquisition module 501 is used to acquire a single view of a three-dimensional object to be generated;

[0147] A normal prediction module 502, used to determine multiple views and predict normal maps of the multiple views to obtain predicted normal maps;

[0148] The model updating module 503 is used to determine an initial three-dimensional model, and iteratively update the initial three-dimensional model according to the predicted normal map, and after obtaining a target three-dimensional model, render the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model;

[0149] Iterative optimization module 504, used to superimpose noise on multiple views, and generate new multiple views according to the multiple views after superimposing noise, the single view and the rendered normal map, and continue to execute the single view acquisition module, the normal prediction module, the model update module and the iterative optimization module until the latest multiple views and the latest target 3D model are obtained when the preset conditions are met;

[0150] The model generation module 505 is used to perform texture mapping on the latest target three-dimensional model based on the latest multi-views to obtain a three-dimensional object model.

[0151] In one embodiment, the normal prediction module includes:

[0152] A first determination submodule is used to determine whether there is a latest multi-view currently, and if so, use the latest multi-view currently as the determined multi-view;

[0153] A second determination submodule is configured to extract information from the single view if it does not exist, obtain description information and key information, input the description information, key information and a preset noise ratio into a preset multi-view diffusion model to generate a multi-view corresponding to the single view, and use the generated multi-view as the determined multi-view;

[0154] The normal prediction submodule is used to input multiple views into a preset normal prediction network to predict the normal maps of the multiple views based on the normal prediction network to obtain predicted normal maps; wherein the normal prediction network is trained using multiple views rendered by real three-dimensional models and their corresponding normal maps as training data.

[0155] In one embodiment, the model updating module includes:

[0156] The normal rendering submodule is used to determine, in one iteration, the initial three-dimensional model corresponding to this iteration, and render the initial three-dimensional model to obtain a rendered normal map corresponding to the initial three-dimensional model;

[0157] A model updating submodule, which is used to construct a loss function in this iteration based on the predicted normal map and the rendered normal map of the initial three-dimensional model, and to update the initial three-dimensional model according to the loss function, and to use the updated initial three-dimensional model as the corresponding initial three-dimensional model in the next iteration round;

[0158] The judgment submodule is used to judge whether the current iteration round reaches the preset iteration round. If not, it enters the next iteration round. If it reaches, the initial three-dimensional model completed by the iteration is determined as the target three-dimensional model.

[0159] In one embodiment, the model updating module includes:

[0160] The normal vector calculation submodule is used to calculate the normal vector corresponding to each vertex of the target three-dimensional model to obtain the normal vector corresponding to the target three-dimensional model;

[0161] The vector projection submodule is used to project the target three-dimensional model and its corresponding normal vector onto a two-dimensional plane to obtain a normal projection map, and render the normal projection map to obtain a rendered normal map.

[0162] In one embodiment, the iterative optimization module includes:

[0163] A noise determination submodule is used to determine the noise ratio to be added in this round, and to superimpose noise on multiple views according to the noise ratio; wherein the noise ratio superimposed in each round decreases as the round increases;

[0164] The information extraction submodule is used to obtain a preset multi-view diffusion model and extract information from a single view to obtain single view information;

[0165] The view generation submodule is used to input the multi-view after superimposing noise, the single view information and the rendered normal map into the multi-view diffusion model, so that the multi-view diffusion model generates a multi-view matching the single view information and the rendered normal map based on the multi-view after superimposing noise, so as to obtain a new multi-view.

[0166] In one embodiment, the noise determination submodule includes:

[0167] A function value acquisition unit, used for acquiring a loss function value when forming the target three-dimensional model when the target three-dimensional model is obtained;

[0168] The noise determination unit is used to obtain a preset mapping function and input the loss function value into the mapping function to determine the noise ratio to be added in this round.

[0169] In one embodiment, the model generation module includes:

[0170] The information acquisition submodule is used to obtain the latest pixel information of each view in the multi-view;

[0171] An information mapping submodule, used to map the pixel information of each view to the corresponding position of the latest target three-dimensional model in turn, so as to obtain the three-dimensional model to be optimized;

[0172] The blind spot judgment submodule is used to judge whether there is a visual blind spot in the three-dimensional model to be optimized. If not, the three-dimensional model to be optimized is determined as a three-dimensional object model;

[0173] The texture filling submodule is used to obtain the texture features of the surrounding area of ​​the viewing blind area in the three-dimensional model to be optimized, if any, and to fill the viewing blind area in the three-dimensional model to be optimized with texture according to the texture features to obtain a three-dimensional object model.

[0174] The division of each module in the above-mentioned three-dimensional object model generation device is only for illustration. In other embodiments, the three-dimensional object model generation device can be divided into different modules as needed to complete all or part of the functions of the above-mentioned three-dimensional object model generation device. Each module in the above-mentioned three-dimensional object model generation device can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0175] In one embodiment, the present application also provides a storage medium storing computer-readable instructions. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the three-dimensional object model generation method as described in any of the above embodiments.

[0176] In one embodiment, the present application also provides a computer device having computer-readable instructions stored therein. When the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the three-dimensional object model generation method as described in any one of the above embodiments.

[0177] Indicatively, if Fig. 9 As shown, Fig. 9 This is a schematic diagram of the internal structure of a computer device provided in an embodiment of the present application. The computer device 600 may be provided as a server. Fig. 9 The computer device 600 includes a processing component 602, which further includes one or more processors, and a memory resource represented by a memory 601, for storing instructions executable by the processing component 602, such as an application. The application stored in the memory 601 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 602 is configured to execute the instructions to perform the three-dimensional object model generation method of any of the above embodiments.

[0178] The computer device 600 may further include a power supply component 603 configured to perform power management of the computer device 600, a wired or wireless network interface 604 configured to connect the computer device 600 to a network, and an input / output (I / O) interface 605. The computer device 600 may operate based on an operating system stored in the memory 601, such as Windows Server TM, Mac OS X TM, Unix TM, Linux TM, Free BSD TM, or the like.

[0179] Those skilled in the art will understand that Fig. 9 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.

[0180] Finally, it should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish an entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "comprise" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only include those elements, but also include other elements that are not clearly listed, or also include elements inherent to such process, method, article or equipment. In the absence of more restrictions, the elements limited by the sentence "comprise one..." do not exclude the existence of other identical elements in the process, method, article or equipment including the elements. Herein, the singular "one", "one" and "described / the" may also include plural forms, unless the context clearly indicates another way. It should also be understood that the terms "include / comprise" or "have" etc. specify the existence of stated features, wholes, steps, operations, components, parts or combinations thereof, but do not exclude the possibility of the existence or addition of one or more other features, wholes, steps, operations, components, parts or combinations thereof. At the same time, the term "and / or" used in this specification includes any and all combinations of the relevant listed items.

[0181] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The various embodiments can be combined as needed, and the same or similar parts can refer to each other.

[0182] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for generating a three-dimensional object model, characterized in that: The method comprises: Obtain a single view of the three-dimensional object to be generated; Determining multiple views, and predicting normal maps of the multiple views to obtain predicted normal maps; Determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model; Superimposing noise on the multiple views, and generating a new multiple view according to the multiple views after superimposing noise, the single view and the rendered normal map, and continuing to perform the step of determining the multiple views and subsequent steps until the latest multiple views and the latest target three-dimensional model are obtained when preset conditions are met; The latest target 3D model is texture mapped based on the latest multi-views to obtain a 3D object model.

2. The method for generating a three-dimensional object model according to claim 1, characterized in that: The determining of multiple views and predicting normal maps of the multiple views to obtain predicted normal maps includes: Determine whether there is a latest multi-view currently, and if so, use the latest multi-view currently as the determined multi-view; If not, extracting information from the single view to obtain description information and key information, and inputting the description information, the key information and a preset noise ratio into a preset multi-view diffusion model to generate a multi-view corresponding to the single view, and using the generated multi-view as the determined multi-view; The multiple views are input into a preset normal prediction network to predict the normal maps of the multiple views based on the normal prediction network to obtain predicted normal maps; wherein the normal prediction network is trained using multiple views rendered by a real three-dimensional model and their corresponding normal maps as training data.

3. The method for generating a three-dimensional object model according to claim 1, wherein: The determining of the initial three-dimensional model and iteratively updating the initial three-dimensional model according to the predicted normal map includes: In one iteration, an initial three-dimensional model corresponding to this iteration is determined, and the initial three-dimensional model is rendered to obtain a rendering normal map corresponding to the initial three-dimensional model; Constructing a loss function in this iteration based on the predicted normal map and the rendered normal map of the initial three-dimensional model, updating the initial three-dimensional model according to the loss function, and using the updated initial three-dimensional model as the corresponding initial three-dimensional model in the next iteration round; It is determined whether the current iteration round reaches the preset iteration round. If not, the next iteration round is entered. If reached, the initial three-dimensional model after the iteration is completed is determined as the target three-dimensional model.

4. The method for generating a three-dimensional object model according to claim 1, characterized in that: The step of rendering the target three-dimensional model to obtain a rendering normal map corresponding to the target three-dimensional model includes: Calculating a normal vector corresponding to each vertex of the target three-dimensional model to obtain a normal vector corresponding to the target three-dimensional model; The target three-dimensional model and its corresponding normal vector are projected onto a two-dimensional plane to obtain a normal projection map, and the normal projection map is rendered to obtain a rendered normal map.

5. The method for generating a three-dimensional object model according to any one of claims 1 to 4, characterized in that: The step of superimposing noise on the multiple views and generating new multiple views according to the multiple views after superimposing noise, the single view and the rendered normal map comprises: Determine the noise ratio to be added in this round, and superimpose noise on the multiple views according to the noise ratio; wherein the noise ratio superimposed in each round decreases as the round increases; Acquire a preset multi-view diffusion model, and extract information from the single view to obtain single view information; The multi-view after superimposing noise, the single view information and the rendered normal map are input into the multi-view diffusion model, so that the multi-view diffusion model generates a multi-view matching the single view information and the rendered normal map based on the multi-view after superimposing noise, so as to obtain a new multi-view.

6. The method for generating a three-dimensional object model according to claim 5, characterized in that: Determining the noise ratio to be added in this round includes: When the target three-dimensional model is obtained, obtaining a loss function value when forming the target three-dimensional model; Obtain a preset mapping function and input the loss function value into the mapping function to determine the noise ratio to be added in this round.

7. The method for generating a three-dimensional object model according to claim 1, characterized in that: The method of performing texture mapping on the latest target three-dimensional model based on the latest multi-view to obtain a three-dimensional object model includes: Get the latest pixel information of each view in the multi-view; Mapping the pixel information of each view to the corresponding position of the latest target three-dimensional model in turn to obtain the three-dimensional model to be optimized; Determine whether the three-dimensional model to be optimized has a blind spot in viewing angle, and if not, determine the three-dimensional model to be optimized as a three-dimensional object model; If it exists, the texture features of the surrounding area of ​​the viewing blind area in the three-dimensional model to be optimized are obtained, and the viewing blind area in the three-dimensional model to be optimized is texture filled according to the texture features to obtain a three-dimensional object model.

8. A three-dimensional object model generation device, characterized in that: The device comprises: A single view acquisition module, used to acquire a single view of the three-dimensional object to be generated; A normal prediction module, used for determining multiple views and predicting normal maps of the multiple views to obtain predicted normal maps; A model updating module, used for determining an initial three-dimensional model, and iteratively updating the initial three-dimensional model according to the predicted normal map, and after obtaining a target three-dimensional model, rendering the target three-dimensional model to obtain a rendered normal map corresponding to the target three-dimensional model; an iterative optimization module, configured to superimpose noise on the multi-views, generate a new multi-view according to the multi-views after superimposing noise, the single view and the rendered normal map, and continue to execute the single view acquisition module, the normal prediction module, the model update module and the iterative optimization module until the latest multi-views and the latest target three-dimensional model are acquired when preset conditions are met; The model generation module is used to perform texture mapping on the latest target three-dimensional model based on the latest multi-views to obtain a three-dimensional object model.

9. A storage medium, characterized in that: The storage medium stores computer-readable instructions, and when the computer-readable instructions are executed by one or more processors, the one or more processors execute the steps of the three-dimensional object model generation method as described in any one of claims 1 to 7.

10. A computer device, characterized in that: include: one or more processors, and memory; The memory stores computer-readable instructions, and when the computer-readable instructions are executed by the one or more processors, the steps of the three-dimensional object model generation method according to any one of claims 1 to 7 are performed.

Images