Model generation method and device, computer program product and electronic equipment

Abstract

The invention relates to the technical field of computers, in particular to a model generation method and device, a computer program product and electronic equipment. The model generation method comprises the steps that according to a two-dimensional image of a target object, global image features and a visual angle image sequence are extracted, and visual angle images in the visual angle image sequence correspond to different visual angles of the target object; extracting local patch features of each view image, performing aggregation processing on the local patch features, and constructing multi-view global aggregation features; and based on the global image features and the global aggregation features, performing noise addition and / or de-noising processing in a three-dimensional space through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object, and generating a target three-dimensional model according to the mesh model containing the texture information. The quality of the generated three-dimensional model can be improved, and the manufacturing cost can be reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and more specifically, to a model generation method, a model generation device, a computer program product, and an electronic device. Background Art

[0002] With the development of computer and Internet technology, more and more fields need to quickly generate the required three-dimensional models. For example, the demand for high-quality three-dimensional models in fields such as games, education, entertainment, and training is increasing, so various methods for making three-dimensional models are gradually emerging.

[0003] Traditional methods of generating 3D models involve complex steps such as modeling and texture processing, and rely on professional techniques and tools. The generated 3D models often require a lot of post-production corrections to meet the requirements, which affects the quality of the generated 3D models and increases production costs.

[0004] It should be noted that the information disclosed in the above background technology section is only used to enhance the understanding of the background of the present disclosure, and therefore may include information that does not constitute the prior art known to ordinary technicians in the field. Summary of the invention

[0005] The purpose of the present disclosure is to provide a model generation method and device, a computer program product and an electronic device, thereby improving the quality of generated three-dimensional models and reducing production costs at least to a certain extent.

[0006] Other features and advantages of the present disclosure will become apparent from the following detailed description, or may be learned in part by the practice of the present disclosure.

[0007] According to one aspect of the present disclosure, a model generation method is provided, comprising: extracting global image features and a perspective image sequence based on a two-dimensional image of a target object, the perspective images in the perspective image sequence respectively corresponding to different perspectives of the target object; extracting local patch features of each of the perspective images, aggregating the local patch features, and constructing global aggregate features of multiple perspectives; based on the global image features and the global aggregate features, performing noise addition and / or denoising processing in three-dimensional space through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object, and generating a target three-dimensional model based on the mesh model containing texture information.

[0008] In an exemplary embodiment of the present disclosure, the local patch features of each of the perspective images are extracted, the local patch features are aggregated, and a global aggregate feature of multiple perspectives is constructed, including: respectively extracting the local patch features corresponding to each of the perspective images; for each voxel in the three-dimensional space, aggregating the local patch features of the voxel corresponding to different perspective images to obtain the voxel aggregate feature of the voxel; and aggregating the voxel aggregate features of each of the voxels to generate the global aggregate feature.

[0009] In an exemplary embodiment of the present disclosure, aggregating the local patch features of the voxels corresponding to different perspective images to obtain the voxel aggregation features of the voxels includes: mapping the voxels to at least one perspective image in the perspective image sequence using a preset projection matrix; aggregating the local patch features of the corresponding mapping positions in the at least one perspective image to obtain the multi-perspective voxel aggregation features of the voxels.

[0010] In an exemplary embodiment of the present disclosure, aggregating the voxel aggregation features of each of the voxels to generate the global aggregation features includes: mapping the voxel aggregation features of each of the voxels to the three-dimensional space, and aggregating the voxel aggregation features mapped to the three-dimensional space to generate the global aggregation features.

[0011] In an exemplary embodiment of the present disclosure, the method of performing denoising and noise removal processing in three-dimensional space based on the global image features and the global aggregation features through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object includes: adding random noise in the three-dimensional space; performing denoising processing in the three-dimensional space through the pre-trained diffusion model according to the global image features and the global aggregation features to generate a three-dimensional occupied volume, wherein the three-dimensional occupied volume describes the basic structure and shape of the target object; dividing the voxels corresponding to the three-dimensional occupied volume into multiple sub-voxels to construct a three-dimensional sparse volume; performing denoising and / or noise removal processing on the three-dimensional sparse volume through the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the mesh model containing texture information.

[0012] In an exemplary embodiment of the present disclosure, the denoising and / or denoising processing is performed on the three-dimensional sparse body according to the global image features and the global aggregation features by using the pre-trained diffusion model to obtain the mesh model containing texture information, including: adding random noise to the three-dimensional sparse body; and denoising the three-dimensional sparse body to which random noise is added by using the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the mesh model containing texture information.

[0013] In an exemplary embodiment of the present disclosure, according to the global image features and the global aggregation features, the three-dimensional sparse body after adding random noise is denoised by the pre-trained diffusion model to obtain the grid model containing texture information, including: determining a closed surface grid based on the three-dimensional sparse body; constructing a signed distance field grid according to the distance between the voxels corresponding to the three-dimensional sparse body and the closed surface, and the signed distance field grid is used to describe the distance between the voxels corresponding to the three-dimensional sparse body and the closed surface.

[0014] In an exemplary embodiment of the present disclosure, the three-dimensional sparse volume to which random noise is added is denoised by the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the grid model containing texture information, including: back-projecting the view image sequence to the three-dimensional space to obtain a three-dimensional point cloud; based on the three-dimensional point cloud, according to the global image features and the global aggregation features, the three-dimensional sparse volume to which random noise is added is denoised by the pre-trained diffusion model to obtain the texture grid, and the texture grid is used to describe the texture information of the voxel.

[0015] In an exemplary embodiment of the present disclosure, the method further comprises: extracting an isosurface based on the signed distance field grid to generate the grid model containing texture information.

[0016] In an exemplary embodiment of the present disclosure, generating a target three-dimensional model based on the mesh model containing texture information includes: UV unfolding the mesh model to obtain a UV unfolding result; rasterizing the mesh model to obtain a two-dimensional plane image; querying a pre-trained texture mesh based on depth information of the two-dimensional plane image, and adjusting the texture information based on the query result; and baking the mesh model based on the adjusted texture information and the UV unfolding result to generate the target three-dimensional model.

[0017] In an exemplary embodiment of the present disclosure, the method further includes: obtaining a target perspective image sequence corresponding to the target three-dimensional model, wherein the perspective image in the target perspective image sequence is extracted by using shooting parameter information of the perspective image sequence corresponding to the two-dimensional image; obtaining a pixel difference between the target perspective image sequence and a corresponding perspective image in the perspective image sequence; and adjusting the texture information according to the pixel difference.

[0018] In an exemplary embodiment of the present disclosure, extracting global image features and a perspective image sequence based on a two-dimensional image of a target object includes: performing feature extraction on the two-dimensional image to obtain the global image features; performing image generation processing on the two-dimensional image according to the global image features to obtain a reference image; based on the reference image, using a pre-trained multi-perspective image generation model to obtain perspective images of the target object at different perspectives to obtain the perspective image sequence.

[0019] In an exemplary embodiment of the present disclosure, the image generation processing is performed on the two-dimensional image according to the global image features to obtain a reference image, including: adding random noise to a default latent space to obtain a noisy latent space; denoising the noisy latent space according to the global image features to determine the latent space features according to the denoising results; and image reconstruction is performed based on the latent space features to obtain the reference image.

[0020] According to one aspect of the present disclosure, a model generation device is provided, including: an image acquisition module, used to extract global image features and a perspective image sequence based on a two-dimensional image of a target object, the perspective images in the perspective image sequence respectively corresponding to different perspectives of the target object; a feature extraction module, used to extract local patch features of each of the perspective images, aggregate the local patch features, and construct a global aggregate feature of multiple perspectives; a model generation module, used to perform noise addition and / or denoising in three-dimensional space through a pre-trained diffusion model based on the global image features and the global aggregate features, to obtain a mesh model containing texture information corresponding to the target object, and generate a target three-dimensional model based on the mesh model containing texture information.

[0021] According to one aspect of the present disclosure, a computer program product is provided, including a computer program, wherein when the computer program is executed by a processor, any one of the methods described above is implemented.

[0022] According to one aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing executable instructions of the processor; wherein the processor is configured to execute any one of the above-mentioned methods by executing the executable instructions.

[0023] The model generation method in the exemplary embodiment of the present disclosure, on the one hand, extracts global image features and view image sequences based on the two-dimensional image of the target object, and extracts local patch features of each view image feature, aggregates the local patch features to construct a multi-view global aggregate feature, and based on the global image features and the global aggregate feature, performs noise and / or denoising on the three-dimensional model through a pre-trained diffusion model to obtain a mesh model containing texture information of the target object. This process integrates multi-dimensional features from different viewpoints to obtain global aggregate features, providing rich multi-view conditions for generating a three-dimensional model, and by combining global image features and global aggregate features, completely extracts the global and local features of the target object, providing rich feature conditions for generating a model, and increasing the fidelity of the generated model from both the viewpoint and the feature. On the other hand, a three-dimensional model can be automatically generated based on an image of the target object, simplifying the process of generating a model and reducing the investment cost of production manpower and tools.

[0024] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] The above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood by reading the detailed description below with reference to the accompanying drawings. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, in which:

[0026] Figure 1 A diagram showing an application environment involved in a model generation method according to an exemplary embodiment of the present disclosure is shown.

[0027] Figure 2 A flow chart of a model generation method according to an exemplary embodiment of the present disclosure is shown.

[0028] Figure 3 A flowchart of an implementation method for constructing multi-perspective global aggregation features according to an exemplary embodiment of the present disclosure is shown.

[0029] Figure 4 A schematic diagram of obtaining voxel aggregation features according to an exemplary embodiment of the present disclosure is shown.

[0030] Figure 5 An exemplary embodiment of the present disclosure is shown. Figure 4 Corresponding schematic diagram of generating global aggregate features based on voxel aggregate features.

[0031] Figure 6 A flowchart of an implementation method of generating a mesh model containing texture information through a diffusion process according to an exemplary embodiment of the present disclosure is shown.

[0032] Figure 7 A flow chart of performing denoising processing on a three-dimensional sparse volume to which random noise is added to obtain a mesh model according to an exemplary embodiment of the present disclosure is shown.

[0033] Figure 8 A flowchart of another method of performing denoising processing on a three-dimensional sparse volume to which random noise is added to obtain a mesh model according to an exemplary embodiment of the present disclosure is shown.

[0034] Fig. 9 A flowchart of an implementation method for acquiring global image features and view image sequences according to an exemplary embodiment of the present disclosure is shown.

[0035] Fig.10 A complete flow chart of a model generation method according to an exemplary embodiment of the present disclosure is shown.

[0036] Fig.11 A schematic diagram of the composition of a model generating device according to an exemplary embodiment of the present disclosure is shown.

[0037] Fig.12 A block diagram of an electronic device according to an exemplary embodiment of the present disclosure is shown.

[0038] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts. DETAILED DESCRIPTION

[0039] The exemplary embodiments will now be described more fully with reference to the accompanying drawings. However, the exemplary embodiments can be implemented in a variety of forms and should not be construed as limited to the examples set forth herein; rather, these embodiments are provided so that the present disclosure will be more comprehensive and complete and the concepts of the exemplary embodiments will be fully conveyed to those skilled in the art. The same reference numerals in the figures represent the same or similar structures, and thus their detailed description will be omitted.

[0040] In addition, the described features, structures or characteristics may be combined in one or more embodiments in any suitable manner. In the following description, many specific details are provided to provide a full understanding of the embodiments of the present disclosure. However, those skilled in the art will appreciate that the technical solutions of the present disclosure may be practiced without one or more of the specific details, or other methods, components, devices, steps, etc. may be adopted. In other cases, known structures, methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the present disclosure.

[0041] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically independent entities. That is, these functional entities may be implemented in software form, or these functional entities or parts of functional entities may be implemented in one or more software hardened modules, or these functional entities may be implemented in different networks and / or processor devices and / or microcontroller devices.

[0042] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0043] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0044] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0045] The technical solution provided by the exemplary embodiments of the present disclosure relates to artificial intelligence machine learning technology, which uses machine learning technology to generate a three-dimensional model based on a two-dimensional image of a target object.

[0046] The model generation method provided by the exemplary embodiments of the present disclosure can be applied to Figure 1In the application environment shown, the terminal 101 communicates with the server 102 via a network. The data storage system can store data that the server 102 needs to process. The data storage system can be integrated on the server 102, or placed on the cloud or other network servers.

[0047] In an exemplary embodiment, the model generation method provided by the exemplary embodiment of the present disclosure may be executed by the server 102, and accordingly, the model generation device is disposed in the server 102. Correspondingly, in this mode of execution by the server 102, the server 102 may start to execute the steps in the technical solution of the exemplary embodiment of the present disclosure in response to a trigger command, wherein the trigger command may be sent by a terminal used by a user, or may be triggered locally by the server in response to some automation events.

[0048] The server 102 may first extract global image features and a view image sequence based on a two-dimensional image of the target object, then extract local patch features of each view image, aggregate the local patch features, and construct a multi-view global aggregate feature; then, based on the global image features and the global aggregate feature, perform noise addition and / or denoising in three-dimensional space through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object, and generate a target three-dimensional model based on the mesh model containing texture information.

[0049] The server 102 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms. The server 102 may perform background tasks.

[0050] Furthermore, in another exemplary embodiment, the terminal 101 may also have similar functions to the server 102 , so as to execute the model generation method provided by the exemplary embodiment of the present disclosure.

[0051] The terminal 101 may be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an IoT device, and a portable wearable device. The IoT device may be a smart speaker, a smart TV, a smart air conditioner, and a smart vehicle-mounted device, etc. The portable wearable device may be a smart watch, a smart bracelet, a head-mounted device, etc. The terminal 101 may also be referred to as a mobile terminal, a terminal device, a mobile device, etc. The exemplary embodiments of the present disclosure do not limit the type of the terminal 101.

[0052] In addition, the technical solution of the exemplary embodiment of the present disclosure can also be executed collaboratively by the terminal 101 and the server 102. In this manner of collaborative execution by the terminal 101 and the server 102, some steps in the technical solution provided by the exemplary embodiment of the present disclosure are executed by the terminal 101, while other steps are executed by the server 102. It should be noted that in this manner of collaborative execution by the terminal 101 and the server 102, the steps respectively executed by the terminal 101 and the server 102 can be dynamically adjusted according to actual conditions, and there is no special limitation on this.

[0053] The terminal 101 and the server 102 may be connected directly or indirectly via wireless communication, and the exemplary embodiments of the present disclosure do not impose any special limitation thereto.

[0054] In the fields of games, entertainment, e-commerce, training, etc., there is a demand for rapid generation of 3D models. For example, in game production, 3D models of game characters are quickly generated, and products are displayed through 3D models on e-commerce platforms, etc. However, the current 3D model generation scheme involves complex steps such as modeling and texture processing, and relies on professional technology and tools. The generated 3D models often require a lot of post-production corrections to meet the needs, which affects the quality of the generated 3D models and increases the production costs.

[0055] Based on this, an exemplary embodiment of the present disclosure provides a model generation method, which can be applied to the scenario of generating a three-dimensional model based on a single image. The exemplary embodiment of the present disclosure can improve the quality of the generated three-dimensional model, simplify the production process, and reduce the production cost.

[0056] refer to Figure 2 FIG. 1 is a flow chart of a model generation method according to an exemplary embodiment of the present disclosure, referring to FIG. Figure 2 As shown, the model generation method includes steps S210 to S230, specifically:

[0057] In step S210, global image features and view image sequences are extracted based on the two-dimensional image of the target object.

[0058] In the exemplary embodiments of the present disclosure, the target object refers to the object for which a three-dimensional model needs to be generated, which may be a person or an object. The two-dimensional image may be an image obtained by photographing the target object through an electronic device, or an image downloaded or received through a network, without limitation.

[0059] Global image features refer to image features extracted directly from two-dimensional images, which are used to describe the content of the target object in detail, including but not limited to type, color, material, pattern and other features. In order to accurately extract global image features, the two-dimensional image can be preprocessed, including but not limited to operations such as cropping, scaling and denoising, to improve the clarity and quality of the two-dimensional image. Among them, a pre-trained multimodal model can be used to perform image recognition on the two-dimensional image to extract global image features. The multimodal model is a model that can process and understand images and texts. Therefore, when extracting global image features, text description information can also be obtained, and the required global image features are extracted based on the text description information, so that the global image features are more in line with actual needs. The multimodal model can be trained using sample data, and the exemplary embodiments of the present disclosure do not specifically limit the training process of the multimodal model.

[0060] Among them, by extracting the global image features of the two-dimensional image based on the multimodal model, the corresponding image description information can be generated to describe the image content of the two-dimensional image. The image description information can be output in a structured format to provide basic data for the subsequent generation of images or three-dimensional models.

[0061] A perspective image sequence refers to a sequence of perspective images of a target object from different perspectives, that is, the perspective images in the perspective image sequence correspond to different perspectives of the target object, and perspective prediction is performed based on the two-dimensional image of the target object to extract the perspective image sequence. For example, the extracted multi-perspective image sequence may be a sequence of six perspective images, and of course it may also include other numbers of perspective images, without limitation.

[0062] The pre-trained multi-view image generation model can be used to obtain the view images of the target object at different view angles. As an example, a pre-trained two-dimensional diffusion model can be selected as a basis, and fine-tuned and trained to obtain a multi-view image generation model.

[0063] Specifically, a two-dimensional diffusion model (such as a stable diffusion model) is trained with image samples with known viewpoint labels, so that the model learns to generate viewpoint images at different viewpoints from a single image of the object.

[0064] The view image sequence in the training sample can be obtained by controlling the camera view data, and the camera view data can be set in the following way: use a fixed absolute altitude angle and relative azimuth angle to locate the camera position of different view images, for example, the absolute altitude angle is alternately 25° and -25°, and the relative azimuth angle starts from 30° and increases by 60° each time, then 6 view images can be generated. Of course, the camera view data here is only exemplary and can be flexibly set according to actual needs.

[0065] The different perspective images can be used to construct a perspective image sequence in a preset order, for example, by tiling the perspective images in sequence, and there is no limitation on this. The different perspective images can be arranged in a composite image or stored separately, and together provide complete visual information of the object from multiple angles.

[0066] Furthermore, the model is trained by using image samples as local input conditions and tiled view image sequences as training target images, by text encoding the image samples as the text encoding input of the image, to provide complete semantic understanding. The decoder part is used to decode the image samples into latent feature images, and the latent feature images are input into the training module of Stable Diffusion UNet.

[0067] In step S220, local patch features of each view image are extracted, and the local patch features are aggregated to construct a multi-view global aggregate feature.

[0068] In an exemplary embodiment of the present disclosure, a local patch feature refers to a local feature extracted from a perspective image. The local feature is extracted by defining a local area. The defined local area contains local information of the perspective image. The details and texture information of the perspective image are captured by the local patch feature. The method for extracting the local patch feature includes but is not limited to using a convolutional neural network (CNN) for feature extraction. The local features in the perspective image are automatically learned through the convolution operation of the CNN. The local features may be edges, corners or other important image elements.

[0069] The local patch features of each view image are aggregated, and the obtained multi-view global aggregation features can ensure that the complete features of the target object observed from different viewpoints are captured, providing accurate and consistent multi-view conditions for generating three-dimensional models.

[0070] In step S230, based on global image features and global aggregation features, noise addition and / or denoising processing is performed in three-dimensional space through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object, so as to generate a target three-dimensional model based on the mesh model containing texture information.

[0071] In an exemplary embodiment of the present disclosure, a 2D global image feature and a multi-view global aggregation feature are combined as guiding conditions, and a 3D diffusion process is performed through a pre-trained diffusion model to generate a mesh model containing texture information of the target object. In the 3D diffusion process based on the pre-trained 3D model, noise addition and / or denoising can be performed according to actual needs.

[0072] After obtaining the mesh model containing texture information, the mesh model containing texture information can be rendered to generate a target three-dimensional model. Optionally, the texture information of the mesh model can be adjusted, and then the model can be baked according to the adjustment result to obtain the target three-dimensional model.

[0073] The model generation method in the exemplary embodiment of the present disclosure, on the one hand, extracts global image features and view image sequences based on the two-dimensional image of the target object, and extracts local patch features of each view image feature, aggregates the local patch features to construct a multi-view global aggregate feature, and based on the global image features and the global aggregate feature, performs noise and / or denoising on the three-dimensional model through a pre-trained diffusion model to obtain a mesh model containing texture information of the target object. This process integrates multi-dimensional features from different viewpoints to obtain global aggregate features, providing rich multi-view conditions for generating a three-dimensional model, and by combining global image features and global aggregate features, completely extracts the global and local features of the target object, providing rich feature conditions for generating a model, and increasing the fidelity of the generated model from both the viewpoint and the feature. On the other hand, a three-dimensional model can be automatically generated based on an image of the target object, simplifying the process of generating a model and reducing the investment cost of production manpower and tools.

[0074] In an exemplary embodiment, a method for constructing a multi-view global aggregation feature is provided. Figure 3 As shown, extract the local patch features of each view image, aggregate the local patch features, and construct the multi-view global aggregation features including:

[0075] Step S310: extracting local patch features corresponding to each perspective image respectively.

[0076] A pre-trained convolutional neural network can be used to extract local patch features from each perspective image, and the local patch features represent the feature information of the local area of ​​the perspective image.

[0077] Step S320: for each voxel in the three-dimensional space, aggregate the local patch features of the voxel corresponding to different viewing angle images to obtain the voxel aggregation features of the voxel.

[0078] Voxel refers to a basic unit in three-dimensional space, similar to a small cube in a three-dimensional grid. The entire three-dimensional space can be composed of multiple voxels, which together describe the distribution of objects in three-dimensional space. Voxels are usually used in model processing to represent the detailed information of objects. Voxels correspond to local patch features of the view image. By mapping voxels to the view image, the local patch features at the mapping positions of the voxels in the view image can be used as the local patch features of the voxels corresponding to the view image.

[0079] Specifically, a preset projection matrix can be used to map voxels to at least one perspective image in a perspective image sequence, and local patch features of corresponding mapping positions in at least one perspective image are aggregated to obtain multi-perspective voxel aggregation features of the voxels.

[0080] The preset projection matrix refers to a predetermined transformation matrix used to map points (such as voxels) in a three-dimensional space to a two-dimensional image plane. The preset projection matrix includes at least the position information, angles (pitch angle, yaw angle, roll angle) and focal length of the camera, and can accurately map the voxels in the three-dimensional space to the position of a specific two-dimensional image (such as a perspective image), so that the corresponding relationship between each perspective image can be accurately calculated.

[0081] When the voxel is mapped to the view image using the preset projection matrix, the spatial coordinates of the voxel can be converted into the coordinates under the camera view, and then the corresponding position of the voxel on the view image is determined using the preset projection matrix, so that the feature of the voxel is mapped to the corresponding position on the view image according to the mapping position. Since the view image sequence includes multiple view images, each voxel may be mapped to multiple view images, so that the feature information of the voxel corresponding to different view angles can be accumulated, so that the voxel aggregation feature of each voxel gathers the local patch features in different view images.

[0082] As an example, Figure 4 A schematic diagram of obtaining voxel aggregation features is shown. The following takes a view image sequence including 6 view images as an example to illustrate the process of obtaining voxel aggregation features of voxels.

[0083] First, the voxel A in the three-dimensional space is mapped to each view image in the view image sequence using the preset projection matrix X, and then the mapping position of voxel A in each view image is determined, and finally the local patch features (represented by A') corresponding to each mapping position are aggregated to obtain the voxel aggregation feature A' of voxel A. Similarly, for voxel B, the mapping position of voxel B in each view image is determined, and the local patch features corresponding to each mapping position are aggregated to obtain the voxel aggregation feature B' of voxel B. The voxel aggregation features of other voxels are obtained in a similar way and are not listed one by one.

[0084] In some optional embodiments, for each voxel, the local patch features corresponding to each mapping position are aggregated by a multi-layer perceptron network and a maximum pooling network with shared weights, thereby ensuring that each voxel can integrate the perspective information from images with different perspectives to obtain a three-dimensional feature representation of the voxel with comprehensive features. The local patch features corresponding to each mapping position are converted into a consistent feature representation by a multi-layer perceptron network, and then the consistent feature representations are integrated by a maximum pooling operation to obtain voxel aggregate features.

[0085] Of course, the exemplary embodiments of the present disclosure may select other networks to implement feature aggregation operations according to actual conditions, and there is no limitation on this.

[0086] Step S330: Aggregate the voxel aggregation features of each voxel to generate a global aggregation feature.

[0087] After obtaining the voxel aggregation features of each voxel, the voxel aggregation features of each voxel are aggregated to form a global aggregation feature containing complete three-dimensional information of the target object.

[0088] The voxel aggregation features of each voxel may be mapped to the three-dimensional space, and the voxel aggregation features mapped to the three-dimensional space may be aggregated to generate a global aggregation feature.

[0089] Specifically, a preset projection matrix may be used to map the voxel aggregation features of each voxel to a three-dimensional space, and then the voxel aggregation features mapped to the same position are aggregated to obtain a global aggregation feature.

[0090] As an example, Figure 5 for Figure 4 The corresponding schematic diagram of generating global aggregation features based on voxel aggregation features is as follows: Figure 5 First, the voxel aggregation feature A' of voxel A is mapped to the three-dimensional space, and the voxel aggregation feature B' of voxel B is mapped to the three-dimensional space, and the voxel aggregation features of other voxels are mapped to the three-dimensional space, and then multiple voxel aggregation features mapped to the same position are aggregated (including but not limited to accumulation processing or averaging processing), so as to generate a global aggregation feature based on all the aggregation results, for example, obtaining 64×64×64 three-dimensional feature volume data.

[0091] The complete three-dimensional information of the target object is represented by global aggregation features, and multi-dimensional features from different perspectives are integrated to provide rich multi-perspective feature conditions for the subsequent generation of three-dimensional models.

[0092] In an exemplary embodiment, a method for generating a mesh model containing texture information through a diffusion process is provided. In this implementation, a two-stage diffusion process is included. Figure 6 As shown, based on global image features and global aggregation features, noise addition and / or denoising processing is performed in three-dimensional space through a pre-trained diffusion model to obtain a grid model containing texture information corresponding to the target object, which may include steps S610 to S640, wherein the first diffusion stage includes steps S610 and S620, and the second diffusion stage includes steps S630 and S640.

[0093] Step S610: adding random noise in the three-dimensional space.

[0094] Random noise can be added in the three-dimensional space as the initial state of the generation process. Among them, normal distribution noise (Gaussian Noise) can be used to add random noise for initialization. Of course, other ways of adding random noise can also be selected according to actual needs, including but not limited to normal distribution noise.

[0095] Step S620: performing denoising processing in three-dimensional space using a pre-trained diffusion model according to global image features and global aggregation features to generate a three-dimensional occupied volume.

[0096] The three-dimensional occupancy volume describes the basic structure and shape of the target object, which is a three-dimensional grid that describes the occupancy of the target object in the three-dimensional space. Based on the pre-trained diffusion model, according to the global image features and global aggregation features, the three-dimensional space is subjected to multi-step denoising processing with the global image features and global aggregation features as the model guidance conditions, and each step uses the output of the previous step as input.

[0097] The purpose of adding noise is to increase the diversity of the data, that is, to simulate creative random possibilities. By introducing random noise in the three-dimensional space, the model is forced to learn how to recover the original information from partially damaged data. The purpose of denoising is to enable the model to learn how to recover a clear and accurate three-dimensional occupancy volume from the noisy three-dimensional space. This process is implemented through a deep learning model (such as a stable diffusion model). The deep learning model requires multiple iterative processing. Each iteration will make the obtained three-dimensional occupancy volume closer and closer to the original, noise-free state, and based on the global image features and global aggregation features, the three-dimensional occupancy volume can accurately capture the basic structure and shape characteristics of the target object.

[0098] In some optional embodiments, in the diffusion process of the first stage, the denoising loss may be achieved as follows:

[0099]

[0100] Where, is the sampling noise, t is the time step, x0 is the data point without noise (i.e., the low-resolution 3D occupied volume of the target), is the output target of the denoising process, and x t is the noisy version, that is, the data point after the noise processing at the preset time step during the diffusion process. "Version" refers to the state of the three-dimensional data point at different stages. The data points of different versions are gradually restored to a three-dimensional occupancy volume x0 close to the original through the denoising process. c is the multi-view condition, including global image features and global aggregation features, f is the UNet network, and N and U are probability distributions respectively.

[0101] Step S630: Divide the voxel corresponding to the three-dimensional occupied volume into a plurality of sub-voxels to construct a three-dimensional sparse volume.

[0102] A 3D sparse volume refers to a 3D grid that only retains high-resolution data at the target location. Using a sparse volume representation, high-resolution data is retained only at the target location, which can improve computational efficiency and detail processing effects.

[0103] Specifically, the voxels corresponding to the three-dimensional occupied volume (including the voxels predicted to be occupied) can be further divided into multiple sub-voxels to construct a three-dimensional sparse volume. Among them, the voxels can be further divided into 8 sub-voxels, which is not limited.

[0104] Step S640: According to the global image features and the global aggregation features, the three-dimensional sparse volume is subjected to denoising and / or denoising processing by using a pre-trained diffusion model to obtain a mesh model containing texture information.

[0105] Among them, random noise can be added to the three-dimensional sparse body first, and then according to the global image features and the global aggregation features, the global image features and the global aggregation features are used as model guidance conditions to guide the pre-trained diffusion model to denoise the three-dimensional sparse body after the random noise is added, so as to obtain a grid model containing texture information.

[0106] By denoising the three-dimensional sparse volume after adding random noise through the pre-trained diffusion model, a signed distance field grid (SDF) (and texture grid) of each voxel can be obtained. The signed distance field grid is used to describe the surface shape of the object. The signed distance field represents the signed distance from the voxel in space to the nearest surface. A positive value indicates that the point is outside the surface, and a negative value indicates that the point is inside the surface. The texture grid is used to describe the texture distribution, such as color distribution.

[0107] Through a two-stage diffusion process, a mesh model containing texture information is generated based on global image features and global aggregation features, so that the generated mesh model contains the core features of the target object and increases the realism of the generated three-dimensional model.

[0108] In an optional embodiment, according to the global image features and the global aggregation features, denoising the three-dimensional sparse volume after adding random noise by using a pre-trained diffusion model to obtain a mesh model containing texture information may include:

[0109] Step S710: Determine a closed surface mesh based on the three-dimensional sparse volume.

[0110] Closed surface mesh refers to a closed manifold, that is, a completely closed surface without holes or discontinuities, which can be used to extract the signed distance field mesh. Among them, the three-dimensional sparse body can be filled with holes or discontinuities through surface repair technology to obtain a closed manifold without holes.

[0111] Step S720: constructing a signed distance field grid according to the distance between the voxels corresponding to the three-dimensional sparse volume and the closed surface, wherein the signed distance field grid is used to describe the distance between the voxels corresponding to the three-dimensional sparse volume and the closed surface.

[0112] The signed distance between each voxel and the closed surface is calculated to generate signed distance field volume data (i.e., signed distance field grid).

[0113] According to the global image features and global aggregation features, the three-dimensional sparse volume after adding random noise is denoised to obtain the signed distance field grid of voxels, so that the grid model containing texture information conforms to the shape and structure of the target object.

[0114] Among them, the surface mesh, that is, the mesh model containing texture information, can be determined by extracting isosurfaces. For example, through Marching Cubes (MC, voxel-level reconstruction algorithm), the mesh model containing texture information is determined according to the isosurfaces in the signed distance field mesh data to ensure the smoothness of the mesh model. For another example, a differentiable isosurface extraction module can be used to extract an explicit mesh surface from an implicit three-dimensional representation to obtain a mesh model containing texture information. The exemplary embodiments of the present disclosure can select a method for determining the surface mesh according to actual needs, and there is no limitation on this.

[0115] In an optional embodiment, according to the global image features and the global aggregation features, denoising the three-dimensional sparse volume after adding random noise by using a pre-trained diffusion model to obtain a mesh model containing texture information may also include:

[0116] Step S810: back-projecting the view image sequence into three-dimensional space to obtain a three-dimensional point cloud.

[0117] Step S820: Based on the three-dimensional point cloud, according to the global image features and the global aggregation features, the three-dimensional sparse volume after adding random noise is denoised by a pre-trained diffusion model to obtain a texture mesh.

[0118] The texture grid is used to describe the texture information of the voxel, and the 3D point cloud is a point cloud containing position information and texture information, for example, a 3D point cloud is a point cloud containing position information and color information. The view images in the view image sequence are back-projected into the 3D space, and the obtained 3D point cloud is the texture volume data, simulating the real texture distribution, where each voxel stores the texture information of its position, such as color information.

[0119] According to the global image features and global aggregation features, the three-dimensional sparse volume after adding random noise is denoised to obtain the texture grid of voxels, so that the grid model containing texture information conforms to the texture distribution of the target object.

[0120] In some optional embodiments, the method for generating a mesh model containing texture information through a diffusion process includes a two-stage diffusion process, which is implemented by respective diffusion networks, for example, both are UNet networks, and the two diffusion networks are trained independently. The model training process involved in the diffusion process is described below.

[0121] First, data preprocessing is performed to process the three-dimensional grid samples to generate closed manifolds.

[0122] Secondly, the closed manifold is converted into a signed distance field volume, that is, by calculating the distance from each voxel to the nearest surface, generating a signed distance function, and obtaining a signed distance field grid.

[0123] Next, the multi-view image samples are back-projected into three-dimensional space to obtain a three-dimensional point cloud, and the three-dimensional point cloud is used to construct a texture volume, where each voxel stores the texture information of its corresponding position.

[0124] Furthermore, a viewpoint image sequence of the image samples is obtained as a conditional input for model training to enhance the model's understanding of three-dimensional shape and texture.

[0125] Finally, the model is trained based on image samples, the signed distance field grids and 3D point clouds corresponding to the image samples, and the view image sequences. The diffusion grids of the two stages are trained independently. By injecting noise and gradually denoising, the model learns the ability to generate a 3D model from a single image.

[0126] An exemplary embodiment of the present disclosure utilizes global aggregation features through a two-stage diffusion process to first generate a low-resolution three-dimensional occupied volume, which is then refined into a high-resolution three-dimensional sparse volume, and gradually generates a high-quality mesh model, ensuring that multi-view features (global aggregation features) and global image features are fully utilized in the generation process, enriching the details of the generated model and increasing realism.

[0127] In an exemplary embodiment, a method for adjusting texture information is also provided. Adjusting the texture information of the mesh model includes:

[0128] First, the mesh model is UV unfolded to obtain the UV unfolding result; then the mesh model is rasterized and rendered to obtain a two-dimensional plane image, and the pre-trained texture mesh is queried according to the depth information of the two-dimensional plane image, and the texture information is adjusted according to the query result; finally, based on the adjusted texture information and UV unfolding result, the mesh model is baked to generate the target three-dimensional model.

[0129] Among them, UV unfolding is the process of unfolding or flattening a 3D geometry into a 2D representation so that each polygon and face of the 3D object is bound to a face in the UV map. Automatic UV unfolding algorithms, such as those based on energy minimization, can be used without restriction.

[0130] In some optional embodiments, after the mesh model is UV unfolded, the UV layout after the unfolding may be adjusted to reduce gaps and deformation.

[0131] Rasterization rendering is the process of converting a 3D model into a 2D image, also known as rasterization or rendering pipeline. Through rasterization rendering, a 3D model can be decomposed into a series of pixels, and the color value of each pixel is calculated to finally generate a 2D plane image. When converting a mesh model into a 2D plane image, the image depth information can be obtained, and the pre-trained texture mesh can be queried based on the image depth information to obtain the texture data of each pixel, and then the texture information of the mesh model can be adjusted based on the query result. By adjusting the texture information of the mesh model, the texture of the rendered image can be made more accurate.

[0132] Bake the mesh model based on the adjusted texture information and UV unfolding results, that is, bake the adjusted texture information onto the mesh model to ensure a consistent appearance at any viewing angle.

[0133] The baking process may be to apply the adjusted texture information to the UV unfolding result (UV map), and generate a texture map through rasterization rendering.

[0134] Based on the aforementioned exemplary embodiments, a method for implementing multi-view image-guided optimization of texture information is provided, and the method may include:

[0135] First, a target perspective image sequence corresponding to a target three-dimensional model is obtained, wherein the perspective image in the target perspective image sequence is extracted by using shooting parameter information of the perspective image sequence corresponding to the two-dimensional image, and then the pixel difference between the target perspective image sequence and the corresponding perspective image in the perspective image sequence is obtained; finally, the texture information is adjusted according to the pixel difference.

[0136] Among them, the shooting parameter information refers to the camera shooting parameters used to obtain multi-view images. The multi-view image sequence corresponding to the two-dimensional image is extracted based on the shooting parameter information. After the target three-dimensional model is generated, the same shooting parameter information is sampled to shoot the target three-dimensional model to obtain the target perspective image sequence. Finally, the pixel difference and loss function can be constructed according to the pixels of the corresponding perspective images in the two groups of multi-view image sequences, so as to optimize and adjust the texture information based on the pixel difference and loss function.

[0137] Exemplarily, the sum of squares of pixel differences between the target view image sequence and the view images in the view image sequence may be calculated as the difference loss, so that the texture information is adjusted based on the difference loss.

[0138] The texture information is adjusted by using the pixel difference between the view images in the two sets of multi-view image sequences before and after the model is generated, so that the texture information is more in line with the multi-view requirements, thereby improving the realism of the generated 3D model.

[0139] In some optional embodiments, a texture map format may be selected, and export parameters may be set to export the texture map so that the export result includes the texture information of the model surface for use in a desired scenario. For example, the generated model and texture map may be imported into a game engine for use by players.

[0140] In an exemplary embodiment, a method for obtaining global image features and view image sequences is also provided. Fig. 9 The extracting of global image features and view image sequences according to the two-dimensional image of the target object comprises:

[0141] Step S910: extract features from the two-dimensional image to obtain global image features.

[0142] A pre-trained multimodal model can be used to extract features from two-dimensional images, including but not limited to color features, material features, pattern features, etc.

[0143] Step S920: performing image generation processing on the two-dimensional image according to the global image features to obtain a reference image.

[0144] Image generation processing refers to the process of processing a two-dimensional image using an image generation model based on global image features, that is, using global image features as model input and as model guidance conditions, to obtain a reference image with a target style and retaining the features of the two-dimensional image. The image generation model can be a stable diffusion model (Stable Diffusion Model), and of course it can also be other models, such as combining a stable diffusion model with a LoRA (Low-Rank Adaptation of Large Language Models) model. The exemplary embodiment of the present disclosure is described by taking the image generation model as a stable diffusion model as an example.

[0145] Among them, performing image generation processing on the two-dimensional image according to the global image feature to obtain the reference image may include:

[0146] First, random noise is added to the default latent space to obtain a noisy latent space; then, the noisy latent space is denoised according to the global image features to determine the latent space features according to the denoising results, and finally, the image is reconstructed based on the latent space features to obtain a reference image. The default latent space is a pre-set latent space.

[0147] Optionally, if the user provides text description information, the text description information represents the user's requirements for generating a reference image, such as a cartoon style. A text encoder can be used to convert the text description information into an embedding vector in a high-dimensional space to obtain semantic information of the text description information, so that the embedding vector is used as a guiding condition at the same time, and the image generation model is used to perform image generation processing on the two-dimensional image to obtain a reference image.

[0148] Among them, the global image features and the parameter files of the stable diffusion model and the LoRA model can be provided to the sampler as input conditions, and the image generation process is performed by the sampler to obtain a reference image.

[0149] When a sampler is used to process a two-dimensional image, it can be implemented by a sampling method in an image generation model, such as a stable diffusion model. In the stable diffusion model, a global sampling process is performed on the two-dimensional image by using the KSampler sampling method. Among them, the stable diffusion model can generate high-quality images based on global image features, and can generate images that match its given global image features through the relationship between the learned image and the global image features. It should be pointed out that a sampling method such as KSampler can also be referred to as a sampler in this article.

[0150] It is worth noting that other deep learning models and samplers can be selected according to needs to perform global sampling processing on two-dimensional images, and there is no limitation on this. The exemplary embodiment of the present disclosure is described by taking the global sampling processing of two-dimensional images by the KSampler sampler in a stable diffusion model as an example.

[0151] Specifically, when adding random noise to the default latent space, noise can be introduced according to the preset random seed and denoising strength. The random seed is used to determine the randomness of the generation process in order to obtain reproducible results. The seed determines the process of noise generation in the algorithm. If the number of seeds is consistent with other input conditions, multiple generations will generally produce almost identical results. The random seed is input by the user or generated randomly, and there is no restriction on this.

[0152] During denoising and detail generation, the image generation model is guided by global image features to remove the added noise and generate new image details where the noise is removed. In other words, this process is equivalent to guiding the image generation model to reconstruct and generate details in a part of the image, so that the image generation processing result retains the global image features while adding new style details. Image reconstruction based on latent space features can be converted into a reference image through a decoder, such as a VAE (Variational Autoencoder) decoder.

[0153] The following takes the image generation model as a stable diffusion model and the sampler as KSampler as an example to illustrate the image generation of a two-dimensional image.

[0154] The inputs to this process include: the image generation model used for denoising; global image features, which are used to guide how the image generation model generates the reference image; the preset latent space; the random seed when creating noise; information that controls the change of the seed after generation (such as control_after_generate); the number of steps used in the denoising process, the more steps, the more accurate the result; the CFG ratio, which is used to determine which features the extracted features are closer to; the sampler name; the scheduler type; and so on.

[0155] Outputs of this process include: Reference images.

[0156] Through the above input and output, the two-dimensional image is globally sampled according to the global image features to obtain a reference image.

[0157] It should also be understood that if the user provides text description information, the text description information can also be used as a guiding condition as an input for the above process.

[0158] Under the guidance of global image features and text description information, image generation processing is performed on the two-dimensional image to obtain a reference image, so that the generated reference image retains the global image features and meets user expectations.

[0159] Step S930: Based on the reference image, use the pre-trained multi-view image generation model to obtain the view images of the target object at different view angles to obtain a view image sequence.

[0160] The pre-trained multi-view image generation model can be obtained by training the to-be-trained model with single image samples and view image sequence samples, without any restriction. Among them, the single image sample can be reversed through the text encoder to generate the text encoding as the text encoding input of the single image sample.

[0161] The exemplary embodiment of the present disclosure generates a reference image under the guidance of global image features, and provides a high-quality reference image that meets the user's expectations for generating a perspective image sequence, thereby providing accurate data support for subsequent generation of a model.

[0162] Fig.10 A complete flow chart of model generation is shown below. Fig.10 , a model generation method of an exemplary embodiment of the present disclosure is described.

[0163] Step S1010: Acquire a two-dimensional image of the target object.

[0164] Among them, the target object can be photographed by an electronic device to obtain a two-dimensional image, or a two-dimensional image can be received from other electronic devices, or a two-dimensional image can be downloaded from the network.

[0165] Step S1020: extract features from the two-dimensional image to obtain global image features.

[0166] Step S1030: performing image generation processing on the two-dimensional image according to the global image features to obtain a reference image.

[0167] If there is text description information provided by the user, the two-dimensional image is subjected to image generation processing under the guidance of the global image features and the text description information.

[0168] Step S1040: Based on the reference image, use a pre-trained multi-view image generation model to obtain view images of the target object at different view angles to obtain a view image sequence.

[0169] Step S1050: extracting local patch features of each viewing angle image, aggregating the local patch features, and constructing multi-view global aggregation features.

[0170] Step S1060: Based on the global image features and the global aggregation features, a pre-trained diffusion model is used to perform noise addition and denoising processing in three-dimensional space to obtain a mesh model containing texture information corresponding to the target object.

[0171] Among them, step S1061 is the first diffusion stage, in which random noise is added in the three-dimensional space, and denoising is performed in the three-dimensional space through a pre-trained diffusion model according to global image features and global aggregation features to generate a three-dimensional occupied volume.

[0172] Step S1062 is the second diffusion stage, in which the voxels corresponding to the three-dimensional occupied volume are divided into multiple sub-voxels to construct a three-dimensional sparse volume, and the three-dimensional sparse volume is denoised and denoised by a pre-trained diffusion model according to the global image features and the global aggregation features to obtain a grid model containing texture information.

[0173] Wherein, the mesh model containing texture information is generated based on the extraction of isosurfaces from the signed distance field mesh.

[0174] Step S1070: adjusting the texture information of the mesh model to generate a target three-dimensional model according to the adjustment result.

[0175] Among them, optionally, the mesh model can be UV unfolded to obtain a UV unfolding result, and the mesh model can be rasterized and rendered to obtain a two-dimensional plane image, and the pre-trained texture mesh can be queried according to the depth information of the two-dimensional plane image, and the texture information can be adjusted according to the query result. Based on the adjusted texture information and the UV unfolding result, the mesh model is baked to generate a target three-dimensional model. Optionally, the texture information can be optimized using multi-view images. Optionally, the mesh model can also be UV unfolded, and the UV layout of the UV unfolding result can be adjusted.

[0176] Based on the adjusted texture information and UV unfolding results, the mesh model is baked to generate the target 3D model.

[0177] It should be noted that the specific description of steps S1010 to S1070 can be found in the above exemplary embodiments and will not be repeated here.

[0178] The model generation method in the exemplary embodiment of the present disclosure, on the one hand, extracts global image features and view image sequences from the two-dimensional image of the target object, and extracts local patch features of each view image feature, aggregates the local patch features to construct a multi-view global aggregate feature, based on the global image features and the global aggregate feature, performs noise and / or denoising on the three-dimensional model through a pre-trained diffusion model, and obtains a mesh model containing texture information of the target object, the process integrates multi-dimensional features from different viewpoints to obtain global aggregate features, provides rich multi-view conditions for generating a three-dimensional model, and by combining the global image features and the global aggregate feature, completely extracts the global and local features of the target object, provides rich feature conditions for generating a model, and increases the fidelity of the generated model from both the viewpoint and the feature. On the other hand, the texture information of the mesh model is adjusted, and the target three-dimensional model is generated according to the adjustment result, and the three-dimensional model is made more realistic through texture optimization, further improving the quality of the generated model. On the other hand, the three-dimensional model can be automatically generated based on an image of the target object, simplifying the process of generating the model, reducing the investment cost of production manpower and tools, and has broad application prospects in the fields of game art production, virtual reality and augmented reality, as well as e-commerce and digital display.

[0179] In an exemplary embodiment of the present disclosure, a model generation device is also provided. Fig.11As shown, the model generation device 1100 may include an image acquisition module 1110, a feature extraction module 1120 and a model generation module 1130. Specifically:

[0180] The image acquisition module 1110 is used to extract global image features and a perspective image sequence based on a two-dimensional image of a target object, wherein the perspective images in the perspective image sequence correspond to different perspectives of the target object respectively; the feature extraction module 1120 is used to extract local patch features of each of the perspective images, aggregate the local patch features, and construct a global aggregate feature of multiple perspectives; the model generation module 1130 is used to perform noise addition and / or denoising in three-dimensional space based on the global image features and the global aggregate features through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object, so as to generate a target three-dimensional model based on the mesh model containing texture information.

[0181] In an exemplary embodiment of the present disclosure, the feature extraction module 1120 is configured to perform: extracting local patch features corresponding to each of the perspective images respectively; for each voxel in the three-dimensional space, aggregating the local patch features of the voxel corresponding to different perspective images to obtain the voxel aggregation feature of the voxel; aggregating the voxel aggregation features of each of the voxels to generate the global aggregation feature.

[0182] In an exemplary embodiment of the present disclosure, the feature extraction module 1120 is configured to perform: mapping the voxel to at least one perspective image in the perspective image sequence using a preset projection matrix; aggregating the local patch features of the corresponding mapping position in the at least one perspective image to obtain the multi-perspective voxel aggregation features of the voxel.

[0183] In an exemplary embodiment of the present disclosure, the feature extraction module 1120 is configured to perform: mapping the voxel aggregation features of each of the voxels to the three-dimensional space, and aggregating the voxel aggregation features mapped to the three-dimensional space to generate the global aggregation features.

[0184] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to execute: adding random noise in the three-dimensional space; performing denoising processing in the three-dimensional space through the pre-trained diffusion model according to the global image features and the global aggregation features to generate a three-dimensional occupied volume, wherein the three-dimensional occupied volume describes the basic structure and shape of the target object; dividing the voxels corresponding to the three-dimensional occupied volume into multiple sub-voxels to construct a three-dimensional sparse volume; performing denoising and / or denoising processing on the three-dimensional sparse volume through the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the mesh model containing texture information.

[0185] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to execute: adding random noise to the three-dimensional sparse body; and denoising the three-dimensional sparse body after adding random noise through the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the mesh model containing texture information.

[0186] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to perform: determining a closed surface mesh based on the three-dimensional sparse body; constructing a signed distance field mesh according to the distance between the voxels corresponding to the three-dimensional sparse body and the closed surface, wherein the signed distance field mesh is used to describe the distance between the voxels corresponding to the three-dimensional sparse body and the closed surface.

[0187] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to execute: back-projecting the view image sequence into the three-dimensional space to obtain a three-dimensional point cloud; based on the three-dimensional point cloud, according to the global image features and the global aggregation features, denoising the three-dimensional sparse volume after adding random noise through the pre-trained diffusion model to obtain a texture mesh, and the texture mesh is used to describe the texture information of the voxel.

[0188] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to execute: extracting isosurfaces based on the signed distance field mesh, and generating the mesh model containing texture information.

[0189] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to execute: UV unfolding the mesh model to obtain a UV unfolding result; rasterizing the mesh model to obtain a two-dimensional plane image; querying a pre-trained texture mesh based on the depth information of the two-dimensional plane image, adjusting the texture information based on the query result, and baking the mesh model based on the adjusted texture information and the UV unfolding result to generate the target three-dimensional model.

[0190] In an exemplary embodiment of the present disclosure, the model generation module 1130 is configured to perform: obtaining a target perspective image sequence corresponding to the target three-dimensional model, wherein the perspective image in the target perspective image sequence is extracted by using shooting parameter information of the perspective image sequence corresponding to the two-dimensional image; obtaining the pixel difference between the target perspective image sequence and the corresponding perspective image in the perspective image sequence; and adjusting the texture information according to the pixel difference.

[0191] In an exemplary embodiment of the present disclosure, the image acquisition module 1110 is configured to execute: extracting features from the two-dimensional image to obtain the global image features; performing image generation processing on the two-dimensional image according to the global image features to obtain a reference image; based on the reference image, obtaining perspective images of the target object at different perspectives using a pre-trained multi-perspective image generation model to obtain the perspective image sequence.

[0192] In an exemplary embodiment of the present disclosure, the image acquisition module 1110 is configured to perform: adding random noise to a default latent space to obtain a noisy latent space; denoising the noisy latent space according to the global image features to determine the latent space features according to the denoising results; and reconstructing the image based on the latent space features to obtain the reference image.

[0193] Since the details of each functional module of the model generating device of the exemplary embodiment of the present disclosure have been described in the exemplary embodiment of the model generating method described above, they will not be repeated here.

[0194] It should be noted that, although several modules or units of the model generation device are mentioned in the above detailed description, this division is not mandatory. In fact, according to the embodiments of the present disclosure, the features and functions of two or more modules or units described above can be embodied in one module or unit. Conversely, the features and functions of one module or unit described above can be further divided into multiple modules or units to be embodied.

[0195] The exemplary embodiments of the present disclosure also provide a computer program product, which includes a computer program, and when the computer program is executed by a processor, the above-mentioned model generation method is implemented.

[0196] In one embodiment, the computer program product may be a tangible product containing a computer program, such as a computer-readable storage medium storing a computer program. The readable storage medium may be a storage medium based on electrical, magnetic, optical, electromagnetic, infrared, or other signals, including but not limited to: random access memory (RAM), read-only memory (ROM), magnetic tape, floppy disk, flash memory (Flash), mechanical hard disk (HDD), solid-state drive (SSD), and the like. Exemplarily, the computer program product may be implemented as a non-volatile storage medium storing a computer program, such as a read-only memory, a NAND flash memory, and the like.

[0197] In one embodiment, the computer program product may be an intangible product including a computer program. Exemplarily, the computer program product may be implemented as a virtual digital product, such as a digital file storing an executable file, an installation package, etc. of the computer program.

[0198] The code of the computer program can be written in one or more programming languages. Programming languages ​​such as C language, Java, C++, etc. The program code can be executed entirely on the user computing device, or partially on the user computing device, or as a separate software package, or partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, such as a local area network (LAN), a wide area network (WAN), etc., or can be connected to an external computing device (e.g., an Internet connection provided by an operator).

[0199] The computer program may be carried or transmitted via electrical, magnetic, optical, electromagnetic, infrared, or other signals. The electronic device may convert the signal carrying the computer program into a digital signal, thereby running the computer program. When the computer program is run on an electronic device, its code is used to enable the electronic device to execute (more specifically, the processor of the electronic device may execute) the method steps of various exemplary embodiments of the present disclosure, such as the above-mentioned model generation method.

[0200] In addition, in an exemplary embodiment of the present disclosure, an electronic device capable of implementing the above method is also provided. It will be appreciated by those skilled in the art that various aspects of the present disclosure may be implemented as a system, method, or program product. Therefore, various aspects of the present disclosure may be specifically implemented in the following forms, namely: a complete hardware embodiment, a complete software embodiment (including firmware, microcode, etc.), or an embodiment combining hardware and software aspects, which may be collectively referred to herein as a "circuit", "module", or "system".

[0201] Refer to the following Fig.12 1200 according to this embodiment of the present disclosure is described. Fig.12 The electronic device 1200 shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0202] like Fig.12 As shown, the electronic device 1200 is in the form of a general computing device. The components of the electronic device 1200 may include, but are not limited to: the at least one processing unit 1210, the at least one storage unit 1220, a bus 1230 connecting different system components (including the storage unit 1220 and the processing unit 1210), and a display unit 1240.

[0203] The storage unit stores a program code, which can be executed by the processing unit 1210, so that the processing unit 1210 performs the steps according to various exemplary embodiments of the present disclosure described in the above “Exemplary Method” section of this specification. For example, the processing unit 1210 can perform the above-mentioned three-dimensional model generation method.

[0204] The storage unit 1220 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 1221 and / or a cache memory unit 1222 , and may further include a read-only memory unit (ROM) 1223 .

[0205] The storage unit 1220 may also include a program / utility 1224 having a set (at least one) of program modules 1225, such program modules 1225 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a network environment.

[0206] Bus 1230 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0207] The electronic device 1200 may also communicate with one or more external devices 1300 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), may also communicate with one or more devices that enable a user to interact with the electronic device 1200, and / or communicate with any device that enables the electronic device 1200 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed via an input / output (I / O) interface 1250. In addition, the electronic device 1200 may also communicate with one or more networks (e.g., local area networks (LANs), wide area networks (WANs), and / or public networks, such as the Internet) via a network adapter 1260. As shown, the network adapter 1260 communicates with other modules of the electronic device 1200 via a bus 1230. It should be understood that, although not shown in the figure, other hardware and / or software modules may be used in conjunction with the electronic device 1200, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage systems.

[0208] Through the description of the above embodiments, it is easy for those skilled in the art to understand that the example embodiments described here can be implemented by software, or by software combined with necessary hardware. Therefore, the technical solution according to the embodiment of the present disclosure can be embodied in the form of a software product, which can be stored in a non-volatile storage medium (which can be a CD-ROM, a USB flash drive, a mobile hard disk, etc.) or on a network, and includes several instructions to enable a computing device (which can be a personal computer, a server, a terminal device, or a network device, etc.) to execute the method according to the embodiment of the present disclosure.

[0209] In addition, the above-mentioned figures are only schematic illustrations of the processes included in the method according to the exemplary embodiments of the present disclosure, and are not intended to be limiting. It is easy to understand that the processes shown in the above-mentioned figures do not indicate or limit the time sequence of these processes. In addition, it is also easy to understand that these processes can be performed synchronously or asynchronously, for example, in multiple modules.

[0210] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure. The description and examples are to be considered exemplary only, and the true scope and spirit of the present disclosure are indicated by the claims.

Claims

1. A model generation method, characterized in that: include: Extracting global image features and a view image sequence according to the two-dimensional image of the target object, wherein the view images in the view image sequence correspond to different view angles of the target object respectively; Extracting local patch features of each of the perspective images, aggregating the local patch features, and constructing multi-perspective global aggregation features; Based on the global image features and the global aggregation features, noise addition and / or denoising processing is performed in three-dimensional space through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object, and a target three-dimensional model is generated based on the mesh model containing texture information.

2. The method according to claim 1, characterized in that The extracting of local patch features of each of the perspective images, aggregating the local patch features, and constructing multi-perspective global aggregation features includes: Extracting local patch features corresponding to each of the perspective images respectively; For each voxel in the three-dimensional space, aggregating local patch features of the voxel corresponding to different viewing angle images to obtain a voxel aggregation feature of the voxel; The voxel aggregation features of each of the voxels are aggregated to generate the global aggregation feature.

3. The method according to claim 2, characterized in that The step of aggregating the local patch features of the voxels corresponding to the different viewing angle images to obtain voxel aggregation features of the voxels includes: Mapping the voxel to at least one perspective image in the perspective image sequence using a preset projection matrix; The local patch features of the corresponding mapping positions in the at least one perspective image are aggregated to obtain multi-perspective voxel aggregation features of the voxel.

4. The method according to claim 2, characterized in that: The step of aggregating the voxel aggregation features of each voxel to generate the global aggregation feature includes: The voxel aggregation features of each of the voxels are mapped to the three-dimensional space, and the voxel aggregation features mapped to the three-dimensional space are aggregated to generate the global aggregation features.

5. The method according to claim 1, characterized in that The method of performing noise addition and / or denoising processing in three-dimensional space based on the global image features and the global aggregation features through a pre-trained diffusion model to obtain a mesh model containing texture information corresponding to the target object includes: adding random noise in the three-dimensional space; According to the global image features and the global aggregate features, denoising is performed in the three-dimensional space by using the pre-trained diffusion model to generate a three-dimensional occupied volume, wherein the three-dimensional occupied volume describes the basic structure and shape of the target object; Dividing the voxel corresponding to the three-dimensional occupied volume into a plurality of sub-voxels to construct a three-dimensional sparse volume; According to the global image features and the global aggregation features, the three-dimensional sparse volume is subjected to denoising and / or denoising processing by using the pre-trained diffusion model to obtain the grid model containing texture information.

6. The method according to claim 5, characterized in that The step of performing denoising and / or denoising processing on the three-dimensional sparse volume by using the pre-trained diffusion model according to the global image feature and the global aggregation feature to obtain the grid model containing texture information includes: adding random noise to the three-dimensional sparse volume; According to the global image features and the global aggregation features, the three-dimensional sparse volume to which random noise is added is denoised by the pre-trained diffusion model to obtain the grid model containing texture information.

7. The method according to claim 6, characterized in that The method of performing denoising processing on the three-dimensional sparse volume after adding random noise by using the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the grid model containing texture information includes: determining a closed surface mesh based on the three-dimensional sparse volume; A signed distance field grid is constructed according to the distance between the voxels corresponding to the three-dimensional sparse volume and the closed surface. The signed distance field grid is used to describe the distance between the voxels corresponding to the three-dimensional sparse volume and the closed surface.

8. The method according to claim 6, characterized in that The method of performing denoising processing on the three-dimensional sparse volume after adding random noise by using the pre-trained diffusion model according to the global image features and the global aggregation features to obtain the grid model containing texture information includes: Back-projecting the view image sequence into the three-dimensional space to obtain a three-dimensional point cloud; Based on the three-dimensional point cloud, according to the global image features and the global aggregation features, the three-dimensional sparse volume after adding random noise is denoised through the pre-trained diffusion model to obtain a texture mesh, and the texture mesh is used to describe the texture information of the voxel.

9. The method according to claim 7, characterized in that: The method further comprises: The isosurface is extracted based on the signed distance field grid to generate the grid model containing texture information.

10. The method according to claim 1, characterized in that The step of generating a target three-dimensional model according to the mesh model containing texture information comprises: Perform UV unfolding on the mesh model to obtain a UV unfolding result; Performing rasterization rendering on the grid model to obtain a two-dimensional plane image; querying a pre-trained texture grid according to the depth information of the two-dimensional plane image, and adjusting the texture information according to the query result; Based on the adjusted texture information and the UV unfolding result, the mesh model is baked to generate the target three-dimensional model.

11. The method according to any one of claims 1 to 10, characterized in that The method further comprises: Acquire a target perspective image sequence corresponding to the target three-dimensional model, wherein the perspective images in the target perspective image sequence are extracted by extracting shooting parameter information of the perspective image sequence corresponding to the two-dimensional image; Acquire a pixel difference between the target perspective image sequence and a corresponding perspective image in the perspective image sequence; The texture information is adjusted according to the pixel difference.

12. The method according to claim 1, characterized in that The step of extracting global image features and view image sequences based on the two-dimensional image of the target object includes: Performing feature extraction on the two-dimensional image to obtain the global image feature; Performing image generation processing on the two-dimensional image according to the global image feature to obtain a reference image; Based on the reference image, a pre-trained multi-view image generation model is used to obtain view images of the target object at different view angles to obtain the view image sequence.

13. The method according to claim 12, characterized in that The performing image generation processing on the two-dimensional image according to the global image feature to obtain a reference image includes: Add random noise to the default latent space to obtain the noisy latent space; De-noising the noisy latent space according to the global image feature to determine the latent space feature according to the de-noising result; Image reconstruction is performed based on the latent space features to obtain the reference image.

14. A model generation device, characterized in that: include: An image acquisition module, used to extract global image features and a view image sequence according to a two-dimensional image of a target object, wherein the view images in the view image sequence correspond to different view angles of the target object respectively; A feature extraction module, used to extract local patch features of each of the view images, aggregate the local patch features, and construct a global aggregate feature of multiple view angles; A model generation module is used to perform noise addition and / or denoising processing in three-dimensional space based on the global image features and the global aggregation features through a pre-trained diffusion model to obtain a grid model containing texture information corresponding to the target object, and generate a target three-dimensional model based on the grid model containing texture information.

15. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 13 is implemented.

16. An electronic device, characterized in that: include: processor; as well as A memory, configured to store executable instructions of the processor; The processor is configured to perform the method of any one of claims 1 to 13 by executing the executable instructions.

Images