Three-dimensional model generation method and device, equipment, storage medium and program product

Abstract

The invention provides a three-dimensional model generation method and device, equipment, a storage medium and a program product. The method comprises the following steps: extracting three-dimensional features of an object in an image, and encoding the image to obtain image encoding features; fusing the three-dimensional feature and the image coding feature to obtain a first feature of the object; determining a second feature of the object based on the three-dimensional feature; and generating a three-dimensional model of the object based on a three-dimensional noise image, the first feature and the second feature. According to the invention, the generation effect of the three-dimensional model can be improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a method, device, equipment, storage medium and program product for generating a three-dimensional model. Background Art

[0002] With the development of artificial intelligence, the application of artificial intelligence in the field of image generation is becoming more and more extensive. With the development of image generation technology, its influence has naturally expanded to the three-dimensional field. In some fields (such as virtual reality (VR), augmented reality (AR), medical imaging, industrial design, etc.), there is an increasing demand for generating three-dimensional models from images. Therefore, how to generate three-dimensional models based on images has become a hot research issue. Summary of the invention

[0003] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for generating a three-dimensional model, which can improve the generation effect of the three-dimensional model.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application provides a method for generating a three-dimensional model, including:

[0006] Extracting three-dimensional features of an object in an image, and encoding the image to obtain image encoding features;

[0007] fusing the three-dimensional feature and the image coding feature to obtain a first feature of the object;

[0008] Based on the three-dimensional feature, determining a second feature of the object;

[0009] A three-dimensional model of the object is generated based on the three-dimensional noise image, the first feature, and the second feature.

[0010] The present application also provides a three-dimensional model generation device, including:

[0011] A feature processing module, used for extracting three-dimensional features of objects in an image, and encoding the image to obtain image coding features;

[0012] A fusion module, used for fusing the three-dimensional feature and the image coding feature to obtain a first feature of the object;

[0013] A determination module, configured to determine a second feature of the object based on the three-dimensional feature;

[0014] A generating module is used to generate a three-dimensional model of the object based on the three-dimensional noise image, the first feature and the second feature.

[0015] The present application also provides an electronic device, including:

[0016] A memory for storing computer executable instructions;

[0017] The processor is used to implement the method for generating a three-dimensional model provided in the embodiment of the present application when executing the computer executable instructions stored in the memory.

[0018] An embodiment of the present application also provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the method for generating a three-dimensional model provided in the embodiment of the present application is implemented.

[0019] An embodiment of the present application also provides a computer program product, including computer executable instructions or a computer program. When the computer executable instructions or the computer program are executed by a processor, the method for generating a three-dimensional model provided in the embodiment of the present application is implemented.

[0020] The embodiments of the present application have the following beneficial effects:

[0021] By using the above-mentioned embodiments of the present application, the three-dimensional features of the object in the image are first extracted, and the image is encoded to obtain the image encoding features, and then the three-dimensional features and the image encoding features are fused to obtain the first feature of the object, and then based on the three-dimensional features, the second feature of the object is determined, and finally based on the three-dimensional noise image, the first feature and the second feature, a three-dimensional model of the object is generated. Here, the three-dimensional features of the object in the image are extracted, and the first feature and the second feature are obtained respectively based on the three-dimensional features, and then the three-dimensional noise image, the first feature and the second feature are combined to generate the three-dimensional model of the object; in this way, the generation of the three-dimensional model is controlled by the three-dimensional features of the object, and the accuracy of the generated three-dimensional model and the consistency with the object in the image are improved, that is, the generation effect of the three-dimensional model is improved by the embodiments of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] Figure 1 It is a schematic diagram of the architecture of a system for generating a three-dimensional model provided in an embodiment of the present application;

[0023] Figure 2 is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0024] Figure 3A It is a first flow chart of a method for generating a three-dimensional model provided in an embodiment of the present application;

[0025] Figure 3B It is a second flow chart of the method for generating a three-dimensional model provided in an embodiment of the present application;

[0026] Figure 3C It is a third flow chart of the method for generating a three-dimensional model provided in an embodiment of the present application;

[0027] Figure 4 is a first schematic diagram of a generation architecture of a three-dimensional model provided in an embodiment of the present application;

[0028] Figure 5 is a training diagram of a three-dimensional feature extraction model provided in an embodiment of the present application;

[0029] Figure 6 This is a second schematic diagram of the generation architecture of the three-dimensional model provided in an embodiment of the present application.

[0030] It should be noted that the above-mentioned “first” and “second” are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0031] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0032] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0033] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described here can be implemented in an order other than that illustrated or described here.

[0034] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the functions of the module or unit.

[0035] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0036] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0037] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0038] 1) Client: An application running in an electronic device to provide various services, such as a client that supports the generation of 3D models.

[0039] 2) In response to: used to indicate the conditions or states on which the executed operations depend. When the dependent conditions or states are met, one or more operations executed may be in real time or have a set delay. Unless otherwise specified, there is no restriction on the order in which the multiple operations executed are executed.

[0040] 3) Voxelization is a method of dividing a continuous three-dimensional space into a finite number of volume elements (voxels), each of which is a cubic unit in space. The advantage of voxelization is that it provides an intuitive way to represent three-dimensional objects and is easy to perform spatial operations such as cutting, moving, and rotating.

[0041] In related technologies, only two-dimensional visual features can be extracted (such as features extracted by models such as CLIP). Occlusion and visual ambiguity problems in two-dimensional images pose challenges to the accurate generation of three-dimensional models, resulting in the lack of accuracy and consistency in the generated three-dimensional models.

[0042] The embodiments of the present application provide a method, device, electronic device, computer-readable storage medium and computer program product for generating a three-dimensional model, which can improve the generation effect of the three-dimensional model. Next, based on the above description of the nouns and terms involved in the embodiments of the present application, the embodiments of the present application are described in detail.

[0043] The following describes a system for generating a three-dimensional model provided in an embodiment of the present application. Figure 1 , Figure 11 is a schematic diagram of the architecture of the three-dimensional model generation system provided in the embodiment of the present application. To support an exemplary application, the three-dimensional model generation system 100 includes: a server 200, a network 300, and a terminal 400. The terminal 400 is connected to the server 200 via the network 300, and the network 300 can be a wide area network or a local area network, or a combination of the two, using a wireless or wired link to achieve data transmission.

[0044] Here, the terminal 400 (for example, running a client that supports generating three-dimensional models) sends a three-dimensional model generation request to the server 200 in response to a three-dimensional model generation instruction; the server 200 receives the three-dimensional model generation request; in response to the three-dimensional model generation request, obtains an image including an object; extracts three-dimensional features of the object in the image, and encodes the image to obtain image coding features; fuses the three-dimensional features and the image coding features to obtain a first feature of the object; based on the three-dimensional features, determines a second feature of the object; based on the three-dimensional features, generates a three-dimensional model of the object;

[0045] In some embodiments, the method for generating a three-dimensional model provided in the embodiments of the present application is implemented by an electronic device, for example, it can be implemented by a terminal alone, it can also be implemented by a server alone, or it can be implemented by a terminal and a server in collaboration. The embodiments of the present application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, instant messaging, user-generated content (UGC), etc. Among them, artificial intelligence involves a variety of specific scenarios, such as virtual assistants, intelligent marketing, virtual people, robots, artificial intelligence generated content (AIGC), conversational interaction, intelligent customer service, etc.

[0046] In some embodiments, the electronic device for implementing the method for generating a three-dimensional model provided in the embodiments of the present application may be various types of terminals or servers. Among them, the server (e.g., server 200) may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content distribution networks (Content Delivery Network, CDN), and big data and artificial intelligence platforms. The terminal (e.g., terminal 400) may be a laptop, a tablet computer, a desktop computer, a smart phone, an intelligent voice interaction device (e.g., an intelligent speaker), a smart home appliance (e.g., a smart TV), a smart watch, a vehicle-mounted terminal, a wearable device, a virtual reality (Virtual Reality, VR) device, an aircraft, etc., but is not limited thereto. The terminal and the server may be directly or indirectly connected by wired or wireless communication, and the embodiments of the present application do not limit this.

[0047] In some embodiments, the terminal or server can implement the generation method of the three-dimensional model provided in the embodiment of the present application by running various computer executable instructions or computer programs. For example, the computer executable instructions can be commands, machine instructions or software instructions at the microprogram level. The computer program can be a native program or software module in the operating system; it can be a local (Native) application (Application, APP), that is, a program that needs to be installed in the operating system to run, such as an APP that supports the generation of a three-dimensional model; it can also be a small program that can be embedded in any APP, that is, a program that can be run only by downloading it to a browser environment. In short, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form.

[0048] The following describes an electronic device for implementing a method for generating a three-dimensional model provided in an embodiment of the present application. Figure 2 , Figure 2 Schematic diagram of the structure of the electronic device provided in the embodiment of the present application. The electronic device 500 provided in the embodiment of the present application can be a terminal or a server. Figure 2 As shown, the electronic device 500 includes: at least one processor 510, a memory 550, at least one network interface 520 and a user interface 530. The various components in the electronic device 500 are coupled together through a bus system 540. It can be understood that the bus system 540 is used to realize the connection and communication between these components. In addition to the data bus, the bus system 540 also includes a power bus, a control bus and a status signal bus. However, for the sake of clarity, the bus system 540 is not used in the following examples. Figure 2 Various buses are labeled as bus system 540 .

[0049] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0050] The user interface 530 includes one or more output devices 531 that enable presentation of media content, including one or more speakers and / or one or more visual display screens. The user interface 530 also includes one or more input devices 532, including user interface components that facilitate user input, such as a keyboard, mouse, microphone, touch screen display, camera, other input buttons and controls.

[0051] The memory 550 may be removable, non-removable, or a combination thereof. The memory 550 may include one or more storage devices physically located away from the processor 510. The memory 550 includes a volatile memory or a non-volatile memory, and may also include both a volatile memory and a non-volatile memory. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 550 described in the embodiments of the present application is intended to include any suitable type of memory.

[0052] In some embodiments, the memory 550 can store data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0053] Operating system 551, including system programs for processing various basic system services and performing hardware-related tasks, such as framework layer, core library layer, driver layer, etc., for implementing various basic services and processing hardware-based tasks;

[0054] A network communication module 552, used to reach other electronic devices via one or more (wired or wireless) network interfaces 520, exemplary network interfaces 520 include: Bluetooth, Wireless Compatibility Certification (WiFi), and Universal Serial Bus (USB), etc.;

[0055] a presentation module 553 for enabling presentation of information via one or more output devices 531 (e.g., display screen, speaker, etc.) associated with the user interface 530 (e.g., a user interface for operating peripherals and displaying content and information);

[0056] The input processing module 554 is used to detect one or more user inputs or interactions from one of the one or more input devices 532 and translate the detected inputs or interactions.

[0057] In some embodiments, the three-dimensional model generation device provided in the embodiments of the present application can be implemented in a software manner. Figure 2 A three-dimensional model generating device 555 stored in a memory 550 is shown, which may be software in the form of a program or plug-in, including the following software modules: a feature processing module 5551, a fusion module 5552, a determination module 5553, and a generating module 5554. These modules are logical, and therefore may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0058] The following describes the method for generating a three-dimensional model provided in the embodiment of the present application. As mentioned above, the method for generating a three-dimensional model provided in the embodiment of the present application is implemented by an electronic device, for example, it can be implemented by a server or a terminal alone, or by a server and a terminal in collaboration. Therefore, the execution subject of each step will not be repeatedly described below. Figure 3A , Figure 3A : is a flow chart of a method for generating a three-dimensional model provided in an embodiment of the present application. The method for generating a three-dimensional model provided in an embodiment of the present application includes:

[0059] Step 101: extract the three-dimensional features of the object in the image, and encode the image to obtain image encoding features.

[0060] Here, firstly, an image for generating a three-dimensional model is obtained, and the image includes an object, such as a person, an animal, an object (such as an airplane, a table, a chair, a building, etc.), etc. The image is a reference image for generating a three-dimensional model, and the generated three-dimensional model is a three-dimensional model of the object. For step 101, the three-dimensional features of the object in the image are extracted, and the three-dimensional features are the three-dimensional information of the object in the image. In some embodiments, the three-dimensional features of the object in the image can be extracted by performing the following steps: extracting at least one of the following three-dimensional features of the object in the image: depth information features, object shape features, object posture features, object light and shadow features, and object texture features. Among them, the depth information features are extracted from the depth map of the image; the object shape features can be detected from the image by edge detection algorithms, etc., and the object shape features can include the structure and contour of the object; the object posture features can be extracted from the image by posture recognition algorithms or models, and the object posture features can include the posture and action of the object; the object light and shadow features can include the illumination and shadow of the object; the object texture features can include the texture (such as the texture of clothes) and the material (such as the material of the object).

[0061] In some embodiments, the three-dimensional features are extracted from the image by a three-dimensional feature extraction model; based on this, see Figure 3B Before extracting the three-dimensional features of the object in the image, a three-dimensional feature extraction model can be obtained by training by executing the following steps 201-203: Step 201, extracting the first sample three-dimensional features of the sample object in the sample image through the initial feature extraction model; Step 202, performing feature extraction on the three-dimensional model of the sample object to obtain the second sample three-dimensional features; Step 203, based on the difference between the first sample three-dimensional features and the second sample three-dimensional features, the initial feature extraction model is trained to obtain the three-dimensional feature extraction model.

[0062] Here, the three-dimensional feature is extracted from the image by a three-dimensional feature extraction model, and the three-dimensional feature extraction model can be obtained by training based on an initial feature extraction model, and the initial feature extraction model can be constructed based on a neural network, such as a convolutional neural network, a deep neural network, a recurrent neural network, etc.; the initial feature extraction model has initial model parameters. When training the initial feature extraction model, a training sample for training can be obtained first, and the training sample includes: a sample image of a sample object, and a three-dimensional model of the sample object, and the three-dimensional model of the sample object can be a voxelized representation of the three-dimensional model. For step 201, the first sample three-dimensional feature of the sample object in the sample image is extracted by the initial feature extraction model. For example, the initial feature extraction model can be a model based on a Transformer architecture, such as a Vision Transformer (ViT) model, and the initial feature extraction model can also be a model based on a convolutional neural network architecture, such as a residual network ResNet model.

[0063] For step 202, feature extraction is performed on the three-dimensional model of the sample object to obtain the second sample three-dimensional feature. In some embodiments, the feature extraction in step 202 can also be implemented by a pre-trained pre-trained feature extraction model, or by the initial feature extraction model, or by an untrained feature extraction model that participates in the training together with the initial feature extraction model. For example, both the pre-trained feature extraction model and the untrained feature extraction model can adopt the structure of the modified data-efficient image transformer (DeiT). Specifically, when performing feature extraction, the DeiT network will first divide the image into a certain number of patches, and then extract the embedding of the patch. Here, the dimension of the input data (i.e., patch) of the DeiT network changes from 2D to 3D, and the convolution kernel of the convolution operation involved is also changed from a 2D convolution kernel to a 3D convolution kernel, and the dimension of the obtained embedding is also 3D.

[0064] For step 203, based on the difference between the first sample three-dimensional feature and the second sample three-dimensional feature, the initial feature extraction model is trained to obtain a three-dimensional feature extraction model. Specifically, the value of the loss function of the initial feature extraction model can be determined based on the difference between the first sample three-dimensional feature and the second sample three-dimensional feature. For example, the loss function can be a contrast loss function; then, based on the value of the loss function, the model parameters of the initial feature extraction model are updated to obtain a three-dimensional feature extraction model. It should be noted that if the second sample three-dimensional feature is extracted by the above-mentioned untrained feature extraction model, then during training, the model parameters of the untrained feature extraction model can also be updated based on the value of the loss function to train the untrained feature extraction model.

[0065] Through steps 201 to 203, based on the difference between the first sample three-dimensional feature of the sample object in the sample image extracted by the initial feature extraction model and the second sample three-dimensional feature extracted from the three-dimensional model of the sample object, the initial feature extraction model is trained so that the first sample three-dimensional feature extracted by the initial feature extraction model and the second sample three-dimensional feature are closer to each other, thereby obtaining a three-dimensional feature extraction model capable of extracting the three-dimensional features of the object from the image, thereby improving the accuracy of three-dimensional model generation.

[0066] For step 101, the image is further encoded to obtain image coding features. The encoding process can be implemented by an image encoder, which can include at least one encoding layer, and the image is encoded by the at least one encoding layer to obtain image coding features.

[0067] Step 102: Fuse the three-dimensional feature and the image coding feature to obtain the first feature of the object.

[0068] For step 102, after obtaining the three-dimensional feature and the image coding feature, the three-dimensional feature and the image coding feature are fused to obtain the first feature of the object. In some embodiments, the three-dimensional feature and the image coding feature can be fused by adding, concatenating, multiplying, etc.

[0069] In some embodiments, the three-dimensional feature and the image coding feature can be fused by performing the following steps to obtain the first feature of the object: mapping the three-dimensional feature to obtain the first mapping feature; adding the first mapping feature and the image coding feature to obtain the first feature.

[0070] Here, the three-dimensional feature is first mapped to obtain the first mapping feature. For example, the three-dimensional feature can be linearly mapped (i.e., linear transformation) to obtain the third mapping feature. The linear mapping can be implemented by a linear layer. The linear layer performs a linear transformation, that is, multiplying the input vector by the weight matrix, and then adding a bias vector to obtain the output vector, which can be expressed as: output = input * W + b, input is the three-dimensional feature, output is the first mapping feature, W is the weight matrix, and b is the bias vector. After obtaining the first mapping feature, the first mapping feature and the image coding feature are added to obtain the first feature. For example, the first mapping feature and the image coding feature can be bitwise added to obtain the first feature. In this way, by linearly mapping the three-dimensional feature, the fusion between the three-dimensional feature and the image coding feature is realized, so that when the three-dimensional model is generated later, both the three-dimensional feature (three-dimensional information) of the object and the image feature (i.e., the original image context information of the image) can be paid attention to, thereby improving the consistency of the generated three-dimensional model and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0071] In some embodiments, the three-dimensional feature can be mapped by executing the following steps to obtain a first mapping feature: mapping the three-dimensional feature into n third mapping features of different feature sizes, where n is an integer greater than 1; and concatenating the n third mapping features to obtain a first mapping feature.

[0072] Here, the three-dimensional feature can be mapped to n third mapping features of different feature sizes, and then the n third mapping features are spliced ​​to obtain the first mapping feature. In some embodiments, when the n third mapping features are spliced, the feature sizes of the n third mapping features can be adjusted to a specific feature size, and then the n third mapping features adjusted to the specific feature size are spliced ​​to obtain the first mapping feature. In this way, by mapping the three-dimensional feature to n third mapping features of different feature sizes, the three-dimensional feature is refined and the feature meanings at different levels are obtained, thereby improving the feature expression ability of the first mapping feature for the three-dimensional information of the object, thereby improving the feature expression ability of the first feature obtained subsequently for the three-dimensional information of the object, thereby improving the consistency between the three-dimensional model generated based on the first feature and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0073] In some embodiments, the mapping process includes m sub-mapping processes, where m is an integer greater than 1; based on this, the three-dimensional feature can be mapped by executing the following steps to obtain a first mapping feature: when i is equal to 1, the three-dimensional feature is sub-mapped to obtain a sub-mapping feature output by the first sub-mapping process; when i is greater than 1 and less than or equal to m, the sub-mapping feature output by the i-1th sub-mapping process is sub-mapped to obtain a sub-mapping feature output by the i-th sub-mapping process; the m sub-mapping features output by the m sub-mapping processes are spliced ​​to obtain a first mapping feature.

[0074] Here, the mapping process may include m sub-mapping processes. Therefore, for each sub-mapping process in the m sub-mapping processes, when i is equal to 1 (i.e., the first sub-mapping process), the three-dimensional feature is sub-mapped to obtain the sub-mapping feature output by the first sub-mapping process; when i is greater than 1 and less than or equal to m (i.e., any sub-mapping process different from the first sub-mapping process in the m sub-mapping processes), the sub-mapping feature output by the i-1th sub-mapping process is sub-mapped to obtain the sub-mapping feature output by the i-th sub-mapping process. It should be noted that the sub-mapping process may be a linear mapping process (i.e., a linear transformation process), which may be implemented by a linear layer (Linear Layer). The linear layer performs a linear transformation process, i.e., multiplying the input vector by a weight matrix, and then adding a bias vector to obtain an output vector. After m sub-mapping processes, m sub-mapping features are obtained, and the m sub-mapping features output by the m sub-mapping processes are concatenated to obtain the first mapping feature. In this way, m sub-mapping features are obtained through m sub-mapping processes, and the first mapping features are obtained by splicing, thereby realizing the refinement of the three-dimensional features and obtaining the feature meanings at different levels, thereby improving the feature expression ability of the first mapping features for the three-dimensional information of the object, thereby improving the feature expression ability of the first features obtained subsequently for the three-dimensional information of the object, thereby improving the consistency between the three-dimensional model generated based on the first feature and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0075] Step 103: Determine a second feature of the object based on the three-dimensional feature.

[0076] For step 103, when determining the second feature of the object based on the three-dimensional feature, the three-dimensional feature can be determined as the second feature of the object; or the three-dimensional feature and the generation parameter information can be fused to obtain the second feature of the object. It should be noted that based on the three-dimensional noise image, the first feature and the second feature, the diffusion model can be called to generate the three-dimensional model of the object. Based on this, the generation parameter information can be the diffusion time step t of the diffusion model. For example, if the total diffusion time step T of the diffusion model is 100, then the generation parameter information can be 100, 99, ..., 2, 1; that is, the generation parameter information is T, T-1, T2, ..., 3, 2, 1.

[0077] In some embodiments, based on the three-dimensional feature, the second feature of the object can be determined by performing the following steps: mapping the three-dimensional feature to obtain a second mapping feature; encoding the generated parameter information to obtain a parameter encoding feature; and fusing the second mapping feature and the parameter encoding feature to obtain a second feature.

[0078] Here, the three-dimensional feature can be first mapped to obtain the second mapping feature. It should be noted that the mapping process can refer to the above-mentioned mapping process of the three-dimensional feature. Specifically, the three-dimensional feature can be mapped to obtain the second mapping feature by executing the following steps: mapping the three-dimensional feature to n fourth mapping features of different feature sizes, where n is an integer greater than 1; splicing the n fourth mapping features to obtain the second mapping feature; or, the mapping process includes m sub-mapping processes, where m is an integer greater than 1; based on this, the three-dimensional feature can be mapped to obtain the second mapping feature by executing the following steps: when i is equal to 1, the three-dimensional feature is sub-mapped to obtain the sub-mapping feature output by the first sub-mapping process; when i is greater than 1 and less than or equal to m, the sub-mapping feature output by the i-1th sub-mapping process is sub-mapped to obtain the sub-mapping feature output by the i-th sub-mapping process; the m sub-mapping features output by the m sub-mapping process are spliced ​​to obtain the second mapping feature. In this way, the refinement of three-dimensional features and the acquisition of feature meanings at different levels are achieved, thereby improving the feature expression capability of the second mapping feature for the three-dimensional information of the object, thereby improving the feature expression capability of the subsequently obtained second feature for the three-dimensional information of the object, thereby improving the consistency between the three-dimensional model generated based on the second feature and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0079] Continuously, the generated parameter information is encoded to obtain a parameter encoding feature. The encoding process may be sine encoding, cosine encoding, etc., which is not limited here. Finally, the second mapping feature and the parameter encoding feature are fused to obtain the second feature. In some embodiments, the second mapping feature and the parameter encoding feature may be fused by adding, concatenating, multiplying, etc.

[0080] Step 104: Generate a three-dimensional model of the object based on the three-dimensional noise image, the first feature and the second feature.

[0081] For step 104, after obtaining the first feature and the second feature, a three-dimensional model of the object can be generated based on the three-dimensional noise image, the first feature and the second feature. In some embodiments, a three-dimensional noise image can be generated in advance. For example, the three-dimensional noise image can be sampled from a target distribution; the target distribution includes a plurality of random data that conform to a specific data distribution type (such as a normal distribution (or Gaussian distribution), a standard normal distribution, and a uniform distribution). The random data that conforms to the specific data distribution type can be generated based on a random data generation algorithm, and the plurality of random data are three-dimensional data. For example, the three-dimensional noise image can be Gaussian noise, Poisson noise, salt and pepper noise, Gaussian white noise, etc. Taking Gaussian noise as an example, a random data generation algorithm can be used to generate three-dimensional random data that conforms to a Gaussian distribution, and then a target random number that conforms to a Gaussian distribution is sampled from the three-dimensional random data that conforms to a Gaussian distribution, and the target random number that conforms to a Gaussian distribution is used as a three-dimensional noise image.

[0082] In some embodiments, the process of generating a three-dimensional model of an object actually includes: based on the first feature and the second feature, denoising the three-dimensional noise image to obtain denoised image features, and then reconstructing the denoised image features (such as decoding processing) to obtain the three-dimensional model of the object. In some embodiments, the three-dimensional model of the object can be a voxelized representation of the three-dimensional model.

[0083] In some embodiments, see Figure 3C Based on the three-dimensional noise image, the first feature and the second feature, a three-dimensional model of the object can be generated by executing the following steps 301-303: Step 301, encoding the three-dimensional noise image to obtain noise image features; Step 302, based on the first feature and the second feature, denoising the noise image features to obtain denoised image features; Step 303, voxelizing the denoised image features to obtain a three-dimensional model.

[0084] Here, for step 301, the three-dimensional noise image is encoded to obtain noise image features. The encoding process can be implemented by an image encoder Encoder, and the image encoder can include at least one encoding layer. The three-dimensional noise image is encoded by the at least one encoding layer to obtain noise image features. For step 302, based on the first feature and the second feature, the noise image feature is denoised to obtain denoised image features. In some embodiments, based on the first feature and the second feature, the noise image feature can be denoised by performing the following steps to obtain denoised image features: based on the first feature and the second feature, the noise image feature is downsampled to obtain a downsampled result; based on the first feature and the second feature, the downsampled result is upsampled to obtain a denoised image feature.

[0085] Here, firstly, based on the first feature and the second feature, the noise image feature is downsampled to obtain a downsampled result. That is, by controlling the downsampling process of the noise image feature through the first feature and the second feature, the downsampled result can be combined with the first feature and the second feature. Then, based on the first feature and the second feature, the downsampled result is upsampled to obtain a denoised image feature. That is, by controlling the upsampling process of the noise image feature through the first feature and the second feature, the denoised image feature can be combined with the first feature and the second feature. In this way, since the denoised image feature is combined with the first feature and the second feature, the consistency between the three-dimensional model generated based on the denoised image feature and the object in the image is improved, and the generation accuracy of the three-dimensional model is improved.

[0086] In practical applications, the downsampling process can be implemented through a downsampling network, and the upsampling process can be implemented through an upsampling network. The upsampling network may include a first convolutional network (for downsampling processing), a self-attention processing layer (SelfAttention) and a cross-attention processing layer (Cross Attention). Based on this, the downsampling process includes: performing a first convolution processing on the noise image feature through the first convolutional network to obtain a first convolution feature; performing a self-attention processing on the first convolution feature through the attention processing layer to obtain a first processing result; fusing the first feature and the second feature to obtain a third feature, and performing a cross-attention processing on the third feature and the first processing result through the cross-attention processing layer to obtain a downsampling result. In this way, the control of the third feature is added in the cross-attention processing process of the downsampling process, thereby improving the downsampling effect. The upsampling network includes a second convolution network (for upsampling processing), a self-attention processing layer (SelfAttention) and a cross-attention processing layer (Cross Attention). Based on this, the upsampling process includes: performing a second convolution processing on the downsampling result through the second convolution network to obtain a second convolution feature; performing self-attention processing on the second convolution feature through the attention processing layer to obtain a second processing result; fusing the first feature and the second feature to obtain a third feature, and performing cross-attention processing on the third feature and the second processing result through the cross-attention processing layer to obtain a denoised image feature. In this way, the control of the third feature is added in the cross-attention processing process of the upsampling process, which improves the upsampling effect, thereby improving the consistency between the three-dimensional model generated based on the denoised image feature and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0087] For step 303, the denoised image features are voxelized to obtain a three-dimensional model. The three-dimensional model is represented by voxelization. Voxelization is a method of dividing a continuous three-dimensional space into a finite number of volume elements (voxels), each voxel being a cubic unit in space. The advantage of voxelization is that it provides an intuitive way to represent three-dimensional objects and is easy to perform spatial operations such as cutting, moving, and rotating.

[0088] In some embodiments, the denoising process includes k denoising processes, where k is an integer greater than 1; based on this, based on the first feature and the second feature, the noise image feature can be denoised by executing the following steps to obtain the denoised image feature: when j is equal to 1, based on the first feature and the second feature, the noise image feature is denoised to obtain the intermediate denoised image feature of the first denoising output; when j is greater than 1 and less than or equal to k, based on the first feature and the second feature, the intermediate denoised image feature of the j-1th denoising output is denoised to obtain the intermediate denoised image feature of the jth denoising output; the intermediate denoised image feature of the kth denoising output is used as the denoised image feature.

[0089] Here, the denoising process includes k denoising processes. For each denoising process in the k denoising processes, when j is equal to 1, the noise image feature is denoised based on the first feature and the second feature to obtain the intermediate denoised image feature of the first denoising output; when j is greater than 1 and less than or equal to k, the intermediate denoised image feature of the j-1 denoising output is denoised based on the first feature and the second feature to obtain the intermediate denoised image feature of the j denoising output; after obtaining the intermediate denoised image feature of the k denoising output, the intermediate denoised image feature of the k denoising output is used as the denoised image feature. In this way, through k denoising processes, the denoising effect can be improved, so that the obtained denoised image feature can better express the three-dimensional information of the object, thereby improving the consistency between the three-dimensional model generated based on the denoised image feature and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0090] In some embodiments, based on the three-dimensional feature, the second feature of the object can be determined by performing the following steps: for the j-th denoising process, determine the generation parameter information corresponding to the j-th denoising process, and fuse the three-dimensional feature and the generation parameter information corresponding to the j-th denoising process to obtain the second feature.

[0091] Here, the generation parameter information may be the diffusion time step t of the diffusion model. For example, if the total diffusion time step T of the diffusion model is 100, then the generation parameter information may be 100, 99, ..., 2, 1; that is, the generation parameter information is T, T-1, T2, ..., 3, 2, 1. For example, for the first denoising process, the generation parameter information corresponding to the first denoising process, i.e., the T value, is determined, and the three-dimensional feature and the generation parameter information corresponding to the first denoising process are fused to obtain the second feature. That is, for the jth denoising process, the generation parameter information corresponding to the jth denoising process, i.e., t, is determined, and the three-dimensional feature and the generation parameter information corresponding to the jth denoising process are fused to obtain the second feature. In this way, combined with the time step t of each denoising process, the denoising effect can be further improved, so that the obtained denoised image features can better express the three-dimensional information of the object, thereby improving the consistency between the three-dimensional model generated based on the denoised image features and the object in the image, and improving the generation accuracy of the three-dimensional model.

[0092] By using the above-mentioned embodiments of the present application, the three-dimensional features of the object in the image are first extracted, and the image is encoded to obtain the image encoding features, and then the three-dimensional features and the image encoding features are fused to obtain the first feature of the object, and then based on the three-dimensional features, the second feature of the object is determined, and finally based on the three-dimensional noise image, the first feature and the second feature, a three-dimensional model of the object is generated. Here, the three-dimensional features of the object in the image are extracted, and the first feature and the second feature are obtained respectively based on the three-dimensional features, and then the three-dimensional noise image, the first feature and the second feature are combined to generate the three-dimensional model of the object; in this way, the generation of the three-dimensional model is controlled by the three-dimensional features of the object, and the accuracy of the generated three-dimensional model and the consistency with the object in the image are improved, that is, the generation effect of the three-dimensional model is improved by the embodiments of the present application.

[0093] The following describes an exemplary application of the embodiment of the present application in an actual application scenario. Deep generative models are increasingly used in the field of image generation. Denoising Diffusion Probabilistic Models (DDPMs) play a core role in this field. They have surpassed Variational Autoencoders (VAEs) and Generative Adversarial Networks (GANs) in both unconditional image generation and text-to-image conversion tasks. A key advantage of DDPMs is that they can ensure the quality and diversity of generated results through conditional guidance. With the development of 2D image generation technology, its influence has naturally extended to the 3D field. In the fields of virtual reality (VR), augmented reality (AR), medical imaging, industrial design, etc., the ability to generate 3D shapes (i.e., three-dimensional models) from 2D images has become particularly important. However, the related art can only extract two-dimensional visual features (such as models such as CLIP), and the occlusion and visual ambiguity problems in 2D images pose challenges to the accurate generation of 3D shapes.

[0094] Based on this, the embodiment of the present application provides a generation architecture of a three-dimensional model (i.e., 3D shape). This architecture can align 2D images and 3D shapes in latent space, ensuring that the 3D model generated by DDPM has higher structural and semantic consistency with the input 2D image, thereby improving the generation quality. This new method is expected to bring breakthrough progress in the field of 3D shape generation, providing more powerful and flexible tools for applications such as VR, AR, medical imaging, and industrial design.

[0095] Due to the occlusion and visual ambiguity problems in 2D images in the related art, the generated 3D shape lacks accuracy and consistency. Therefore, in the embodiment of the present application, a large number of data sets are used to train a 2D image encoder with prior knowledge. This prior knowledge can introduce 3D information in the high-dimensional space during the encoding process of the image encoding, so that the encoding result is accompanied by certain 3D shape information, which helps to solve this problem and generate a more accurate and consistent 3D shape.

[0096] The embodiments of the present application specifically include: 1) A novel algorithm architecture is proposed, which first realizes the alignment of 2D image and 3D shape in latent space, and then uses the denoising diffusion probability model (DDPM) to generate 3D shape, allowing the model to more effectively utilize 2D image information in the generation process. 2) 3D knowledge graph integration method: An innovative 3D knowledge integration technology is designed, which can effectively fuse 3D shape and 2D image information in latent space. This multimodal feature integration method can provide richer contextual information for 3D shape generation. Continue to explain next.

[0097] like Figure 4 ( Figure 4 3D-CLIP represents a pre-trained model, which is obtained by training on 2D image and 3D shape data using contrastive learning. DDPM represents a denoising diffusion model. i represents the input 2D image (i.e., the image including the object), S e represents the encoding in the latent space (the encoding corresponds to the input 2D image or 3D shape), V o is the generated 3Dshape.

[0098] The following is an explanation of the 3D-CLIP training process. Figure 5 , Figure 5 3D-CLIP is a training diagram of a three-dimensional feature extraction model provided in an embodiment of the present application. Here, 3D-CLIP includes an image encoder E i (i.e. the above three-dimensional feature extraction model) and the 3D shape encoder E s In terms of training data, the input of the embodiment of the present application is a 2D image and a corresponding 3D shape (voxel representation). In the image branch, a Transformer architecture model (such as ViT) or a CNN architecture model (such as ResNet) is used, and the image encoder E is obtained after training. i On the 3Dshape branch, the modified DeiT structure is used to adapt to the processing of 3D data. After training, the 3Dshape encoder (i.e. the feature extraction model mentioned above) is obtained. s Specifically, when the DeiT network extracts features, it will first divide the image into a certain number of patches, and then extract the embedding of the patches. Here, the dimension of the input data (i.e., patch) of the DeiT network changes from 2D to 3D, and the convolution kernel involved in the convolution operation also changes from a 2D convolution kernel to a 3D convolution kernel, and the dimension of the obtained embedding is also 3D. In terms of loss function, the contrast loss function is used. Through 3D-CLIP, the image encoder Ei (i.e. the above-mentioned three-dimensional feature extraction model) can obtain 3D information related to 3D shape.

[0099] When the image encoder E is trained i After that, the next step is to train the DDPM part, that is, to generate the 3D shape part of the model. Figure 6 , Figure 6 is a second schematic diagram of a generation architecture of a three-dimensional model provided in an embodiment of the present application. i correspond Figure 5 The trained E i , represents the image encoder; through E i Process the 2D image (i.e., the image including the object) to obtain a feature vector E o,i (i.e. the three-dimensional features mentioned above), and then undergo two linear calculations, i.e. linear transformations, which map the original feature vectors to other feature spaces.

[0100] Among them, o,i After the first Linear processing, we get E p,i , and then perform feature fusion with Time embedding (i.e., encoding the time steps of the diffusion model, i.e., the above-mentioned generated parameter features). The fusion method here is concat, i.e., splicing, to obtain a multimodal conditional vector E cond (That is, the second feature mentioned above).

[0101] Among them, o,i After the second linear processing, we get T i (T i The dimension and T c The same can be a hyperparameter, which can be controlled manually). Here, Linear is a plurality of linear mapping layers. The purpose of using multiple Linear is to represent detailed features. o,i It is an image encoding with 3D information in a 2D image. In order to refine the feature representation, the image encoding is subjected to multiple Linear transformations so that the feature vectors obtained after each Linear transformation focus on different meanings on the 3D shape.

[0102] E c Indicates that it is different from E i An image encoder, through E c The ability to introduce the original context information of the 2D image (i.e., the above-mentioned image encoding features) helps to improve the consistency between the generated 3D shape and the 2D image. c is an image encoder whose output is T c , dimension and T iConsistent. i and T c A sum operation will be performed to aggregate the two feature representations to obtain a conditional feature T that is rich in image context information, image structure and other 3D information. cond .

[0103] Continue, Figure 6 X0 in represents the initialization noise image (i.e. the above three-dimensional noise image). Since the generated object is a 3D shape, which is a 3D data, X0 is also a 3D noise image. t represents the graph of the t-th step in the denoising process of the diffusion model, X t+1 Represents the graph at step t+1. The input of the DDPM model consists of three parts: two conditions and a noise map. The conditions are: E cond and T cond , initialize the noise image to: X0; the output is a 3D image representation X t+1 .X t+1 After Voxelization transformation, a 3D shape (3D model) represented by voxels can be obtained.

[0104] The training process consists of two stages, which can be summarized as follows:

[0105] 1) Phase 1: 3D-CLIP model training. First, a pre-trained model called 3D-CLIP is trained. The model is based on the principle of contrastive learning and is trained on 2D images and 3D shapes (represented in voxelized form) data. The 3D-CLIP model contains two branches: the image branch and the 3D shape branch. The image branch uses the Transformer architecture, such as VisionTransformer (ViT), to extract image features and generate an image encoder E. i The 3D shape branch uses a modified version of the Data-efficient image Transformer (DeiT) to adapt the voxel data through 3D convolution to generate the shape encoder E s The present invention adopts the same contrast loss function as CLIP for training, aiming to enable the image encoder to capture the characteristic information of 3D shape.

[0106] 2) Second stage: DDPM model training. After the 3D-CLIP model training is completed, the second stage is to train the denoising diffusion model (DDPM) to generate 3D shapes. Specifically, the image encoder E in 3D-CLIP is used. i, through two linear transformations (Linear), the image features are mapped to the new latent space. The features after the first linear transformation are concatenated with the time embedding (Time embedding) to form a multimodal conditional vector E cond The second linear transformation refines the feature representation and generates conditional features T i In addition, another image encoder E is introduced c , its output T c With T i The bitwise sum is performed to integrate the image context information and structural information. In this way, a conditional feature T that integrates image and 3D information is obtained. cond In the DDPM model, the initialization noise map is used as the starting point X0, and the voxel representation of the 3D shape is generated through an iterative process. The input of DDPM includes the conditional vector E cond and T cond , and the noise map X0. The output of the model is a voxelized 3D shape representation, which can be further processed by Voxelization to obtain the desired 3D model.

[0107] By using the above embodiment of the present application, firstly, the 3D-CLIP model is trained by contrastive learning to obtain an image encoder and a shape encoder that can understand 3D shape information; secondly, these encoders are used to train a conditional 3D DDPM model that can generate a 3D shape consistent with the input image from noise. The 3D DDPM in the embodiment of the present application processes three-dimensional features, which improves the accuracy and consistency of 3D shape generation.

[0108] The following is a description of an exemplary structure of a three-dimensional model generation device 555 provided in an embodiment of the present application implemented as a software module. In some embodiments, Figure 2 As shown, the software modules in the three-dimensional model generation device 555 stored in the memory 550 may include: a feature processing module 5551, used to extract the three-dimensional features of the object in the image, and encode the image to obtain the image coding features; a fusion module 5552, used to fuse the three-dimensional features and the image coding features to obtain the first features of the object; a determination module 5553, used to determine the second features of the object based on the three-dimensional features; a generation module 5554, used to generate a three-dimensional model of the object based on the three-dimensional noise image, the first features and the second features.

[0109] In some embodiments, the three-dimensional feature is extracted from the image by a three-dimensional feature extraction model; the feature processing module 5551 is also used to extract a first sample three-dimensional feature of a sample object in the sample image through an initial feature extraction model before extracting the three-dimensional feature of the object in the image; perform feature extraction on the three-dimensional model of the sample object to obtain a second sample three-dimensional feature; and train the initial feature extraction model based on the difference between the first sample three-dimensional feature and the second sample three-dimensional feature to obtain the three-dimensional feature extraction model.

[0110] In some embodiments, the fusion module 5552 is further used to perform mapping processing on the three-dimensional feature to obtain a first mapping feature; and add the first mapping feature and the image coding feature to obtain the first feature.

[0111] In some embodiments, the fusion module 5552 is further used to map the three-dimensional feature into n third mapping features of different feature sizes, where n is an integer greater than 1; and concatenate the n third mapping features to obtain the first mapping feature.

[0112] In some embodiments, the mapping processing includes m sub-mapping processing, where m is an integer greater than 1; the fusion module 5552 is also used to perform sub-mapping processing on the three-dimensional feature when i is equal to 1, to obtain the sub-mapping feature output by the first sub-mapping processing; when i is greater than 1 and less than or equal to m, perform sub-mapping processing on the sub-mapping feature output by the i-1th sub-mapping processing to obtain the sub-mapping feature output by the i-th sub-mapping processing; and splice the m sub-mapping features output by the m sub-mapping processing to obtain the first mapping feature.

[0113] In some embodiments, the determination module 5553 is further used to map the three-dimensional feature to obtain a second mapping feature; encode the generated parameter information to obtain a parameter coding feature; and fuse the second mapping feature and the parameter coding feature to obtain the second feature.

[0114] In some embodiments, the generation module 5554 is also used to encode the three-dimensional noise image to obtain noise image features; based on the first features and the second features, denoise the noise image features to obtain denoised image features; and voxelize the denoised image features to obtain the three-dimensional model.

[0115] In some embodiments, the generating module 5554 is further used to downsample the noise image feature based on the first feature and the second feature to obtain a downsampled result; and upsample the downsampled result based on the first feature and the second feature to obtain the denoised image feature.

[0116] In some embodiments, the denoising process includes k denoising processes, where k is an integer greater than 1; the generating module 5554 is further used to, when j is equal to 1, denoise the noise image feature based on the first feature and the second feature to obtain the intermediate denoised image feature of the first denoising output; when j is greater than 1 and less than or equal to k, denoise the intermediate denoised image feature of the j-1th denoising output based on the first feature and the second feature to obtain the intermediate denoised image feature of the jth denoising output; and use the intermediate denoised image feature of the kth denoising output as the denoised image feature.

[0117] In some embodiments, the determination module 5553 is further used to determine the generation parameter information corresponding to the j-th denoising process, and fuse the three-dimensional feature with the generation parameter information corresponding to the j-th denoising process to obtain a second feature.

[0118] In some embodiments, the feature processing module 5551 is also used to extract at least one of the following three-dimensional features of the object in the image: depth information feature, object shape feature, object posture feature, object light and shadow feature, and object texture feature.

[0119] It should be noted that the description of the device embodiment in this application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, which will not be described in detail here. The unfinished technical details in the device for generating a three-dimensional model provided in the embodiment of this application can be understood based on the description of the technical details in the method embodiment above.

[0120] The embodiment of the present application also provides a computer program product, which includes computer executable instructions or computer programs, and the computer executable instructions or computer programs are stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instructions or computer program from the computer-readable storage medium, and the processor executes the computer executable instructions or computer program, so that the electronic device executes the method for generating a three-dimensional model provided in the embodiment of the present application.

[0121] An embodiment of the present application also provides a computer-readable storage medium, which stores computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the three-dimensional model generation method provided in the embodiment of the present application.

[0122] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0123] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0124] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0125] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0126] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A method for generating a three-dimensional model, characterized in that: The method comprises: Extracting three-dimensional features of an object in an image, and encoding the image to obtain image encoding features; fusing the three-dimensional feature and the image coding feature to obtain a first feature of the object; Based on the three-dimensional feature, determining a second feature of the object; A three-dimensional model of the object is generated based on the three-dimensional noise image, the first feature, and the second feature.

2. The method according to claim 1, characterized in that The three-dimensional features are extracted from the image by a three-dimensional feature extraction model; before extracting the three-dimensional features of the object in the image, the method further includes: Extracting a first sample three-dimensional feature of a sample object in a sample image by using an initial feature extraction model; Extracting features from the three-dimensional model of the sample object to obtain a second three-dimensional feature of the sample; Based on the difference between the first sample three-dimensional feature and the second sample three-dimensional feature, the initial feature extraction model is trained to obtain the three-dimensional feature extraction model.

3. The method according to claim 1, characterized in that The fusing the three-dimensional feature and the image coding feature to obtain the first feature of the object includes: Performing mapping processing on the three-dimensional feature to obtain a first mapping feature; The first mapping feature and the image coding feature are added to obtain the first feature.

4. The method according to claim 3, characterized in that The mapping process is performed on the three-dimensional feature to obtain a first mapping feature, including: Mapping the three-dimensional feature into n third mapping features of different feature sizes, where n is an integer greater than 1; The n third mapping features are concatenated to obtain the first mapping feature.

5. The method according to claim 3, characterized in that The mapping process includes m sub-mapping processes, where m is an integer greater than 1; the mapping process is performed on the three-dimensional feature to obtain a first mapping feature, including: When i is equal to 1, sub-mapping processing is performed on the three-dimensional feature to obtain a sub-mapping feature output by the first sub-mapping processing; When i is greater than 1 and less than or equal to m, performing sub-mapping processing on the sub-mapping feature output by the (i-1)th sub-mapping processing to obtain the sub-mapping feature output by the (i)th sub-mapping processing; The m sub-mapping features outputted from the m sub-mapping processes are concatenated to obtain the first mapping feature.

6. The method according to claim 1, characterized in that The determining, based on the three-dimensional feature, a second feature of the object comprises: Performing mapping processing on the three-dimensional feature to obtain a second mapping feature; Encode the generated parameter information to obtain parameter encoding features; The second mapping feature and the parameter encoding feature are fused to obtain the second feature.

7. The method according to claim 1, characterized in that The step of generating a three-dimensional model of the object based on the three-dimensional noise image, the first feature and the second feature comprises: Encoding the three-dimensional noise image to obtain noise image features; Based on the first feature and the second feature, denoising the noise image feature to obtain a denoised image feature; The denoised image features are voxelized to obtain the three-dimensional model.

8. The method according to claim 7, characterized in that The step of performing denoising on the noise image feature based on the first feature and the second feature to obtain the denoised image feature includes: Based on the first feature and the second feature, down-sampling the noise image feature to obtain a down-sampling result; Based on the first feature and the second feature, the down-sampling result is up-sampled to obtain the denoised image feature.

9. The method according to claim 7, characterized in that The denoising process includes k denoising processes, where k is an integer greater than 1; the denoising process is performed on the noise image feature based on the first feature and the second feature to obtain the denoised image feature, including: When j is equal to 1, the noise image feature is denoised based on the first feature and the second feature to obtain an intermediate denoised image feature of a first denoising output; When j is greater than 1 and less than or equal to k, based on the first feature and the second feature, denoising the intermediate denoised image feature output by the j-1th denoising to obtain the intermediate denoised image feature output by the jth denoising; The intermediate denoised image feature outputted by the kth denoising is used as the denoised image feature.

10. The method according to claim 9, characterized in that The determining, based on the three-dimensional feature, a second feature of the object comprises: For the j-th denoising process, generation parameter information corresponding to the j-th denoising process is determined, and the three-dimensional feature and the generation parameter information corresponding to the j-th denoising process are fused to obtain a second feature.

11. The method according to any one of claims 1 to 10, characterized in that: The step of extracting three-dimensional features of an object in an image comprises: Extract at least one of the following three-dimensional features of the object in the image: Depth information features, object shape features, object posture features, object light and shadow features, and object texture features.

12. A three-dimensional model generation device, characterized in that: The device comprises: A feature processing module, used for extracting three-dimensional features of objects in an image, and encoding the image to obtain image coding features; A fusion module, used for fusing the three-dimensional feature and the image coding feature to obtain a first feature of the object; A determination module, configured to determine a second feature of the object based on the three-dimensional feature; A generating module is used to generate a three-dimensional model of the object based on the three-dimensional noise image, the first feature and the second feature.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions; A processor, configured to implement the method for generating a three-dimensional model as described in any one of claims 1 to 11 when executing the computer executable instructions stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the method for generating a three-dimensional model described in any one of claims 1 to 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the method for generating a three-dimensional model described in any one of claims 1 to 11 is implemented.

Images