Method and system for generating 3D model

Abstract

The invention provides a 3D model generation method and system, and the method comprises the steps: obtaining a target intersection point between a target ray of a target object and the surface of the target object when the target object is imaged at m visual angles and n domains under each visual angle, determining a target physical material value of the target object according to the target intersection point, and inverse rendering is carried out on the target physical material values and the target intersection points based on physical constraints, a 3D model of the target object is obtained, and the physical constraints are used for constraining that pixel values corresponding to the target intersection points have consistency in physical attributes. According to the method, the consistency of the generated 3D model in physical attributes can be ensured, so that the generated 3D model well reproduces the imaging environment when the target object is imaged, and the 3D model is more vivid in the aspects of illumination, shadow, reflection and the like.

Description

Technical Field

[0001] The present specification relates to the field of artificial intelligence technology, and in particular to a method and system for generating a 3D model. Background Art

[0002] The traditional 3D modeling process is highly dependent on manual labor, which has largely hindered the application of 3D content in industries such as the Internet and games.

[0003] In the related art, a 3D model can be generated by directly generating a 3D model, which is mainly based on 3D training data, and performs encoding-decoding and training to generate a model in a 3D space.

[0004] However, this solution is time-efficient and can better express the geometric features of simple objects, but it cannot depict complex details and cannot meet the requirements for generating physical materials.

[0005] It is worth noting that the contents of the above-mentioned related technologies are merely information known to the inventor personally, which does not mean that the above-mentioned information has entered the public domain before the filing date of this specification, nor does it mean that it can become the prior art of this specification. Summary of the invention

[0006] This specification provides a method and system for generating a 3D model to avoid at least one of the above-mentioned technical problems.

[0007] In a first aspect, this specification provides a method for generating a 3D model, comprising:

[0008] Obtaining a target intersection point between a target ray of the target object and a surface of the target object when the target object is imaged at m viewing angles and n domains at each viewing angle, wherein both m and n are integers greater than 1;

[0009] Determining a target physical material value of the target object according to the target intersection point;

[0010] The target physical material value and the target intersection are inversely rendered based on physical constraints to obtain a 3D model of the target object, wherein the physical constraints are used to constrain the pixel values ​​corresponding to each target intersection to be consistent in physical properties.

[0011] In a second aspect, the present specification provides a 3D model generation system, comprising:

[0012] at least one storage medium storing at least one set of instructions for generating a 3D model;

[0013] At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, it reads the at least one instruction set and executes the method as described in the first aspect according to the instructions of the at least one instruction set.

[0014] In a third aspect, the present specification provides a computer-readable non-temporary storage medium, wherein the computer-readable non-temporary storage medium stores at least one instruction set, and the at least one instruction set is executed by at least one processor to implement the method described in the first aspect.

[0015] It can be seen from the above technical solutions that the 3D model generation method and system provided in this specification can obtain accurate geometric information of the target object by determining the target intersection, which represents a specific position on the surface of the target object; through m perspectives, n domains under each perspective, and target physical material values, more comprehensive and rich three-dimensional structural information and physical properties (such as material, etc.) of the target object can be provided, which helps to generate a more complete, more detailed, and more realistic 3D model; through inverse rendering based on physical constraints, it can ensure that the generated 3D model is consistent in physical properties, and the generated 3D model can well reproduce the imaging environment when the target object is imaged, so that the 3D model is more realistic in terms of lighting, shadows, reflections, etc.

[0016] Other functions of the 3D model generation method and system provided in this specification will be partially listed in the following description. The creative aspects of the 3D model generation method and system provided in this specification can be fully explained by practicing or using the methods, devices and combinations described in the following detailed examples. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the embodiments of this specification, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this specification. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0018] Figure 1 A schematic diagram of an application scenario of the method for generating a 3D model provided in an embodiment of this specification;

[0019] Figure 2 A schematic diagram of the structure of a 3D model generation system provided in an embodiment of this specification;

[0020] Figure 3 A schematic diagram of a process for generating a 3D model according to an embodiment of the present invention;

[0021] Figure 4 A schematic diagram of the principle of the method for generating a 3D model provided in the embodiments of this specification;

[0022] Figure 5 A schematic diagram of the principle of the hybrid representation method provided in the embodiments of this specification;

[0023] Figure 6 A schematic diagram of the principle of training two basic MLPs provided in the embodiments of this specification;

[0024] Figure 7 A schematic diagram of the principle of generating a multi-view image by a multi-view image generation model provided in an embodiment of this specification;

[0025] Figure 8 A schematic diagram of the training principle of the multi-view image generation model provided in the embodiments of this specification;

[0026] Fig. 9 A schematic diagram of the principle of generating a 3D asset based on a 3D model provided in an embodiment of this specification. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with this specification. Instead, they are merely examples of devices and methods consistent with some aspects of this specification as detailed in the appended claims.

[0028] It should be understood that the terms "including" and "having" and any variations thereof in the embodiments of this specification are intended to cover but not exclude inclusion. For example, a product or device comprising a series of components is not necessarily limited to those components explicitly listed, but may include other components not explicitly listed or inherent to these products or devices.

[0029] In the embodiments of this specification, the term "and / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship.

[0030] The term "plurality" in the embodiments of the present specification refers to two or more than two, and other quantifiers are similar.

[0031] The terms "first", "second", "third", etc. in this specification are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise indicated. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, for example, they can be implemented in an order other than those given in the diagrams or descriptions of the embodiments of this specification.

[0032] The term "unit / module" as used in this specification refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0033] To facilitate the reader's understanding of this specification, at least some of the terms in this specification are explained as follows:

[0034] A generative model is a machine learning model that generates new data samples whose distribution is similar to the training data. For example, by learning the characteristics of the training data, the generative model can create new samples such as images, audio, or text.

[0035] The Diffusion Model is a generative model that is often used for image generation. It generates data based on the process of gradually adding and denoising Gaussian noise. It generates images by modeling the gradual change process of data distribution.

[0036] Physically Based Rendering (PBR) is a rendering technology that aims to simulate the real interaction between light and materials to achieve more realistic visual effects, including lighting, shadows, and material reflections.

[0037] Albedo is the ability of an object's surface to reflect light. It usually refers to the intensity of visible light reflected by an object under white light conditions, which affects the color and brightness of the object.

[0038] Roughness is a parameter that describes the microscopic unevenness of an object's surface, which affects the degree of reflection of light on the surface, thereby affecting the glossiness and effect of the reflection.

[0039] Metallic describes the degree to which a material's surface displays metallic properties, which usually affects the reflection and color of light. Metal surfaces have higher reflections and specific color mapping.

[0040] Signed Distance Function (SDF) is a function that represents the distance from a point in space to the surface of an object. A positive distance means it is outside the object, and a negative distance means it is inside the object. It is usually used in computer graphics and physical simulations.

[0041] 3D Mesh is a structure used in computer graphics to represent three-dimensional objects. It is usually composed of vertices, edges, and faces to form the geometric shape of the object.

[0042] Transformer is a deep learning architecture for processing sequence data, which mainly consists of a self-attention mechanism and a feedforward neural network. It is widely used in natural language processing and computer vision tasks.

[0043] Inverse Rendering is a technique for inferring the geometry, material properties, and lighting conditions of a scene from an image, mainly used in computer vision and graphics. Its goal is to restore the real physical properties of the scene.

[0044] To facilitate readers' understanding of this specification, the application scenarios of this specification are now introduced.

[0045] The technical solution provided in this specification is applicable to scenarios where a 3D model (such as a 3D mesh) of a target object needs to be generated.

[0046] Figure 1 FIG. 1 is a schematic diagram of an application scenario of a method for generating a 3D model according to an embodiment of the present specification (hereinafter referred to as a generating method), wherein the generating method of the present specification can be applied to: Figure 1 The application scenario 100 shown. Figure 1 As shown, the application scenario 100 may include a target user 101 , a client 102 , a server 103 , and a network 104 .

[0047] The target user 101 may be a user who triggers the generation of a 3D model of the target object. For example, the target user 101 may perform a target operation on the client 102 to trigger the generation of the 3D model.

[0048] The client 102 may be an electronic device that provides interactive functions to the target user 101. For example, the client 102 may provide an interactive interface to the target user 101, and the target user 101 may perform interactive operations in the interactive page. In some embodiments, the client 102 executes the generation method described in this specification in response to detecting the generation operation of the 3D model triggered by the target user 101. At this time, the client 102 may store data or instructions for executing the generation method described in this specification, and may execute or be used to execute the data or instructions. In some embodiments, the client 102 may include a hardware device with a data information processing function and the necessary program required to drive the hardware device to work, so as to execute the generation method described in this specification.

[0049] In some embodiments, the client 102 may include a mobile device, a tablet computer, a laptop computer, a built-in device of a motor vehicle, or the like, or any combination thereof. In some embodiments, the mobile device may include a smart home device, a smart mobile device, a virtual reality device, an augmented reality device, or the like, or any combination thereof. In some embodiments, the smart home device may include a smart TV, a desktop computer, or the like, or any combination thereof. In some embodiments, the smart mobile device may include a smart phone, a personal digital assistant, a gaming device, a navigation device, or the like, or any combination thereof. In some embodiments, the built-in device in a motor vehicle may include an onboard computer, an onboard TV, or the like.

[0050] In some embodiments, the client 102 may be installed with one or more applications (Application, APP). APP can provide the target user 101 with the ability and interface to interact with the outside world through the network 104. APP includes but is not limited to: web browser APP programs, search APP programs, chat APP programs, shopping APP programs, video APP programs, financial management APP programs, instant messaging tools, email clients, social platform software, etc. In some embodiments, the client 102 may be installed with a target APP.

[0051] like Figure 1 As shown, the client 102 can be connected to the server 103 for communication. The server 103 can be connected to one client 102 for communication, or can be connected to multiple clients 102 for communication. In some embodiments, the client 102 can interact with the server 103 through the network 104 to receive or send messages, etc.

[0052] The server 103 may be a server that provides various services. For example, the server 103 may be a cloud server or a local server. The server 103 may be connected to a client 102 for communication and receive data sent by the client 102, or may be connected to multiple clients 102 for communication and receive data sent by each client 102.

[0053] In some embodiments, the generation method described in this specification can be executed on the server 103. In this case, the server 103 can store data or instructions for executing the generation method described in this specification, and can execute or be used to execute the data or instructions. The server 103 can include a hardware device with a data information processing function and a necessary program required to drive the hardware device to work.

[0054] The network 104 is a medium for providing a communication connection between the client 102 and the server 103. The network 104 can facilitate the exchange of information or data. Figure 1 As shown, the client 102 and the server 103 can be connected to the network 104 respectively, and transmit information or data to each other through the network 104.

[0055] In some embodiments, the network 104 may be any type of wired or wireless network, or a combination thereof. For example, the network 104 may include a cable network, a wired network, a fiber optic network, a telecommunications network, an intranet, the Internet, a local area network (LAN), a wide area network (WAN), a wireless local area network (WLAN), a metropolitan area network (MAN), a public switched telephone network (PSTN), a Bluetooth network TM, a short-range wireless network (ZigBee TM), a near field communication (NFC) network, or a similar network.

[0056] In some embodiments, the network 104 may include one or more network access points. For example, the network 104 may include a wired or wireless network access point, such as a base station or an Internet exchange point, through which one or more components of the client 102 and the server 103 can connect to the network 104 to exchange data or information.

[0057] It is worth mentioning that Figure 1The number of clients 102, servers 103, and networks 104 in the example are merely illustrative. Depending on the implementation requirements, there may be any number of clients 102, servers 103, and networks 104. The generation method provided in this specification may be executed entirely on the client 102, entirely on the server 103, or partially on the client 102 and partially on the server 103.

[0058] That is to say, Figure 1 and for Figure 1 The above description is only used to illustrate the application scenarios to which the generation method of this specification may be applicable, and should not be understood as a limitation on the application scenarios.

[0059] Figure 2 The hardware structure diagram of a generation system 200 provided according to an embodiment of the present specification is shown. The generation system 200 can execute the generation method described in the present specification. The generation method is introduced in other parts of the present specification. When the generation method is executed on the client 102, the generation system 200 can be the client 102. When the generation method is executed on the server 103, the generation system 200 can be the server 103. When the generation method is partially executed on the client 102 and partially executed on the server 103, the generation system 200 can be a system including the client 102 and the server 103.

[0060] like Figure 2 As shown, the generation system 200 may include at least one storage medium 203 and at least one processor 202. In some embodiments, the generation system 200 may further include a communication port 204 and an internal communication bus 201. The generation system 200 may further include an I / O component 205.

[0061] The internal communication bus 201 can connect different system components. For example, the internal communication bus 201 can connect the storage medium 203, the processor 202, the communication port 204 and the I / O component 205.

[0062] I / O component 205 supports input / output between generation system 200 and other components.

[0063] The communication port 204 is used to generate data communication between the system 200 and the outside world. For example, the communication port 204 can be used to generate data communication between the system 200 and the network 104. The communication port 204 can be a wired communication port or a wireless communication port.

[0064] The storage medium 203 may include a data storage device. The data storage device may be a non-temporary storage medium or a temporary storage medium. For example, the data storage device may include one or more of a disk 2031, a read-only storage medium (ROM) 2032, or a random access storage medium (RAM) 2033. The storage medium 203 also includes at least one instruction set stored in the data storage device. The instruction set includes computer program code, and the computer program code may include a program, a routine, an object, a component, a data structure, a process, a module, etc. for executing the generation method provided in this specification.

[0065] At least one processor 202 may be connected in communication with at least one storage medium 203. At least one processor 202 is used to execute the above-mentioned at least one instruction set. When the generation system 200 is running, at least one processor 202 reads the at least one instruction set and executes the generation method provided in this specification according to the instructions of the at least one instruction set. The processor 202 may execute all the steps included in the generation method. The processor 202 may be in the form of one or more processors. In some embodiments, the processor 202 may include one or more hardware processors, such as a microcontroller, a microprocessor, a reduced instruction set computer (RISC), an application-specific integrated circuit (ASIC), an application-specific instruction set processor (ASIP), a central processing unit (CPU), a graphics processing unit (GPU), a physical processing unit (PPU), a microcontroller unit, a digital signal processor (DSP), a field programmable gate array (FPGA), an advanced RISC machine (ARM), a programmable logic device (PLD), any circuit or processor capable of performing one or more functions, etc., or any combination thereof.

[0066] For illustration purposes only, only one processor 202 is shown in the generation system 200 in the accompanying drawings. However, it should be noted that the generation system 200 in this specification may also include multiple processors, and therefore, the operations and / or method steps disclosed in this specification may be performed by one processor or by multiple processors in combination. For example, if the processor 202 of the generation system 200 is described in this specification as performing step A and step B, it should be understood that step A and step B may also be performed jointly or separately by two different processors 202 (e.g., the first processor performs step A, the second processor performs step B, or the first and second processors perform steps A and B together).

[0067] See also Figure 3 , Figure 3 This is a flow chart of the generation method provided in the embodiment of this specification. Figure 3 As shown, the method includes the following S301 to S303:

[0068] S301: Obtaining a target intersection point between a target ray of the target object and a surface (iso-surface) of the target object when the target object is imaged at m viewing angles and n domains at each viewing angle, wherein m and n are both integers greater than 1.

[0069] It is understandable that this embodiment introduces the technical solution for generating a 3D model from the inference stage, and the following also includes an introduction to the technical solution for the training stage. Therefore, in order to distinguish different stages, the "target" in this embodiment and the "sample" in the following are used to distinguish the same type of technical features, which cannot be understood as a limitation of the technical features.

[0070] For example, the "target" in this embodiment and the "sample" below are intended to facilitate the distinction between the same type of technical features "object" and "intersection" in the reasoning stage and the training stage, and should not be understood as a limitation on "object" and "intersection".

[0071] The target object can be a person or an object. Figure 4 As shown, the target object may be a safety helmet.

[0072] It is worth noting that the imaging of the target object can be understood as: the target object is imaged at multiple viewing angles, and when the imaging is performed at each viewing angle, specifically, multiple domains are imaged at the viewing angle.

[0073] Exemplarily, if m and n are both 6, the imaging of the target object can be understood as: the target object is imaged at 6 viewing angles, and imaging at any viewing angle among the 6 viewing angles, specifically imaging at the viewing angle to obtain images of 6 domains.

[0074] For example, Figure 4 The images of six domains under one viewing angle are shown as an example. The images of the six domains include: color image, normal image, depth image, albedo image, roughness image, and metalness image.

[0075] It can be understood that an image of a domain includes multiple target rays. For each target ray, there is a relationship between the target ray and the surface of the target object. Figure 4 The target intersection is shown.

[0076] Accordingly, this step can be understood as that, for each image of each domain at each viewing angle, the generation system determines the target intersection point between the target ray of the target object and the surface of the target object based on uniform sampling of the preset coordinate space.

[0077] In some embodiments, S301 may include the following steps 11 to 13:

[0078] Step 11: Obtain k sampling points on the target ray based on implicit surface sampling, where k is an integer greater than 1.

[0079] Exemplarily, the generation system performs sampling on the target ray based on an implicit representation (such as SDF) to obtain multiple sampling points.

[0080] For example, Figure 5 As shown, the generation system obtains five sampling points on the target ray based on implicit surface sampling: point 1 to point 5.

[0081] Step 12: Determine two truncation points among the k sampling points, wherein the two truncation points are two adjacent points among the k sampling points and are located on two sides of the surface.

[0082] Continue to refer to 5, the two truncation points are point 2 and point 3. Point 2 is a point inside the surface, and point 3 is a point outside the surface.

[0083] Step 13: Sample between the two cutoff points to obtain the target intersection point.

[0084] Combining the above examples with Figure 5 ,After determining the two truncation points (point 2 and point 3), the generation system further samples between point 2 and point 3 to obtain the target intersection point.

[0085] It is worth noting that, since one of the two cutoff points is located outside the surface and the other is located inside the surface, and the target intersection point is a point located on the surface, the generation system obtains the target intersection point by resampling between the two cutoff points, so that the determined target intersection point can have higher accuracy and reliability.

[0086] In some embodiments, step 13 may include the following steps 131 and 132:

[0087] Step 131: uniformly sample between two truncation points to obtain p sampling points, where p is an integer greater than 1.

[0088] Combined with the above analysis, the target intersection point is located between the two truncation points. Therefore, the generation system performs uniform sampling between the two truncation points to obtain multiple sampling points.

[0089] Exemplarily, the generation system may uniformly sample between two cutoff points based on adaptive sampling to obtain p sampling points. For example, the generation system may adjust the sampling density according to the local features of the surface of the target object so that the complex area has a higher resolution.

[0090] like Figure 5 As shown, p is 4. That is, the generating system samples between point 2 and point 3 to obtain 4 sampling points, namely point 6 to point 9.

[0091] Step 132: interpolate the p sampling points according to the signed distance function SDF values ​​and the cumulative distribution function CDF values ​​of the p sampling points to obtain the target intersection point.

[0092] Exemplarily, the generation system calculates the SDF values ​​of p sample points, calculates weights based on the p SDF values, and interpolates the p sample points based on the CDF with a preset probability (such as 0.5) to improve the accuracy of the difference, thereby accurately and reliably determining Figure 5 The target intersection point is shown in .

[0093] In some embodiments, the generation system may introduce additional smoothness constraints to optimize the calculation of the target intersection point to ensure that the target intersection point is close to reality and smoother.

[0094] Combined with the above analysis, it can be seen that the number of target intersections is multiple. Accordingly, after obtaining the target intersections, the generation system can also verify whether each target intersection conforms to the geometric information in the sparse view image to maintain overall consistency.

[0095] Combined with the above analysis of step 131 and step 132, it can be seen that in this embodiment, the generating device resamples between two truncation points to interpolate the target intersection point among the resampled points based on the SDF value and CDF value of the resampled points, which can further improve the effectiveness and accuracy of the determined target intersection point.

[0096] That is to say, in this embodiment, a hybrid representation method is used to determine the target intersection. Among them, the hybrid representation method includes an implicit representation method (i.e., an implicit sampling method) and an explicit representation method (i.e., an explicit sampling method). The explicit representation method can be understood as a method of sampling to obtain the target intersection after the implicit sampling method in the above example. The target intersection is determined by the hybrid representation method, while achieving differentiability, stability, and compatibility.

[0097] S302: Determine a target physical material value of the target object according to the target intersection point.

[0098] The target physical material value may be understood as a value characterizing the physical material of the target object. Exemplarily, the target physical material value includes albedo, roughness, and metalness.

[0099] In some embodiments, the target intersection point is predicted based on a pre-trained first multi-layer perceptron MLPs, and the target physical material value is predicted based on a pre-trained second MLPs.

[0100] The trained first MLPs and the trained second MLPs are obtained based on training samples, where the training samples include images of the target object at m perspectives, and each perspective image includes images of n domains.

[0101] Exemplarily, the generation system or other system may train the first MLPs and the second MLPs based on the training samples to train the first MLPs to learn the ability to predict the target intersection points and train the second MLPs to learn the ability to predict the target physical material values, thereby obtaining the trained first MLPs and the trained second MLPs. The technical solution for training is described in other parts of this specification.

[0102] For example, combined with Figure 4 It can be seen that the trained first MLPs (not shown in the figure) can be used to predict the SDF value to determine the target intersection point on this basis. Figure 4 The input of the MLPs shown in FIG. 1 is the target intersection point, specifically the coordinates of the target intersection point, and the output is the target physical material value (not shown in the figure).

[0103] In this embodiment, by combining MLPs, the efficiency, accuracy and reliability of predicting the target intersection point and the target physical property value can be improved.

[0104] S303: Inversely render the target physical material value and the target intersection point based on the physical constraint to obtain a 3D model of the target object, wherein the physical constraint is used to constrain the pixel values ​​corresponding to each target intersection point to be consistent in physical properties.

[0105] As can be seen from the above examples, the number of target intersection points is multiple. For example, when m and n are both 6, the target intersection points are all target intersection points between each target ray in the 36 domain images and the surface of the target object.

[0106] In the process of inverse rendering of the generation system to obtain the 3D model, the inverse rendering process is constrained based on physical constraints so that the inverse rendering process conforms to the laws of physics, so that the pixel values ​​of all target intersections between each target ray and the surface of the target object in the 36 domain images are consistent in physical properties, so that the 3D model can truly and reliably restore the characteristics of the target object in the imaging environment.

[0107] The physical attributes can be understood as the attributes of the imaging environment when the target object is imaged, and the attributes of the target object itself. For example, the physical attributes may include illumination, geometry, material attributes, and the like.

[0108] Combined with the above analysis of S301 to S303, it can be seen that in this embodiment, by determining the target intersection point, which represents a specific position on the surface of the target object, the precise geometric information of the target object can be obtained; through m perspectives, n domains under each perspective, and the target physical material value, more comprehensive and rich three-dimensional structural information and physical properties (such as material, etc.) of the target object can be provided, which is conducive to generating a more complete, more detailed and more realistic 3D model; through inverse rendering based on physical constraints, it can be ensured that the generated 3D model is consistent in physical properties, so that the generated 3D model can well reproduce the imaging environment when the target object is imaged, so that the 3D model is more realistic in terms of lighting, shadows, reflections, etc.

[0109] In some embodiments, S303 may include: inputting the physical material value and the target intersection point into a physically based rendering (PBR) lighting equation, so as to obtain a 3D model by inverse rendering based on the PBR lighting equation.

[0110] Among them, the PBR lighting equation is used to determine the light value reflected by the target intersection under the physical material value after being illuminated by the light source in the imaging environment, and the pixel value of the pixel point obtained by reflecting the light value.

[0111] For example, the PBR lighting equation can be expressed by Formula 1:

[0112]

[0113] Where Ω represents the hemispherical domain for integration, ω O is the viewing direction, n is the normal, x is the surface point with the normal, L i is the incident light, ω i is the incident direction, d is the differential, f r is the bidirectional reflectance distribution function.

[0114] The integral in Formula 1 can be calculated by discrete integral or closed form approximation. Relatively speaking, in order to improve efficiency, a closed form can be used. For example, the generation system can approximate the illumination in the imaging environment as a spherical Gaussian distribution, and determine the integral in Formula 1 based on the spherical Gaussian distribution.

[0115] In this embodiment, the PBR lighting equation is used as an additional constraint to ensure that the pixel values ​​of the generated pixels conform to the physical lighting laws and other physical laws. Accordingly, when the pixel values ​​of the pixels corresponding to each target intersection are obtained, the image of the domain composed of the pixels corresponding to the corresponding target intersection can be restored, so that the 3D model of the target object that conforms to the physical lighting laws and other physical laws can be restored.

[0116] For example, Figure 4 As shown, the target object is a helmet, and the generation system can be obtained as follows Figure 4 A 3D model of the hard hat shown.

[0117] Combined with the above analysis, it can be seen that the generation system can predict the target intersection point with the help of the trained first MLPs and predict the target physical attribute value with the help of the trained second MLPs. The technical scheme of the training process of the two trained MLPs is now described. Among them, the training process includes the following steps 1 to 5:

[0118] Step 1: obtaining sample intersection points between sample rays of the target object and the surface of the target object when imaging the target object based on the first basic MLPs, wherein the imaging of the target object includes imaging the target object in m viewing angles, and each viewing angle includes images of n domains.

[0119] It can be understood that in order to avoid cumbersome description, the technical features of this embodiment that are the same or similar to those in the above examples will not be described in detail in this embodiment.

[0120] For example, regarding the implementation principle of step 1, please refer to the above S301, which will not be repeated here.

[0121] It is worth noting that in the above S301, the generation system samples coordinates from the preset coordinate space by uniform sampling to determine the target intersection point. During the training process, the generation system samples coordinates from the preset coordinate space by random sampling to determine the sample intersection point so as to iteratively update the parameters of the two basic MLPs.

[0122] That is to say, Figure 6 As shown, this step can be understood as that, for each sample ray of the image in each domain at each viewing angle, the generation system determines the sample intersection point between the sample ray of the target object and the surface of the target object based on the random sampling of the preset coordinate space by the basic first MLPs.

[0123] Step 2: Based on the basic second MLPs, the sample physical material value of the target object is predicted according to the sample intersection points.

[0124] Continue reading Figure 6 ,The generation system inputs the sample intersection points into the ,base second MLPs, and outputs the sample physical material values ​​,such as albedo, roughness, and metalness.

[0125] Step 3: Determine the sample color value of the target object based on the sample intersection point and the sample physical material value.

[0126] Combining the above examples with Figure 6, the generation system inputs the sample intersection and sample physical material value into the PBR lighting equation to obtain the sample color value.

[0127] Step 4: Determine the sample normal and sample depth of the target object based on the sample intersection point.

[0128] Exemplarily, the sample intersection has coordinate information, such as sample coordinates. The generation system can calculate according to the sample coordinates to obtain the corresponding Figure 6 Sample normal and sample depth shown.

[0129] Step 5: Taking minimizing the true values ​​corresponding to the sample physical material value, sample color value, sample normal, sample depth and the images of the n domains as the training target, the parameters of the basic first MLPs and the basic second MLPs are updated to obtain the trained first MLPs and the trained second MLPs.

[0130] Combining the above examples with Figure 6 The generation system can respectively construct loss functions between the predicted sample color value, sample normal, sample depth, sample albedo, sample roughness, and sample metalness and the color value, normal, depth, albedo, roughness, and metalness of the images in the six domains of the training samples, and take minimizing the sum of these loss functions as the training goal, and update the parameters of the basic first MLPs and the basic second MLPs to obtain the trained first MLPs and the trained second MLPs.

[0131] Combined with the above analysis of steps 1 to 5, it can be seen that in this embodiment, by determining the prediction results based on two basic MLPs and minimizing the difference between the prediction results and the true values, it is ensured that the generated physical material values, color values, normals, and depths are as close to the actual situation as possible, thereby improving the prediction ability of the two trained MLPs.

[0132] In some embodiments, the technical solution for obtaining training samples for training two basic MLPs includes the following steps 1 and 2:

[0133] Step 1: Obtain a target single-view image of the target object.

[0134] like Figure 7 As shown, the target object may be a helmet. Accordingly, the target single-view image is an image of the helmet taken from a certain perspective. It should be understood that Figure 7 The target single-view image shown in the figure is just an example of a certain view angle, and cannot be understood as a limitation of the view angle. That is, the input target single-view image can be an image of any view angle.

[0135] The following example can be used to obtain a single-view image of the target:

[0136] In one example, the generation system may be connected to an image acquisition device and receive a target single-view image sent by the image acquisition device.

[0137] In another example, the generation system may provide a tool for loading images, and the user may transfer the target single-view image to the generation system through the tool for loading images.

[0138] Among them, the tool for loading images can be an interface for connecting to an external device, such as an interface for connecting to other storage devices, through which the target monoscopic image transmitted by the external device is obtained; the tool for loading images can also be a display device, such as the generation system can output an interface of the image loading function on the display device, and the user can import the target monoscopic image into the generation system through this interface.

[0139] Step 2: Input the target single-view image into the pre-trained multi-view image generation model, and output the target object's m-view images, where one view image includes n domain images. The domain images of the target object have consistency in physical properties.

[0140] Exemplarily, the multi-view image generation model can be pre-trained by a training system. The training system can be the same system as the generation system or a different system, which is not limited in this embodiment. The multi-view generation model is trained to predict a multi-view image based on a single-view image.

[0141] Accordingly, in this embodiment, the multi-view image generation model performs prediction based on the target single-view image to obtain images of m views. Moreover, for each of the m views, the image of the view includes images of multiple domains.

[0142] This embodiment has no other restrictions on m and n. For example, m can be equal to n; m can also be greater than n; m can also be less than n. For example, taking m and n as equal and 6 as an example, the generation system can generate 36 domain images based on the multi-view image generation model. Similarly, this embodiment does not limit the specific perspectives in the multi-view.

[0143] It is worth noting that, in this embodiment, the domain images generated by the generation system are consistent in physical properties, that is, the domain images under different viewing angles can correctly reflect the real physical properties of the target object.

[0144] Exemplarily, the physical attributes may include illumination, geometry, and material attributes. That is, each domain image can truly and consistently reflect the illumination in the imaging environment of the target object, the geometric features of the target object, and the physical material features of the target object.

[0145] This means that domain images under different viewing angles can correctly reflect the material and lighting characteristics of the target object, etc. That is, each domain image can realistically and consistently reflect the lighting effects, geometric characteristics, and material performance.

[0146] Based on the above analysis of step 1 and step 2, it can be known that in this embodiment, a multi-perspective image corresponding to the target single-perspective image is obtained by predicting based on a multi-perspective image generation model, so as to expand from a single perspective to a multi-perspective and provide richer visual information; by ensuring that the physical properties of each domain image in the multi-perspective image are consistent, realistic lighting and material performance can be ensured, thereby making the details of the generated domain image richer and more accurate.

[0147] In some embodiments, the n domains include multiple ones of: color, normal, depth, albedo, roughness, and metallic.

[0148] Exemplarily, combined with the above analysis, n>1, therefore, the domain image of the target object at one viewing angle can be two of the color image, normal image, depth image, albedo image, roughness image, and metalness image, or it can be three or more, up to all six.

[0149] In some embodiments, n is an integer greater than 2, and the n domains are used to characterize geometric properties and / or physical material properties of the target object.

[0150] Exemplarily, for each of the m viewing angles, the generation system generates at least three domain images under the viewing angle. The three domain images can represent the geometric properties of the target object; can also represent the physical material properties of the target object; or can represent both the geometric properties of the target object and the physical material properties of the target object.

[0151] For example, n can be 6. Figure 7 As shown, the images of the six domains of a viewing angle may include a color image, a normal image, a depth image, an albedo image, a roughness image, and a metalness image at the viewing angle.

[0152] In some embodiments, step 2 may include steps 1 to 3 as follows:

[0153] Step 1: Input the target single-view image into the multi-view image generation model to extract key features representing the target object from the target single-view image based on the multi-view image generation model.

[0154] The key features characterizing the target object may be understood as image features in the target single-view image that are used to characterize the characteristics of the target object.

[0155] Exemplary, combined Figure 7 It can be seen that the multi-view image generation model includes a residual block (ResBlock). The residual block can extract key features that characterize the target object from the target single-view image.

[0156] Step 2: Based on the extracted key features characterizing the target object, determine the features used to characterize the interdependencies between the target object in different viewpoints and different domains of the m viewpoints.

[0157] In some embodiments, step 2 may include the following steps 21 and 22:

[0158] Step 21: Based on the extracted key features characterizing the target object, determine features characterizing the dependency relationship between different viewpoints of the target object in the m viewpoints.

[0159] Continue reading Figure 7 , the multi-view image generation model can also include multi-view self-attention. The multi-view self-attention mechanism is used to capture the dependencies between different perspectives. That is, through the self-attention mechanism, the multi-view image generation model can determine the global dependencies between images of different perspectives, ensuring that the generated images are consistent under different perspectives.

[0160] Step 22: According to the determined features characterizing the dependency relationship of the target object between different viewpoints of the m viewpoints, determine features characterizing the mutual dependency relationship of the target object under different viewpoints of the m viewpoints and between different domains.

[0161] In some embodiments, step 22 may include the following steps 221 and 222:

[0162] Step 221: According to the determined features characterizing the dependencies between different viewpoints of the target object in the m viewpoints, features characterizing the global dependencies between different domains of the target object in the same viewpoint of the m viewpoints are determined.

[0163] Continue reading Figure 7 , the multi-view image generation model can also include cross-domain self-attention. Cross-domain self-attention is used to process the dependencies between different domains. That is, through cross-domain self-attention, the multi-view image generation model can learn the global dependencies between images in different domains and ensure that the generated images are consistent between different domains.

[0164] In some embodiments, the cross-domain self-attention includes a trained domain switching parameter. Through the trained switching parameter, the cross-domain self-attention determines which different domains need to extract global dependencies.

[0165] For example, if the switching parameter of two domains is 1, it means that the cross-domain self-attention of the two domains needs to be calculated to extract the dependency relationship; conversely, if the switching parameter of a domain is 0, it means that the attention of the domain and other domains is calculated.

[0166] In some embodiments, among the n domains, a main domain may be included. The main domain is used to characterize the need to calculate cross-domain attention between the domain and other domains separately. That is, the generation system can guide the generation of other domains based on the main domain as a reference. This not only improves the quality of the generated images, but also ensures their physical rationality. In this way, the multi-view image generation model is able to generate realistic multi-view images while maintaining geometric and material details.

[0167] Step 222: According to the determined features of the global dependency relationship between different domains of the target object at the same viewpoint of the m viewpoints, features for characterizing the mutual dependency relationship between different domains of the target object at different viewpoints of the m viewpoints are determined.

[0168] Continue reading Figure 7 , the multi-view image generation model can also include cross attention. Cross attention processes the interdependence between images from different perspectives and different domains. That is, through cross attention, the multi-view image generation model can determine the interdependence between images from different perspectives and different domains, ensuring that the generated images are consistent between different perspectives and different domains.

[0169] Step 3: Generate images of the target object from m perspectives according to the determined features of the target object at different perspectives of the m perspectives and the interdependence between different domains.

[0170] Continue reading Figure 7 , the multi-view image generation model can also include UNet. The input of UNet is the output of cross attention; the output of UNet is the image of the target object from m perspectives. For example, the output of UNet includes Figure 7 Images of 6 domains under one viewing angle are shown.

[0171] Based on the above analysis of steps 1 to 3, it can be seen that in this embodiment, by analyzing the interdependence between different perspectives, different domains under the same perspective, and different domains under different perspectives, it is ensured that the generated multi-view images are consistent in physical properties. For example, the normal image should maintain the same geometric structure under different perspectives, the color image should maintain the same material color under different perspectives, and the depth image should maintain a consistent spatial position relationship under different perspectives, etc., which are not listed here one by one. In addition, by analyzing the complex interactions between images in different domains (such as color, normal, depth, etc.), the multi-view image generation model can capture these dependencies and ensure that the generated images are coordinated and consistent between various domains. For example, changes in the normal image will affect the lighting effect, and changes in the color image will affect the material performance. The multi-view image generation model can ensure that these changes are coherent under different perspectives. Moreover, by analyzing the interdependence between different perspectives, different domains under the same perspective, and different domains under different perspectives, the multi-view image generation model can better understand the three-dimensional structure and physical properties of the target object, thereby generating more realistic multi-view images.

[0172] In some embodiments, the multi-view image generation model is trained based on physical constraints, and the physical constraints are used to constrain the generation of images of each domain of the target object to follow physical laws so as to truly and consistently represent the target object.

[0173] Physical constraints can be understood as introducing a series of physical laws and rules during the training process of the multi-view image generation model to guide and restrict the multi-view images generated by the multi-view image generation model to conform to the physical properties of the real world. For example, physical constraints may include geometric consistency, lighting consistency, material consistency, motion consistency, etc.

[0174] Geometric consistency can be understood as the fact that depth images from different perspectives should maintain a consistent spatial structure. For example, the shape, size, and position of an object should remain unchanged from different perspectives.

[0175] Lighting consistency can be understood as the lighting effects at different viewing angles should follow the same lighting model. For example, the position, intensity, and direction of the light source should be consistent in all viewing angles, and the shadow projection and reflection effects should also conform to physical laws.

[0176] Material consistency can be understood as the material properties (such as color, roughness, transparency, etc.) under different viewing angles should remain consistent to ensure that the appearance of the object under different viewing angles is realistic and credible.

[0177] Motion consistency can be understood as, if it involves dynamic scenes, the motion trajectory and speed of the target object should conform to the laws of physical dynamics to ensure the realism of the motion.

[0178] In this embodiment, by introducing physical constraints, the images generated by the multi-view image generation model strictly follow the laws of physics in terms of geometry, lighting, material, etc., making the generated images more realistic and credible. Users can observe objects from different perspectives without feeling any obvious unnaturalness or disobedience.

[0179] In order to further understand the implementation principle of the multi-view image generation model to generate multi-view images based on single-view images, the training process of the multi-view image generation model is now described in detail from the training stage. The training process includes the following steps 1 to 3:

[0180] Step 1: Obtain a sample single-view image of a sample object.

[0181] Step 2: Input the sample single-view image into the basic network model to predict the images of the sample object from m perspectives and the images of n domains corresponding to each of the images from the m perspectives of the sample object based on the basic network model according to the physical constraints that follow the laws of physics, wherein m and n are both integers greater than 1, and the physical constraints are used to constrain the images of the sample object from m perspectives and the images of n domains corresponding to each of the images from the m perspectives of the sample object to have consistency in physical properties.

[0182] Exemplarily, a single-view image is input into a basic network model so that the basic network model predicts images of m viewpoints of the sample object and images of n domains corresponding to each viewpoint (such as color images, normal images, depth images, albedo images, roughness images, metalness images, etc.) according to pre-set physical constraints.

[0183] By introducing physical constraints, we ensure that the generated multi-view images and images in each domain have consistency in physical properties. For example, the normal image should follow the geometric structure of the object surface, the depth image should reflect the spatial position relationship of the object, the color image should maintain the color characteristics of the material, and so on.

[0184] Step 3: Taking minimizing the difference between the images of the sample object from m perspectives and the preset true values ​​corresponding to each domain image as the training goal, the parameters of the basic network model are updated to obtain a multi-view image generation model.

[0185] By minimizing the difference between the predicted results (including images of m perspectives of sample objects and domain images generated based on the basic network model) and the preset true values ​​(which can be marked in advance), the generated images and domain images are ensured to be as close to the actual situation as possible.

[0186] In some embodiments, step 2 may include steps 1 to 3 as follows:

[0187] Step 1: Based on the basic network model, extract the features of the sample objects under different perspectives of m viewpoints and the interdependence between different domains.

[0188] In some embodiments, step 1 may include: extracting features of global dependencies between different domains of sample objects at the same perspective of m perspectives based on the basic network model, and determining features of mutual dependencies between sample objects at different perspectives of m perspectives and between different domains based on the extracted features of global dependencies between different domains of sample objects at the same perspective of m perspectives.

[0189] Combined with the above examples (such as Figure 7 corresponding examples) and Figure 8 It can be seen that the basic network model can include: residual block, multi-view self-attention, cross-domain self-attention, and cross-attention. The principles of each component in the basic network model can be found in the above examples and will not be repeated here.

[0190] Correspondingly, the training system inputs the sample single-view image into the basic network model, and the basic network model determines the characteristics of the interdependence between the sample objects in m different perspectives and different domains through sequentially connected residual blocks, multi-view self-attention, cross-domain self-attention, and cross-attention.

[0191] It is worth noting that, in some embodiments, the parameters of the basic network model include a domain switching parameter; the domain switching parameter is used to determine the domain from which the basic network model needs to extract features of the global dependency relationship.

[0192] Exemplarily, the cross-domain self-attention includes a switching parameter. Through the switching parameter, the cross-domain self-attention determines which different domains need to extract global dependencies.

[0193] For example, if the switching parameter of two domains is 1, it means that the cross-domain self-attention of the two domains needs to be calculated to extract the dependency relationship; conversely, if the switching parameter of a domain is 0, it means that the attention of the domain and other domains is calculated.

[0194] In some embodiments, the n domains may include a main domain, which is used to indicate that cross-domain attention between the domain and other domains needs to be calculated separately.

[0195] For example, the main domain can be a color domain. So as to reflect the information of the appearance of the object relatively and most completely based on the color domain. That is to say, through the technical solution provided by this embodiment, it is possible to use the color image as a reference to guide the generation of other domains while ensuring the geometric consistency between different domains. This not only improves the quality of the generated images, but also ensures their physical rationality. In this way, the basic network model can learn to generate realistic multi-view images while maintaining geometric and material details.

[0196] It is worth noting that when the cross-domain self-attention includes domain switching parameters and the main domain, the cross-domain attention can be first calculated through the main domain, and then weighted fused with the switching parameters to obtain the features finally captured by the cross-domain self-attention.

[0197] Step 2: Based on the extracted features of the sample object at different viewing angles of the m viewing angles and the interdependence between different domains, an image of the sample object in n domains at each of the m viewing angles is generated.

[0198] Step 3: Perform physical based rendering (PBR) on the images of the n domains at each of the m viewing angles according to the physical constraints to obtain images of the sample object at m viewing angles.

[0199] Exemplary, combined Figure 8 It can be seen that the basic network model can also include UNet and PBR lighting equation (PBRCondition).

[0200] Among them, the input of UNet is the output of cross attention, and the output of UNet is the image of n domains of the sample object in each of the m perspectives. For example, the output of UNet is 36 domain images.

[0201] The input of the PBR lighting equation is the output of UNet. For example, the input of the PBR lighting equation is 36 domain images. Accordingly, the PBR lighting equation performs inverse rendering on the 36 domain images based on physical constraints, and restores the images under 6 perspectives corresponding to the sample single-view image (that is, 6 images are obtained).

[0202] For the description of the PBR lighting equation, please refer to the above example, which will not be repeated here.

[0203] Based on the above analysis of steps 1 to 3, it can be seen that in this embodiment, the PBR lighting equation is used as an additional constraint to ensure that the generated image complies with the physical lighting law and other physical laws, such as making the six images truly and consistently represent the sample object.

[0204] In some embodiments, step 3 may include steps 1 to 3 as follows:

[0205] Step 1: Calculate the image loss between the images of the sample object at m perspectives and the corresponding preset true values.

[0206] Combined with the above example, if m is 6, the image loss is the sum of the losses corresponding to each of the 6 images.

[0207] Step 2: Calculate the domain loss between each domain image of the sample object and the corresponding preset true value.

[0208] Combined with the above example, if m and n are both 6, there are 36 domain images in total, and the domain loss is the sum of the losses corresponding to each of the 36 domain images.

[0209] Step 3: Taking minimizing image loss and domain loss as the training goal, the parameters of the basic network model are updated to obtain a multi-view image generation model, where the weight of image loss is smaller than the weight of domain loss.

[0210] Exemplarily, the image and the domain have weights respectively, and the weight of the domain is greater than the weight of the image. Accordingly, the parameters of the basic network model are updated with minimizing the weighted loss of the image and the domain as the training goal, such as including updating the domain switching parameters in the above example, to obtain a multi-view image generation model.

[0211] In some embodiments, after the 3D model of the target object is constructed, the generation method may further include: performing retopology processing on the 3D model to obtain a processed 3D model.

[0212] For example, Fig. 9 As shown, after obtaining the 3D model, the generation system can perform retopology processing on the 3D model, such as mesh quadrification, to simplify the 3D model into a low-poly model with a more reasonable and optimized topological structure while retaining the key features and details of the 3D model.

[0213] That is, in this embodiment, the generation system optimizes the 3D model by retopology to improve the editability, rendering quality, animation and deformation capabilities of the processed 3D model, while maintaining the shape and details of the 3D model before processing.

[0214] In some embodiments, after the processed 3D model is obtained through retopology optimization, the generation method may further include: performing texture coordinate UV unfolding and baking on the processed 3D model to obtain a two-dimensional texture map of the target object.

[0215] Among them, the processed 3D models and two-dimensional images can be called 3D assets.

[0216] For example, see Fig. 9 After obtaining the processed 3D model, the generation system may perform UV Unwrapping on the processed 3D model so as to map the surface of the processed 3D model onto a 2D plane, thereby applying a texture map to the 3D model.

[0217] For example, the vertices of the processed 3D model have corresponding UV coordinates, which specify the position of the vertex in the 2D texture image. Through UV unfolding, it can be ensured that the texture image can be correctly mapped to the surface of the processed 3D model.

[0218] Continue reading Fig. 9 After the processed 3D model is UV-unwrapped, the generation system can bake the result of the UV unwrapping so that certain properties of the processed 3D model (such as lighting, shadows, normals, colors, etc.) are calculated and stored as a two-dimensional map. The result of the baking is one or more 2D images that can be directly applied to the model without recalculating these properties at runtime.

[0219] In this embodiment, the generation system can ensure that the texture image can be correctly mapped onto the surface of the processed 3D model through UV unfolding and baking, and can pre-calculate and store complex lighting, shadows, normals and other information as maps, thereby reducing the amount of calculation.

[0220] It is worth noting that the above examples are only used to illustrate possible implementations of the generation method of this specification, and cannot be understood as limiting the implementation of the generation method of this specification. Exemplarily, some of the technical features in the above examples can be combined to obtain a new embodiment; new technical features can be added on the basis of the above examples to obtain a new embodiment; some technical features can be reduced on the basis of the above examples to obtain a new embodiment; some technical features in the above examples can be replaced with other technical features; some technical features and their order in the above examples can be adjusted to obtain a new embodiment, and so on.

[0221] Take "combining some of the above technical features to obtain a new embodiment" as an example:

[0222] Combining the above analysis, we can see that Fig. 9 The technical features shown and Figure 4 The technical features shown in the figure are combined to obtain the corresponding embodiments; combined with the above analysis, it can be seen that Figure 4 The technical features shown and Figure 7 The technical features shown in the figure are combined to obtain the corresponding embodiments. Correspondingly, the above two examples can also be combined to obtain a new example. Figure 4 , Figure 7 , Fig. 9 The technical features shown are combined to obtain corresponding embodiments.

[0223] It is worth noting that by including Figure 4 , Figure 7 , Fig. 9 The technical solution obtained by combining the technical features shown can greatly simplify the workflow of the traditional 3D design industry (including 2D plan design, 3D view design, 3D modeling, texture mapping design, etc.), shortening its work cycle from several days to several minutes. In addition, it also provides high-quality generation results to meet the needs of relevant 3D designers, graphics workers and other groups.

[0224] Take "replacing some of the technical features in the above examples with other technical features" as an example:

[0225] For example, you can Fig. 9 The processed 3D model shown in is replaced with the 3D model to obtain the corresponding embodiment. That is, in some embodiments, the generation system can perform UV unwrapping and baking on the processed 3D model after retopology of the 3D model as described in the above examples. In other embodiments, the generation system can also perform UV unwrapping and baking on the 3D model after obtaining the 3D model.

[0226] For example, in the Figure 4 and Figure 7 In the example, if Figure 7 The network architecture of the multi-view image generation model in is a diffusion model, then Figure 7 The network architecture shown in the example can be replaced by a generative adversarial network (GAN) and an automatic regression generator.

[0227] For example, the above MLPs can also be replaced by other models, such as Transformer.

[0228] Other ways of determining new embodiments are not listed here one by one.

[0229] According to the above technical concept, this specification also provides a computer-readable non-temporary storage medium, in which at least one instruction set is stored. When the at least one instruction set is executed by a processor, the steps of the generation method described in this specification are implemented.

[0230] In some possible implementations, the various aspects of this specification can also be implemented as a form of a program product, which includes program code. When the program product is run on the generation system 200, the program code is used to make the generation system 200 perform the steps of the generation method described in this specification. The program product for implementing the above method can adopt a portable compact disk read-only memory (CD-ROM) to include program code, and can be run on the generation system 200. However, the program product of this specification is not limited to this. In this specification, the readable storage medium can be any tangible medium containing or storing a program, which can be used by the instruction execution system or used in combination with it. The program product can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples of readable storage media include: an electrical connection with one or more conductors, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, wherein a readable program code is carried. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above. Program code for performing the operations of the present specification may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the generation system 200, partially on the generation system 200, as a stand-alone software package, partially on the generation system 200 and partially on a remote generation system, or entirely on a remote generation system 200.

[0231] The above is a description of a specific embodiment of the present specification. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps recorded in the claims can be performed in an order different from that in the embodiments and still achieve the desired results. In addition, the processes depicted in the drawings do not necessarily require a specific order or a continuous order to achieve the desired results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.

[0232] In summary, after reading this detailed disclosure, those skilled in the art will appreciate that the foregoing detailed disclosure may be presented only by way of example and may not be limiting. Although not explicitly stated herein, those skilled in the art will appreciate that this specification requires various reasonable changes, improvements and modifications to the embodiments. These changes, improvements and modifications are intended to be proposed by this specification and are within the spirit and scope of the exemplary embodiments of this specification.

[0233] In addition, certain terms in this specification have been used to describe embodiments of this specification. For example, "one embodiment", "an embodiment" and / or "some embodiments" mean that a particular feature, structure or characteristic described in conjunction with the embodiment may be included in at least one embodiment of this specification. Therefore, it can be emphasized and should be understood that two or more references to "an embodiment" or "one embodiment" or "an alternative embodiment" in various parts of this specification do not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics may be appropriately combined in one or more embodiments of this specification.

[0234] It should be understood that in the foregoing description of the embodiments of this specification, in order to help understand a feature and for the purpose of simplifying this specification, this specification combines various features in a single embodiment, figure or its description. However, this does not mean that the combination of these features is necessary. When reading this specification, it is entirely possible for a person skilled in the art to mark out some of the devices as separate embodiments. In other words, the embodiments in this specification can also be understood as the integration of multiple secondary embodiments. And the content of each secondary embodiment is also valid when it is less than all the features of a single aforementioned disclosed embodiment.

[0235] Each patent, patent application, patent application publication, and other material, such as articles, books, specifications, publications, documents, literature, etc., cited herein (excluding any historical review documents related thereto) is hereby incorporated by reference for all purposes relevant to this document, such as in the specification and claims herein. However, if there is any inconsistency or conflict between the descriptions, definitions, and / or terminology of the aforementioned materials and the descriptions, definitions, and / or terminology used herein, the descriptions, definitions, and / or terminology used herein shall prevail.

[0236] Finally, it should be understood that the embodiments of the application disclosed herein are explanations of the principles of the embodiments of this specification. Other modified embodiments are also within the scope of this specification. Therefore, the embodiments disclosed in this specification are only used as examples and not as limitations. Those skilled in the art can adopt alternative configurations according to the embodiments in this specification to implement the applications in this specification. Therefore, the embodiments of this specification are not limited to the embodiments accurately described in the application.

Claims

1. A method for generating a 3D model, comprising: Obtaining a target intersection point between a target ray of the target object and a surface of the target object when the target object is imaged at m viewing angles and n domains at each viewing angle, wherein both m and n are integers greater than 1; Determining a target physical material value of the target object according to the target intersection point; The target physical material value and the target intersection are inversely rendered based on physical constraints to obtain a 3D model of the target object, wherein the physical constraints are used to constrain the pixel values ​​corresponding to each target intersection to be consistent in physical properties.

2. The method according to claim 1, wherein: The step of obtaining a target intersection point between a target ray of the target object and a surface of the target object when imaging the target object at m viewing angles and n domains at each viewing angle comprises: Obtaining k sampling points on the target ray based on implicit surface sampling, wherein k is an integer greater than 1; Determine two truncation points among the k sampling points, wherein the two truncation points are two adjacent points among the k sampling points and are located on two sides of the surface; The target intersection point is obtained by sampling between the two truncation points.

3. The method according to claim 2, wherein: The sampling between the two truncation points to obtain the target intersection point includes: Uniformly sampling between the two truncation points to obtain p sampling points, wherein p is an integer greater than 1; The p sampling points are interpolated according to the signed distance function SDF values ​​and the cumulative distribution function CDF values ​​of the p sampling points to obtain the target intersection point.

4. The method according to claim 1, wherein: The inverse rendering of the physical material value and the intersection point based on the physical constraint to obtain the 3D model of the target object includes: Inputting the physical material value and the target intersection point into a physical based rendering (PBR) lighting equation to obtain the 3D model by inverse rendering based on the PBR lighting equation; Among them, the PBR lighting equation is used to determine the light value reflected by the target intersection under the physical material value after being illuminated by the light source in the imaging environment, and the pixel value of the pixel point obtained by reflecting the light value.

5. The method according to claim 1, wherein: The target intersection point is predicted based on a pre-trained first multi-layer perceptron MLPs, and the target physical material value is predicted based on a pre-trained second MLPs; wherein, The trained first MLPs and the trained second MLPs are obtained through training based on training samples, wherein the training samples include images of the target object at the m viewing angles, and the images at each viewing angle include images of the n domains.

6. The method according to claim 5, wherein: The training step of obtaining the trained first MLPs and the trained second MLPs includes: When obtaining the target object imaging based on the basic first MLPs, the sample intersection points between the sample rays of the target object and the surface of the target object, wherein the target object imaging includes imaging the target object in the m viewing angles, and each viewing angle includes the images of the n domains; Predicting the sample physical material value of the target object based on the sample intersections based on the basic second MLPs; Determine a sample color value of the target object according to the sample intersection point and the sample physical material value; Determine a sample normal and a sample depth of the target object according to the sample intersection point; Taking minimizing the true values ​​corresponding to the sample physical material value, the sample color value, the sample normal, the sample depth and the images of the n domains as the training goal, the parameters of the basic first MLPs and the basic second MLPs are updated to obtain the trained first MLPs and the trained second MLPs.

7. The method according to claim 5, wherein: Obtaining the training sample comprises: Obtaining a target single-view image of the target object; The target single-view image is input into a pre-trained multi-view image generation model, and images of the target object from m perspectives are output, where an image from one perspective includes images from n domains, and the images of the target object in each domain are consistent in physical properties.

8. The method according to claim 1, wherein: The target physical material values ​​include albedo, roughness, and metalness.

9. The method according to any one of claims 1 to 8, wherein: The method further comprises: The 3D model is subjected to retopology processing to obtain a processed 3D model.

10. The method according to claim 9, wherein: The method further comprises: The processed 3D model is subjected to texture coordinate UV expansion and baking to obtain a two-dimensional map of the target object.

11. A 3D model generation system, comprising: at least one storage medium storing at least one set of instructions for generating a 3D model; At least one processor is communicatively connected to the at least one storage medium, wherein when the at least one processor is running, the at least one instruction set is read, and the method as described in any one of claims 1 to 10 is executed according to the instructions of the at least one instruction set.

Images