Three-dimensional modeling method, electronic device, storage medium, and computer program product

Abstract

The application discloses a three-dimensional modeling method, an electronic device, a storage medium and a computer program product, comprising: obtaining a target object to be modeled in a monocular image and physical state information of the target object; the physical state information represents a plurality of key physical states of the target object in the appearance evolution process; the plurality of key physical states represent one or more object states corresponding to the target object when the morphology or spatial attribute changes in the growth or motion cycle; obtaining a first model sequence of the target object, the first model sequence comprising a plurality of 3D models corresponding to the key physical states; performing multi-view image generation processing on the plurality of 3D models to obtain first image data; generating first video data based on the first image data, the first video data comprising a plurality of multi-view videos of the target object in the appearance evolution; generating a second model sequence based on the first video data, the second model sequence comprising a plurality of 3D models of the target object in the appearance evolution.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and more particularly to a three-dimensional modeling method, electronic device, storage medium, and computer program product. Background Technology

[0002] Currently, generating the dynamic appearance changes of a single object in 3D space from a single object image is an important research direction in computer graphics and vision. After inputting a single object image, existing techniques first predict the appearance changes from the corresponding viewpoint based on a diffusion model and generate a monocular video. Then, based on the diffusion model, multi-view videos are generated from the monocular video, with each video describing the object's appearance changes from different viewpoints. Finally, based on neural radiation fields or Gaussian splashing techniques, the appearance changes of the object in 3D space are predicted from the multi-view videos. However, in the process of generating monocular videos, existing techniques suffer from several problems: the appearance changes lack key physical state constraints, leading to the generated results easily deviating from real physical laws; and when generating multi-view videos from monocular videos, the consistency of appearance between different viewpoints at the same moment is insufficient, resulting in distorted or physically inconsistent object shapes in the subsequent 3D reconstruction. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a three-dimensional modeling method, an electronic device, a storage medium, and a computer program product.

[0004] In a first aspect, embodiments of this application provide a three-dimensional modeling method, the method comprising: Acquire the target object to be modeled and its physical state information in a monocular image; wherein the physical state information represents multiple key physical states of the target object during its appearance evolution process; the multiple key physical states represent one or more object states corresponding to changes in the morphology or spatial attributes of the target object during its growth or movement cycle. Obtain a first model sequence of the target object, the first model sequence including multiple 3D models that correspond one-to-one with key physical states; Multiple 3D models are processed to generate images from multiple perspectives to obtain the first image data; Based on the first image data, first video data is generated, which includes multi-view videos of the target object as its appearance evolves from multiple perspectives. Based on the first video data, a second model sequence is generated, which includes multiple 3D models of the target object as its appearance evolves.

[0005] Secondly, embodiments of this application provide an electronic device, the electronic device comprising: Memory is used to store executable instructions for a computer; The processor, when executing computer-executable instructions stored in the memory, implements the three-dimensional modeling method provided in the embodiments of this application.

[0006] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program or computer-executable instructions for implementing the three-dimensional modeling method provided in embodiments of this application when executed by a processor.

[0007] Fourthly, embodiments of this application provide a computer program product, including a computer program or computer executable instructions, which, when executed by a processor, implement the three-dimensional modeling method provided in embodiments of this application.

[0008] In the technical solution of this application embodiment, the target object to be modeled and its physical state information are obtained from a monocular image. The physical state information represents multiple key physical states of the target object during its appearance evolution. These key physical states represent one or more object states corresponding to changes in morphology or spatial attributes during the target object's growth or movement cycle. A first model sequence of the target object is obtained, comprising multiple 3D models corresponding one-to-one with the key physical states. Multi-view image generation processing is performed on the multiple 3D models to obtain first image data. Based on the first image data, first video data is generated, comprising multi-view videos corresponding to multiple perspectives during the appearance evolution of the target object. Based on the first video data, a second model sequence is generated, comprising multiple 3D models of the target object during its appearance evolution. Thus, by utilizing the multi-view images rendered from the key physical states of the object, the consistency of the multi-view images at the corresponding physical state can be constrained, minimizing the inconsistency problem caused by multi-view generation. Furthermore, using the 3D models of the key physical states as constraints guides the self-evolution of the 3D object's appearance, ensuring that the object's appearance changes conform to expectations. Attached Figure Description

[0009] Figure 1 This is a schematic diagram of the architecture of the 3D modeling system provided in the embodiments of this application; Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application; Figure 3 This is a schematic diagram illustrating the principle of the three-dimensional modeling method provided in the embodiments of this application; Figure 4 This is a first flowchart illustrating the three-dimensional modeling method provided in this application embodiment; Figure 5 This is a schematic diagram of the second process of the three-dimensional modeling method provided in the embodiments of this application; Figure 6 This is a schematic diagram illustrating the principle of the virtual camera settings provided in the embodiments of this application; Figure 7 This is a schematic diagram of the third process of the three-dimensional modeling method provided in the embodiments of this application; Figure 8 This is a schematic diagram of the fourth process of the three-dimensional modeling method provided in the embodiments of this application. Detailed Implementation

[0010] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0011] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0012] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0013] In the embodiments of this application, the terms "module" or "unit" refer to a computer program or part of a computer program that has a predetermined function and works with other related parts to achieve a predetermined goal, and can be implemented wholly or partially using software, hardware (such as processing circuitry or memory), or a combination thereof. Similarly, a processor (or multiple processors or memory) can be used to implement one or more modules or units. Furthermore, each module or unit can be part of an overall module or unit that includes the functionality of that module or unit.

[0014] Unless otherwise defined, all technical and scientific terms used in the embodiments of this application have the same meaning as commonly understood by one of ordinary skill in the art. The terminology used in the embodiments of this application is for the purpose of describing the embodiments of this application only and is not intended to limit this application.

[0015] In the implementation of this application, the collection and processing of relevant data should strictly comply with the requirements of relevant national laws and regulations, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of laws and regulations and the authorization of the personal information subject.

[0016] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0017] 1) Monocular images are two-dimensional images obtained by a single camera or a single viewpoint, which do not contain depth information or multi-view geometric constraints.

[0018] 2) 3D modeling refers to the process of reconstructing the shape, structure and appearance of an object in three-dimensional space based on two-dimensional images or multi-view information using geometric, optical and physical models.

[0019] 3) Appearance evolution refers to the process of appearance change of an object over time, including but not limited to continuous or phased changes in visual attributes such as shape, texture, color, and posture.

[0020] 4) Image rendering refers to the process of generating a two-dimensional image from the geometric model, material, lighting and other information in a three-dimensional scene through calculation, which can simulate the imaging effect of a real camera.

[0021] 5) A virtual camera is a virtual imaging unit that simulates the intrinsic parameters (such as focal length and principal point) and extrinsic parameters (position and orientation) of a real camera in computer graphics or visual computing. It is used to render or sample images from any specified viewpoint.

[0022] 6) Monocular video refers to a sequence of images continuously captured from the same viewpoint by a fixed or moving virtual / real camera, used to describe the changes in the appearance of an object over time from that viewpoint.

[0023] 7) Multi-view video refers to multiple video streams that are simultaneously captured or generated by multiple virtual / real cameras with different perspectives. Each video corresponds to a fixed observation perspective and together describe the changes in the appearance of an object at the same time or in consecutive moments.

[0024] 8) Image-generated diffusion model refers to a diffusion probability model that takes an image as a conditional input. It learns the reverse process of gradually denoising from noise to generate a target image, thereby realizing the function of generating new images or video frames based on the input image.

[0025] In related technologies, generating the dynamic appearance changes of an object in three-dimensional space from a single object image includes the following steps: First, input a single object image, use a diffusion model to predict the appearance changes of the object from the corresponding viewpoint, and generate a monocular video describing the object's evolution over time; second, based on the diffusion model, convert the monocular video into a multi-view video, where each video corresponds to a different viewing perspective, collectively describing the object's appearance changes at various times; finally, use Neural Radiance Field (NeRF) or Gaussian Splatting techniques to reconstruct the object's appearance change process in three-dimensional space from the multi-view video.

[0026] However, monocular video generation in related technologies lacks physical state constraints, resulting in appearance changes that do not meet expectations. When generating monocular videos from a single image, existing diffusion models mainly rely on data-driven appearance learning, lacking constraints on key physical states during object changes. For example, given an initial image of a sapling, when predicting its growth process, the model struggles to guarantee the normal phenological sequence (such as germination, growth, flowering, fruiting, and flower fall), leading to significant deviations between the generated dynamic sequence and actual physical laws. Furthermore, multi-view video generation lacks cross-view consistency, resulting in flaws in the 3D reconstruction results. When generating multi-view videos from monocular videos, existing diffusion models generate video frames independently from different perspectives, making it difficult to ensure appearance consistency across different perspectives at the same moment. This inconsistency directly impacts subsequent 3D reconstruction stages, causing errors in neural radiation field or Gaussian splash models when fusing multi-view information. The resulting 3D object shape is prone to appearance distortion and physical inconsistencies. For example, when predicting the appearance of a flower at a certain moment from four perspectives, the petal details in the frontal view may differ slightly from the petal shape and leaf outline in the left view, causing local distortion in the 3D model.

[0027] To address the aforementioned issues, embodiments of this application provide a 3D modeling method, electronic device, storage medium, and computer program product, which can improve the accuracy of 3D modeling and ensure that the appearance evolution of objects conforms to expectations.

[0028] The following describes exemplary applications of the devices provided in the embodiments of this application. The electronic devices provided in the embodiments of this application can be implemented as various types of terminals such as laptops, tablets, desktop computers, set-top boxes, smartphones, smart speakers, smartwatches, smart TVs, and vehicle terminals, or they can be implemented as servers.

[0029] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the 3D modeling system provided in the embodiments of this application, for example. Figure 1The system involves server 100, terminal device 200, and network 300. Terminal device 200 is connected to server 100 through network 300, which can be a wide area network (WAN), a local area network (LAN), or a combination of both.

[0030] In some embodiments, the present application embodiments can be implemented collaboratively by a server and a terminal device. For example, the terminal device 200 acquires the target object to be modeled and its physical state information in a monocular image, obtains a first model sequence of the target object, and sends the first model sequence to the server 100. The server 100 generates a second model sequence using the 3D modeling method provided in the present application embodiments, and sends the second model sequence to the terminal device 200. The terminal device 200 displays the second model sequence in a renderer to present the appearance evolution of the target object.

[0031] In other embodiments, the embodiments of this application can be implemented by a terminal device alone. The terminal device 200 acquires the target object to be modeled and the physical state information of the target object in a monocular image, acquires a first model sequence of the target object, and generates a second model sequence using the three-dimensional modeling method provided in the embodiments of this application. The terminal device 200 displays the effect of the second model sequence in a renderer to present the appearance evolution of the target object.

[0032] In some embodiments, the terminal device or server can implement the 3D modeling method provided in this application by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be microprogram-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application program, module, or plug-in. Terminal devices include, but are not limited to, mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle terminals, and aircraft.

[0033] See Figure 2 , Figure 2 This is a schematic diagram of the structure of the electronic device provided in the embodiments of this application. Figure 2 The electronic device 400 shown can be either the server 100 or the terminal device 200 mentioned above. Figure 2 The illustrated electronic device 400 includes at least one processor 410, a memory 430, and at least one network interface 420. The various components of the electronic device 400 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2The general labeled all buses as Bus System 440.

[0034] Processor 410 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0035] The memory 430 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard disk drives, optical disk drives, etc. The memory 430 may optionally include one or more storage devices physically located away from the processor 410.

[0036] The memory 430 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 430 described in this application embodiment is intended to include any suitable type of memory.

[0037] In some embodiments, memory 430 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0038] Operating system 431 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks; The network communication module 432 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, exemplary network interfaces 420 including: Bluetooth, WiFi, and Universal Serial Bus (USB), etc. In some embodiments, the apparatus provided in this application can be implemented in software. Figure 2 A 3D modeling apparatus 433 stored in memory 430 is shown. This apparatus can be software in the form of programs and plugins, and includes the following software modules: a data acquisition module 4331 and a data processing module 4332. These modules are logically linked and can therefore be arbitrarily combined or further separated according to the functions they implement. The functions of each module will be described below.

[0039] In other embodiments, the apparatus provided in this application can be implemented in hardware. As an example, the apparatus provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the three-dimensional modeling method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0040] See Figure 3 , Figure 3 This is a schematic diagram illustrating the principle of 3D modeling provided in this application embodiment. First, 3D models of key physical states are created. Next, multi-view images of the key physical states are generated. Further, multi-view, multi-view video is generated. Finally, a 3D model sequence is generated based on the multi-view, multi-view video. (See [link]). Figure 4 , Figure 4 This is a schematic diagram of the first process of the three-dimensional modeling method provided in the embodiments of this application, which will be described in detail below.

[0041] In step 401, the target object to be modeled and its physical state information in the monocular image are obtained.

[0042] Among them, physical state information represents multiple key physical states of the target object during the appearance evolution process; multiple key physical states represent one or more object states corresponding to changes in the morphology or spatial attributes of the target object during its growth or movement cycle.

[0043] In this embodiment, the monocular image can be a two-dimensional image obtained by a single camera or a single viewpoint, without containing depth information or multi-view geometric constraints. The target object can be the object to be modeled in the monocular image, such as plants, animals, deformable objects, etc. Physical state information characterizes multiple key physical states of the target object during its appearance evolution process. Key physical states refer to one or more discrete states corresponding to significant changes in the morphological or spatial attributes (such as size, posture, texture, and structure) of an object during its growth or movement cycle, such as germination, growth, flowering, fruiting, and flower fall of plants, or deformation, movement, and rotation of objects. Appearance evolution refers to the process of appearance changes that occur in an object over time, including continuous or staged changes in visual attributes such as shape, texture, color, and posture.

[0044] In this embodiment, an input monocular image is acquired, and the target object to be modeled is extracted from it using image segmentation or object detection algorithms. The system also identifies or receives user-specified physical state information. This physical state information can be predefined or automatically matched based on the object category. For example, for plant objects, physical state information includes stages such as germination, flowering, and fruiting; for rigid body motion, it includes states such as initial, deformation, and reset. The physical state information describes the key nodes that the target object must experience throughout its appearance evolution, with each node corresponding to a specific geometric and appearance feature. Thus, by explicitly acquiring and utilizing physical state information, discrete constraints conforming to real physical laws are provided for subsequent 3D appearance evolution, avoiding the generation of changes that do not conform to physical common sense by purely data-driven methods, and improving the realism and controllability of the evolution process.

[0045] In step 402, a first model sequence of the target object is obtained, the first model sequence including multiple 3D models that correspond one-to-one with key physical states.

[0046] In this embodiment of the application, the first model sequence can be a collection of 3D models that correspond one-to-one with multiple key physical states. Taking the target object as a fruit tree as an example, the key physical states corresponding to the entire evolution process can include "sapling-grown tree-flowering-fruiting-fruit falling". The first model sequence includes 5 3D models, and each 3D model corresponds to a key physical state.

[0047] Each 3D model is a three-dimensional geometric and appearance representation of a target object in its corresponding physical state. A 3D model refers to the three-dimensional digital representation of an object, which can be in formats such as mesh, point cloud, implicit neural field (such as neural radiation field), or Gaussian point cloud (Gaussian splash). 3D modeling is the process of reconstructing the shape, structure, and appearance of an object in three-dimensional space based on two-dimensional images or multi-view information, using geometric, optical, and physical models.

[0048] In this embodiment, the physical state information of the target object is obtained, and then a 3D model corresponding to each key physical state is pre-created or acquired to form a first model sequence. Creation methods include, but are not limited to, manual modeling, generation based on physical simulation, or matching from an existing 3D scanning library. Each 3D model conforms to the true geometric shape, size proportion, and spatial position of that physical state. For example, for plant growth, the model for the germination state is small, the model for the flowering state is medium-sized, and the model for the fruiting state reaches its maximum size. All models have a unified world coordinate origin, ensuring correct positional relationships during subsequent rendering.

[0049] In some embodiments, the 3D model corresponding to the key physical states can be created using a large language model (or image diffusion model). For example, image data of the key physical states of the target object can be generated using a large language model (or image diffusion model). The text description of the key physical states of the target object is used as input to the large language model to generate image data of the key physical states of the target object. Then, a graph-based 3D algorithm (such as Trellis2) is used, inputting the image data of the key physical states of the target object generated in the previous step to generate the 3D model corresponding to the key physical states of the target object. Finally, manual refinement is performed according to the actual situation to ensure that the final 3D model conforms to physical common sense and is accurate in detail. The generation of image data of the key physical states of the target object can be achieved in the following way: First, obtain the text description of the key physical states of the target object. Taking a fruit tree as the object, the key physical states of the fruit tree can include "germination-growth-flowering-fruiting-fruit falling". The text description is used as the conditional input. Next, the text description is fed into a pre-trained large language model or image diffusion model (such as DALL). Models such as E, Stable Diffusion, and Imagen semantically encode text and extract feature vectors related to physical states. Next, a diffusion model starts with random Gaussian noise and uses a progressive denoising reverse diffusion process, guided by text feature vectors at each step, to generate initial image data corresponding to key physical states. Finally, the generated initial image data undergoes post-processing, including resolution enhancement, detail sharpening, color correction, and removal of artifacts, resulting in clear image data of key physical states that conforms to physical principles. After obtaining the image data, manual selection or local refinement can be performed based on actual physical laws to ensure the accuracy of the shape, texture, and spatial relationships of objects in the image.

[0050] In other embodiments, the 3D models corresponding to the key physical states can be created by artists entirely by hand, for example, based on common sense. It should be noted that the embodiments of this application are not limited to the specific implementation methods for creating 3D models of key physical states.

[0051] In this way, by pre-establishing a sequence of 3D models bound to key physical states, abstract physical laws are transformed into concrete and controllable three-dimensional assets, providing high-quality anchor points for the subsequent generation of videos and dynamic 3D models that conform to physical logic. This fundamentally solves the problem in existing technologies where the lack of physical constraints leads to unintended appearance evolution.

[0052] In step 403, multiple 3D models are processed to generate images from multiple perspectives to obtain the first image data.

[0053] In this embodiment of the application, a rendering engine or renderer is used to synthesize a two-dimensional image from multiple different perspectives (such as front view, left view, right view, rear view, etc.) for the 3D model corresponding to each key physical state, and further generate first image data. The first image data contains a set of rendered images of all key physical states in multiple perspectives, and each physical state corresponds to a set of multi-view images.

[0054] For example, a rendering engine can be a software system that integrates scene management, lighting calculation, material processing, camera control, and rendering output, such as Unreal Engine, Unity, Maya, etc., or a dedicated engine that supports volumetric rendering / Gaussian rendering. It is important to note that a rendering engine refers to any engine, software, or tool capable of rendering 3D models, such as general-purpose rendering engines like Unreal Engine and Maya software; this application does not limit the specific rendering engine.

[0055] For example, each 3D model in the first model sequence is imported into the rendering engine, and lighting conditions are set to be basically consistent with the real physical environment. For each physical state model, a virtual camera is configured to generate images from specified multiple viewpoints. Two strategies can be used to obtain the rendered images: the first strategy is to fix the camera position and orientation to ensure that the largest model is fully imaged, while smaller models occupy a smaller proportion of the image; the second strategy is to fix the camera orientation but change the position to ensure that the model in each state occupies a basically consistent proportion of the rendered image. During rendering, the 3D model of each key physical state is visible individually, while the 3D models corresponding to other key physical states are hidden. The final first image data includes M viewpoints for each of the N key physical states.

[0056] In some embodiments, see Figure 5 , Figure 4 Step 403 shown can be implemented through steps 4031 to 4032, as explained in detail below.

[0057] In step 4031, rendered images of each 3D model from multiple perspectives are acquired using a virtual camera.

[0058] In step 4032, the rendered images from multiple perspectives of each 3D model are rendered and synthesized to obtain first image data, which includes the multi-view rendered images corresponding to each key physical state.

[0059] Here, a virtual camera refers to a virtual imaging unit within the rendering engine that simulates the intrinsic parameters (focal length, principal point) and extrinsic parameters (position, orientation) of a real camera, used to render images from any specified viewpoint. The process of acquiring multiple viewpoint rendering images of each 3D model can refer to the virtual camera image rendering process, which involves calculating and generating 2D images from the geometric models, materials, lighting, and other information in a 3D scene, simulating the imaging effect of a real camera. Based on the rendering image data acquired by the virtual camera, multi-view images are generated, including frontal, left, right, and rear view images under different key physical states. Note that when generating images under a specified key physical state, the rendering engine only presents the 3D model under that physical state; 3D models under other physical states are not visible. The final first image data consists of multi-view images corresponding to multiple key physical states of the target object, with each physical state having its corresponding multi-view image.

[0060] In some embodiments, before performing step 4031, the method further includes: setting a virtual camera based on a first strategy, wherein the spatial position and observation orientation of the virtual camera remain unchanged; or setting a virtual camera based on a second strategy, wherein the observation orientation of the virtual camera remains unchanged.

[0061] Furthermore, when the first strategy is used, the proportion of the target object in the multi-view rendered image under each key physical state is positively correlated with the size of the 3D model corresponding to each key physical state; when the second strategy is used, the proportion of the target object in the multi-view rendered image is the same under each key physical state.

[0062] For example, see Figure 6 The first strategy involves keeping the virtual camera's position and orientation fixed. The virtual camera's orientation ensures that all key physical state 3D models can be seen from the frontal view within the virtual camera. If the sizes of key physical states are inconsistent during the object's appearance evolution, the virtual camera's position must ensure that the final rendered image from that view can present the full picture of all key physical states. Taking the evolution of a flower seedling's appearance as an example, the frontal view image must fully present the full picture of the seedling's largest size (flower blooming) during its appearance evolution. The final rendering result is that the seedling occupies a smaller proportion in the rendered image when it sprouts, and a larger proportion when it grows into a blooming flower.

[0063] For example, see Figure 6The second strategy involves keeping the virtual camera's orientation fixed, but its position can change. The virtual camera's orientation ensures that it can see the images of all key physical state 3D models from a direct viewpoint. For 3D models with different key physical states, the virtual camera's position can be moved, but it must be ensured that the 3D models of different key physical states are roughly the same size in the final rendered image. Taking the evolution of a seedling's appearance as an example, when the seedling sprouts and the virtual camera is close to the 3D model, the sprouting part of the seedling occupies half of the rendered image. When the seedling grows into a flower and the virtual camera is farther away from the 3D model, the flowering part still occupies approximately half of the rendered image.

[0064] Thus, through controllable multi-view rendering, accurate multi-view images are generated for each physical state, and these images have consistent geometric realism and lighting consistency. Because the camera parameter scheme (fixed or variable position) can be flexibly selected, it can adapt to scenes with large variations in object size, providing high-quality, well-aligned visual constraints for subsequent multi-view video generation.

[0065] In step 404, first video data is generated based on the first image data. The first video data includes multi-view videos corresponding to multiple perspectives when the target object's appearance evolves.

[0066] In this embodiment, the first video data includes multi-view videos corresponding to multiple perspectives as the appearance of the target object evolves. Multi-view videos refer to multiple video streams synchronously generated by multiple virtual / real cameras with different perspectives. Each video corresponds to a fixed viewing perspective, collectively describing the changes in the appearance of the object at various times.

[0067] In some embodiments, for each viewpoint, the rendered images of different physical states under that viewpoint and their corresponding frame positions (manually specified, e.g., germination at frame 0, flowering at frame 30, and fruiting at frame 60) are input into the video generation model as constraints. The model generates a complete monocular video under that viewpoint, ensuring that the image at the specified frame position in the video is strictly consistent with the corresponding rendered image. The above process is repeated for all viewpoints to obtain a multi-view video (i.e., a collection of multiple monocular videos). To ensure multi-view consistency, the frame positions of the same physical state in videos from different viewpoints must remain consistent (e.g., flowering is at frame 30 in all viewpoint videos). The more key physical states there are, the higher the consistency of the multi-view appearance of the generated video at any given time. Thus, by embedding the rendered images of key physical states as hard constraints into the video generation process, not only is the physical rationality of object evolution in each viewpoint video guaranteed, but the consistency of appearance at corresponding times is also forced by the unified frame position, thereby solving the technical problem in the prior art where cross-viewpoint inconsistency in multi-view videos leads to distortion in subsequent 3D reconstruction.

[0068] In some embodiments, a video generation model generates first video data based on the first image data. For example, a diffusion probability model with an image as conditional input learns the inverse process of generating a target image by progressively denoising from noise, thus enabling the generation of new images or video frames based on the input image. See specifically... Figure 7 , Figure 4 Step 404 shown can be implemented through steps 4041 to 4042, as explained in detail below.

[0069] In step 4041, for the rendered images corresponding to multiple key physical states at each viewpoint, a monocular video is generated at each viewpoint using a graph-diffusion model. The monocular video is a video showing the appearance evolution of the target object at that viewpoint.

[0070] In step 4042, first video data is generated based on the video of the target object's appearance evolution from each viewpoint. The first video data includes multi-view videos of the target object's appearance evolution from multiple viewpoints. The rendered images of multiple physical states from each viewpoint have the same frame position in the first video data.

[0071] Here, the video generation model can be a graph-based diffusion model, a diffusion probability model that takes an image as a conditional input. It learns the inverse process of gradually denoising from noise to generate the target image, thus generating video based on the input image. Taking front-view video generation as an example, rendered images of different key physical states from the front view are used as constraints for the video generation model to generate a video of the object's appearance evolution from the front view. The specific process of video generation using the video generation model is as follows: the rendered images of different key physical states from the front view and their corresponding frame positions in the final generated video are input as constraints into the video generation model, where the frame positions can be manually specified. This ensures that the image at the corresponding frame position in the final generated video is the 3D rendered image of the corresponding physical state. Based on the same generation logic, multi-view videos from different perspectives are generated.

[0072] It's important to note that to ensure better consistency in the appearance of multi-view videos at any given moment, the frame positions of the rendered images of different key physical states at each viewpoint must remain consistent in the final generated video. In other words, the frame positions of the rendered images of the same physical state from different viewpoints must be consistent across all viewpoints. The more key physical states of the object being created, the higher the consistency in the appearance of any frame in the generated multi-view video.

[0073] In step 405, a second model sequence is generated based on the first video data. The second model sequence includes multiple 3D models of the target object as its appearance evolves.

[0074] In this embodiment of the application, the second model sequence is a collection of 3D models corresponding to each moment (or each frame) of the target object during the appearance evolution process, reflecting continuous three-dimensional dynamic changes.

[0075] In some embodiments, see Figure 8 , Figure 4 Step 405 shown can be implemented through steps 4051 to 4053, as explained in detail below.

[0076] In step 4051, the multi-view image corresponding to each frame in the first video data is obtained.

[0077] In step 4052, a 3D model corresponding to each frame is generated or reconstructed based on the multi-view image corresponding to each frame.

[0078] In step 4053, a second model sequence is generated based on the 3D model corresponding to each frame.

[0079] For example, for each frame in the first video data, extract all viewpoint images corresponding to that frame.

[0080] In some embodiments, after extracting all viewpoint images corresponding to the frame, the following steps are performed: The multi-viewpoint image coverage of the frame is determined. If there are only sparse viewpoints (e.g., only the front, left, right, and back directions, with low overlap between adjacent viewpoints), a 3D data generation model based on sparse viewpoints (e.g., conditional diffusion model or multi-viewpoint generative adversarial network) is invoked to generate a 3D model for the frame. If there are dense viewpoints (e.g., dozens of surrounding viewpoints with high overlap), a 3D reconstruction model based on dense viewpoints (e.g., neural radiation field, Gaussian splash, or traditional stereo matching) is invoked to reconstruct a high-precision 3D model. This operation is repeated for all frames to obtain a second model sequence.

[0081] Here, the 3D data generation / reconstruction model refers to an algorithm for generating or reconstructing 3D models based on multi-view images. For sparse viewpoints, generative model-based methods, such as single-image to 3D diffusion models, can be used; for dense viewpoints, reconstruction-based methods, such as neural radiation fields, Gaussian splashing, and multi-view stereo vision, can be used. Thus, the most suitable 3D generation or reconstruction method is adaptively selected based on the sparsity or density of available viewpoints, maximizing the accuracy of the 3D model while ensuring efficiency. Since step 404 ensures the consistency and physical plausibility of images from different viewpoints at the same moment in the multi-view video, the 3D model sequence generated in this step can accurately reflect the true 3D shape of the object at each moment, avoiding shape distortion or errors in physical facts.

[0082] In some embodiments, in response to the target object appearance evolution operation, the second model sequence is displayed in the renderer to present the appearance evolution of the target object.

[0083] Here, the second model sequence is displayed in a renderer to achieve the self-evolution of the object's appearance. For example, based on the data format of the 3D models in the second model sequence, the corresponding renderer (general-purpose graphics engine, volumetric renderer, or Gaussian renderer) is used to display the effects, thus presenting a continuous and complete evolution process of the object's three-dimensional appearance. Here, a renderer refers to a software or hardware module used to convert three-dimensional model data into a two-dimensional display image. Different data formats require different renderers; for example, a mesh uses a general-purpose graphics engine, a neural radiation field requires a renderer that supports volumetric rendering, and a Gaussian point cloud requires a renderer that supports Gaussian rendering.

[0084] It should be noted that different 3D data generation / reconstruction models employ different underlying technologies, resulting in different 3D model data formats and display methods. If the generated result uses a mesh representation, the second model sequence can be displayed in general graphics engines such as Unreal Engine, Unity, and Maya. If the generated result uses a neural network (based on neural radiation field technology), the second model sequence needs to be displayed in a renderer that supports volumetric rendering. If the generated result uses a Gaussian point cloud (based on Gaussian splashing technology), the second model sequence needs to be displayed in a renderer that supports Gaussian rendering.

[0085] The following description continues to illustrate the exemplary structure of the 3D modeling 433 provided in the embodiments of this application as a software module. In some embodiments, such as Figure 2 As shown, the software modules stored in the 3D modeling device 433 in the memory 430 may include: The data acquisition module 4331 is used to acquire the target object to be modeled in the monocular image and the physical state information of the target object; wherein, the physical state information represents multiple key physical states of the target object in the process of appearance evolution; the multiple key physical states represent one or more object states corresponding to changes in morphology or spatial attributes of the target object during its growth or movement cycle; and to acquire a first model sequence of the target object, the first model sequence including multiple 3D models that correspond one-to-one with the key physical states.

[0086] The data processing module 4332 is used to perform multi-view image generation processing on multiple 3D models to obtain first image data; based on the first image data, generate first video data, the first video data including multi-view videos corresponding to multiple perspectives when the target object's appearance evolves; based on the first video data, generate a second model sequence, the second model sequence including multiple 3D models of the target object when its appearance evolves.

[0087] In some embodiments, the data processing module 4332 is further configured to acquire rendered images of multiple perspectives of each 3D model through a virtual camera; perform rendering and compositing processing on the rendered images of multiple perspectives of each 3D model to obtain the first image data, wherein the first image data includes the multi-view rendered images corresponding to each key physical state.

[0088] In some embodiments, the data processing module 4332 is further configured to set the virtual camera based on a first strategy, wherein the spatial position and observation orientation of the virtual camera remain unchanged; or, to set the virtual camera based on a second strategy, wherein the observation orientation of the virtual camera remains unchanged.

[0089] In some embodiments, when the first strategy is used, the proportion of the target object in the multi-view rendered image under each key physical state is positively correlated with the size of the 3D model corresponding to each key physical state; when the second strategy is used, the proportion of the target object in the multi-view rendered image is the same under each key physical state.

[0090] In some embodiments, the data processing module 4332 is further configured to generate monocular videos for each viewpoint based on the rendered images corresponding to multiple key physical states at each viewpoint using a video generation model. The monocular videos are videos of the appearance evolution of the target object at that viewpoint. Based on the videos of the appearance evolution of the target object at each viewpoint, the first video data is generated. The first video data includes multi-view videos corresponding to multiple viewpoints when the target object's appearance evolves. The rendered images of multiple physical states at each viewpoint have the same frame position in the first video data.

[0091] In some embodiments, the data processing module 4332 is further configured to acquire multi-view images corresponding to each frame in the first video data; generate or reconstruct a 3D model corresponding to each frame based on the multi-view images corresponding to each frame; and generate the second model sequence based on the 3D model corresponding to each frame.

[0092] In some embodiments, the data processing module 4332 is further configured to display the second model sequence in a renderer in response to the target object appearance evolution operation, so as to present the appearance evolution of the target object.

[0093] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the three-dimensional modeling method described above in this application.

[0094] This application provides a computer-readable storage medium storing computer-executable instructions or a computer program. When the computer-executable instructions or the computer program are executed by a processor, the processor will execute the 3D modeling method provided in this application. For example, ... Figure 4 The three-dimensional modeling method is shown.

[0095] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of devices including one or any combination of the above-mentioned memories.

[0096] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0097] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a Hyper Text Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0098] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0099] In summary, through the embodiments of this application, for a target object in a monocular image, a 3D model of the key physical states during the object's appearance evolution is pre-obtained, and rendered images from multiple perspectives are generated based on the 3D model. Then, for each perspective, based on the rendered images of the 3D models of different key physical states and their corresponding frame positions in the final generated video, a video corresponding to that perspective is generated using a video generation model. The frame positions of the rendered images of the same physical state from different perspectives are consistent across all perspective videos. Images at the same frame positions are extracted from multiple perspective videos, and a 3D model corresponding to that frame is generated based on the 3D data generation model. Using the same logic, a 3D model corresponding to each frame is generated, forming a 3D model sequence. Thus, by using the multi-view images rendered from the object's key physical states, the consistency of the multi-view images in the multi-view video at the corresponding physical state can be constrained, minimizing the inconsistency problem caused by multi-view generation. The 3D models based on the key physical states serve as constraints, guiding the self-evolution of the 3D object's appearance, ensuring that the object's appearance changes as expected.

[0100] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A three-dimensional modeling method, characterized in that, The method includes: Acquire the target object to be modeled and its physical state information in a monocular image; wherein the physical state information represents multiple key physical states of the target object during its appearance evolution process; the multiple key physical states represent one or more object states corresponding to changes in the morphology or spatial attributes of the target object during its growth or movement cycle. Obtain a first model sequence of the target object, the first model sequence including multiple 3D models that correspond one-to-one with key physical states; Multiple 3D models are processed to generate images from multiple perspectives to obtain the first image data; Based on the first image data, first video data is generated, which includes multi-view videos of the target object as its appearance evolves from multiple perspectives. Based on the first video data, a second model sequence is generated, which includes multiple 3D models of the target object as its appearance evolves.

2. The method according to claim 1, characterized in that, The process of generating images from multiple 3D models from multiple perspectives to obtain first image data includes: Rendered images of each 3D model from multiple perspectives are obtained using a virtual camera; The rendered images from multiple perspectives of each 3D model are rendered and composited to obtain the first image data, which includes the multi-view rendered images corresponding to each key physical state.

3. The method according to claim 2, characterized in that, Before acquiring multiple viewpoint rendering images of each 3D model using a virtual camera, the method further includes: The virtual camera is set based on a first strategy, wherein the spatial position and viewing orientation of the virtual camera remain unchanged; or, the virtual camera is set based on a second strategy, wherein the viewing orientation of the virtual camera remains unchanged.

4. The method according to claim 3, characterized in that, When the first strategy is used, the proportion of the target object in the multi-view rendered image under each key physical state is positively correlated with the size of the 3D model corresponding to each key physical state. When the second strategy is used, the target object has the same screen proportion in the multi-view rendered image for each key physical state.

5. The method according to claim 2, characterized in that, The step of generating first video data based on the first image data includes: For the rendered images corresponding to multiple key physical states at each viewpoint, a video generation model is used to generate monocular videos for each viewpoint. The monocular videos are videos of the appearance evolution of the target object at that viewpoint. The first video data is generated based on the video of the appearance evolution of the target object from each viewpoint. The first video data includes multi-view videos of the target object from multiple viewpoints during the appearance evolution. In this context, the rendered images of multiple physical states from each viewpoint are at the same frame position in the first video data.

6. The method according to claim 1, characterized in that, The step of generating a second model sequence based on the first video data includes: Obtain the multi-view image corresponding to each frame in the first video data; Based on the multi-view images corresponding to each frame, generate or reconstruct the 3D model corresponding to each frame. The second model sequence is generated based on the 3D model corresponding to each frame.

7. The method according to claim 1, characterized in that, The method further includes: In response to the target object appearance evolution operation, the second model sequence is displayed in the renderer to present the appearance evolution of the target object.

8. An electronic device, characterized in that, include: The processor and the memory used to store computer programs that can run on the processor. When the processor is used to run the computer program, it performs the steps of the method according to any one of claims 1 to 7.

9. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 7.

Images