Three-dimensional reconstruction method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a three-dimensional reconstruction method and device, electronic equipment, a storage medium and a program product. The method comprises the steps of generating a first image set of a target object based on a to-be-processed image including the target object; the first image set comprises a plurality of first images, and the viewing angles of the target objects in different first images are different; based on the first image set, performing visual angle interpolation on the at least one first image to obtain a second image set comprising at least one second image, the visual angle of the target object in the second image being different from the visual angle of the target object in the first image set; and performing three-dimensional reconstruction on the target object based on the first image set and the second image set to obtain a three-dimensional model of the target object. According to the invention, the visual quality of three-dimensional reconstruction can be effectively improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a three-dimensional reconstruction method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] 3D object reconstruction is a hot research topic in computer vision and computer graphics. It involves extracting information from a set of images to infer the geometric model of an object in three-dimensional space. 3D object reconstruction has wide applications in many fields, including virtual reality, augmented reality, computer-aided design, medical image analysis, and robot navigation. Through 3D reconstruction technology, the real world can be digitally replicated, providing rich 3D model resources for various application scenarios.

[0003] In related technologies, the reconstruction of three-dimensional objects is usually based directly on a single image containing two-dimensional objects. By making assumptions, the parts of the two-dimensional objects that are not visible in the image are inferred to obtain multi-view images of the two-dimensional objects. Due to the uncertainty of the inference assumptions, there are viewpoint jumps in the multi-view images, which in turn leads to geometric deformation of the three-dimensional objects reconstructed based on the multi-view images, resulting in poor visual quality of the three-dimensional reconstruction. Summary of the Invention

[0004] This application provides a three-dimensional reconstruction method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the visual quality of three-dimensional reconstruction.

[0005] The technical solution of this application embodiment is implemented as follows:

[0006] This application provides a three-dimensional reconstruction method, including:

[0007] Based on the image to be processed, which includes the target object, a first image set of the target object is generated;

[0008] The first image set includes multiple first images, and the target object in different first images has a different perspective;

[0009] Based on the first image set, at least one of the first images is interpolated by perspective to obtain a second image set containing at least one second image, wherein the perspective of the target object in the second image is different from the perspective of the target object in the first image set;

[0010] Based on the first image set and the second image set, the target object is reconstructed in three dimensions to obtain a three-dimensional model of the target object.

[0011] This application provides a three-dimensional reconstruction device, including:

[0012] The generation module is used to generate a first image set of the target object based on the image to be processed, which includes the target object; wherein the first image set includes multiple first images, and the target object has a different perspective in the different first images;

[0013] A perspective interpolation module is used to perform perspective interpolation on at least one of the first images based on the first image set to obtain a second image set containing at least one second image, wherein the perspective of the target object in the second image is different from the perspective of the target object in the first image set;

[0014] The 3D reconstruction module is used to perform 3D reconstruction of the target object based on the first image set and the second image set to obtain a 3D model of the target object.

[0015] In the above scheme, the above generation module is further used to extract features from the image to be processed to obtain image features of the image to be processed; call the view prediction model to predict the target object based on the image features to obtain an initial image set of the target object, the initial image set including multiple first images; and determine the first image set based on the initial image set.

[0016] In the above scheme, the generation module is further configured to obtain the number of first images in the initial image set; when the number is greater than the target number, select the target number of first images from the initial image set to obtain the first image set; when the number is less than or equal to the target number, determine the initial image set as the first image set.

[0017] In the above scheme, the viewpoint interpolation module is further configured to select at least one first image from the first image set as an image to be interpolated; for each image to be interpolated, viewpoint interpolation is performed on the image to be interpolated based on a reference image to obtain a second image corresponding to the image to be interpolated; wherein, the reference image is a first image in the first image set other than the image to be interpolated; and the second images corresponding to each image to be interpolated are used to construct a second image set.

[0018] In the above scheme, the viewpoint interpolation module is further configured to obtain the first camera pose of the target object in the image to be interpolated, and perform the following processing on each of the reference images of the image to be interpolated: based on the first camera pose and the reference image, perform viewpoint interpolation on the image to be interpolated to obtain a third image corresponding to the image to be interpolated; obtain the second camera pose of the target object in the reference image, and based on the second camera pose and the reference image, perform viewpoint interpolation on the image to be interpolated to obtain a fourth image corresponding to the image to be interpolated; and determine the third image and the fourth image as the second image corresponding to the image to be interpolated.

[0019] In the above scheme, the viewpoint interpolation module is further used to extract features from the reference image to obtain reference image features, and to extract features from the image to be interpolated to obtain image features to be interpolated; to compress the reference image features to obtain compressed image features of the reference image features; and to perform viewpoint interpolation on the image to be interpolated based on the image features to be interpolated, the compressed image features of the reference image features, and the first camera pose to obtain the third image; wherein, the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the image to be interpolated is a first difference, and the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the reference image is a second difference, and the first difference is less than the second difference.

[0020] In the above scheme, the viewpoint interpolation module is further used to extract features from the reference image to obtain reference image features, and to extract features from the image to be interpolated to obtain image features to be interpolated; to compress the image features to be interpolated to obtain compressed image features of the image features to be interpolated; and to perform viewpoint interpolation on the image to be interpolated based on the reference image features, the compressed image features of the image features to be interpolated, and the second camera pose to obtain the fourth image; wherein, the difference between the viewpoint of the target object in the fourth image and the viewpoint of the target object in the image to be interpolated is the third difference, the difference between the viewpoint of the target object in the fourth image and the viewpoint of the target object in the reference image is the fourth difference, and the third difference is greater than the fourth difference.

[0021] In the above scheme, the viewpoint interpolation module is further used to obtain the first camera pose of the target object in the image to be interpolated, and the second camera pose of the target object in the reference image; to determine the average value of the first camera pose and the second camera pose as the third camera pose; and to perform viewpoint interpolation on the image to be interpolated based on the third camera pose and the reference image to obtain the second image corresponding to the image to be interpolated.

[0022] In the above scheme, the viewpoint interpolation module is further used to call the feature extraction layer of the viewpoint interpolation model to extract features from the reference image and the image to be interpolated, respectively, to obtain reference image features and image to be interpolated features; call the feature compression layer of the viewpoint interpolation model to compress the reference image features to obtain compressed image features corresponding to the reference image features; and call the viewpoint interpolation layer of the viewpoint interpolation model to perform viewpoint interpolation on the image to be interpolated based on the pose of the third camera, the compressed image features corresponding to the reference image features, and the image to be interpolated features, to obtain the second image.

[0023] In the above scheme, the viewpoint interpolation is achieved through a viewpoint interpolation model. The above-mentioned three-dimensional reconstruction device further includes: a training module, which is used to call the initial viewpoint interpolation model, perform viewpoint interpolation on image samples carrying image labels to obtain an interpolated image; and train the initial viewpoint interpolation model based on the interpolated image and the image labels to obtain the viewpoint interpolation model.

[0024] In the above scheme, the above-mentioned three-dimensional reconstruction module is further used to extract features from the first image set to obtain a first set feature of the first image set, and to extract features from the second image set to obtain a second set feature of the second image set; to fuse the first set feature and the second set feature to obtain a fused feature; and to perform three-dimensional reconstruction on the target object based on the fused feature to obtain a three-dimensional model of the target object.

[0025] In the above scheme, the aforementioned 3D reconstruction is achieved through a 3D reconstruction model. The training module is further used to acquire fused feature samples carrying image labels of the sample object, and call the initial 3D reconstruction model. Based on the fused feature samples, the sample object is reconstructed in 3D to obtain a predicted 3D model of the sample object. The predicted 3D model of the sample object is image acquired to obtain an acquired image of the predicted 3D model. Based on the acquired image and the image labels, the initial 3D reconstruction model is trained to obtain the 3D reconstruction model.

[0026] In the above scheme, the fused feature samples carry image labels of multiple image types of the sample object, and the acquired images correspond one-to-one with the image labels. The training module is also used to determine the similarity between the image label and the corresponding acquired image for each image label; determine the loss value of the initial three-dimensional reconstruction model based on each similarity; and train the initial three-dimensional reconstruction model based on the loss value to obtain the three-dimensional reconstruction model.

[0027] In the above scheme, the above-mentioned three-dimensional reconstruction device further includes: a virtual scene module, used to display the three-dimensional model in an initial virtual scene, and in response to an editing operation on the three-dimensional model, edit the three-dimensional model to obtain an edited three-dimensional model; and update the three-dimensional model in the initial virtual scene to the edited three-dimensional model to obtain an updated virtual scene.

[0028] This application provides an electronic device, including:

[0029] Memory is used to store executable instructions or computer programs.

[0030] The processor, when executing computer-executable instructions or computer programs stored in the memory, implements the three-dimensional reconstruction method provided in the embodiments of this application.

[0031] This application provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the three-dimensional reconstruction method provided in this application.

[0032] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the three-dimensional reconstruction method described above in this application.

[0033] The embodiments of this application have the following beneficial effects:

[0034] By generating a first image set of the target object based on the images to be processed, including the target object, and then performing viewpoint interpolation on at least one of the first images to obtain a second image set containing at least one second image, the target object is reconstructed in 3D based on the first and second image sets to obtain a 3D model of the target object. Thus, by generating a first image set of the target object based on the images to be processed, where different first images show different viewpoints of the target object, and by performing viewpoint interpolation on at least one of the first images to obtain a second image set containing at least one second image, the target object is created using the first image set as a transitional viewpoint between different first images. This second image provides a smooth transition between different viewpoints of the target object between the different first images. By performing 3D reconstruction on the target object based on the first and second image sets, a 3D model of the target object is obtained. The combined participation of the first and second image sets effectively reduces abrupt changes in viewpoint, thereby effectively reducing geometric deformation and singularities in the 3D model of the target object obtained from viewpoint jumps, and thus effectively improving the visual quality of the 3D reconstruction. Attached Figure Description

[0035] Figure 1 This is a schematic diagram of the architecture of the three-dimensional reconstruction system provided in the embodiments of this application;

[0036] Figure 2 This is a schematic diagram of the structure of an electronic device for three-dimensional reconstruction provided in an embodiment of this application;

[0037] Figure 3 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 1 ;

[0038] Figure 4 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 2 ;

[0039] Figure 5 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 3 ;

[0040] Figure 6 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 4 ;

[0041] Figure 7 This is a schematic diagram illustrating the effect of three-dimensional object reconstruction provided in an embodiment of this application;

[0042] Figure 8This is a schematic diagram illustrating the principle of the three-dimensional reconstruction method provided in the embodiments of this application;

[0043] Figure 9 This is a schematic diagram illustrating the principle of perspective interpolation provided in the embodiments of this application;

[0044] Figure 10 This is a schematic diagram of the view prediction model provided in the embodiments of this application. Detailed Implementation

[0045] To make the objectives, technical solutions, and advantages of this application clearer, the application will be further described in detail below with reference to the accompanying drawings. The described embodiments should not be regarded as limitations on this application. All other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0046] In the following description, references are made to “some embodiments,” which describe a subset of all possible embodiments. However, it is understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0047] In the following description, the terms "first, second, third" are used merely to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first, second, third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein.

[0048] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0049] Before providing a further detailed description of the embodiments of this application, the nouns and terms involved in the embodiments of this application will be explained, and the nouns and terms involved in the embodiments of this application shall be interpreted as follows.

[0050] 1) Tri-plane Representation: This is a method used in computer vision and graphics to describe the structure of 3D objects. Tri-plane representation refers to a technique that represents a 3D object as a set of images of three mutually perpendicular 2D planes. These three planes are typically the object's horizontal plane (XY plane), vertical plane (XZ plane), and side plane (YZ plane). Each plane is a 2D slice of the object from a specific angle, capturing structural information about the object in that direction. By capturing information on each plane separately, tri-plane representation breaks down complex 3D information into more manageable 2D information. The three planes are mutually perpendicular, ensuring that the object is covered from three different directions, minimizing information loss. The information provided by each plane is complementary to the other two planes; combining the information from these three planes allows for the reconstruction or approximation of the complete 3D shape.

[0051] 2) Two-dimensional objects: A two-dimensional object is an object with only two spatial dimensions—length and width—and no thickness. In mathematics and computer graphics, two-dimensional objects are typically composed of basic geometric elements such as points, line segments, and curves, which define the object's shape and outline on a two-dimensional plane. In the real world, when we observe a two-dimensional object, we are actually seeing a projection of that object into three-dimensional space. In computer vision, processing two-dimensional objects often involves image analysis techniques such as edge detection, shape recognition, and image segmentation.

[0052] 3) Three-dimensional objects: Three-dimensional objects are objects with three spatial dimensions: length, width, and height. These objects exist in the real world and have a three-dimensional structure and volume. In computer graphics, three-dimensional objects can be represented by various mathematical models, such as polygonal meshes, voxel representations, or point clouds. Computer models of three-dimensional objects can be used in fields such as rendering, animation, virtual reality, and 3D printing.

[0053] 4) Viewpoint: Viewpoint refers to the angle of observation when viewing an object or scene from a specific position and direction. In image processing and computer vision, viewpoint defines the relative positional relationship between the camera or observer and the observed object, determining the appearance and presentation of the object in the image. Different viewpoints refer to the different viewing angles from which a two-dimensional object is presented in an image. For example, an object can be viewed from the front, side, or top, each angle producing a different viewpoint image.

[0054] During the implementation of the embodiments of this application, the applicant discovered the following problems with the related technology:

[0055] In related technologies, the reconstruction of three-dimensional objects is usually based directly on a single image containing two-dimensional objects. By making assumptions, the parts of the two-dimensional objects that are not visible in the image are inferred to obtain multi-view images of the two-dimensional objects. Due to the uncertainty of the inference assumptions, there are view jumps in the multi-view images, which in turn leads to geometric deformation of the three-dimensional objects reconstructed from the multi-view images, resulting in poor visual quality of the reconstructed three-dimensional objects.

[0056] This application provides a three-dimensional reconstruction method, apparatus, electronic device, computer-readable storage medium, and computer program product, which can effectively improve the visual quality of three-dimensional reconstruction. The exemplary application of the three-dimensional reconstruction system provided in this application is described below.

[0057] See Figure 1 , Figure 1 This is a schematic diagram of the architecture of the three-dimensional reconstruction system 100 provided in the embodiments of this application. The terminal (terminal 400 is shown as an example) is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0058] Terminal 400 is used by a user to access client 410 and display a 3D model on a graphical interface 410-1 (graphical interface 410-1 is shown as an example). Terminal 400 and server 200 are interconnected via wired or wireless network.

[0059] In some embodiments, server 200 can be a standalone physical server, a server cluster or business system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Terminal 400 can be a smartphone, tablet, laptop, desktop computer, smart speaker, smart TV, smartwatch, in-vehicle terminal, etc., but is not limited to these. The electronic device provided in this application embodiment can be implemented as a terminal or a server. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited in this application embodiment.

[0060] In some embodiments, server 200 generates a first image set of the target object based on the image to be processed including the target object, performs viewpoint interpolation on at least one first image based on the first image set to obtain a second image set including at least one second image, and sends the first image set and the second image set to terminal 400. Terminal 400 performs three-dimensional reconstruction of the target object based on the first image set and the second image set to obtain a three-dimensional model of the target object.

[0061] In other embodiments, the terminal 400 generates a first image set of the target object based on the image to be processed, including the target object, and performs viewpoint interpolation on at least one of the first images based on the first image set to obtain a second image set containing at least one second image. The terminal 400 then sends the first image set and the second image set to the server 200. The server 200 performs three-dimensional reconstruction of the target object based on the first image set and the second image set to obtain a three-dimensional model of the target object.

[0062] See Figure 2 , Figure 2 This is a schematic diagram of the structure of an electronic device 500 for three-dimensional reconstruction provided in an embodiment of this application, wherein, Figure 2 The electronic device 500 shown can be Figure 1 Server 200 or terminal 400 in the middle, Figure 2 The illustrated electronic device 500 includes at least one processor 430, a memory 450, and at least one network interface 420. The various components in the electronic device 500 are coupled together via a bus system 440. It is understood that the bus system 440 is used to implement communication between these components. In addition to a data bus, the bus system 440 also includes a power bus, a control bus, and a status signal bus. However, for clarity, ... Figure 2 The general labeled all buses as Bus System 440.

[0063] Processor 430 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0064] The memory 450 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state storage, hard disk drives, optical disk drives, etc. The memory 450 may optionally include one or more storage devices physically located away from the processor 430.

[0065] The memory 450 may include volatile memory or non-volatile memory, or both. The non-volatile memory may be read-only memory (ROM), and the volatile memory may be random access memory (RAM). The memory 450 described in this application embodiment is intended to include any suitable type of memory.

[0066] In some embodiments, memory 450 is capable of storing data to support various operations, examples of which include programs, modules, and data structures or subsets or supersets thereof, as illustrated below.

[0067] Operating system 451 includes system programs for handling various basic system services and performing hardware-related tasks, such as the framework layer, core library layer, driver layer, etc., for implementing various basic business functions and handling hardware-based tasks;

[0068] The network communication module 452 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 420, such as Bluetooth, WiFi, and Universal Serial Bus (USB).

[0069] In some embodiments, the three-dimensional reconstruction apparatus provided in this application can be implemented in software. Figure 2 A 3D reconstruction apparatus 455 stored in memory 450 is shown. This apparatus can be software in the form of programs and plugins, including the following software modules: a generation module 4551, a viewpoint interpolation module 4552, and a 3D reconstruction module 4553. These modules are logically linked and can therefore be arbitrarily combined or further separated according to their implemented functions. The functions of each module will be described below.

[0070] In other embodiments, the three-dimensional reconstruction device provided in this application can be implemented in hardware. As an example, the three-dimensional reconstruction device provided in this application can be a processor in the form of a hardware decoding processor, which is programmed to execute the three-dimensional reconstruction method provided in this application. For example, the processor in the form of a hardware decoding processor can be one or more application-specific integrated circuits (ASICs), DSPs, programmable logic devices (PLDs), complex programmable logic devices (CPLDs), field-programmable gate arrays (FPGAs), or other electronic components.

[0071] In some embodiments, the terminal or server can implement the 3D reconstruction method provided in this application by running a computer program or computer-executable instructions. For example, the computer program can be a native program in the operating system (e.g., a dedicated 3D reconstruction program) or a software module, such as a 3D reconstruction module that can be embedded in any program (e.g., an instant messaging client, a photo album program, an electronic map client, a navigation client); or it can be a native application (APP), i.e., a program that needs to be installed in the operating system to run. In summary, the above-mentioned computer program can be any form of application, module, or plugin.

[0072] The three-dimensional reconstruction method provided in this application will be described in conjunction with exemplary applications and implementations of the server or terminal provided in the embodiments of this application.

[0073] See Figure 3 , Figure 3 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 1 , will combine Figure 3 Steps 101 to 103 are described below. The three-dimensional reconstruction method provided in this application embodiment can be implemented by the server or terminal alone, or by the server and terminal working together. The following description will take the implementation by the server alone as an example.

[0074] In step 101, a first image set of the target object is generated based on the image to be processed, including the target object.

[0075] In some embodiments, the first image set includes multiple first images, the target object in different first images has a different perspective, and the perspective of the target object in each first image in the first image set is different from the perspective of the target object in the image to be processed.

[0076] In some embodiments, the target object is the object in the image to be processed that needs to be reconstructed in 3D, which may be a person, object, scene, etc. The image to be processed is the original image that needs to be reconstructed in 3D, and the first image set is a collection of a series of first images generated based on the image to be processed. Each first image is a single image in the first image set, and the target object has a different perspective in each of these images.

[0077] In some embodiments, perspective refers to the position and orientation of an observer or camera relative to a target object. In the context of image processing and computer vision, perspective describes the projection of a target object onto a two-dimensional image plane, which is determined by the geometric position and orientation of the camera or scanning device relative to the object in three-dimensional space.

[0078] In some embodiments, in each first image of the first image set, the target object is presented from a different orientation or angle. These orientations or angles differ from the orientation or angle at which the camera captured the image to be processed. The viewpoint of the target object in each first image differs from the viewpoint of the target object in the image to be processed: in each first image, the viewpoint of the target object differs from the viewpoint in the original image to be processed, thus providing a different angle view of the target object.

[0079] As an example, the viewpoint of the target object in the image to be processed is viewpoint A1. Based on the image to be processed including the target object, the first image set of the target object generated includes first image B1, first image B2 and first image B3. The viewpoint of the target object in first image B1 is viewpoint A2, the viewpoint of the target object in first image B2 is viewpoint A3, and the viewpoint of the target object in first image B3 is viewpoint A4. Viewpoints A1, A2, A3 and A4 are all different.

[0080] In some embodiments, see Figure 4 , Figure 4 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 2 , Figure 3 Step 101 shown can be achieved through Figure 4 Steps 1011 to 1013 shown are implemented.

[0081] In step 1011, feature extraction is performed on the image to be processed to obtain the image features of the image to be processed.

[0082] In some embodiments, feature extraction is the process of identifying and extracting features that are important for solving a problem from raw data, i.e., the image to be processed. In image processing, feature extraction refers to using algorithms to select and compute unique information from an image that represents its essential attributes. This information can be colors, shapes, textures, edges, corners, etc., in the image. The purpose of feature extraction is to reduce the dimensionality of the data while retaining information that is crucial for tasks such as image recognition, classification, and detection, so that subsequent image analysis and processing are more efficient and accurate.

[0083] In some embodiments, image features refer to unique attributes or patterns in an image that can represent its content. Image features can be local information of the image, such as edges, corners, and textures, or global information, such as color histograms and the overall shape of the image. The image features of the image to be processed are vector representations of the image.

[0084] In step 1012, the view prediction model is invoked to predict the target object based on image features, thereby obtaining an initial set of images of the target object.

[0085] In some embodiments, the initial image set includes multiple first images, the target object has a different perspective in the different first images, and the perspective of the target object in each of the first images in the initial image set is different from the perspective of the target object in the image to be processed.

[0086] In some embodiments, the viewpoint prediction model is a machine learning model trained to predict different viewpoints or orientations of a target object. For example, the viewpoint prediction model can be obtained by training a diffusion model, which may include an encoding layer and a viewpoint prediction layer. A diffusion model is a generative model that generates data, such as images and audio, by simulating the diffusion and denoising processes of data distribution. A diffusion model is a deep learning framework that includes two main processes: diffusion and denoising. In computer vision, diffusion models are commonly used to generate high-quality images.

[0087] As an example, see Figure 10 , Figure 10 This is a schematic diagram of the view prediction model provided in the embodiments of this application. Figure 10The view prediction model shown includes an encoding layer 1 and a view prediction layer 2. The above-mentioned view prediction model is called to predict the target object based on image features and obtain the initial image set of the target object. This can be achieved in the following way: the encoding layer 1 of the view prediction model is called to encode the image features and obtain the encoding result; the view prediction layer 2 of the view prediction model is called to predict the target object based on the encoding result and obtain the initial image set of the target object.

[0088] As an example, in the application scenario of 3D modeling and rendering, the outline and key features of an object are determined by feature extraction from a 2D image of the object. Then, the view prediction model generates 2D images of the object from different viewpoints. These images are then used to generate a 3D model of the object, which can be subjected to various rendering processes.

[0089] As an example, in multimedia editing and feature creation applications, a multi-angle video frame of a target object is uploaded, the image features of the key frames are extracted, and a view prediction model is used to generate other view images of the object. These images are then used to create complex visual effects, such as rotation, scaling, or object replacement.

[0090] As an example, in robot navigation and perception applications, a robot uses a camera to capture an image containing a target obstacle. Through feature extraction and view prediction models, the robot can predict different viewpoints of the obstacle and thus plan a path to avoid it.

[0091] In step 1013, a first image set is determined based on the initial image set.

[0092] In some embodiments, the above-mentioned determination of the first image set based on the initial image set can be achieved in the following manner: obtaining the number of first images in the initial image set; when the number is greater than the target number, selecting the target number of first images from the initial image set to obtain the first image set; when the number is less than or equal to the target number, determining the initial image set as the first image set.

[0093] In some embodiments, the target number may be the desired number of first images in the first image set, and the target number may be preset according to actual conditions.

[0094] In some embodiments, the more first images in the first image set, the more pronounced the viewpoint fragmentation becomes. When the first image set contains a large number of first images generated from different viewpoints, these first images may have significant differences between viewpoints, leading to viewpoint fragmentation. This can affect the overall or visual coherence of the target object. If there are many predictions, a subset may be selected. If the viewpoint prediction model generates many images, it may only need to select the most representative or suitable subset to avoid the clutter or computational burden caused by too many images. If there are few predictions, all predictions may be used as the first image set. If the model generates a small number of images, all predictions may be used, as even a small number of images can provide different viewpoint information about the target object.

[0095] As an example, obtain the number of first images in the initial image set, for example, 10 images; when the number is greater than the target number (for example, 4 images), select the target number (4 images) of first images from the initial image set, and determine the 4 selected first images as the first image set.

[0096] Thus, when the number of images exceeds the target number, carefully selecting the first set of images within the target number avoids resource waste, reduces storage and computation costs, and ensures the quality and relevance of the image set. When the number of images is insufficient, all predicted images are directly used as the first image set, ensuring information integrity and enabling subsequent applications to make decisions or analyses based on the most comprehensive possible viewpoint data. Furthermore, controlling the size of the first image set helps reduce the sense of fragmentation between viewpoints, avoiding visual clutter caused by too many viewpoints, thereby improving user experience and visual presentation. This quantity control strategy not only improves efficiency and performance but also enhances the practicality and usability of the image set in various application scenarios.

[0097] In step 102, based on the first image set, at least one first image is interpolated by perspective to obtain a second image set containing at least one second image.

[0098] In some embodiments, the perspective of the target object in the second image is different from the perspective of the target object in the first image set.

[0099] In some embodiments, viewpoint interpolation refers to the algorithmic inference of new images from an existing image set (such as a first image set) based on the viewpoint information of at least one image. These new images depict the appearance of a target object from other viewpoints between known viewpoints. Viewpoint interpolation generates additional viewpoints in 3D modeling to create more complete 3D models, provides smoother viewpoint transitions in virtual reality (VR), or is used in image editing to create new viewpoints to enhance visual effects. Through viewpoint interpolation, new viewpoints can be synthesized without actually capturing or generating multi-angle images, providing users with a richer visual experience.

[0100] As an example, in a game application scenario, suppose in a role-playing game, the player-controlled character stands in front of a castle. The game already provides several static images of the castle from different angles (the first image set). When the player surrounds themselves and tries to observe the castle from different angles, the game engine can use perspective interpolation technology to generate new perspectives based on the existing castle images (the second image set). In this way, even if the player does not see the castle directly from a specific angle, they can still appreciate different aspects of the castle through the new images (the second images).

[0101] As an example, in an augmented reality (AR) application, an AR app allows users to place virtual objects in the real world. The user sees the virtual object placed on a table through their phone screen, but can only observe it from a limited perspective. Using multiple perspectives of the object in a first image set, the application can perform perspective interpolation to generate new perspective images (a second image set), allowing the user to view the virtual object from even more angles, just like a real object, even though these perspectives are not directly captured when the user is using their mobile device.

[0102] As an example, in virtual reality (VR) applications, in VR games, players might interact with a complex 3D model, such as a dinosaur, within a virtual environment. While the dinosaur's 3D model is created with information from multiple perspectives, in certain situations, it may be necessary to display a new perspective of the model. Through perspective interpolation, the VR system can generate new perspective images (a second set of images) based on the dinosaur's existing perspective images (a first set of images). This allows players to see more natural and continuous changes in perspective as they move around the dinosaur, enhancing immersion and realism.

[0103] In some embodiments, see Figure 5 , Figure 5 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 3 , Figure 3 Step 102 shown can be achieved through Figure 5 Steps 1021 to 1023 shown are implemented.

[0104] In step 1021, at least one first image is selected from the first image set as the image to be interpolated.

[0105] In some embodiments, step 1021 above can be implemented in the following way: determining each first image in the first image set as the image to be interpolated, or taking any one first image in the first image set as the image to be interpolated.

[0106] As an example, suppose there is a first set of images containing the following four images: image A, image B, image C, and image D. All images in the set are selected as the images to be interpolated, meaning the interpolated images will include images A, B, C, and D. Then, any one image is randomly selected from the set as the interpolated image; which image is chosen depends on the specific requirements or scenario.

[0107] In step 1022, for each image to be interpolated, based on the reference image of the image to be interpolated, viewpoint interpolation is performed on the image to be interpolated to obtain the second image corresponding to the image to be interpolated.

[0108] In some embodiments, the reference image is a first image in the first image set other than the image to be interpolated.

[0109] In some embodiments, before performing viewpoint interpolation on the image to be interpolated based on a reference image to obtain a second image corresponding to the image to be interpolated, a reference image may be selected from the first images in the first image set other than the image to be interpolated.

[0110] As an example, the first image set includes first image A, first image B and first image C, and the image to be interpolated is first image A. Then the reference image of the image to be interpolated can be at least one of first image B and first image C.

[0111] As an example, the first image set includes: first image A, first image B, first image C, and first image D. Taking the image to be interpolated as first image A, the reference images for the image to be interpolated (first image A) can be: first image B, first image C, and first image D. Taking the image to be interpolated as first image B, the reference images for the image to be interpolated (first image B) can be: first image C and first image D. Since the reference image for the image to be interpolated (first image A) is first image B, the reference image for the image to be interpolated (first image B) can be excluded, thereby avoiding redundant calculations and effectively improving computational efficiency.

[0112] In some embodiments, step 1022 above can be implemented as follows: obtain the first camera pose of the target object in the image to be interpolated, and perform the following processing on each reference image of the image to be interpolated: perform viewpoint interpolation on the image to be interpolated based on the first camera pose and the reference image to obtain the third image corresponding to the image to be interpolated; obtain the second camera pose of the target object in the reference image, and perform viewpoint interpolation on the image to be interpolated based on the second camera pose and the reference image to obtain the fourth image corresponding to the image to be interpolated; determine the third image and the fourth image as the second image corresponding to the image to be interpolated.

[0113] In some embodiments, the first camera pose is a set of parameters describing the spatial position and orientation of the target object in the image to be interpolated. The first camera pose includes the position and orientation of the target object in the image to be interpolated. The position of the target object in the image to be interpolated represents its position in three-dimensional space, typically represented by three coordinates (x, y, z), indicating the horizontal, vertical, and longitudinal positions of the target object relative to the camera coordinate system. The orientation of the target object in the image to be interpolated represents its orientation or direction, typically described by a rotation vector or rotation matrix, indicating the rotation of the target object relative to the camera coordinate system.

[0114] In some embodiments, the second camera pose of the target object in the reference image is a set of parameters describing the spatial position and orientation of the target object in the reference image. The second camera pose includes the position and orientation of the target object in the reference image. The position of the target object in the reference image represents its position in three-dimensional space, typically represented by three coordinates (x, y, z), indicating the horizontal, vertical, and longitudinal positions of the target object relative to the camera coordinate system. The orientation of the target object in the reference image represents its orientation or direction, typically described by a rotation vector or rotation matrix, indicating the rotation of the target object relative to the camera coordinate system.

[0115] As an example, in a 3D game development application scenario, the scene needs to be dynamically rendered based on the player's perspective. The first camera pose is assumed to be at coordinates (10, 5, 10), facing north. The third image is obtained by interpolating the player's first camera pose with a reference image from a southern perspective, resulting in an intermediate image (the third image) viewed from the player's perspective looking south. The second pose is obtained from the reference image from the southern perspective, for example, (20, 5, 10), facing north. The fourth image is obtained by interpolating the second camera pose from the southern perspective with the reference image, resulting in an intermediate image (the fourth image) viewed from the southern perspective looking towards the player's perspective. Both the third and fourth images are then used as the second image.

[0116] As an example, in the application scenario of multi-view reconstruction in medical imaging, doctors need to observe the patient's 3D scan data from multiple angles. First camera pose: Assuming the current view pose is (0, 0, 0), facing forward of the patient. Third image: Using the first camera pose and a side reference image, viewpoint interpolation is performed to obtain a side view image (third image). Second camera pose: The camera pose is obtained from the side reference image, for example, (0, 5, 0), facing forward of the patient. Fourth image: Using the side second camera pose and the reference image, viewpoint interpolation is performed to obtain a front-to-back view image (fourth image). Using both the third and fourth images as the second image helps doctors observe the patient's internal structures more comprehensively.

[0117] Thus, by performing viewpoint interpolation on the image to be interpolated based on the first camera pose, the resulting third image can more accurately reflect the appearance and position of the target object in the image to be interpolated, improving the realism and visual effect of the image. By obtaining the second camera pose of the target object in the reference image, and performing viewpoint interpolation on the image to be interpolated to obtain the fourth image, the third and fourth images are determined as the second images corresponding to the image to be interpolated. Multiple viewpoint interpolations are then performed using the image to be interpolated and the reference image to obtain the third and fourth images, effectively improving the interpolation effect of viewpoint interpolation, enhancing image quality, and providing a more reliable foundation for subsequent image processing and analysis.

[0118] In some embodiments, the above-mentioned method of performing viewpoint interpolation on the image to be interpolated based on the first camera pose and the reference image to obtain a third image corresponding to the image to be interpolated can be implemented in the following way: extracting features from the reference image to obtain reference image features, and extracting features from the image to be interpolated to obtain features of the image to be interpolated; compressing the features of the reference image to obtain compressed image features of the reference image features; and performing viewpoint interpolation on the image to be interpolated based on the features of the image to be interpolated, the compressed image features of the reference image features, and the first camera pose to obtain a third image.

[0119] In some embodiments, the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the image to be interpolated is a first difference, and the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the reference image is a second difference, wherein the first difference is less than the second difference.

[0120] In some embodiments, feature compression refers to the process of reducing the dimensionality or size of features through mathematical transformations or algorithms after feature extraction. This step is typically used to reduce computational complexity, improve processing speed, and reduce storage requirements. Feature compression can be implemented in various ways, such as principal component analysis (PCA), singular value decomposition (SVD), or autoencoders. When processing image features, feature compression may remove some redundant information while striving to maintain the discriminative power of the original features. For example, feature compression of a reference image can yield a more compact set of image feature representations, which can be used to perform viewpoint interpolation more efficiently.

[0121] As an example, see Figure 9 For the reference image ( Figure 9 Feature extraction is performed on the main view image 1 shown to obtain the features of the reference image, and the interpolated image ( ) is then used. Figure 9 Feature extraction is performed on the front view image 2 shown to obtain the features of the image to be interpolated; the features of the reference image ( Figure 9 Feature compression is performed on the image features of the main viewpoint image 1 shown (via...). Figure 9 The feature compression layer shown is used to obtain compressed image features of the reference image features; based on the features of the image to be interpolated, the compressed image features of the reference image features, and the pose of the first camera, viewpoint interpolation is performed on the image to be interpolated (through...). Figure 9 The viewpoint interpolation layer shown is implemented to obtain the third image. Figure 9 The interpolated viewpoint image shown is 2).

[0122] In some embodiments, the viewpoint interpolation layer is a functional layer in computer vision and image processing responsible for generating new viewpoint images between two or more known viewpoints. The role of the viewpoint interpolation layer is particularly important in fields such as 3D reconstruction and virtual reality. After feature extraction from the reference image and the image to be interpolated, the viewpoint interpolation layer uses these features and camera pose information to estimate and generate a new viewpoint image. By analyzing the differences between the features of the reference image and the features of the image to be interpolated, and combining this with the camera's position and orientation information, the layer infers the content that the image should present under the new viewpoint.

[0123] In some embodiments, the feature compression layer is a processing layer in image processing and machine learning used to reduce the dimensionality of feature representations while preserving as much important information as possible from the original features. Compression of reference image features reduces data complexity and computational burden while retaining sufficient information for subsequent processing. The feature compression layer reduces the dimensionality of the feature space by applying techniques such as principal component analysis, autoencoders, or convolutional neural networks (CNN).

[0124] In some embodiments, since the third image is generated based on the image to be interpolated and the pose of the first camera, this means that the viewpoint of the third image is closer to the viewpoint of the image to be interpolated. Therefore, the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the image to be interpolated (the first difference) will naturally be smaller than the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the reference image (the second difference), because the viewpoint of the reference image may have a larger deviation from the viewpoint of the image to be interpolated.

[0125] In this way, the features of the target object in the image to be interpolated can be accurately captured, and a third image that is closer to the viewpoint of the image to be interpolated can be effectively generated through feature compression of the reference image and camera pose information. Since the third image is created based on the original features and pose information of the image to be interpolated, it can better preserve the viewpoint features of the image to be interpolated, resulting in a smaller difference (first difference) between the viewpoint of the target object in the third image and the viewpoint in the image to be interpolated. Meanwhile, due to the potentially large viewpoint difference between the reference image and the image to be interpolated, and the errors that may be introduced by feature compression, the viewpoint difference (second difference) between the third image and the reference image is relatively large. This difference indicates that the generated third image is visually closer to the image to be interpolated, thereby improving the realism and visual effect of the image interpolation.

[0126] In some embodiments, the above-mentioned method of performing viewpoint interpolation on the image to be interpolated based on the second camera pose and the reference image to obtain the fourth image corresponding to the image to be interpolated can be implemented in the following way: extracting features from the reference image to obtain reference image features, and extracting features from the image to be interpolated to obtain features of the image to be interpolated; compressing the features of the image to be interpolated to obtain compressed image features of the image to be interpolated; and performing viewpoint interpolation on the image to be interpolated based on the reference image features, the compressed image features of the image to be interpolated, and the second camera pose to obtain the fourth image.

[0127] In some embodiments, the difference between the viewpoint of the target object in the fourth image and the viewpoint of the target object in the image to be interpolated is the third difference, and the difference between the viewpoint of the target object in the fourth image and the viewpoint of the target object in the reference image is the fourth difference, wherein the third difference is greater than the fourth difference.

[0128] In some embodiments, the above-mentioned viewpoint interpolation of the image to be interpolated based on the second camera pose and the reference image to obtain the fourth image corresponding to the image to be interpolated can be implemented in the following way: calling the feature extraction layer of the viewpoint interpolation model to extract features from the reference image to obtain reference image features, and calling the feature extraction layer of the viewpoint interpolation model to extract features from the image to be interpolated to obtain features of the image to be interpolated; calling the feature compression layer of the viewpoint interpolation model to compress the features of the image to be interpolated to obtain compressed image features of the features of the image to be interpolated; calling the viewpoint interpolation layer of the viewpoint interpolation model to perform viewpoint interpolation of the image to be interpolated based on the reference image features, the compressed image features of the features of the image to be interpolated, and the second camera pose to obtain the fourth image.

[0129] In some embodiments, the compressed features of the image to be interpolated may lose some detailed information, which is crucial for accurately reconstructing or interpolating the viewpoint of the image. Therefore, viewpoint interpolation based on the compressed features of the image to be interpolated (resulting in the third image) may be less accurate than interpolation based on the features of the complete reference image (resulting in the fourth image). This may result in the third difference (the viewpoint difference between the third image and the image to be interpolated) being greater than the fourth difference (the viewpoint difference between the fourth image and the reference image).

[0130] In some embodiments, the second camera pose is obtained from a reference image, which may contain estimation errors. These errors are amplified during interpolation, leading to a larger difference in viewpoint (third difference) between the fourth image obtained based on the second camera pose and the image to be interpolated. Since the reference image itself is used to calculate the second camera pose, the difference in viewpoint (fourth difference) between the fourth image and the reference image is relatively small.

[0131] As an example, see Figure 9 For the reference image ( Figure 9 Feature extraction is performed on the main view image 1 shown to obtain the features of the reference image, and the interpolated image ( ) is then used. Figure 9 Feature extraction is performed on the front view image 2 shown to obtain the features of the image to be interpolated; the features of the image to be interpolated ( Figure 9 The image features of the main viewpoint image 2 shown are compressed to obtain the compressed image features of the image to be interpolated; based on the reference image features, the compressed image features of the image to be interpolated, and the pose of the second camera, viewpoint interpolation is performed on the image to be interpolated (through...). Figure 9 The viewpoint interpolation layer shown is implemented to obtain the fourth image. Figure 9 The interpolated viewpoint image shown is 1).

[0132] In this way, based on the features of the reference image and the compressed features of the image to be interpolated, combined with the pose information of the second camera, a viewpoint interpolation result (fourth image) can be generated. Although the viewpoint difference between the fourth image and the image to be interpolated (third difference) is greater than the viewpoint difference with the reference image (fourth difference), this characteristic allows the fourth image to maintain a high similarity with the reference image while more finely adjusting the viewpoint of the image to be interpolated, thus achieving a more natural and smooth transition visually. This not only enriches the viewpoint representation of the image but also improves the realism and coherence of the visual effect.

[0133] In other embodiments, step 1022 above can be implemented as follows: obtain the first camera pose of the target object in the image to be interpolated, and the second camera pose of the target object in the reference image; determine the average of the first camera pose and the second camera pose as the third camera pose; and perform viewpoint interpolation on the image to be interpolated based on the third camera pose and the reference image to obtain the second image corresponding to the image to be interpolated.

[0134] As an example, the expression for the pose of the third camera mentioned above can be:

[0135]

[0136] Where W is used to indicate the pose of the third camera, w1 is used to indicate the pose of the first camera, and w2 is used to indicate the pose of the second camera.

[0137] In some embodiments, the above-mentioned method of performing viewpoint interpolation on the image to be interpolated based on the pose of the third camera and the reference image to obtain a second image corresponding to the image to be interpolated can be implemented as follows: The feature extraction layer of the viewpoint interpolation model is invoked to extract features from the reference image and the image to be interpolated, respectively, to obtain reference image features and image to be interpolated features; the feature compression layer of the viewpoint interpolation model is invoked to compress the reference image features, to obtain compressed image features corresponding to the reference image features; the viewpoint interpolation layer of the viewpoint interpolation model is invoked to perform viewpoint interpolation on the image to be interpolated based on the pose of the third camera, the compressed image features corresponding to the reference image features, and the image to be interpolated features, to obtain the second image.

[0138] In other embodiments, the above-mentioned method of performing viewpoint interpolation on the image to be interpolated based on the pose of the third camera and the reference image to obtain a second image corresponding to the image to be interpolated can also be implemented in the following way: calling the feature extraction layer of the viewpoint interpolation model to extract features from the reference image and the image to be interpolated, respectively, to obtain reference image features and image to be interpolated features; calling the feature compression layer of the viewpoint interpolation model to compress the features of the image to be interpolated, to obtain compressed image features corresponding to the features of the image to be interpolated; calling the viewpoint interpolation layer of the viewpoint interpolation model to perform viewpoint interpolation on the image to be interpolated based on the pose of the third camera, the compressed image features corresponding to the features of the image to be interpolated, and the image features to be interpolated, to obtain a second image.

[0139] In some embodiments, the viewpoint interpolation model is a computer vision algorithm specifically designed to generate new viewpoint images from a given image sequence. The viewpoint interpolation model includes a feature extraction layer, a feature compression layer, and a viewpoint interpolation layer. The feature extraction layer extracts features from the input reference image and the image to be interpolated. The purpose of feature extraction is to identify unique information from the image that helps represent the image content, such as keypoints, edges, textures, and shapes. These features represent important visual information of the image and provide a basis for subsequent processing steps. After feature extraction, the feature compression layer of the viewpoint interpolation model compresses the extracted reference image features. This step reduces data complexity by reducing the dimensionality of the features while preserving as much useful information as possible from the original features. Feature compression helps improve computational efficiency, reduce storage requirements, and may reduce the impact of noise. The viewpoint interpolation layer performs viewpoint interpolation on the image to be interpolated based on the pose of a third camera (which may be a virtual or calculated camera position and orientation), the compressed reference image features, and the features of the image to be interpolated. The goal of the viewpoint interpolation layer is to synthesize a new image (the second image) that visually represents an intermediate viewpoint between the image to be interpolated and the reference image.

[0140] As an example, virtual reality (VR) game developers are creating a game scene that requires a high degree of immersion. In the game, players can observe virtual buildings, environments, and objects in the scene from different perspectives. The feature extraction layer of the perspective interpolation model is invoked to extract features from each reference image, resulting in reference image features. Since some details in the scene may not be as important in the VR experience, the developers decide to compress the reference image features. This step reduces the amount of data processed and transmitted while preserving key visual information. The result of the feature compression layer is the compressed reference image features. Using the perspective interpolation model, the developers define a third camera pose, which is a camera position that does not actually exist in the virtual scene. Then, using the perspective interpolation layer, the developers combine the third camera pose, the compressed reference image features, and the features of the image to be interpolated (possibly a snapshot of the player's current position in the game) to perform perspective interpolation. In the game, the player is currently at the top of a tower. The developers want to provide a perspective looking down at the bottom of the tower. Using the perspective interpolation model described above, the developers can extract features from the reference image of the view from the bottom of the tower, compress these features, and then use the player's perspective image in the game for perspective interpolation to generate a second image looking down from the top of the tower. This image can be rendered and displayed to players in real time, making them feel as if they are truly standing on top of the tower looking down.

[0141] In this way, features of the reference image and the image to be interpolated can be accurately extracted, and the feature compression layer can effectively reduce the feature dimensionality, thereby reducing computational resources and storage requirements while preserving key image information. Based on this, the viewpoint interpolation layer uses the pose of the third camera, the compressed features of the reference image, and the features of the image to be interpolated to perform viewpoint interpolation, obtaining the second image. This process not only improves the efficiency of image processing but also helps generate high-quality new viewpoint images, which have greater realism and coherence in visual effects.

[0142] In step 1023, the second images corresponding to each image to be interpolated are constructed into a second image set.

[0143] As an example, in a first-person game, a player is exploring an unknown area. The game design needs to provide multiple perspectives to enhance the player's immersion and exploration experience. The game development team has collected a set of first images taken from different angles, covering different viewpoints of the player's area. From this set of first images, the developers select at least one image as the interpolation image. This image could be a snapshot of the player's current perspective or a screenshot of a specific key moment in the game. For this interpolation image, the developers use perspective interpolation technology, interpolating the perspective based on its corresponding reference image (which could be an image of the same scene taken from a different angle). The purpose of this step is to create a transitional image from the player's current perspective (the interpolation image) to the reference image's perspective, i.e., a second image. By performing the above perspective interpolation process on each interpolation image, the developers are able to generate a series of second images. These second images together constitute a set of second images.

[0144] In some embodiments, the above-mentioned viewpoint interpolation is implemented through a viewpoint interpolation model. Before step 102, the viewpoint interpolation model can also be trained in the following way: call the initial viewpoint interpolation model, perform viewpoint interpolation on the image samples carrying image labels to obtain the interpolated image; train the initial viewpoint interpolation model based on the interpolated image and the image labels to obtain the viewpoint interpolation model.

[0145] In some embodiments, the above-mentioned training of the initial viewpoint interpolation model based on the interpolated image and image labels to obtain the viewpoint interpolation model can be achieved as follows: determining the similarity between the interpolated image and the image labels, determining the similarity as the loss value of the initial viewpoint interpolation model, and updating the model parameters of the initial viewpoint interpolation model in reverse based on the loss value of the initial viewpoint interpolation model to obtain the viewpoint interpolation model.

[0146] This effectively expands and enriches the visual content in games or virtual reality experiences. Through perspective interpolation, new perspective images can be created, filling visual gaps in game scenes and enhancing player immersion. The construction of a second image set further enhances scene diversity, making the game environment more realistic and vivid. This not only improves the player's visual enjoyment but may also reduce reliance on large amounts of original image resources, thereby lowering storage requirements and optimizing game performance. This approach provides game development with a powerful tool to expand the visual expression of the game world in an innovative and efficient way.

[0147] In step 103, the target object is reconstructed in three dimensions based on the first image set and the second image set to obtain a three-dimensional model of the target object.

[0148] In some embodiments, 3D reconstruction is a research area in computer vision and graphics, referring to the process of recovering the shape and position information of an object in three-dimensional space from a set of two-dimensional images. This process involves multiple technical aspects such as image processing, geometric modeling, camera calibration, and optimization algorithms. In 3D reconstruction, images from multiple perspectives are typically used to obtain sufficient information to address issues such as missing depth information, occlusion, and noise, ultimately constructing a 3D representation of the object.

[0149] In some embodiments, a 3D model is a mathematical representation of an object in three-dimensional space, containing information about the object's shape, size, and surface. 3D models can be created in various ways, such as 3D scanning, computer-aided design (CAD), or reconstruction from 2D images using 3D reconstruction techniques. In computer graphics, 3D models are commonly used for rendering and animation; they can contain detailed texture and color information, making rendered images appear more realistic. 3D models have wide applications in game development, film production, virtual reality, and augmented reality.

[0150] In some embodiments, the images in the first image set and the second image set are images from different perspectives of the target object. By using images from different perspectives of the target object, a three-dimensional reconstruction of the target object is performed to obtain a three-dimensional model of the target object, thereby enabling the obtained three-dimensional model of the target object to achieve an accurate three-dimensional representation of the target object from each different perspective.

[0151] In some embodiments, see Figure 6 , Figure 6 This is a flowchart illustrating the three-dimensional reconstruction method provided in the embodiments of this application. Figure 4 , Figure 3 Step 103 shown can be achieved through Figure 6 Steps 1031 to 1033 shown are implemented.

[0152] In step 1031, feature extraction is performed on the first image set to obtain the first set features of the first image set, and feature extraction is performed on the second image set to obtain the second set features of the second image set.

[0153] In some embodiments, the above feature extraction can be achieved through feature extraction methods such as Scale Invariant Feature Transform (SIFT), Speed-Up Robust Feature Transform (SURF), ORB (Oriented Fast and Rotated BRIEF), and HOG (Histogram of Oriented Gradients). The first set of features is a vector representation of the first image set, and the second set of features is a vector representation of the second image set.

[0154] In step 1032, the features of the first set and the features of the second set are fused to obtain the fused features.

[0155] In some embodiments, the features of the first set and the features of the second set are fused to obtain fused features. This can be achieved by concatenating the features of the first set and the features of the second set along their dimensions to form a new feature vector. Alternatively, it can be achieved by weighted summation of the features of the two feature sets. Another approach is to use a neural network layer (such as a fully connected layer or a convolutional layer) to transform the features of the two feature sets into the same space and then perform element-wise operations (such as addition). A cross-self-attention mechanism can also be used to fuse the features of the two feature sets. Furthermore, an interaction mechanism can be introduced at the feature level, for example, by calculating the outer product between the two feature sets and then processing it through a neural network layer.

[0156] In some embodiments, the above fusion can be achieved through a feature fusion network: the feature fusion network is invoked to fuse the features of the first set and the features of the second set to obtain fused features.

[0157] In some embodiments, the feature fusion network described above can be a cross-self-attention model. Cross-self-attention models are a commonly used attention mechanism in deep learning, particularly in natural language processing (NLP) and computer vision. A cross-self-attention model is a mechanism for calculating the dependencies between elements in an input sequence. In a cross-self-attention model, each element of an input sequence is compared with each element of another sequence, typically using dot product or cosine similarity. These similarity scores are then converted into probability distributions using a softmax function, representing the importance of each element's contribution to the output. Finally, these probability distributions are used to weightedly aggregate sequence elements to form a fused feature vector.

[0158] As an example, the feature fusion network is invoked to fuse the features from the first set and the features from the second set to obtain the fused features. This process can be represented as follows:

[0159] f^F=p+A_cm(p,f^M⊕f^I) (2)

[0160] Where p represents the position embedding, signifying channel-wise concatenation, and A_cm represents the feature fusion network. p, after projection, serves as the query vector. f^M and f^I represent the first and second sets of features, respectively, and are projected as key and value vectors. f^F represents the obtained feature embedding, i.e., the fused features.

[0161] In step 1033, the target object is reconstructed in three dimensions based on the fusion features to obtain a three-dimensional model of the target object.

[0162] In some embodiments, features from different sources (e.g., image features from different viewpoints or data features from different sensors) are fused using specific algorithms or network models to form fused features. These fused features contain more comprehensive information about the target object and can better reflect the object's true shape and texture. Then, using these fused features, the target object is reconstructed in 3D through 3D reconstruction (such as multi-view stereo matching, point cloud generation, surface reconstruction, etc.) to finally obtain an accurate 3D model.

[0163] Thus, by extracting features and fusing features from the first and second image sets, the resulting fused features can provide richer information for 3D reconstruction, significantly improving the accuracy and completeness of the reconstruction. This helps the model capture multidimensional information of the target object from images from different perspectives, thereby generating a more accurate 3D model and enhancing the reconstruction effect and applicability.

[0164] In some embodiments, the three-dimensional reconstruction is achieved through a three-dimensional reconstruction model, which includes a conversion layer and a three-dimensional reconstruction layer. Step 1033 can be implemented as follows: the conversion layer is invoked to perform a three-plane representation conversion on the fused features to obtain a three-plane representation of the fused features; the three-dimensional reconstruction layer is invoked to perform three-dimensional reconstruction on the target object based on the three-plane representation to obtain a three-dimensional model of the target object.

[0165] In some embodiments, the aforementioned 3D reconstruction model can be a Large Reconstruction Model (LRM). A LRM is a 3D reconstruction algorithm used to reconstruct high-quality 3D models from 2D images from multiple perspectives. The core idea of ​​this algorithm is to utilize deep learning techniques, particularly the Vision Transformer (ViT) encoder, to extract image features and combine them with camera pose information to achieve accurate 3D structure reconstruction. The ViT encoder is an image processing model based on the Transformer architecture, capable of effectively extracting image feature information and capturing global dependencies within the image. In LRM, not only are image features extracted, but the corresponding camera pose information (i.e., the camera's position and orientation) is also encoded. This information is crucial for the reconstruction algorithm because it provides the spatial relationships between images.

[0166] In some embodiments, during 3D reconstruction, the fused features contain key information extracted from images from different viewpoints, which is crucial for accurately reconstructing the target object. The transformation layer further processes and transforms the fused features, which are obtained by fusing features from a first set of images and a second set of images. These features may exist in vector form and contain multi-angle visual information. Triplane representation is a technique that maps the fused features onto three independent planes for processing. These three planes may be based on different feature dimensions or attributes; for example, they may represent information such as color, depth, and normal vectors. This transformation helps decompose complex fused features into more easily processed components for subsequent 3D reconstruction. The 3D reconstruction layer receives the features transformed by the triplane representation and uses these features to recover the 3D structure of the target object. This process involves understanding and interpreting the features, as well as transforming the features into a geometric representation in 3D space.

[0167] Thus, by performing a three-plane representation transformation on the fused features, complex information can be decomposed and simplified. This not only improves the efficiency and accuracy of the 3D reconstruction algorithm but also helps the model better understand feature information from different dimensions. The implementation of this method makes the 3D reconstruction process more stable and reliable, enabling the generation of more accurate and detailed 3D models.

[0168] In some embodiments, 3D reconstruction is achieved through a 3D reconstruction model. Before performing step 1033 above, the 3D reconstruction model can also be trained in the following way: obtain fused feature samples carrying image labels of sample objects, and call the initial 3D reconstruction model. Based on the fused feature samples, perform 3D reconstruction on the sample objects to obtain a predicted 3D model of the sample objects; acquire images of the predicted 3D model of the sample objects to obtain the acquired images of the predicted 3D model; and train the initial 3D reconstruction model based on the acquired images and image labels to obtain the 3D reconstruction model.

[0169] In some embodiments, the aforementioned 3D reconstruction model is a model in computer vision and graphics that can recover the shape and position information of an object in 3D space from a 2D image. It refers to an algorithmic framework that accepts fused feature samples as input, outputs a 3D model of the target object through a series of processing steps, and continuously improves reconstruction accuracy through iterative training.

[0170] In some embodiments, the fused feature samples carry image labels for multiple image types of the sample object, and the acquired images correspond one-to-one with the image labels.

[0171] In some embodiments, the above-mentioned training of the initial 3D reconstruction model based on the acquired images and image labels to obtain the 3D reconstruction model can be achieved in the following way: for each image label, determine the similarity between the image label and the corresponding acquired image; based on each similarity, determine the loss value of the initial 3D reconstruction model, and based on the loss value, train the initial 3D reconstruction model to obtain the 3D reconstruction model.

[0172] In some embodiments, the loss value of the initial 3D reconstruction model determined based on each similarity can be achieved by weighted summation of the similarities to obtain the loss value of the initial 3D reconstruction model. For example, the expression for the loss value of the initial 3D reconstruction model can be:

[0173] L=α1S1+α2S2+…+α n S n (3)

[0174] Where L indicates the loss value of the initial 3D reconstruction model, S1…S n Used to indicate the similarity between each image label and the corresponding acquired image, α1…α n Weights used to indicate the similarity levels.

[0175] In some embodiments, the loss value of the initial 3D reconstruction model based on each similarity can be determined by weighted summation of the norms of each similarity to obtain the loss value of the initial 3D reconstruction model.

[0176] In some embodiments, the loss of the RGB image, the loss of the mask image, the loss of the depth image, and the loss of the normal vector image can be specifically calculated as follows:

[0177]

[0178] Where I_i and I^gt_i are the RGB image labels and the corresponding RGB acquired images, respectively.

[0179]

[0180] Where M_i and M^gt_i are the mask image label and the corresponding mask acquisition image, respectively.

[0181]

[0182] Where D_i and D^gt_i are the depth image label and the corresponding depth acquisition image, respectively.

[0183]

[0184] Where N_i and N^gt_i are the normal vector image label and the corresponding normal vector acquired image, respectively.

[0185] In some embodiments, the acquired images can be various image types corresponding to the 3D reconstruction model, such as RGB images, mask images, depth images, and normal vector images. An RGB image is a common color image, composed of three color channels: red, green, and blue. Each channel contains color information for each pixel in the image. A mask image is a special type of image that contains binary (black and white) or multi-level grayscale pixels of the same size as the original image, used to specify which areas are important and which are irrelevant. A depth image, also known as a depth map or distance map, is an image containing distance information from each pixel to the object's surface. A normal vector image is an image containing the direction of the plane normal to each pixel. Normal vectors are commonly used to represent the directionality of a surface in 3D space; they can be used to calculate the curvature of an object, identify surface features, or describe surface smoothness during lighting simulation and shadow calculation.

[0186] Thus, by analyzing the similarity between image labels and acquired images, the training effect of the 3D reconstruction model can be effectively improved. This process helps ensure that the model focuses on data that matches the features of real images, thereby optimizing the calculation of loss values ​​and promoting the precise adjustment of model parameters. As a result, the trained 3D reconstruction model can not only capture the 3D structure of objects more accurately, but also enhance the model's generalization ability and adaptability to different scenarios, ultimately achieving higher quality and more efficient reconstruction results.

[0187] In some embodiments, after performing step 103 above, the following processing may also be performed: displaying a 3D model in the initial virtual scene; in response to an editing operation on the 3D model, editing the 3D model to obtain an edited 3D model; updating the 3D model in the initial virtual scene to the edited 3D model to obtain an updated virtual scene.

[0188] In some embodiments, a virtual scene refers to a visual environment created using computer technology that simulates the real world or an imagined world. A virtual scene is a computer-generated environment that simulates a scene in the real world or constructs a completely fictional world. This environment can be an indoor room, an outdoor landscape, a city planning model, or even a surreal or science fiction world. Virtual scenes can simulate the physical properties and visual effects of the real world, providing users with an experience similar to a real environment. Users can typically interact with virtual scenes, whether through simple observation or more complex actions and behaviors such as walking and manipulating objects. Virtual scenes have applications in many fields, such as virtual reality (VR), augmented reality (AR), game development, film production, architectural visualization, and education and training. Virtual reality is a simulated environment created using computer technology, in which users can be fully immersed through devices such as head-mounted displays (HMDs). Augmented reality is a technology that overlays virtual content onto the real world, enhancing the user's visual experience by adding computer-generated images, text, and other information to the user's real environment.

[0189] In some embodiments, the aforementioned editing operations can refer to a series of modifications and adjustments made to a 3D model in a virtual scene to change the model's geometry, appearance attributes, or other features. Editing operations refer to a series of interactive operations performed on a 3D model displayed in the initial virtual scene, designed to modify certain aspects of the model according to the user's needs or design purposes. For example: Movement operation: The user selects the 3D model and moves it along the X, Y, and Z axes to change the model's position in the scene. Rotation operation: The user selects the 3D model and rotates it around an axis to change the model's orientation. Scaling operation: The user adjusts the size of the 3D model, making it larger or smaller. Material adjustment: The user changes the color or texture of the model's surface to create different visual effects. Collision detection: During editing, collisions between the model and other models or scene elements are detected to prevent unnatural overlap. Detail enhancement: The user adds or removes detailed features of the model, such as edge smoothness, surface bumps, etc.

[0190] As an example, in an augmented reality application, a user takes an image containing a target object, and a first set of images of the target object is generated from this image. This set may include multiple photos taken from different angles. Viewpoint interpolation is performed on one image from the first set to generate a new second image, simulating the user viewing the target object from different perspectives. Using the first and second image sets, the target object is reconstructed in 3D to obtain a 3D model, which is then overlaid onto a real-world scene in augmented reality. The user edits the 3D model in the AR application, such as resizing, rotating, or changing its color. The virtual scene is updated, and the edited 3D model is displayed on the user's real-world view, enhancing the augmented reality experience.

[0191] As an example, in game development, designers might need to create a collection of multi-angle images of a character for use in game animations and visual effects. To create smooth animations or close-ups in the game engine, perspective interpolation might be performed on an original image of the character to generate additional perspective images. The image collection is then used to reconstruct the character's 3D model, creating a realistic appearance in the game. The character's design can be modified, such as adjusting clothing or facial expressions. In the game, the character's 3D model is updated and displayed in real-time in the game world, providing players with a fresh gaming experience.

[0192] As an example, the cameras of an autonomous driving system capture images of the road and surrounding environment, generating a first set of images containing target objects such as road signs and traffic lights. To detect traffic signs in different directions, the images are interpolated to generate new perspective images. A 3D model of the road signs is reconstructed from the image set so that the autonomous driving system can more accurately identify and locate these signs. If different traffic sign recognition algorithms need to be tested, the 3D model may be edited. In autonomous driving simulation tests, the 3D model is updated and used to simulate the test environment to help verify performance.

[0193] Thus, by generating a first image set of the target object based on the image to be processed, including the target object, and then performing viewpoint interpolation on at least one of the first images based on the first image set to obtain a second image set containing at least one second image, the target object is reconstructed in 3D based on the first and second image sets to obtain a 3D model of the target object. Similarly, by generating a first image set of the target object based on the image to be processed, where different first images show different viewpoints of the target object, and then performing viewpoint interpolation on at least one of the first images based on the first image set to obtain a second image set containing at least one second image, the first image set is used to create images of transitional viewpoints between different first images, i.e., second images. These second images provide images of different viewpoints of the target object that smoothly transition between different first images. By performing 3D reconstruction on the target object based on the first and second image sets, a 3D model of the target object is obtained. The combined participation of the first and second image sets effectively reduces abrupt changes in viewpoints, thereby effectively reducing geometric deformation and singularities in the 3D model of the target object obtained from the 3D reconstruction due to viewpoint jumps, thus effectively improving the visual quality of the 3D reconstruction.

[0194] The following will describe an exemplary application of the embodiments of this application in a real-world three-dimensional object reconstruction scenario.

[0195] Reconstructing a 3D model of an object from a single image can be applied to numerous fields such as games, AR / VR, and AIGC. These fields often require massive amounts of 3D assets. 3D object reconstruction is a research hotspot in computer vision and computer graphics, involving extracting information from a set of images to infer the geometric model of an object in 3D space. 3D object reconstruction has wide applications in multiple fields, including virtual reality, augmented reality, computer-aided design, medical image analysis, and robot navigation. Through 3D reconstruction technology, the digital replication of the real world can be achieved, providing rich 3D model resources for various application scenarios.

[0196] The embodiments of this application often achieve good 3D object reconstruction results through LRM-based 3D reconstruction. Given a single image as input, the embodiments of this application first train a network for generating main view images (front, back, left, and right) based on a diffusion model. Then, the diffusion model is used to perform pairwise view interpolation between the main view images to train and generate interpolated view images with multi-view consistency. Finally, all views are trained and encoded into a three-plane representation. Through the three-plane representation decoder and FlexiCubes 3D reconstruction, excellent 3D reconstruction results are achieved.

[0197] In some embodiments, see Figure 7, Figure 7 This is a schematic diagram of the effect of three-dimensional object reconstruction provided in the embodiment of this application. By inputting a single image 71 of the object, the three-dimensional model 72 of the object can be effectively reconstructed through the embodiment of this application.

[0198] In some embodiments, see Figure 8 , Figure 8 This is a schematic diagram illustrating the principle of the 3D reconstruction method provided in this application embodiment. It is based on multi-view generation using a diffusion model and 3D model reconstruction based on three planes. After the entire scheme is trained, only a single image of an object needs to be input. The multi-view generation module generates a main view image and a corresponding interpolated view image. Then, the 3D model corresponding to the image can be obtained through image feature extraction and 3D reconstruction.

[0199] In some embodiments, see Figure 8 This multi-view generation method, based on a diffusion model, takes a single image as input and generates four main viewpoint images (i.e., the first image set described above) (front, back, left, and right) through the diffusion model. The diffusion model can be any single-image-to-multi-image model, including Wonder3D, Zero123++, etc.

[0200] In some embodiments, see Figure 8 and Figure 9 , Figure 9 This is a schematic diagram illustrating the principle of perspective interpolation provided in this application embodiment. After obtaining the main perspective image, the perspective interpolation module can obtain a series of interpolated perspectives between pairs of main perspective images, which can be represented as:

[0201] I^I_i=f(I^M_1, I^M_2, Π) (8)

[0202] Where I^I_i is any interpolated view image, I^M_1 and I^M_2 are the two corresponding main view images, Π is the camera pose corresponding to these three images (each image corresponds to a different camera pose), and f is... Figure 9 The view interpolation module shown.

[0203] In some embodiments, see Figure 8 and Figure 9In the viewpoint interpolation module, the main viewpoint image 1 is used as the input to UNet. The main viewpoint image 2 is processed through the CLIP module (i.e., the feature compression layer described above) to obtain an embedding, which, together with the camera pose of the main viewpoint image 1, is used as the condition input to UNet (i.e., the viewpoint interpolation model described above), thereby obtaining the interpolated viewpoint image 1 (i.e., the second image set described above). For the main viewpoint image 2, the main viewpoint image 2 is used as the input to UNet, and the main viewpoint image 1 is processed through the CLIP module to obtain an embedding, which, together with the camera pose, is used as the condition input to UNet to obtain the interpolated viewpoint image 2.

[0204] In some embodiments, the MSE loss function L_MSE is used in the generation of the main view map and the training of view interpolation. The loss function L can be expressed as (the loss for Unet):

[0205] L=L_MSE(Noise_pred, Noise_gt) (9)

[0206] Here, noise is a noise map sample.

[0207] In some embodiments, see Figure 8 Three-plane-based 3D model reconstruction: After completing the multi-view generation training based on the diffusion model, we can obtain multiple view images of the object. Using these as supervision signals and the rendered images of the object from multiple viewpoints as input, we can train a three-plane-based 3D model reconstruction. The input is a set of multi-view images of the object, denoted as I^M for the main viewpoint image set and I^I for the interpolated viewpoint image set. First, we obtain the image feature tokens through ViT:

[0208] [f^M, f^I)]=ViT([I^M, I^I]) (10)

[0209] Where f is the feature token obtained corresponding to image group I, f^M is the feature corresponding to image group I^M (that is, the first set of features described above); f^I is the feature corresponding to image group I^I (that is, the second set of features described above).

[0210] After obtaining the image features, they are combined with the location embeddings (f^M or f^I or f^M⊕f^I) through a cross-attention mechanism, and further feature embeddings are obtained through residual connections. This process can be represented as:

[0211] f^F=p+A_cm(p,f^M⊕f^I) (11)

[0212] Where p is the position embedding, representing channel-wise concatenation, and A_cm represents the cross-attention mechanism. The projected p is used as the query vector, and the projected f^M⊕f^I is used as the key and value vectors. f^F is the obtained feature embedding. The obtained feature embedding is encoded into three planes to obtain three planes. These three planes are used to construct the three planes, and then FlexiCubes is used for the final 3D model reconstruction.

[0213] In some embodiments, the loss function of the reconstruction process (the loss within the red box) can be expressed as:

[0214] L=L_rgb+lambda_mask×L_mask+lambda_depth×L_depth+lambda_normal×L_normal (12)

[0215] Where lambda represents the weights of each loss term, and L_rgb, L_mask, L_depth, and L_normal represent the RGB image loss, mask image loss, depth image loss, and normal image loss for generating the 3D model rendering and the ground truth model rendering, respectively. The RGB image, mask image, depth image, and normal image are obtained by image acquisition from the reconstructed 3D model, and their specific calculation methods can be expressed as follows:

[0216]

[0217] Where I_i and I^gt_i are the rendered RGB images of the generative model and the ground truth model, respectively.

[0218]

[0219] Where M_i and M^gt_i are the rendering masks for the generative model and the ground truth model, respectively.

[0220]

[0221] Where D_i and D^gt_i are the rendering depth maps of the generative model and the ground truth model, respectively, and \otimes indicates element-wise production, that is, only the part within the ground truth mask is considered.

[0222]

[0223] Where N_i and N^gt_i are the rendering normal vector maps of the generative model and the ground truth model, respectively.

[0224] In some embodiments, after training the multi-view image generation and triplane reconstruction, during the testing phase, a single image of the object is input, and the multi-view image generation module can obtain the main view image for 3D reconstruction. Furthermore, stable view interpolation is performed to obtain more consistent interpolated view images. Then, the corresponding 3D reconstruction result of the object can be obtained through ViT feature encoding and triplane reconstruction.

[0225] In this way, a 3D reconstruction of an object can be obtained by inputting a single image. This effectively solves the problem of inconsistencies in multiple viewpoints caused by generating multiple images from a single image, which impairs the 3D reconstruction results and leads to serious geometric distortions, singularities, or geometric blurring. It significantly improves the accuracy and diversity of asset generation, reduces asset production costs, and provides designers with entrepreneurial inspiration.

[0226] It is understood that in the embodiments of this application, data such as images to be processed are involved. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0227] The following description continues to illustrate the exemplary structure of the three-dimensional reconstruction device 455 provided in the embodiments of this application as a software module. In some embodiments, such as... Figure 2 As shown, the software modules stored in the 3D reconstruction device 455 in the memory 450 may include: a generation module, used to generate a first image set of the target object based on an image to be processed including the target object; wherein the first image set includes multiple first images, and the viewpoint of the target object is different in different first images; a viewpoint interpolation module, used to perform viewpoint interpolation on at least one of the first images based on the first image set to obtain a second image set including at least one second image, wherein the viewpoint of the target object in the second image is different from the viewpoint of the target object in the first image set; and a 3D reconstruction module, used to perform 3D reconstruction of the target object based on the first image set and the second image set to obtain a 3D model of the target object.

[0228] In some embodiments, the above-described generation module is further configured to extract features from the image to be processed to obtain image features of the image to be processed; call a view prediction model to predict the target object based on the image features to obtain an initial image set of the target object, wherein the initial image set includes multiple first images; and determine the first image set based on the initial image set.

[0229] In some embodiments, the generation module is further configured to obtain the number of first images in the initial image set; when the number is greater than a target number, select the target number of first images from the initial image set to obtain the first image set; when the number is less than or equal to the target number, determine the initial image set as the first image set.

[0230] In some embodiments, the above-described perspective interpolation module is further configured to select at least one of the first images from the first image set as an image to be interpolated; for each image to be interpolated, perform perspective interpolation on the image to be interpolated based on a reference image of the image to be interpolated to obtain a second image corresponding to the image to be interpolated; wherein, the reference image is a first image in the first image set other than the image to be interpolated; and construct a second image set from the second images corresponding to each image to be interpolated.

[0231] In some embodiments, the above-described viewpoint interpolation module is further configured to obtain a first camera pose of the target object in the image to be interpolated, and perform the following processing on each of the reference images of the image to be interpolated: perform viewpoint interpolation on the image to be interpolated based on the first camera pose and the reference image to obtain a third image corresponding to the image to be interpolated; obtain a second camera pose of the target object in the reference image, and perform viewpoint interpolation on the image to be interpolated based on the second camera pose and the reference image to obtain a fourth image corresponding to the image to be interpolated; and determine the third image and the fourth image as the second image corresponding to the image to be interpolated.

[0232] In some embodiments, the above-mentioned viewpoint interpolation module is further configured to extract features from the reference image to obtain reference image features, and extract features from the image to be interpolated to obtain image features to be interpolated; compress the reference image features to obtain compressed image features of the reference image features; and perform viewpoint interpolation on the image to be interpolated based on the image features to be interpolated, the compressed image features of the reference image features, and the first camera pose to obtain the third image; wherein, the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the image to be interpolated is a first difference, and the difference between the viewpoint of the target object in the third image and the viewpoint of the target object in the reference image is a second difference, and the first difference is less than the second difference.

[0233] In some embodiments, the above-mentioned viewpoint interpolation module is further configured to extract features from the reference image to obtain reference image features, and extract features from the image to be interpolated to obtain image features to be interpolated; compress the image features to be interpolated to obtain compressed image features of the image features to be interpolated; and perform viewpoint interpolation on the image to be interpolated based on the reference image features, the compressed image features of the image features to be interpolated, and the second camera pose to obtain the fourth image; wherein, the difference between the viewpoint of the target object in the fourth image and the viewpoint of the target object in the image to be interpolated is a third difference, the difference between the viewpoint of the target object in the fourth image and the viewpoint of the target object in the reference image is a fourth difference, and the third difference is greater than the fourth difference.

[0234] In some embodiments, the above-described view interpolation module is further configured to obtain a first camera pose of the target object in the image to be interpolated and a second camera pose of the target object in the reference image; determine the average value of the first camera pose and the second camera pose as a third camera pose; and perform view interpolation on the image to be interpolated based on the third camera pose and the reference image to obtain a second image corresponding to the image to be interpolated.

[0235] In some embodiments, the above-mentioned viewpoint interpolation module is further configured to call the feature extraction layer of the viewpoint interpolation model to extract features from the reference image and the image to be interpolated, respectively, to obtain reference image features and image to be interpolated features; call the feature compression layer of the viewpoint interpolation model to compress the reference image features to obtain compressed image features corresponding to the reference image features; and call the viewpoint interpolation layer of the viewpoint interpolation model to perform viewpoint interpolation on the image to be interpolated based on the pose of the third camera, the compressed image features corresponding to the reference image features, and the image to be interpolated features, to obtain the second image.

[0236] In some embodiments, the viewpoint interpolation is implemented through a viewpoint interpolation model. The above-mentioned three-dimensional reconstruction device further includes: a training module, used to call an initial viewpoint interpolation model, perform viewpoint interpolation on image samples carrying image labels to obtain an interpolated image; and train the initial viewpoint interpolation model based on the interpolated image and the image labels to obtain the viewpoint interpolation model.

[0237] In some embodiments, the above-mentioned three-dimensional reconstruction module is further configured to extract features from the first image set to obtain a first set of features from the first image set, and extract features from the second image set to obtain a second set of features from the second image set; fuse the first set of features and the second set of features to obtain fused features; and perform three-dimensional reconstruction on the target object based on the fused features to obtain a three-dimensional model of the target object.

[0238] In some embodiments, the above-described 3D reconstruction is achieved through a 3D reconstruction model. The training module is further configured to acquire fused feature samples carrying image labels of the sample object, and call an initial 3D reconstruction model to perform 3D reconstruction on the sample object based on the fused feature samples, thereby obtaining a predicted 3D model of the sample object; to acquire images of the predicted 3D model of the sample object, thereby obtaining an acquired image of the predicted 3D model; and to train the initial 3D reconstruction model based on the acquired image and the image labels, thereby obtaining the 3D reconstruction model.

[0239] In some embodiments, the fused feature samples carry image labels of multiple image types of the sample object, and the acquired images correspond one-to-one with the image labels. The training module is further configured to determine the similarity between the image label and the corresponding acquired image for each image label; determine the loss value of the initial 3D reconstruction model based on each similarity; and train the initial 3D reconstruction model based on the loss value to obtain the 3D reconstruction model.

[0240] In some embodiments, the above-described three-dimensional reconstruction apparatus further includes: a virtual scene module, configured to display the three-dimensional model in an initial virtual scene, edit the three-dimensional model in response to an editing operation on the three-dimensional model to obtain an edited three-dimensional model, and update the three-dimensional model in the initial virtual scene to the edited three-dimensional model to obtain an updated virtual scene.

[0241] This application provides a computer program product, which includes a computer program or computer-executable instructions stored in a computer-readable storage medium. A processor of an electronic device reads the computer-executable instructions from the computer-readable storage medium and executes the computer-executable instructions, causing the electronic device to perform the three-dimensional reconstruction method described above in this application.

[0242] This application provides a computer-readable storage medium storing computer-executable instructions. When these computer-executable instructions are executed by a processor, they cause the processor to execute the three-dimensional reconstruction method provided in this application. For example, ... Figure 3 The three-dimensional reconstruction method is shown.

[0243] In some embodiments, the computer-readable storage medium may be a memory such as FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or it may be a variety of electronic devices including one or any combination of the above-mentioned memories.

[0244] In some embodiments, computer-executable instructions may take the form of programs, software, software modules, scripts, or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as stand-alone programs or as modules, components, subroutines, or other units suitable for use in a computing environment.

[0245] As an example, computer-executable instructions may, but do not necessarily, correspond to files in a file system. They may be stored as part of a file that holds other programs or data, for example, in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple co-located files (e.g., files that store one or more modules, subroutines, or code sections).

[0246] As an example, computer-executable instructions can be deployed to execute on a single electronic device, or on multiple electronic devices located at one location, or on multiple electronic devices distributed across multiple locations and interconnected via a communication network.

[0247] In summary, the embodiments of this application have the following beneficial effects:

[0248] (1) A first image set of the target object is generated based on the image to be processed, including the target object. Based on the first image set, at least one first image is interpolated by perspective to obtain a second image set containing at least one second image. Based on the first image set and the second image set, the target object is reconstructed in three dimensions to obtain a three-dimensional model of the target object. Thus, a first image set of the target object is generated based on the image to be processed, including the target object. The target object has different perspectives in different first images in the first image set. By interpolating the perspective of at least one first image based on the first image set, a second image set containing at least one second image is obtained. The first image set is used to create images of transitional perspectives between different first images, i.e., second images. The second images form images of different perspectives of the target object that smoothly transition between different first images. By reconstructing the target object in three dimensions based on the first image set and the second image set, a three-dimensional model of the target object is obtained. Due to the combined participation of the first image set and the second image set, the abrupt changes between perspectives are effectively reduced, thereby effectively reducing the geometric deformation and singularity of the three-dimensional model of the target object obtained by the three-dimensional reconstruction due to the jump in perspective, thus effectively improving the visual quality of the three-dimensional reconstruction.

[0249] (2) When the number of images exceeds the target number, carefully selecting the first set of images within the target number avoids wasting resources, reduces storage and computation costs, and ensures the quality and relevance of the image set. When the number of images is insufficient, directly using all predicted images as the first image set ensures the integrity of the information, enabling subsequent applications to make decisions or analyses based on the most comprehensive viewpoint data possible. Furthermore, controlling the size of the first image set helps reduce the sense of fragmentation between viewpoints, avoiding visual clutter caused by too many viewpoints, thereby improving user experience and visual presentation. This quantity control strategy not only improves efficiency and performance but also enhances the practicality and usability of the image set in various application scenarios.

[0250] (3) Based on the first camera pose, viewpoint interpolation is performed on the image to be interpolated. The resulting third image can more accurately reflect the appearance and position of the target object in the image to be interpolated, improving the realism and visual effect of the image. By obtaining the second camera pose of the target object in the reference image, the fourth image is obtained by viewpoint interpolation of the image to be interpolated. The third and fourth images are determined as the second images corresponding to the image to be interpolated. Thus, multiple viewpoint interpolations are performed using the image to be interpolated and the reference image to obtain the third and fourth images, thereby effectively improving the interpolation effect of viewpoint interpolation, improving image quality, and providing a more reliable foundation for subsequent image processing and analysis.

[0251] (4) It can accurately capture the features of the target object in the image to be interpolated, and effectively generate a third image that is closer to the viewpoint of the image to be interpolated by using feature compression of the reference image and camera pose information. Since the third image is created based on the original features and pose information of the image to be interpolated, it can better preserve the viewpoint features of the image to be interpolated, resulting in a smaller difference (first difference) between the viewpoint of the target object in the third image and the viewpoint in the image to be interpolated. At the same time, due to the possible large viewpoint difference between the reference image and the image to be interpolated, as well as the errors that may be introduced by feature compression, the viewpoint difference (second difference) between the third image and the reference image is relatively large. This difference indicates that the generated third image is visually closer to the image to be interpolated, thereby improving the realism and visual effect of image interpolation.

[0252] (5) Based on the features of the reference image and the compression features of the image to be interpolated, combined with the pose information of the second camera, a viewpoint interpolation result (fourth image) can be generated. Although the viewpoint difference (third difference) between the fourth image and the image to be interpolated is greater than the viewpoint difference (fourth difference) between the fourth image and the reference image, this characteristic allows the fourth image to maintain a high similarity with the reference image while more finely adjusting the viewpoint of the image to be interpolated, thus achieving a more natural and smooth transition visually. This not only enriches the viewpoint representation of the image but also improves the realism and coherence of the visual effect.

[0253] (6) It can accurately extract features from the reference image and the image to be interpolated, and effectively reduce the feature dimension through a feature compression layer, thereby reducing computational resources and storage requirements while preserving key image information. Based on this, the viewpoint interpolation layer uses the pose of the third camera, the compressed features of the reference image, and the features of the image to be interpolated to perform viewpoint interpolation to obtain the second image. This process not only improves the efficiency of image processing, but also helps to generate high-quality new viewpoint images, which have higher realism and coherence in visual effects.

[0254] (7) It can effectively expand and enrich the visual content in games or virtual reality experiences. Through perspective interpolation, new perspective images can be created to fill the visual gaps in the game scene and enhance the player's immersion. The construction of the second image set further enhances the diversity of the scene, making the game environment more realistic and vivid. This not only improves the player's visual enjoyment but may also reduce the dependence on a large number of original image resources, thereby reducing the storage requirements of game resources and optimizing game performance. This processing method provides a powerful tool for game production to expand the visual expression of the game world in an innovative and efficient way.

[0255] (8) By analyzing the similarity between image labels and acquired images, the training effect of the 3D reconstruction model can be effectively improved. This process helps ensure that the model focuses on data that matches the features of real images, thereby optimizing the calculation of loss values ​​and promoting the precise adjustment of model parameters. As a result, the trained 3D reconstruction model can not only capture the 3D structure of objects more accurately, but also enhance the model's generalization ability and adaptability to different scenarios, ultimately achieving higher quality and more efficient reconstruction results.

[0256] (9) By extracting features and fusing features from the first and second image sets, the resulting fused features can provide richer information for 3D reconstruction, significantly improve the accuracy and completeness of reconstruction, help the model capture multidimensional information of the target object from images from different perspectives, thereby generating a more accurate 3D model and enhancing the reconstruction effect and applicability.

[0257] (10) After training the multi-view image generation and three-plane reconstruction, during the testing phase, a single image of the object is input. Through the multi-view image generation module, a main view image for 3D reconstruction can be obtained, and further stable view interpolation can be performed to obtain more consistent interpolated view images. Then, through ViT feature encoding and three-plane reconstruction, the corresponding 3D reconstruction result of the object can be obtained. In this way, the corresponding 3D reconstruction result of the object can be obtained by inputting a single image. This effectively solves the problem of inconsistency in multiple views caused by generating multiple images from a single image, which damages the 3D reconstruction result of the object and causes serious geometric deformation, singularity, or geometric blur. It greatly improves the accuracy and diversity of asset generation, reduces the cost of asset production, and provides designers with entrepreneurial inspiration.

[0258] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Any modifications, equivalent substitutions, and improvements made within the spirit and scope of this application are included within the scope of protection of this application.

Claims

1. A three-dimensional reconstruction method, characterized by, The method comprises: generating a first image set of a target object based on a to-be-processed image comprising the target object; wherein the first image set comprises multiple first images, and the target object has different viewing angles in different first images; performing viewing angle interpolation on at least one first image based on the first image set to obtain a second image set comprising at least one second image, the target object having different viewing angles in the second image set and the first image set; performing three-dimensional reconstruction on the target object based on the first image set and the second image set to obtain a three-dimensional model of the target object.

2. The method of claim 1, wherein, The method comprises: performing feature extraction on the to-be-processed image to obtain image features of the to-be-processed image; calling a viewing angle prediction model to predict the target object based on the image features to obtain an initial image set of the target object, the initial image set comprising multiple first images; determining the first image set based on the initial image set.

3. The method of claim 2, wherein, The method comprises: obtaining the number of first images in the initial image set; when the number is greater than a target number, selecting the target number of first images from the initial image set to obtain the first image set; when the number is less than or equal to the target number, determining the initial image set as the first image set.

4. The method of claim 1, wherein, The method comprises: selecting at least one first image from the first image set as a to-be-interpolated image; performing viewing angle interpolation on each to-be-interpolated image based on a reference image of the to-be-interpolated image to obtain a second image corresponding to the to-be-interpolated image; wherein the reference image is a first image in the first image set other than the to-be-interpolated image; constructing each second image corresponding to the to-be-interpolated image into the second image set.

5. The method of claim 4, wherein, The method comprises: obtaining a first camera pose of the target object in the to-be-interpolated image, and performing the following processing on each reference image of the to-be-interpolated image respectively: performing viewing angle interpolation on the to-be-interpolated image based on the first camera pose and the reference image to obtain a third image corresponding to the to-be-interpolated image; obtaining a second camera pose of the target object in the reference image, and performing viewing angle interpolation on the to-be-interpolated image based on the second camera pose and the reference image to obtain a fourth image corresponding to the to-be-interpolated image; determining the third image and the fourth image as the second image corresponding to the to-be-interpolated image.

6. The method of claim 5, wherein, The view angle interpolation of the to-be-interpolated image based on the first camera pose and the reference image comprises: feature extraction is performed on the reference image to obtain reference image features, and feature extraction is performed on the to-be-interpolated image to obtain to-be-interpolated image features; feature compression is performed on the reference image features to obtain compressed image features of the reference image features; view angle interpolation is performed on the to-be-interpolated image based on the to-be-interpolated image features, the compressed image features of the reference image features, and the first camera pose to obtain the third image; wherein a difference between the view angle of the target object in the third image and the view angle of the target object in the to-be-interpolated image is a first difference, and a difference between the view angle of the target object in the third image and the view angle of the target object in the reference image is a second difference, and the first difference is smaller than the second difference.

7. The method of claim 5, wherein, The view angle interpolation of the to-be-interpolated image based on the second camera pose and the reference image comprises: feature extraction is performed on the reference image to obtain reference image features, and feature extraction is performed on the to-be-interpolated image to obtain to-be-interpolated image features; feature compression is performed on the to-be-interpolated image features to obtain compressed image features of the to-be-interpolated image features; view angle interpolation is performed on the to-be-interpolated image based on the reference image features, the compressed image features of the to-be-interpolated image features, and the second camera pose to obtain the fourth image; wherein a difference between the view angle of the target object in the fourth image and the view angle of the target object in the to-be-interpolated image is a third difference, and a difference between the view angle of the target object in the fourth image and the view angle of the target object in the reference image is a fourth difference, and the third difference is greater than the fourth difference.

8. The method of claim 4, wherein, The view angle interpolation of the to-be-interpolated image based on the reference image of the to-be-interpolated image comprises: a first camera pose of the target object in the to-be-interpolated image and a second camera pose of the target object in the reference image are obtained; an average value of the first camera pose and the second camera pose is determined as a third camera pose; view angle interpolation is performed on the to-be-interpolated image based on the third camera pose and the reference image to obtain the second image corresponding to the to-be-interpolated image.

9. The method of claim 8, wherein, The view angle interpolation of the to-be-interpolated image based on the third camera pose and the reference image to obtain the second image corresponding to the to-be-interpolated image comprises: a feature extraction layer of a view angle interpolation model is called to perform feature extraction on the reference image and the to-be-interpolated image respectively to obtain reference image features and to-be-interpolated image features; a feature compression layer of the view angle interpolation model is called to perform feature compression on the reference image features to obtain compressed image features corresponding to the reference image features; The view interpolation layer of the view interpolation model is called to perform view interpolation on the to-be-interpolated image based on the third camera pose, the compressed image feature corresponding to the reference image feature, and the to-be-interpolated image feature, to obtain the second image.

10. The method of claim 1, wherein, Before the view interpolation is performed on at least one of the first images based on the first image set to obtain the second image set containing at least one second image, the method further includes: calling an initial view interpolation model to perform view interpolation on an image sample carrying an image label to obtain an interpolated image; training the initial view interpolation model based on the interpolated image and the image label to obtain the view interpolation model.

11. The method of claim 1, wherein, The three-dimensional reconstruction of the target object based on the first image set and the second image set to obtain the three-dimensional model of the target object includes: performing feature extraction on the first image set to obtain first set features of the first image set, and performing feature extraction on the second image set to obtain second set features of the second image set; fusing the first set features and the second set features to obtain fused features; performing three-dimensional reconstruction of the target object based on the fused features to obtain the three-dimensional model of the target object.

12. The method of claim 11, wherein, Before the three-dimensional reconstruction of the target object based on the fused features to obtain the three-dimensional model of the target object is performed, the method further includes: obtaining a fused feature sample carrying an image label of a sample object, and calling an initial three-dimensional reconstruction model to perform three-dimensional reconstruction of the sample object based on the fused feature sample to obtain a predicted three-dimensional model of the sample object; performing image acquisition on the predicted three-dimensional model of the sample object to obtain an acquired image of the predicted three-dimensional model; training the initial three-dimensional reconstruction model based on the acquired image and the image label to obtain the three-dimensional reconstruction model.

13. The method of claim 12, wherein, The fused feature sample carries image labels of multiple image types of the sample object, the acquired image corresponds to the image label one-to-one, and the training of the initial three-dimensional reconstruction model based on the acquired image and the image label to obtain the three-dimensional reconstruction model includes: determining a similarity between the image label and the corresponding acquired image for each image label; determining a loss value of the initial three-dimensional reconstruction model based on each similarity, and training the initial three-dimensional reconstruction model based on the loss value to obtain the three-dimensional reconstruction model.

14. The method of claim 1, wherein, After the three-dimensional reconstruction of the target object based on the first image set and the second image set to obtain the three-dimensional model of the target object is performed, the method further includes: displaying the three-dimensional model in an initial virtual scene, editing the three-dimensional model in response to an editing operation on the three-dimensional model to obtain an edited three-dimensional model; updating the three-dimensional model in the initial virtual scene to the edited three-dimensional model to obtain an updated virtual scene.

15. A three-dimensional reconstruction apparatus, characterized by comprising: The device comprises: a generating module configured to generate a first image set of a target object based on a to-be-processed image comprising the target object, wherein the first image set comprises a plurality of first images, and the target object has different viewing angles in different first images; a viewing angle interpolation module configured to perform viewing angle interpolation on at least one first image based on the first image set to obtain a second image set comprising at least one second image, and the target object has different viewing angles in the second image set and the first image set; a three-dimensional reconstruction module configured to perform three-dimensional reconstruction on the target object based on the first image set and the second image set to obtain a three-dimensional model of the target object.

16. An electronic device, comprising: The electronic device comprises: a memory configured to store computer executable instructions or computer programs; a processor configured to execute the computer executable instructions or computer programs stored in the memory to implement the three-dimensional reconstruction method in any one of claims 1 to 14.

17. A computer-readable storage medium storing computer-executable instructions or a computer program, wherein the computer-executable instructions or the computer program comprise the steps of claim 16. The computer executable instructions or computer programs are executed by the processor to implement the three-dimensional reconstruction method in any one of claims 1 to 14.

18. A computer program product comprising computer programs or computer executable instructions, characterized in that, The computer program or computer executable instructions are executed by the processor to implement the three-dimensional reconstruction method in any one of claims 1 to 14.

Images